Managing your git repositories with ghq

The more you work with git to manage your code (or other stuff), the messier your directory structure becomes and after a few years you will maybe end up with something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
├── code
│   ├── blog
│   └── website
├── repos
│   ├── coolproject
│   └── demo
├── repositories
│   └── website
└── src
    ├── foo
    └── project1337

At least for me this happened. Most of these directories were at least actual git repositories but some were not. Obviously this is a mess and if each of the directories contains a few dozens of repositories it is getting hard to find the right one.

So I was looking for a better way to organize my repositories and a friend told me about ghq. ghq makes it easy to manage all your git repositories and enforces an organized directory structure at the same time. If you work with go you will already know the directory structure as it is exactly the same.

Install ghq

Installing ghq is easy. If you use a Linux distribution you can check if ghq is available with your package manger. Otherwise you have a few other options. You can download a current release from GitHub or if you have go installed you can run go get github.com/motemen/ghq. In case you use macOS you can run brew install ghq. To verify your installation just run:

1
2
$ ghq root
/home/max/.ghq

Change the ghq root directory

By default ghq organizes all git repositories in ~/.ghq but this can be changed in your ~/.gitconfig file. For example, I do not like the idea of putting all my repositories in a hidden directory like .ghq. Instead I use ~/repos, so the config entry in my ~/.gitconfig file looks like this:

1
2
3
[ghq]
  root = ~/repos
  root = ~/go/src

By defining root=... multiple times ghq uses all directories to look for repositories. This is particularly helpful if you work with go as it organizes repositories in the same way but puts them in ~/go/src by default. If you run ghq list to display all your repositories ghq looks in ~/repos and ~/go/src as well.

Clone a new repository

Instead of running git clone https://github.com/githubtraining/hellogitworld.git to checkout a new repository you now have to run ghq get https://github.com/githubtraining/hellogitworld.git and ghq will use the repository URL to create a proper directory structure:

1
2
3
4
5
6
7
8
9
repos/
└── github.com
    └── githubtraining
        └── hellogitworld
            ├── build.gradle
            ├── fix.txt
            ├── pom.xml
            ├── README.txt
            └ ...

If you clone more repositories from different servers and from different users ghq will create directories to keep everything organized. For example an excerpt of my ~/repos directory looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
├── aur.archlinux.org
│   └── heluxup
│       └── PKGBUILD
├── git.centralserver.de
│   └── max
│       ├── bachelor-thesis
│       ├── dotfiles
│       ├── fotoallerlei
│       ├── k8s
│       ├── telegram-stickers
│       └── wiki
└ github.com
    ├── ekeih
    │   ├── dwdpollen
    │   ├── FrundenBot
    │   ├── hello-github-actions
    │   ├── heluxup
    │   ├── hetzner-deployment
    │   ├── i3-renameworkspace
    │   ├── icinga2telegram
    │   ├── InfiniteWisdomBot
    │   ├── tado
    │   ├── tado-influxdb
    │   ├── taustakuva
    │   └── webhook
    ├── fluxcd
    │   └── helm-operator
    └── grandchild
        └── arch-cleaner

ghq look and what’s bad about it

Personally, I find this way of organization already a huge benefit of ghq but the major advantage is the way ghq helps to directly jump to the right repository:

1
2
3
4
5
$ pwd
/home/max
$ ghq look hello
$ pwd
/home/max/repos/github.com/githubtraining/hellogitworld

This way it is possible to switch between different repositories fast and without the need to know the full paths of them. Unfortunately, this has a downside… the current working directory of a process is part of its environment and a process environment can not be modified by its child processes. If you want to read some more technical background about this I recommend this Stack Overflow answer which explains it in more detail. In the end it means that ghq can not really change the current working directory of your shell… instead it starts a new subshell in the new directory. Technically this works but I find it very annoying that each ghq look ... adds an additional nested shell process end exiting the shell means to return to the previous shell.

The developer of ghq suggested to use a shell function to wrap ghq to solve this issue in a GitHub issue. This approach may seem a bit weird but it actually works great and is probably the only solution to really change the current working directory by running cd which is a shell builtin (shell builtins run directly in the shell process and not in a new one, therefore they can modify the environment of the shell process).

I extended the function a bit and added it to my ~/.bashrc:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ghq () {
  if [ "$1" = look -a -n "$2" ]; then
    local repos=($(command ghq list -p "$2"))
    case ${#repos[@]} in
      0)
        echo 'No repo found.'
        return 1
        ;;
      1)
        cd "${repos[0]}"
        return
        ;;
      *)
        local PS3="Select repo: "
        select reponame in ${repos[@]}; do
          cd "${reponame}"
          return
        done
    esac
  elif [ "$1" = get -a -n "$2" ]; then
    command ghq "$@"
    cd $(command ghq list -e -p "$2")
    return
  fi
  command ghq "$@"
}

Remember to open a new shell after modifying your ~/.bashrc or run source ~/.bashrc to load the changes. So what does it do? Each time you run ghq your shell does not run the real ghq binary (e.g. in /usr/bin/ghq) directly, instead it runs the ghq shell function with the same name.

Line 2-19 wrap the ghq look ... call which means that when you call ghq look foobar the function searches for a repository with foobar in its name and if it finds exactly one it runs cd ... to switch to the directory of it. In case multiple repositories match foobar it offers a selection and then switches to the selected repository.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ ghq look foobar
No repo found.
$ ghq look fotoallerlei
$ pwd
/home/max/repos/git.centralserver.de/max/fotoallerlei
$ ghq look heluxup
1) /home/max/repos/aur.archlinux.org/heluxup
2) /home/max/repos/github.com/ekeih/heluxup
Select repo: 2
$ pwd
/home/max/repos/github.com/ekeih/heluxup

Line 20-25 wrap the ghq get ... call to switch to the new repository after it is cloned successfully. If ghq is called neither with look nor with get the original ghq binary is called directly.

Wrap-Up

After moving my existing repositories to the directory structure of ghq I started to use it every day. It took me a few days to remember to use ghq get ... instead of git clone ... and ghq look ... instead of cd ~/repos/..., but I like this new workflow very much and can recommend it to everyone who uses multiple git repositories. The shell function to wrap the ghq call to avoid nested shell processes makes it a lot easier to use, so please take a second and add it to your ~/.bashrc if you start to use ghq.

icinga2telegram

The Telegram bot API makes it really easy to use Telegram to send automated messages to yourself (or other people). A while ago I started to use this to send Icinga2 alerts to me.

Several other people implemented Icinga2-Telegram notifications by writing a simple Bash script. Most of the time this works but there are a few things I do not like about the Bash solutions:

  1. They use two scripts with almost the same content.
  2. Bash can screw up really bad when your alert output contains special characters.
  3. The Bash solutions are based on environment variables. This does not work natively with Icinga2 director as it only supports arguments in command definitions.

As a solution I wrote a small Python tool to handle Icinga2 notifications: icinga2telegram.

Screenshot of icinga2telegram

Obviously the Python implementation has a bigger footprint than the Bash solution. It requires a Python interpreter and installs several Python packages as dependencies. This could be a problem for an embedded system with very limited resources but most Icinga2 instances run on powerful machines and the icinga2telegram footprint does not matter there. In return you get a single maintainable and robust script that you can use for host and service notifications.

If you want to learn more about icinga2telegram you can check out the source code and documentation at GitHub.

OSMC 2018

From the 4th of November to the 7th of November two coworkers and I attended the OSMC 2018 in Nuremberg. We started the first day with a Prometheus workshop by Julien Pivotto. He gave us a nice overview how Prometheus works and we were able to try everything out in real time.

After the workshop we had two days of interesting talks about different monitoring related topics followed by a hackathon on the last day. During the hackathon I was able to work on a new IcingaWeb2 feature and hopefully it will find its way into the upstream code :)

During the second day of the conference I had the chance to give a lecture about our Icinga2 setup at SysEleven: Scaling Icinga2 with many heterogeneous projects – and still preserving configurability.

My slides are also available at SlideShare and all videos and photos of the amazing conference are available at the OSMC archive.

Record a GIF with Linux

I found a cool tool called Peek to record a GIF with linux. Unfortunately it does not work well with i3.

So I ended up using byzanz-record instead. My only issue with byzanz is that it requires you to pass x and y coordinates when you want to record a selection of your screen and usually I do not like to count pixels before every recording…

Luckily someone wrote xrectsel, a simple tool to get the geometry of a selected screen region. By combining xrectsel and byzanz it is possible to record a nice GIF with i3 and probably all other window managers:

1
2
3
4
5
6
7
byzanz-record                                        \
  --cursor                                           \
  --verbose                                          \
  --delay=2                                          \
  --duration=10                                      \
  $(xrectsel "--x=%x --y=%y --width=%w --height=%h") \
  /tmp/recording.gif

To make my life easier I put it in a small shell script called gif.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
#!/usr/bin/env sh

[ $# -lt 2 ] && \
  echo 'You have to pass a duration in seconds and a filename: "gif.sh 10 /tmp/record.gif"' && \
  exit 1

byzanz-record                                        \
  --cursor                                           \
  --verbose                                          \
  --delay=2                                          \
  --duration=${1}                                    \
  $(xrectsel "--x=%x --y=%y --width=%w --height=%h") \
  "${2}"

Record a GIF

Hello World 2.0

Hello again, I am back!

In 2011 I started my blog fotoallerlei.com. In the beginning it was more about photography but over the years my focus shifted more and more to tech related blog posts. In 2018 the GDPR came along and because I did not want to spend my time with all the new internet rules I decided to take my blog offline.

After a few months I was finally able to catch up with the new regulations and I also realized that I miss my blog. So I replaced my previous Wordpress installation with Hugo - a simple static site generator. Maybe I will repost some of the old content if it is still relevant.

From now on I will use my blog again to dump my personal notes for future use. And maybe they will be useful for some strangers in the internet. Most of the content will be tech related but we will see what the future holds.

34C3 - I talked about OmNomNom

Between Christmas and New Year’s Eve I attended the 34C3 in Leipzig. I visited the congress before but it was the first time that I gave a presentation (at the congress) myself. At the last day of the 34C3 I talked about OmNomNom, my open source Telegram canteen chatbot, as part of the Lightning Talks. I explained the architecture of the software and the infrastructure I use to run and deploy the bot.

All talks were recorded, you can check out my talk in the embedded video. It starts at 01:24:45.

If the video does not work you can check YouTube or media.ccc.de. Despite the fact that my talk was a little too long for the five minutes time slot it was a nice experience to talk on such a big stage and luckily I was done with the important part of my talk when the time was up.