[Howto] Rebase feature branches in Git/Github

Git-Icon-1788CUpdating a feature branch to the actual state of the upstream main branch can be troublesome. Here is a workflow that works – at least for me.

Developing with Git is amazing, due to the possibilities to work with feature branches, remote repositories and so on. However, at some point, after some hours of development, the base of a feature branch will be outdated and it makes sense to update it before a pull request is send upstream. This is best done via rebasing. Here is a short work flow for a typical feature branch rebase I often need when developing for example Ansible modules.

  1. First, checkout the main branch, here devel.
  2. Update the main branch from the upstream repository.
  3. Rebase the local copy of the main branch.
  4. Push it to the remote origin, most likely your personal fork of the Git repo.
  5. Check out the feature branch
  6. Rebase the feature branch to the main branch.
  7. Force push the new history to the remote feature branch, most likely again your personal fork of the Git repo.

In terms of code this means:

$ git checkout devel
$ git fetch upstream devel
$ git rebase upstream/devel
$ git push
$ git checkout feature_branch
$ git rebase origin/devel
$ git push -f

This looks rather clean and easy – but I have to admit it took me quite some errors and Git cherry picking to finally get what is needed and what actually works.

Advertisements

[Short Tip] Git, cherry-pick and push

Git-Icon-1788C

Much too often, when working with Git and working on long time pull requests, I tend to screw up the Git history. For that reason, I often have to cherry pick a commit into a new branch and push that one upstream to the feature branch – with force.

First, identify the actual commits you want to cherry pick. You need to get the correct hash there. Then, create a new branch, cherry pick the commit, force the new branch upstream, delete the old branch, and rename the new one.

git log --author liquidat
git branch mywork_feature_tmp
git cherry-pick abcdefgh123456
git push --force origin HEAD:mywork_feature
git checkout devel
git branch -D mywork_feature
git branch -m mywork_feature_tmp mywork_feature

My hope is that in some point in the future I will be able to fix such broken Git repos at that point without cherry-pick. But until then, the current way works for me…

[Howto] Git history cleanup

920839987_135ba34fffGit is great. It stores everything you hand over to it. Unfortunately it also stores stuff you, later on, realize you should better not have handed over, for example due to security concerns. Here are two short ways to remove stuff from git, to cleanup the history.

Most people using Git in their daily routine sooner or later stumble in a situation where you realize that you have committed files which should not be in the repository anymore. In my (rather special case I admit) I was working in a Git repo and created a new branch to add some further stuff in a new sub-directory. Later on, however, when I had to clone the content of the new branch to another remote location I realized that there were some old files in the repo (and thus also in the new branch) which could not be exported to another location due to security concerns. They had to be removed beforehand!

So, I had to screw around with Git – but since Git is awesome, there are ways. One way I found under the marvelous title git rocks even when it sucks is to go through the entire commit history and rewrite each and every commit by deleting everything related to the given file name. This can be done by using git filter-branch.

For example, given that I had to remove a folder called “Calculation” the command is:

$ git filter-branch -f --index-filter 'git rm -rf --cached --ignore-unmatch Calculation' -- --all
Rewrite 5089fb36c64934c1b7a8301fe346a214a7cccdaa (360/365)rm 'Calculation'
Rewrite cc232788dfa60355dd6db6c672305700717835b4 (361/365)rm 'Calculation'
Rewrite 33d1782fdd6de5c75b7db994abfe228a028d7351 (362/365)rm 'Calculation'
Rewrite 7416d33cac120fd782f75c6eb91157ce8135590b (363/365)rm 'Calculation'
Rewrite 81e77acb22bd08c9de743b38a02341682ca369dd (364/365)rm 'Calculation'
Rewrite 2dce54592832f333f3ab947b020c0f98c94d1f51 (365/365)rm 'Calculation'

Ref 'refs/heads/documentation' was rewritten
Ref 'refs/remotes/origin/master' was rewritten
Ref 'refs/remotes/origin/documentation' was rewritten
WARNING: Ref 'refs/remotes/origin/master' is unchanged

The folder was removed entirely! However, old commit logs still there, so you better not have any relevant data in the commit messages! And as mentioned in the linked blog post, to really get rid of all traces of the files the best is to clone the repository again once afterwards.

In my case an even simpler way was to take the new subdirectory, make it the new root or the repository and rewrite everything regarding to the root. All other files not under the new root are discarded in such case. Here is the proper command, given that I have added my new content under the subdir “documentation”:

$ git filter-branch --subdirectory-filter documentation -- --all
Rewrite dd1d03f648e983208b1acd9a9db853ee820129b9 (34/34)
Ref 'refs/heads/documentation' was rewritten
WARNING: Ref 'refs/remotes/origin/master' is unchanged
Ref 'refs/remotes/origin/documentation' was rewritten
WARNING: Ref 'refs/remotes/origin/master' is unchanged

Please note that in both cases you have to be extra careful when you renamed the directories in the meantime. If you are not sure, better check all files which have ever been in the repository:

$ for commit in `git log --all --pretty=format:%H`; do git ls-tree -r -l $commit; done |awk '{print $5}'
documentation/foobar.txt
documentation/barfoo.txt
Calculation/example.txt
Calculation/HiglySecureStuff.txt
...

[Howto] Managing dotfiles with dfm

920839987_135ba34fffMost system administrators have a set of personalized dotfiles like .vimrc and .bashrc. Taking these files with you from host to host and keeping them up2date everywhere can be a quite wearisome task. There are various tools to ease the pain, and I like to shed some light on one of them: dfm – the dotfile manager.

My background

On my machines I usually keep a set of personalized dotfiles which I don’t want to miss on any other server I have to administrate:

.screenrc
.bashrc
.inputrc
.vimrc
.vim/colors/jellybeans.vim
.vim/colors/desert256.vim

I need these files on all machines which I regularly work on – and since there are quite some customer machines I have access to regularly I wrote my own, git backed Python script years ago to keep these files synced and up2date on each machine. While it was fun to write the script, I always knew that it did not cover all my use cases regarding dotfiles, and it was not really flexible in terms of complex directory structures and so on. Also, I knew there must be other people with the same problem out there – and thus I was sure better solutions already existed.

And boy, there are so many of them!

Some interesting solutions for dotfile management

Many people have looked at this problem before – and solved it in their own ways. Most often the basic principle is that the files are stored and tracked via git in a hidden directory, and the tool of your choise manages symlinks between the files in the store and in $HOME.

For example, a very interesting idea is to use GNU Stow to manage dotfiles. It tracks the necessary files in subdirectories and of course links the files from there to the ‘real’ places in $HOME. I like reusing existing tools, so the idea of using GNU Stow appealed immediately. Also, the ‘packages’ or ‘group’ support it offers is tempting. Unfortunately, on most systems GNU Stow is not installed by default, and I cannot install new software on customer machines.

The problem of necessary software installation is also relevant for another often mentioned solution: Homesick. Homesick is Ruby based, and works similar to the GNU Stow solution mentioned above: files are stored in a hidden subdirectory, tracked with git, and linked in $HOME. The main feature here is that it can keep the configuration files in various git repositories, called ‘castles’, so you can integrate the work of projects like oh-my-zsh.
While Homesick does offer quite some features, it is Ruby based – and I cannot expect a working Ruby environment on each system, so it is out of question. I can go with Perl or Python, but that’s about it.

Other people had the same Ruby problem and created Homeshick – a Homesick clone spelled with an additional ‘h’ and besides written in Bash. It is quite straight forward and offers all necessary features like listing and tracking various git repositories as source for dotfiles, linking the actual dotfiles to your home, and so on. This one is almost my favorite! I wouldn’t be surprised if it is the favorite for most of the users out there.

But Homeshick is only almost my favorite – meet dfm – a Utility to Manage Dotfiles! It is written in Perl and mainly does the same as mentioned above, even minus the support for more than one repository. But on the plus side it has the capability of ensuring file rights via chmod. I haven’t seen that in any other solution. Additionally it supports arbitrary scripts executed during the update process for example for host specific commands. And last but not least, using a three letter program feels, somehow, right 😉

Starting with dfm

So, first of course you have to get dfm. If you are hosting your dotfiles on github anyway, just fork the dfm starter repo and clone it. Otherwise, if you later want to host it yourself, clone the main dfm repo and change the remote URL. My choice was the second way:

$ git clone git@github.com:justone/dotfiles.git .dotfiles
Cloning into '.dotfiles'...
remote: Counting objects: 3212, done.
remote: Compressing objects: 100% (1531/1531), done.
remote: Total 3212 (delta 1413), reused 3096 (delta 1397)
Receiving objects: 100% (3212/3212), 4.22 MiB | 202 KiB/s, done.
Resolving deltas: 100% (1413/1413), done.

Next I configured the just cloned repository to use my own URL since my dotfiles are not on github:

$ cd .dotfiles/
$ git remote -v
origin  git@github.com:justone/dotfiles.git (fetch)
origin  git@github.com:justone/dotfiles.git (push)
$ git remote set-url origin git@git.example.net:dotfiles
$ git push origin master
Counting objects: 402, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (139/139), done.
Writing objects: 100% (402/402), 58.03 KiB, done.
Total 402 (delta 207), reused 389 (delta 195)
To git@git.example.net:dotfiles
 * [new branch]      master -> master

You now have the repository up and ready. So let’s install dfm as a tool available in $PATH, meaning creating a symlink between ~/bin and ~/.dotfiles/bin and also extending the $PATH variable in .bashrc.load, which is added to .bashrc:

$ ./.dotfiles/bin/dfm
INFO: Installing dotfiles...
INFO:   Symlinking bin (.dotfiles/bin).
INFO:   Symlinking .bashrc.load (.dotfiles/.bashrc.load).
INFO: Appending loader to .bashrc

The .bashrc is hardly modified:

$ tail -n 1 .bashrc
. $HOME/.bashrc.load

As a side node, I am not sure if I really want to drop all my customizations on the bashrc loader, but the reasoning behind that move from the dfm author is rationale:

Why .bashrc.load instead of .bashrc?

Each OS or distribution generally has its own way of populating a default .bashrc in each new user’s home directory. This file works with the rest of the OS to load in special things like bash completion scripts or aliases. The idea behind using .bashrc.load is that dotfiles should add new things to a system rather than overwriting built-in funcitonality.

For instance, if a system sources bash completion files for you, and your dotfiles overwrites the system-provided .bashrc, then you would have to replicate that functionality on your own.

But no matter if you agree with it or not, the next step is to add further files to your dfm repository, which is quite easy because dfm comes along with an import function:

$ dfm import .vimrc
INFO: Importing .vimrc from /home/liquidat into /home/liquidat/.dotfiles
INFO:   Symlinking .vimrc (.dotfiles/.vimrc).
INFO: Committing with message 'importing .vimrc'
[master d7de67a] importing .vimrc
 1 file changed, 29 insertions(+)
 create mode 100644 .vimrc

The usage is pretty straightforward, and supports directories as well:

$ dfm import .vim
INFO: Importing .vim from /home/liquidat into /home/liquidat/.dotfiles
INFO:   Symlinking .vim (.dotfiles/.vim).
INFO: Committing with message 'importing .vim'
[master e9bd60a] importing .vim
 3 files changed, 875 insertions(+)
 create mode 100644 .vim/colors/desert256.vim
 create mode 100644 .vim/colors/jellybeans.vim

Using dfm on a new system

Using dfm on a new system is straightforward as well: clone the repo, invocate dfm, and you are done:

$ git clone git@git.example.com:dotfiles .dotfiles
Cloning into '.dotfiles'...
remote: Counting objects: 418, done.
remote: Compressing objects: 100% (142/142), done.
remote: Total 418 (delta 211), reused 401 (delta 207)
Receiving objects: 100% (418/418), 66.83 KiB, done.
Resolving deltas: 100% (211/211), done.
$ ./.dotfiles/bin/dfm
INFO: Installing dotfiles...
INFO:   Backing up .vimrc.
INFO:   Symlinking .vimrc (.dotfiles/.vimrc).
INFO:   Backing up bin.
INFO:   Symlinking bin (.dotfiles/bin).
INFO:   Symlinking .bashrc.load (.dotfiles/.bashrc.load).
INFO:   Backing up .inputrc.
INFO:   Symlinking .inputrc (.dotfiles/.inputrc).
INFO:   Backing up .vim.
INFO:   Symlinking .vim (.dotfiles/.vim).
INFO: Appending loader to .bashrc

As you see quite some files are backed up, that just means they are moved to .backup, so in worst case you know where to look.

Now lets see what happens when you change something.

$ cd ~/bin
$ ln -s /usr/bin/gnome-terminal gt
$ dfm add bin/gt
$ dfm commit -m "Added gt symlink for gnome-terminal."
[master 441c067] Added gt symlink for gnome-terminal.
 1 file changed, 1 insertion(+)
 create mode 120000 bin/gt
$ dfm push
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 363 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
To git@sinoda:dotfiles
   b28dc11..441c067  master -> master

As you see, dfm supports git pass through: git commands are directly handed over to git. The changes where added to the git repository, and the repository was pushed to the remote URL.

So, to get the changes onto the other system you just have to ask dfm to update the files via dfm umi. In this case I called it after I made changes to .screenrc:

$ dfm umi
[...]
INFO: re-installing dotfiles
INFO: Installing dotfiles...
INFO:   Symlinking .screenrc (.dotfiles/.screenrc).

dfm special features

As mentioned above, the strongest feature of dfm is to be able to ensure file system rights and to start scripts after an update. The first option comes in handy when you are sharing files in your ssh config directory. The second is useful whenever you have to alter files or do anything based for example on host names. Imagine that you have various build machines to build rpm files, but you have to use different packages names on each build environment (think of customer specific e-mail addresses here).

It should be possible to create a script that would fill in the necessary details in the rpmmacros file based on IP or hostname. I haven’t given that a try, but it should be worth it…

Keeping dfm up2date

Last but not least, it is of course desirable to keep dfm itself up2date. The dfm wiki proposes the following workflow for that:

$ dfm remote add upstream git://github.com/justone/dotfiles.git
$ dfm checkout master
$ dfm fetch upstream 
$ dfm merge upstream/master

It is a pretty neat way, using git tools as they should be used, and is still easy enough to handle.

Summary

So, summarizing I can say dfm offers a quite neat and easily understandable solution for managing dotfiles while not relying on languages or tools you probably cannot install on the systems you are working on. However, Homeshick comes in as a close second, and I might give that one a try at some other point in the future. In the end, both solutions are much better than self written solutions – or no solution at all.

Pass – A perfect shell based password manager

920839987_135ba34fffPass is a tool to store and manage passwords and other data securely and on command line – even with built in support for Git and remote Git repositories. Thus it is a welcomed alternative for existing password managers which often require a GUI, or do not provide repository support.

What it is

Pass is a shell based password manager to store passwords and login data – or anything you want, actually. The name “the standard unix password manager” however is pretty misleading: the author wanted to stress that it only uses standard Unix tools, but failed to highlight that with a catchy name and instead just created confusion.

But the author is right with his main point: pass is in fact just gluing together already well known and tested Unix tools: the encryption of all information is ensured by GPG, passwords are queried using gpg-agent, the version control and remote repository support is done by Git, and the tool itself is written in shell code. Thus you have features you can rely on – in fact, if you want you can directly access the Git repository and the Gnupg files, you do not have to use Pass at all.

Pass stores information in simple files, which can be grouped in folders. While the main idea of Pass is to store one password in one file you can actually access each file with editors to store as many information in it as you want. Each file is encrypted with the gpg key which was defined during the initial setup of Pass. As a result the Pass database is nothing else but a folder full of other folders and gpg encrypted files:

$ ls -1 $HOME/.password-store
business
commerce
financial
$ ls -1 $HOME/.password-store/business/
linkedin.com.gpg
example.com.gpg
important.com.gpg

Pass is included in all major distributions like Fedora, Ubuntu, Debian, and so on, and thus can be installed with the usual package management tools.

How it works

If you call Pass without any further options, it just outputs the content of its password store:

$ pass
Password Store
|-- business
|   |-- linkedin.com
|   |-- example.com
|   `-- important.com
|-- commerce
|   `-- amazon
|-- financial
|   |-- badbank.com
|   `-- mybank.com

The file type ending “gpg” is not shown here to not confuse users (I guess).

Showing the content of a file is straight forward:

$ pass business/example.com
login:  example
pass:   password

Adding new entries can be done with the command pass insert $FOLDER/$FILENAME. But it might be more convenient to just use the default editor to edit a new file: pass edit $FOLDER/$FILENAME. That way multi line information can be added more easily.

However, the real strength of Pass is that after each change – like adding a new password – git-add and git-commit are called: the new file is automatically committed to a local git repository:

$ pass edit business/example.com
[master 4c09c76] Added password for business/example.com using /usr/bin/vim.
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 business/example.com.gpg

As a result all changes are automatically under version control and can be reverted. But it gets better: Pass forwards arbitrary options and commands to Git itself. Thus it is possible to access the full functionality of Git – and to push the files to an online repository:

$ pass git push
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 823 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
To git@example.net:pass
   aa2aff7..2011296  master -> master

That way the password store can be shared with any remote Git repository – and thus can be re-used by other clients, given that they have the proper GPG key.

Missing pieces

As shown above Pass is almost perfect if you need a way to manage passwords (or any other data at all) on command line level, including repository and encryption support.

But while Pass replaced all my other password managers literally in a few minutes there is still one big feature I miss: the support for GUI tools! It would be nice if Pass support could be included in the major Desktop Environments and major GUI programs used in the Linux desktop world:

  • KDE’s Kwallet
  • Gnome’s Keyring
  • Android
  • Firefox
  • Chrome/Chromium

To summarize it: Pass is great, but would be even better if it could server as a backend for the usual GUI tools and desktop environments. There is already an experimental iOS client, so there is at least hope for an Android client…