Howto: Managing dotfiles with dfm

Most system administrators have a set of personalized dotfiles like .vimrc and .bashrc. Taking these files with you from host to host and keeping them up2date everywhere can be a quite wearisome task. There are various tools to ease the pain, and I like to shed some light on one of them: dfm – the dotfile manager.

My background

On my machines I usually keep a set of personalized dotfiles which I don’t want to miss on any other server I have to administrate:

.screenrc
.bashrc
.inputrc
.vimrc
.vim/colors/jellybeans.vim
.vim/colors/desert256.vim

I need these files on all machines which I regularly work on – and since there are quite some customer machines I have access to regularly I wrote my own, git backed Python script years ago to keep these files synced and up2date on each machine. While it was fun to write the script, I always knew that it did not cover all my use cases regarding dotfiles, and it was not really flexible in terms of complex directory structures and so on. Also, I knew there must be other people with the same problem out there – and thus I was sure better solutions already existed.

And boy, there are so many of them!

Some interesting solutions for dotfile management

Many people have looked at this problem before – and solved it in their own ways. Most often the basic principle is that the files are stored and tracked via git in a hidden directory, and the tool of your choise manages symlinks between the files in the store and in $HOME.

For example, a very interesting idea is to use GNU Stow to manage dotfiles. It tracks the necessary files in subdirectories and of course links the files from there to the ‘real’ places in $HOME. I like reusing existing tools, so the idea of using GNU Stow appealed immediately. Also, the ‘packages’ or ‘group’ support it offers is tempting. Unfortunately, on most systems GNU Stow is not installed by default, and I cannot install new software on customer machines.

The problem of necessary software installation is also relevant for another often mentioned solution: Homesick. Homesick is Ruby based, and works similar to the GNU Stow solution mentioned above: files are stored in a hidden subdirectory, tracked with git, and linked in $HOME. The main feature here is that it can keep the configuration files in various git repositories, called ‘castles’, so you can integrate the work of projects like oh-my-zsh.
While Homesick does offer quite some features, it is Ruby based – and I cannot expect a working Ruby environment on each system, so it is out of question. I can go with Perl or Python, but that’s about it.

Other people had the same Ruby problem and created Homeshick – a Homesick clone spelled with an additional ‘h’ and besides written in Bash. It is quite straight forward and offers all necessary features like listing and tracking various git repositories as source for dotfiles, linking the actual dotfiles to your home, and so on. This one is almost my favorite! I wouldn’t be surprised if it is the favorite for most of the users out there.

But Homeshick is only almost my favorite – meet dfm – a Utility to Manage Dotfiles! It is written in Perl and mainly does the same as mentioned above, even minus the support for more than one repository. But on the plus side it has the capability of ensuring file rights via chmod. I haven’t seen that in any other solution. Additionally it supports arbitrary scripts executed during the update process for example for host specific commands. And last but not least, using a three letter program feels, somehow, right

Starting with dfm

So, first of course you have to get dfm. If you are hosting your dotfiles on github anyway, just fork the dfm starter repo and clone it. Otherwise, if you later want to host it yourself, clone the main dfm repo and change the remote URL. My choice was the second way:

$ git clone git@github.com:justone/dotfiles.git .dotfiles
Cloning into '.dotfiles'...
remote: Counting objects: 3212, done.
remote: Compressing objects: 100% (1531/1531), done.
remote: Total 3212 (delta 1413), reused 3096 (delta 1397)
Receiving objects: 100% (3212/3212), 4.22 MiB | 202 KiB/s, done.
Resolving deltas: 100% (1413/1413), done.

Next I configured the just cloned repository to use my own URL since my dotfiles are not on github:

$ cd .dotfiles/
$ git remote -v
origin  git@github.com:justone/dotfiles.git (fetch)
origin  git@github.com:justone/dotfiles.git (push)
$ git remote set-url origin git@git.example.net:dotfiles
$ git push origin master
Counting objects: 402, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (139/139), done.
Writing objects: 100% (402/402), 58.03 KiB, done.
Total 402 (delta 207), reused 389 (delta 195)
To git@git.example.net:dotfiles
 * [new branch]      master -> master

You now have the repository up and ready. So let’s install dfm as a tool available in $PATH, meaning creating a symlink between ~/bin and ~/.dotfiles/bin and also extending the $PATH variable in .bashrc.load, which is added to .bashrc:

$ ./.dotfiles/bin/dfm
INFO: Installing dotfiles...
INFO:   Symlinking bin (.dotfiles/bin).
INFO:   Symlinking .bashrc.load (.dotfiles/.bashrc.load).
INFO: Appending loader to .bashrc

The .bashrc is hardly modified:

$ tail -n 1 .bashrc
. $HOME/.bashrc.load

As a side node, I am not sure if I really want to drop all my customizations on the bashrc loader, but the reasoning behind that move from the dfm author is rationale:

Why .bashrc.load instead of .bashrc?

Each OS or distribution generally has its own way of populating a default .bashrc in each new user’s home directory. This file works with the rest of the OS to load in special things like bash completion scripts or aliases. The idea behind using .bashrc.load is that dotfiles should add new things to a system rather than overwriting built-in funcitonality.

For instance, if a system sources bash completion files for you, and your dotfiles overwrites the system-provided .bashrc, then you would have to replicate that functionality on your own.

But no matter if you agree with it or not, the next step is to add further files to your dfm repository, which is quite easy because dfm comes along with an import function:

$ dfm import .vimrc
INFO: Importing .vimrc from /home/liquidat into /home/liquidat/.dotfiles
INFO:   Symlinking .vimrc (.dotfiles/.vimrc).
INFO: Committing with message 'importing .vimrc'
[master d7de67a] importing .vimrc
 1 file changed, 29 insertions(+)
 create mode 100644 .vimrc

The usage is pretty straightforward, and supports directories as well:

$ dfm import .vim
INFO: Importing .vim from /home/liquidat into /home/liquidat/.dotfiles
INFO:   Symlinking .vim (.dotfiles/.vim).
INFO: Committing with message 'importing .vim'
[master e9bd60a] importing .vim
 3 files changed, 875 insertions(+)
 create mode 100644 .vim/colors/desert256.vim
 create mode 100644 .vim/colors/jellybeans.vim

Using dfm on a new system

Using dfm on a new system is straightforward as well: clone the repo, invocate dfm, and you are done:

$ git clone git@git.example.com:dotfiles .dotfiles
Cloning into '.dotfiles'...
remote: Counting objects: 418, done.
remote: Compressing objects: 100% (142/142), done.
remote: Total 418 (delta 211), reused 401 (delta 207)
Receiving objects: 100% (418/418), 66.83 KiB, done.
Resolving deltas: 100% (211/211), done.
$ ./.dotfiles/bin/dfm
INFO: Installing dotfiles...
INFO:   Backing up .vimrc.
INFO:   Symlinking .vimrc (.dotfiles/.vimrc).
INFO:   Backing up bin.
INFO:   Symlinking bin (.dotfiles/bin).
INFO:   Symlinking .bashrc.load (.dotfiles/.bashrc.load).
INFO:   Backing up .inputrc.
INFO:   Symlinking .inputrc (.dotfiles/.inputrc).
INFO:   Backing up .vim.
INFO:   Symlinking .vim (.dotfiles/.vim).
INFO: Appending loader to .bashrc

As you see quite some files are backed up, that just means they are moved to .backup, so in worst case you know where to look.

Now lets see what happens when you change something.

$ cd ~/bin
$ ln -s /usr/bin/gnome-terminal gt
$ dfm add bin/gt
$ dfm commit -m "Added gt symlink for gnome-terminal."
[master 441c067] Added gt symlink for gnome-terminal.
 1 file changed, 1 insertion(+)
 create mode 120000 bin/gt
$ dfm push
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 363 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
To git@sinoda:dotfiles
   b28dc11..441c067  master -> master

As you see, dfm supports git pass through: git commands are directly handed over to git. The changes where added to the git repository, and the repository was pushed to the remote URL.

So, to get the changes onto the other system you just have to ask dfm to update the files via dfm umi. In this case I called it after I made changes to .screenrc:

$ dfm umi
[...]
INFO: re-installing dotfiles
INFO: Installing dotfiles...
INFO:   Symlinking .screenrc (.dotfiles/.screenrc).

dfm special features

As mentioned above, the strongest feature of dfm is to be able to ensure file system rights and to start scripts after an update. The first option comes in handy when you are sharing files in your ssh config directory. The second is useful whenever you have to alter files or do anything based for example on host names. Imagine that you have various build machines to build rpm files, but you have to use different packages names on each build environment (think of customer specific e-mail addresses here).

It should be possible to create a script that would fill in the necessary details in the rpmmacros file based on IP or hostname. I haven’t given that a try, but it should be worth it…

Keeping dfm up2date

Last but not least, it is of course desirable to keep dfm itself up2date. The dfm wiki proposes the following workflow for that:

$ dfm remote add upstream git://github.com/justone/dotfiles.git
$ dfm checkout master
$ dfm fetch upstream 
$ dfm merge upstream/master

It is a pretty neat way, using git tools as they should be used, and is still easy enough to handle.

Summary

So, summarizing I can say dfm offers a quite neat and easily understandable solution for managing dotfiles while not relying on languages or tools you probably cannot install on the systems you are working on. However, Homeshick comes in as a close second, and I might give that one a try at some other point in the future. In the end, both solutions are much better than self written solutions – or no solution at all.

Short Tip: egrep – using grep with more than one expression

I stumbled across an old blog post of mine about using grep with more than one expression: in the old days I used -e several times, one for each new expression. But as stressed in the comments that way is neither convenient nor reliable on ll platforms. And I have developed as well, so today I usually use egrep if I need to grep for several expressions. Thus, here are some short notes about using it.

The multiple arguments you are searching for a passed to egrep separated by pipes. For example, if you want to grep the output of lspci for all audio and video controllers, the correct command is:

$ lspci|egrep -i 'audio|vga'
00:05.0 Audio device: NVIDIA Corporation MCP61 High Definition Audio (rev a2)
00:0d.0 VGA compatible controller: NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2)

( Yes, I know, I write my blog post on pretty old hardware right now )

egrep does understand more than two expressions, so you can use the option like $STRING_1|$STRING_2|$STRING_3|.... But don’t forget to include the high tics ' in the command: these ensure that the pipe is used as a separator instead of being interpreted by your shell.

Skype is following your links – that’s proprietary for you

Yesterday it was reported that Skype, owned by Microsoft these days, seems to automatically follow each exchanged https link. Besides the fact that this is a huge security and personal rights problem in its own it again shows how important it is to not trust a proprietary system.

The problem, skin deep

Heise reported yesterday that Skype follows https links which have been exchanged in chats on a regular basis. First and foremost, this is a privacy issue: it looks like Skype, and thus Microsoft, scans your chat history and acts based on these findings on a regular base. That cannot be explained by “security measures” or anything like it and is not acceptable. My personal data are mine, and Microsoft should not have anything to do with as long as there is no need!

Second, there is the security problem: imagine you are exchanging private links, or even links containing passwords and usernames for direct access (you shouldn’t, but sometimes you have to). Microsoft does follows these links -and therefore gains full access to all data hidden there. Imagine these are sensitive data (private or business), you have no idea what Microsoft is going to do with them.

Third, there is the disturbing part: Microsoft only follows the https links, only the encrypted URLs. If this action would be a security thing, they would surely follow the http links as well. So there must be another explanation – but which one? It is disturbing to know that Microsoft has a motivation to regularly follow links to specifically secured content.

The problem, profound

While these news are shocking, the root problem is not Skype or the behavior of Microsoft – I am pretty sure that their Licence Agreement will cover such actions. And it is most likely that others like WhatsApp, Facebook Chat or whatnot do behave in similar ways. So the actual problem is handing over all your data to a company which you have no inside to. You have no idea what they are doing, you have no control about it, and you cannot even be sure that nothing bad is done with it. Also, most vendors try to lock you in with your service, so that switching away from them is painfully due to used workflows, tools and social networks.

The solution

From my point of view, my personal perfect solution is hosting such sensitive services on my own. However, that cannot be a solution for everyone, and I for myself cannot provide for example the SLAs others need.

Thus I guess the best solution is to be conscious about what you do – and what the consequences are. Try to avoid proprietary solutions where possible. For example for chats, try to use open protocols like XMPP. Google Talk is a good example here: company based, but still using open protocols, they even push the development forward (Jingle, …). Or, if you upload files to web services, make sure you have local backup. Also, try not to upload sensitive data – if you have to, encrypt it beforehand. And if you use social networks, try to not depend on one of them too much, use cross posts for various services at the same time if possible.

And, last but not least: ask your service providers to establish transparency and rules for a responsible and acceptable usage of your data. After all, they depend on the users trust, and if enough users are requesting such changes, they will have to follow.

Howto: Changing the expiry date of GPG keys

GnuPG keys can have an expiry date. When the key expires, it cannot be used to encrypt data anymore, and thus is a good way to enforce security measures. However, what most people does not seem to know is that this expiry date can be changed quite easily.

Setting an expiry date for a GPG key is usually a good thing: it makes sure that even if you forget the password and do not have a revocation certificate the key will not be valid at some point in the future. Additionally it might force users to replace keys ever so often to enforce specific security measures. Last but not least it forces the key owner to think about his or her own GPG infrastructure and if changes are needed.

Still, there might be times where it makes sense to change the expiry date – if only because you realized that your GPG keys are all fine.

First, you need to know the key ID, in this example ABCDEF12:

$ gpg --list-keys liquidat@example.com
pub   2048R/ABCDEF12 2012-09-10 [expires: 2032-09-10]
uid                  liquidat <liquidat@example.com>
sub   2048R/BCDEF123 2012-09-10 [expires: 2032-09-10]]

With that ID at hand you can now edit the key:

gpg --edit-key ABCDEF12
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

pub  2048R/ABCDEF12  created: 2012-09-10  expires: 2013-09-10  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/BCDEF123  created: 2012-09-10  expires: 2013-09-10  usage: E   
[ultimate] (1). liquidat <liquidat@example.com>

gpg>

As you see this key is going to expire in fall 2013. The gpg> indicates a prompt, so you are basically at a gpg specific shell. So, let’s actually change the expiry date:

gpg> expire
Changing expiration time for the primary key.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 3y
Key expires at Fri May  6 15:45:42 2016 CEST
Is this correct? (y/N) y

You need a passphrase to unlock the secret key for
user: "liquidat <liquidat@example.com>"
2048-bit RSA key, ID ABCDEF12, created 2012-09-10

The passphrase is usually queried by standard means, so on a desktop systems a pop up windows should come up asking you for the passphrase.

Afterwards, list the key again to check the new expiry date:

gpg> list
pub  2048R/ABCDEF12  created: 2012-09-10  expires: 2016-09-10  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/BCDEF123  created: 2012-09-10  expires: 2013-09-10  usage: E   
[ultimate] (1). liquidat <liquidat@example.com>

gpg>

As you see, the expiry date has only changed for the first key, but not for the pub key. The edit procedure is always for one key only. Thus, change the focus from the first key, called “key 0″, to the sub key, “key 1″. A star sign * will indicate the focus on the subkey:

gpg> key 1
pub  2048R/ABCDEF12  created: 2012-09-10  expires: 2016-09-10  usage: SC  
                     trust: ultimate      validity: ultimate
sub*  2048R/BCDEF123  created: 2012-09-10  expires: 2013-09-10  usage: E   
[ultimate] (1). liquidat <liquidat@example.com>

gpg> expire

Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 3y
Key expires at Fri May  6 15:45:42 2016 CEST
Is this correct? (y/N) y

You need a passphrase to unlock the secret key for
user: "liquidat <liquidat@example.com>"
2048-bit RSA key, ID BCDEF123, created 2012-09-10

gpg> list
pub  2048R/ABCDEF12  created: 2012-09-10  expires: 2016-09-10  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048R/BCDEF123  created: 2012-09-10  expires: 2016-09-10  usage: E   
[ultimate] (1). liquidat <liquidat@example.com>

As you see, you are done, both dates are changed now. The changes finally need to be saved:

gpg> save

And, last but not least, don’t forget to upload the updated public key to the key servers:

$ gpg --keyserver pgp.mit.edu --send-keys ABCDEF12
gpg: sending key ABCDEF12 to hkp server pgp.mit.edu

Short Tip: Fix qdbus problems during a Kubuntu upgrade to 13.04

I just performed an upgrade of my Kubuntu installation from 12.10 to 13.04 and had problems with the starting KDE seesion: it wasn’t able to bring up dbus, asked if I were able to call qdbus and quit afterwards.

A short test on the command line calling qdbus brought up a strange error: the binary was there and could be called, but looked for another binary called qdbus in /usr/lib/x86_64-linux-gnu/qt4/bin/qdbus which wasn’t there. However, there was a binary called /usr/lib/i386-linux-gnu/qt4/bin/qdbus, and thus I realized that the i386 version of qdbus was installed, rather than the x86_64 version I needed. Thus the fix was easy:

apt-get install qdbus

The i386 version was automatically removed, the x86_64 bit version was installed, and KDE was able to start up properly.

Howto: Installing Owncloud News, a self hosted RSS reader

Owncloud News, a RSS news reader for the self hosting cloud service Owncloud, is available in an Alpha version. That comes right at the time Google Reader is bound to see its end soon.

I must admit that I do not understand why Google decided to shut down the Google Reader service. Social media with their unstructured news areas are nice, but no match to a well structured news feed full of read and unread news. But, there are replacements, and one pretty wise choice would be to not depend on yet another web service, but to host it yourself.

In comes Owncloud: it can already host your addresses, calendars, files and musik and can be integrated with your desktop as well. Now a RSS reader app, Owncloud News was released as an Alpha version, and indeed already looks promising:

The installation is pretty smooth as well. The requirements are a running Owncloud 5 version, so 4.5 won’t do it. The installation itself basically consists of two steps: installing and activating the so called App Framework, which is supposed to be the foundation for other Owncloud apps in the future, and afterwards installing the news app itself:

# cd /var/www
# git clone https://github.com/owncloud/appframework.git
Cloning into 'appframework'...
[...]
# git clone https://github.com/owncloud/news.git
Cloning into 'news'...
[...]

I choose /var/www here because it is recommended in the manual and because there the appropriate user has the necessary access rights. But it could be any dir, since you only link the plugins anyway:

# ln -s /var/www/appframework /var/www/owncloud/apps
# ln -s /var/www/news /var/www/owncloud/apps

Speaking about rights, make sure the web server can write cache files:

# sudo chown -R www-data:www-data /var/www/news/cache

Afterwards, login to your owncloud, and active the plugins: first the framework, followed by the actual application. Add feeds, play around, as you will see it works pretty nice.

What is still missing right now is an Android news reader which could sync with the server. When that is available as well, Owncloud News might become *the* Google Reader descendant.

Short Tip: Check configured virtual hosts in Apache

Whenever you have to debug virtual host setups in Apache, checking the actual running virtual host configuration is a good first step. This can be done with the -S option used on the Apache binary: It lists all running virtual hosts and performs a syntax check.

On Fedora, RHEL, CentOS the Apache binary can be found on /usr/sbin/httpd:

# /usr/sbin/httpd -S
VirtualHost configuration:                                                        
1.2.3.4:80      me.example.net (/etc/httpd/conf.d/me.conf:5)
2.3.4.5:80      others.example.net (/etc/httpd/conf.d/others.conf:1)
2.3.4.5:443     others.example.net (/etc/httpd/conf.d/others.conf:38)
Syntax OK

On Debian systems the call is almost the same, you just have to source the environment variables upfront, and the binary has a different name for historical reasons:

# source /etc/apache2/envvars
# /usr/sbin/apache2 -S
VirtualHost configuration:                                                        
1.2.3.4:80      me.example.net (/etc/apache2/sites-enabled/me.conf:5)
2.3.4.5:80      others.example.net (/etc/apache2/sites-enabled/others.conf:1)
2.3.4.5:443     others.example.net (/etc/apache2/sites-enabled/others.conf:38)
Syntax OK

you might run into an error about user names, in such cases it is helpful to call upfront.