Short Tip: egrep – using grep with more than one expression

920839987_135ba34fff
I stumbled across an old blog post of mine about using grep with more than one expression: in the old days I used -e several times, one for each new expression. But as stressed in the comments that way is neither convenient nor reliable on ll platforms. And I have developed as well, so today I usually use egrep if I need to grep for several expressions. Thus, here are some short notes about using it.

The multiple arguments you are searching for a passed to egrep separated by pipes. For example, if you want to grep the output of lspci for all audio and video controllers, the correct command is:

$ lspci|egrep -i 'audio|vga'
00:05.0 Audio device: NVIDIA Corporation MCP61 High Definition Audio (rev a2)
00:0d.0 VGA compatible controller: NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2)

( Yes, I know, I write my blog post on pretty old hardware right now 😉 )

egrep does understand more than two expressions, so you can use the option like $STRING_1|$STRING_2|$STRING_3|.... But don’t forget to include the high tics ' in the command: these ensure that the pipe is used as a separator instead of being interpreted by your shell.

[Howto] Fixing unstable clocksource in virtualised CentOS

920839987_135ba34fffMaintaining the correct time in virtual machines can be quite troublesome. If problems occur, a change of the clock source might be the right step.

Before the age of virtualization time was measured by tick counting: the operating system initializes a device which sends interrupts – called ticks – at a certain, fixed rate. The OS counts these interrupts, for example 100 a second, and thus knows how much time has passed.

However, in case you have a running virtual machine it cannot be guaranteed that the virtual machine has the proper resources to generate the ticks at fixed rate. Imagine a server hosting dozens of virtual guests, it might happen that at the moment the virtual machine for a specific guest does not get the resource to generate a tick. A backlog of ticks is build, and might grow more if the server machine is under heavy load. Thus the clock on the vm guest runs behind. If the backlog is too large, the ticks might even been dropped, so the clock source for the vm is unstable and the vm guest clock is further behind.

Linux tries to find out if the clock source is unstable and reports if that is the case. The error message in such cases looks like:

Clocksource tsc unstable (delta = -102057770 ns).  Enable clocksource failover by adding clocksource_failover kernel parameter.

The best is to check first what clock sources are available and which one is currently used:

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock tsc hpet acpi_pm
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

The problem of an unstable clock source can be fixed most often by adding another, failover clock source, for example hpet or acpi_pm. These and other clock sources are explained in detail in “Understanding the Linux Kernel, 3rd Edition” by Daniel P. Bovet, Marco Cesati, btw.

The failover clock source has to be added to the kernel boot options in /etc/grub.conf:

clocksource_failover=acpi_pm in /etc/grub.conf

If you want further information about time keeping problems in virtual machines, there is a great techpaper by vmware exploring these problems and shedding more light on the various problems: Timekeeping in VMware Virtual Machines.

UUIDs and Linux: Everything you ever need to know [Update]

920839987_135ba34fffWhat is so special about UUIDs in Linux? I don’t know! But here is everything you ever need to know about UUIDs on Linux.

A single, simple short tip about looking up UUIDs in Linux from 2007 is one of the most successful posts I ever wrote. And is still looked up by hundreds each day! So I decided: Feed the masses.

This list covers everything about Linux and UUIDs. And it is feature complete. Of course. *cough*

Background

UUIDs are 128 bit long numbers represented by 32 hexadecimal digits and which are used in software development to uniquely identify information with no further context. They are described in RFC 4122, an example UUID is:

13152fae-d25a-4d78-b318-74397eb08184

UUIDs are probably best known in Linux as identifier for block devices. The Windows world knows UUIDs in the form of Microsoft’s globally unique identifiers, GUID, which are used in Microsoft’s Component Object Model.

The UUIDs are generated in various variants: originally most of them were derived from the Computer’s MAC, later hash sums of names were used. And about the question, how many UUIDs there are and how big the chance is that you will generate a a number you already own, here are some numbers from Wikipedia’s UUID article:

After generating 1 billion UUIDs every second for the next 100 years, the probability of creating just one duplicate would be about 50%. The probability of one duplicate would be about 50% if every person on earth owns 600 million UUIDs.

Usage in fstab

As mentioned UUIDs are most often used in Linux to identify block devices. Imagine, you have a couple of hard disks attached via USBs, than there is no persistent, reliable naming of the devices: sometimes the first USB hard disk is named “sda”, sometimes it is named “sdb”. So to uniquely address the right disk for example in your /etc/fstab, you have to add an entry like:

UUID=9043278a-1817-4ff5-8145-c79d8e24ea79 /boot ext3 defaults 0 2

For the block device itself, the uuid is stored in the superblock.

Beware however that UUIDs should not be used in fstab when you work with LVM snapshots. See the section “When not to use them” below for more details.

Linux implementation and generation

In Linux UUIDs are generated in /drivers/char/random.c?id=refs/tags/v3.8, and you can generate new ones via proc:

$ cat /proc/sys/kernel/random/uuid
eaf3a162-d770-4ec9-a819-ec96d429ea9f

There is also the library libuuid which is used by uuidgen and especially by the ext2/3/4 tools E2fsprogs to generate UUIDs:

$ uuidgen
f81cc383-aa75-4714-aa8a-3ce39e8ad33c

How to get them, bash style

The most interesting part in UUIDs is most likely how to get the current UUIDs of the hard disks. As already mentioned years ago, there are two major ways to retrieve them: a simple ls call in a special directory, and the tool blkid.

So, first the ls call which has to be made in the directory /dev/disk/by-uuid. The directory contains links named after the UUIDs and pointing to the “real” block device files. Pretty handy if you are on a system where hardly anything is installed.

$ ls -l /dev/disk/by-uuid
lrwxrwxrwx 1 root root 10 11. Okt 18:02 53cdad3b-4b01-4a6c-a099-be1cdf1acf6d -> ../../sda2

The second call uses the tool blkid which is part of the util-linux package. It provides a real interface to actually query for certain devices and also supports searching for labels.

$ blkid /dev/sda1
/dev/sda1: LABEL="/" UUID="ee7cf0a0-1922-401b-a1ae-6ec9261484c0" SEC_TYPE="ext2" TYPE="ext3"

And there are even more ways! Let’s install hwinfo:

$ hwinfo --block
[...]
  UDI: /org/freedesktop/Hal/devices/volume_uuid_3e953ee0_79f2_4d94_98b3_5f49ad652b7c
[...]
  Device Files: [...] /dev/disk/by-uuid/3e953ee0-79f2-4d94-98b3-5f49ad652b7c
[...]

As you see hwinfo lists huge amounts of data about your hardware – among them are the UUIDs of the devices. Use it when you are grabbing for more data about the block devices anyway.

Or how about udevadm? It is the udev provided tool for querying data from the udev database. This database contains all the information udev has about the system, so the UUID info is just one among many, many other data. If you are writing a “modern” script which integrates with Linux standard tools nicely, I guess I would go with udev. But for pure, quick and dirty command line utilization, it produces a bit too many information, just like hwinfo.

$ udevadm info -q all -n /dev/sda1|grep uuid
S: disk/by-uuid/9043278a-1817-4ff5-8145-c79d8e24ea79
E: [...] /dev/disk/by-uuid/9043278a-1817-4ff5-8145-c79d8e24ea79
E: ID_FS_UUID=9043278a-1817-4ff5-8145-c79d8e24ea79
E: ID_FS_UUID_ENC=9043278a-1817-4ff5-8145-c79d8e24ea79

In this context the tool udevinfo is also mentioned sometimes. However, that is deprecated, most distributions don’t ship it anymore. Also, another often mentioned way to retrieve the UUIDs is the program /lib/vol/vol_id. But as described in bug redhat#476379 vol_id is only a private udev function. It should not be used by outside programs (or people) since the application interface is not stable. Also, the entire program might be removed in the future and in fact is already removed on some distributions.

Another tool which contains many disk data is dumpe2fs. It prints the super block and blocks group information for the selected filesystem and thus has also knowledge about the uuid:

$ sudo dumpe2fs /dev/dm-0|grep UUID
dumpe2fs 1.44.2 (14-May-2018)
Filesystem UUID: 11aef60d-3d80-49d8-aa7b-bf3510d946cb

Thanks to Greg for that tip!

And not to forget is lsblk which can has a nice structured output in the shell, providing not only the UUID but also the directory, available storage and so on. The advantage of this command is that it is very easy to identify which disk carries which UUID:

$ lsblk -f
NAME FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
nvme0n1
│                                                                     
├─nvme0n1p1
│    vfat         6972-1366                             633,8M     3% /boot/efi
├─nvme0n1p2
│    ext4         5b62c867-016d-492d-b8a0-e50baf353952  715,1M    20% /boot
├─nvme0n1p3
│    crypto       f1859e2a-acfa-49a8-b9e4-bf3d9dec4208                
│ └─luks-f1859e2a-acfa-49a8-b9e4-bf3d9dec4208
│    ext4         4e385121-f4b0-4d91-86fe-26119c25c8b4  141,2G    14% /home

Thanks to __ :~) __bernard for the tip.

How to get them, GUI style

If you are afraid of shells, there is of course a KDE-GUI tool available as well to look up the UUID: /usr/bin/kcmshell4 devinfo

KCM-Shell-devinfo

Setting a UUID

As mentioned in the comment section, it can be also interesting to set a UUID. Since the UUID is part of the superblock the way to set it depends on the used file system. For ext file systems you can use tune2fs:

# tune2fs -U new_uuid /dev/sdaX

When not to use them

As mentioned by Zhenech it is not always advisable to use UUIDs everywhere.

Since it is not possible to mount two file systems with the same UUID with most file systems, extra care need to be taken when LVM snapshots (or cloned disks) are used in an environment: mounting might fail due to duplicate UUIDs.

XFS: Filesystem dm-2 has duplicate UUID - can't mount

One way to deal with this is by the way to change the UUID during creation or afterwards, another way is to mount with the nouuid option.

Note that Btrfs is different here, since you might mount multiple sub-volumes, all of them from a different main UUID, as Ben pointed out.

Anything else

If you have any other, further information, please post them in the comments! I will happily add them here. After all, my years old short tip already got me 250 k visits, I wonder how many a comprehensive list will bring me…

Stunned by the friendliness of a stranger

Since I decided to blog again a couple of days ago I was always asked by WordPress to publish my posts in twitter as well. However, I didn’t have a twitter account and thus never really gave it any thought.

Today I had some spare time, and decided to go for it and looked for twitter.com/liquidat – and it was taken. The account was abandoned, the last tweet was from years ago, and it was obvious that the company behind it already used another, better fitting twitter account. But, nevertheless, the name was taken.

So what to do? I use my nick name “liquidat” almost everywhere, from Wikipedia over WordPress to GitHub and whatnot, and somehow I didn’t want to use another nick name for twitter. So I decided to write the people behind the twitter account if they somehow would be willing to let me have the twitter name. I went to company website, used the contact form and asked kindly – not expecting any response, and not at all a positive one, since this is a company on another continent, thousands of kilometers away.

But today I got the answer – and it was positive:

No problem. Is a pleasure help you.
[…]
I wish your success.

And in a second mail, it became even better:

Hi! This is a chain. I do well for you, you do good for someone and that
someone does for someone else, and one day your turn will come you again.
Be happy

I am stunned. And speechless. And can hardly believe the fact that this person actually decided to help me. And that the reason behind it was a reason I try to live myself: helping others where you can so that they help others, to make this blue marble a better place. To actually help someone you never met and most likely will never meet who is living thousands of kilometers away, is a beautiful thing to do. And just gave me a bit more faith in humanity.

So: I am now on twitter as liquidat. And that is due to the kindliness and friendliness of the people at liquidation.com.br. I wish them all the best, and best regards!

So there are people who want to make this a better place. I like that =)

There and back again

Hej there – two years ago I left the bloggosphere for good. I had a new job, and hardly any time for blogging. Two years later a lot of things have changed. I mean, I still have the job. And I have even less time for blogging!

But: I make a lot of experiences using Linux and Open Source Software in business critical environments, and I managed quite some larger Open Source projects myself (yes, I was promoted to “Project Manager / Senior Consultant” – Hurray for fancy job titles! 😉 ). And: I simply need a place to write down these experiences, what was really good, what was bad, what I learned from them, what others could learn and so on. And why should

So, as a result, you might find one or the other new blog post here in the future. You might. =)