[Howto] Fixing unstable clocksource in virtualised CentOS

Maintaining the correct time in virtual machines can be quite troublesome. If problems occur, a change of the clock source might be the right step.

Before the age of virtualization time was measured by tick counting: the operating system initializes a device which sends interrupts – called ticks – at a certain, fixed rate. The OS counts these interrupts, for example 100 a second, and thus knows how much time has passed.

However, in case you have a running virtual machine it cannot be guaranteed that the virtual machine has the proper resources to generate the ticks at fixed rate. Imagine a server hosting dozens of virtual guests, it might happen that at the moment the virtual machine for a specific guest does not get the resource to generate a tick. A backlog of ticks is build, and might grow more if the server machine is under heavy load. Thus the clock on the vm guest runs behind. If the backlog is too large, the ticks might even been dropped, so the clock source for the vm is unstable and the vm guest clock is further behind.

Linux tries to find out if the clock source is unstable and reports if that is the case. The error message in such cases looks like:

Clocksource tsc unstable (delta = -102057770 ns).  Enable clocksource failover by adding clocksource_failover kernel parameter.

The best is to check first what clock sources are available and which one is currently used:

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock tsc hpet acpi_pm
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock

The problem of an unstable clock source can be fixed most often by adding another, failover clock source, for example hpet or acpi_pm. These and other clock sources are explained in detail in “Understanding the Linux Kernel, 3rd Edition” by Daniel P. Bovet, Marco Cesati, btw.

The failover clock source has to be added to the kernel boot options in /etc/grub.conf:

clocksource_failover=acpi_pm in /etc/grub.conf

If you want further information about time keeping problems in virtual machines, there is a great techpaper by vmware exploring these problems and shedding more light on the various problems: Timekeeping in VMware Virtual Machines.