So you think offline systems need no updates?

offlineOften customers run offline systems and claim that such machines do not need updates since they are offline. But this is a fallacy: updates do not only close security holes but also deliver bug fixes – and they can be crucial.

Background

Recently a customer approached me with questions regarding an upgrade of a server. During the discussion, the customer mentioned that the system never got upgrades:

“It is an offline system, there is no need.”

That’s a common misconception. And a dangerous one.

Many people think that updates are only important to fix security issues, and that bugfixes are not really worth considering – after all, the machine works, right?

Wrong!

Software is never perfect. Errors happen. And while security issues might be uncomfortable, bugs in the program code can be a much more serious issue than “mere” security problems.

Example One: Xerox

To pick an example, almost each company out there has one type of system which hardly ever gets updated: copy machines. These days they are connected to the internet and can e-mail scanned documents. They are usually never updated, after all it just works, right?

In 2013 it was discovered that many Xerox WorkCentres had a serious software bug, causing them to alter scanned numbers. It took quite some weeks and analysis until finally a software update fixed the issue. During that time it turned out that the bug was at least 8 years old. So millions and millions of faulty scans have been produced over the years. In some cases the originals were destroyed in the meantime. It can hardly be estimated what impact that will have, but for sure it’s huge and will accompany us for a long time. And it was estimated that even today many scanners are still not patched – because it is not common to patch such systems. Offline, right?

So yes, a security issue might expose your data to the world. But it’s worse when the data is wrong to begin with.

Example two: Jails

Another example hit the news just recently: the US Washington State Department of Correction released inmates too early – due to a software bug. Again the software bug was present for years and years, releasing inmates too early all the time.

Example three: Valve

While Valve’s systems are often per definition online, the Valve Steam for Linux bug showed that all kinds of software can contain, well, all kinds of bugs: if you moved the folder of your Steam client, it could actually delete your entire (home) directory. Just like that. And again: this bug did not happen all the time, but only in certain situations and after quite some time.

# deletes your files when the variable is not set anymore
rm -rf "$STEAMROOT/"*

Example four: Office software

Imagine you have a bug in your calculating software – so that numbers are not processed or displayed correctly. The possible implications are endless. Two famous bugs which shows that bugfixes are worth considering are the MS Office multiplication bug from 2007 and the MS Office sum bug from a year later.

Example five: health

Yet another example surfaced in 2000 when a treatment planning system at a radiotherapy department was found to calculate wrong treatment times for patients and thus the patients were exposed to much more radiation than was good for them. It took quite some time until the bug was discovered – too lat for some patients whose

“deaths were probably radiation related”.

Conclusion

So, yes, security issues are harmful. They must be taken serious, and a solid and well designed security concept should be applied. Multiple layers, different zones, role based access, update often, etc.

But systems which are secured by air gaps need to be updated as well. The above mentioned examples do not show bugs in highly specific applications, but also in software components used in thousands and millions of machines. So administrators should at least spend few seconds reading into each update and check if its relevant. Otherwise you might ignore that you corrupt your data over years and years without realizing it – until its too late.

[Howto] Upgrading CentOS 6 to CentOS 7

CentOS LogoCentOS 7 is out. With systemd, Docker integration and many more fixes and new features I wanted to see if the new remote upgrade possibility already works. I’ve already posted the procedure to Vexxhost’s Blog and wanted to share it here as well. But beware: don’t try this on production servers!

CentOS 7 was released only few weeks after Red Hat Enterprise Linux 7, including the same exciting features RHEL ships. Besides the long awaited Systemd and the right now much discussed Docker this release also features the possibility to perform upgrades from version 6 to version 7 automatically without the need of the installation images. And although the upgrade still requires a reboot and thus is not a live upgrade as such, it comes in very handy for servers which can only be reached remotely.

Red Hat has already released and documented the necessary tools. The CentOS team didn’t have time yet to import, test and rebuild the tools but the developers are already on it – and they provide untested binaries.

Please, note: Since the packages are not tested yet you should not, by any means, try these on anything else than on spare test machines you can easily re-deploy and which do not have any valuable data. Do not try this on your production machines!

But if you want to get a first idea of how the tools do basically work, I recommend to set up a simple virtual machine with a fully updated CentOS 6 and as few packages as possible. Next, install the rpms from the CentOS repository mentioned above. Among these is the Preupgrade Assistant, which can be run on a system with no harm: preupg just analyses the system and gives hints what to look out for during an upgrade without performing any tasks. Since I only tested with systems with hardly any services installed I got no real results from preupg. Even a test run on a system with more services installed brought the same output (only showing some examples of the dozens and dozens of lines):

$ sudo preupg
Preupg tool doesn't do the actual upgrade.
Please ensure you have backed up your system and/or data in the event of a failed upgrade
 that would require a full re-install of the system from installation media.
Do you want to continue? y/n
y
Gathering logs used by preupgrade assistant:
All installed packages : 01/10 ...finished (time 00:00s)
All changed files      : 02/10 ...finished (time 00:48s)
Changed config files   : 03/10 ...finished (time 00:00s)
All users              : 04/10 ...finished (time 00:00s)
...
042/100 ...done    (samba shared directories selinux)
043/100 ...done    (CUPS Browsing/BrowsePoll configuration)
044/100 ...done    (CVS Package Split)
...
|samba shared directories selinux          |notapplicable  |
|CUPS Browsing/BrowsePoll configuration    |notapplicable  |
|CVS Package Split                         |notapplicable  |
...

As mentioned above the Preupgrade Assistant only helps evaluating what problems might come up during the upgrade – the real step must be done with the tool redhat-upgrade-tool-cli. For that to work the CentOS 7 key must be imported first:

$ sudo rpm --import http://isoredirect.centos.org/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-7

Afterwards, the actual upgrade tool can be called. As options it takes the future distribution version and a URL to pull the data from. Additionally I had to add the option --force since the tool complained that preupg was not run previously – although it was. As soon as the upgrade tool is called, it starts downloading all necessary information, packages and images, and afterwards asks for a reboot – the reboot does not happen automatically.

$ sudo /usr/bin/redhat-upgrade-tool-cli --force --network 7 --instrepo=http://mirror.centos.org/centos/7/os/x86_64
setting up repos...
.treeinfo                                                                                                                                            | 1.1 kB     00:00     
getting boot images...

After the reboot the machine updates itself with the help of the downloaded packages. Note that this phase does take some time, depending on the speed of the machine, expect minutes, not seconds. However, if everything turns out right, the next login will be into a CentOS 7 machine:

$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

Concluding it can be said that the upgrade tool worked quite nicely. While it is not comparable to a real live upgrade if offers a decent way to upgrade remote servers. I’ve tested it with a clean VM and also with a bare metal, remote server, and it worked surprisingly good. The analysis tool unfortunately did not perform how I expected it to work, but that might be due to the untested state or I was not using it properly. I’m looking forward what how that develops and improves over time.

But, again, and as mentioned before – don’t try this on your own prod servers.

Short Tip: Fix qdbus problems during a Kubuntu upgrade to 13.04

135712502612
I just performed an upgrade of my Kubuntu installation from 12.10 to 13.04 and had problems with the starting KDE seesion: it wasn’t able to bring up dbus, asked if I were able to call qdbus and quit afterwards.

A short test on the command line calling qdbus brought up a strange error: the binary was there and could be called, but looked for another binary called qdbus in /usr/lib/x86_64-linux-gnu/qt4/bin/qdbus which wasn’t there. However, there was a binary called /usr/lib/i386-linux-gnu/qt4/bin/qdbus, and thus I realized that the i386 version of qdbus was installed, rather than the x86_64 version I needed. Thus the fix was easy:

apt-get install qdbus

The i386 version was automatically removed, the x86_64 bit version was installed, and KDE was able to start up properly.