Ansible Tower 3.1 – screenshot tour

Ansible LogoAnsible Tower 3.1 was just released. Time to have a closer look at some of the new features like the workflow editor.

Just a few days ago, Ansible Tower 3.1 was released. Besides the usual bug fixes, refinements of the UI and similar things this Tower version comes with major new feature: a workflow editor, scale out clustering, integration with logging providers and a new job details page.

The basic idea of a workflow is to link multiple job templates coming one after the other. They may or may not share inventory, playbooks or even permissions. The links can be conditional: if job template A succeeds, job template B is automatically executed afterwards, but in case of failure, job template C will be run. And the workflows are not even limited to job templates, but can also include project or inventory updates.

This enables new applications for Tower: besides the rather simple execution of prepared job templates, now different workflows can build upon each other. Imagine the networking team which creates a playbooks with their own content, in their own Git repository and even targeting their own inventory, while the operations team also has their own repos, playbooks and inventory. With older Tower versions there would be no simple way to bring these totally separated ways together – with 3.1 this can be done even with a graphical editor.

Workflows can be created right from the job template page. As can be seen that page got an overhaul:


The button to add a new template offers a small arrow to get a menu from which a workflow can be set up.

Afterwards, the workflow needs to be defined – name, organization, etc. This is a necessary step, before the actual links can be created:


As shown in the screenshot above from this screen on the actual editor can be started. And I must admit that I was surprised of how simple but yet rather elegant the editor looks like and works. It takes hardly any time to get used to, and the result is visually appealing and easily understandable:


The above screenshot shows the major highlights: links depending on the result of the previous job template in red and green, blue links which are executed every time, a task in the workflow to update a project (indicated by the “P”), and the actual editor.

As mentioned at the beginning, there are more features in this new Tower release. The clustering feature is an explicitly interesting feature for load balancing and HA setups, though I have not tested it yet. Another possibility is the integration of logging providers right into the UI:


As shown above a logstash logging provider  was configured to gather all the Tower logs. Other possible providers are  splunk, and in general everything which understands REST calls.

A change I yet have to get familiar with is the new view on the jobs page, showing running or completed jobs:

The new view is much more tailored to the output of ansible-playbook, showing the time at each task. Also, a search bar has been added which can be used to search through the results rather easily. Each taks can be clicked at to get much more details about the task. However, in the old view I liked the possibility to simply click through a play and the single tasks, getting the list of hosts adjusted automatically, etc. I can already see that the change will be for the better – but I have to get used to it first 😉

Overall the new release is pretty impressive. Especially the workflow editor will massively help bringing different teams even closer in automation (DevOps, anyone?). Also, the cluster feature will certainly help create stable, HA like setups of Tower. The UI might take some time to get used to, but that’s ok, since there will be a benefit at the end.

So, it is a great release – get started now!

So you think offline systems need no updates?

offlineOften customers run offline systems and claim that such machines do not need updates since they are offline. But this is a fallacy: updates do not only close security holes but also deliver bug fixes – and they can be crucial.


Recently a customer approached me with questions regarding an upgrade of a server. During the discussion, the customer mentioned that the system never got upgrades:

“It is an offline system, there is no need.”

That’s a common misconception. And a dangerous one.

Many people think that updates are only important to fix security issues, and that bugfixes are not really worth considering – after all, the machine works, right?


Software is never perfect. Errors happen. And while security issues might be uncomfortable, bugs in the program code can be a much more serious issue than “mere” security problems.

Example One: Xerox

To pick an example, almost each company out there has one type of system which hardly ever gets updated: copy machines. These days they are connected to the internet and can e-mail scanned documents. They are usually never updated, after all it just works, right?

In 2013 it was discovered that many Xerox WorkCentres had a serious software bug, causing them to alter scanned numbers. It took quite some weeks and analysis until finally a software update fixed the issue. During that time it turned out that the bug was at least 8 years old. So millions and millions of faulty scans have been produced over the years. In some cases the originals were destroyed in the meantime. It can hardly be estimated what impact that will have, but for sure it’s huge and will accompany us for a long time. And it was estimated that even today many scanners are still not patched – because it is not common to patch such systems. Offline, right?

So yes, a security issue might expose your data to the world. But it’s worse when the data is wrong to begin with.

Example two: Jails

Another example hit the news just recently: the US Washington State Department of Correction released inmates too early – due to a software bug. Again the software bug was present for years and years, releasing inmates too early all the time.

Example three: Valve

While Valve’s systems are often per definition online, the Valve Steam for Linux bug showed that all kinds of software can contain, well, all kinds of bugs: if you moved the folder of your Steam client, it could actually delete your entire (home) directory. Just like that. And again: this bug did not happen all the time, but only in certain situations and after quite some time.

# deletes your files when the variable is not set anymore
rm -rf "$STEAMROOT/"*

Example four: Office software

Imagine you have a bug in your calculating software – so that numbers are not processed or displayed correctly. The possible implications are endless. Two famous bugs which shows that bugfixes are worth considering are the MS Office multiplication bug from 2007 and the MS Office sum bug from a year later.

Example five: health

Yet another example surfaced in 2000 when a treatment planning system at a radiotherapy department was found to calculate wrong treatment times for patients and thus the patients were exposed to much more radiation than was good for them. It took quite some time until the bug was discovered – too lat for some patients whose

“deaths were probably radiation related”.


So, yes, security issues are harmful. They must be taken serious, and a solid and well designed security concept should be applied. Multiple layers, different zones, role based access, update often, etc.

But systems which are secured by air gaps need to be updated as well. The above mentioned examples do not show bugs in highly specific applications, but also in software components used in thousands and millions of machines. So administrators should at least spend few seconds reading into each update and check if its relevant. Otherwise you might ignore that you corrupt your data over years and years without realizing it – until its too late.

[Howto] Upgrading CentOS 6 to CentOS 7

CentOS LogoCentOS 7 is out. With systemd, Docker integration and many more fixes and new features I wanted to see if the new remote upgrade possibility already works. I’ve already posted the procedure to Vexxhost’s Blog and wanted to share it here as well. But beware: don’t try this on production servers!

CentOS 7 was released only few weeks after Red Hat Enterprise Linux 7, including the same exciting features RHEL ships. Besides the long awaited Systemd and the right now much discussed Docker this release also features the possibility to perform upgrades from version 6 to version 7 automatically without the need of the installation images. And although the upgrade still requires a reboot and thus is not a live upgrade as such, it comes in very handy for servers which can only be reached remotely.

Red Hat has already released and documented the necessary tools. The CentOS team didn’t have time yet to import, test and rebuild the tools but the developers are already on it – and they provide untested binaries.

Please, note: Since the packages are not tested yet you should not, by any means, try these on anything else than on spare test machines you can easily re-deploy and which do not have any valuable data. Do not try this on your production machines!

But if you want to get a first idea of how the tools do basically work, I recommend to set up a simple virtual machine with a fully updated CentOS 6 and as few packages as possible. Next, install the rpms from the CentOS repository mentioned above. Among these is the Preupgrade Assistant, which can be run on a system with no harm: preupg just analyses the system and gives hints what to look out for during an upgrade without performing any tasks. Since I only tested with systems with hardly any services installed I got no real results from preupg. Even a test run on a system with more services installed brought the same output (only showing some examples of the dozens and dozens of lines):

$ sudo preupg
Preupg tool doesn't do the actual upgrade.
Please ensure you have backed up your system and/or data in the event of a failed upgrade
 that would require a full re-install of the system from installation media.
Do you want to continue? y/n
Gathering logs used by preupgrade assistant:
All installed packages : 01/10 ...finished (time 00:00s)
All changed files      : 02/10 ...finished (time 00:48s)
Changed config files   : 03/10 ...finished (time 00:00s)
All users              : 04/10 ...finished (time 00:00s)
042/100 ...done    (samba shared directories selinux)
043/100 ...done    (CUPS Browsing/BrowsePoll configuration)
044/100 ...done    (CVS Package Split)
|samba shared directories selinux          |notapplicable  |
|CUPS Browsing/BrowsePoll configuration    |notapplicable  |
|CVS Package Split                         |notapplicable  |

As mentioned above the Preupgrade Assistant only helps evaluating what problems might come up during the upgrade – the real step must be done with the tool redhat-upgrade-tool-cli. For that to work the CentOS 7 key must be imported first:

$ sudo rpm --import

Afterwards, the actual upgrade tool can be called. As options it takes the future distribution version and a URL to pull the data from. Additionally I had to add the option --force since the tool complained that preupg was not run previously – although it was. As soon as the upgrade tool is called, it starts downloading all necessary information, packages and images, and afterwards asks for a reboot – the reboot does not happen automatically.

$ sudo /usr/bin/redhat-upgrade-tool-cli --force --network 7 --instrepo=
setting up repos...
.treeinfo                                                                                                                                            | 1.1 kB     00:00     
getting boot images...

After the reboot the machine updates itself with the help of the downloaded packages. Note that this phase does take some time, depending on the speed of the machine, expect minutes, not seconds. However, if everything turns out right, the next login will be into a CentOS 7 machine:

$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID_LIKE="rhel fedora"
PRETTY_NAME="CentOS Linux 7 (Core)"

Concluding it can be said that the upgrade tool worked quite nicely. While it is not comparable to a real live upgrade if offers a decent way to upgrade remote servers. I’ve tested it with a clean VM and also with a bare metal, remote server, and it worked surprisingly good. The analysis tool unfortunately did not perform how I expected it to work, but that might be due to the untested state or I was not using it properly. I’m looking forward what how that develops and improves over time.

But, again, and as mentioned before – don’t try this on your own prod servers.