[Short Tip] Use Ansible with managed nodes running Python3

Ansible Logo

Python 3 is becoming the default Python version on more and more distributions. Fedora 28 ships Python 3, and RHEL 8 is expected to ship Python 3 as well.

With Ansible this can lead to trouble: some of these distributions do not ship a default /usr/bin/python but instead insist on picking either /usr/bin/python2 or /usr/bin/python3 thus leading to errors when Ansible is called to manage such machines:

TASK [Gathering Facts] 
fatal: [116.116.116.202]: FAILED! => {"changed": false, "module_stderr": "Connection to 116.116.116.202 closed.\r\n", "module_stdout": "/bin/sh: /usr/bin/python: No such file or directory\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

The fix is to define the Python interpreter in additional variables. They can even be provided on the command line:

$ ansible-playbook -i 116.116.116.202, mybook.yml -e ansible_python_interpreter="/usr/bin/python3"

Monitoring OpenVPN ports and the ways of Open Source

openvpnYears ago I wrote a small OpenVPN port monitoring script for my former employer. Over the time, it got multiple contributions from various users, evolving it into quite some sophisticated piece of software. For me, this is a powerful example how Open Source works even in small ways.

Years ago I created a small Python script to monitor OpenVPN ports for my employer of that time, credativ – under an Open Source license, of course. To be frank: for me it was one of my first Nagios/Icinga monitoring scripts, and one of my first serious Python attempts, thus the code was rather simple. Others probably would have done it in half the time with much better code. But it worked, it met the requirements, the monitoring people were happy.

Over the years, even when I left credativ and stopped working regularly on monitoring environments I carried on to be the maintainer of the code base.

And in fact over time it evolved quite a bit and got many more features:

  • IPv6 support
  • Python 3 support
  • UDP retries
  • response validation
  • dynamic HMAC digests
  • proper Python packaging structure

There were even packages created for Gentoo.

All those features and additions were not written by me, but by multiple contributors. This was only possible because the script was released under an Open Source license, here MIT, to begin with.

For me this rather small, simple example shows one particular way of how Open Source can work: different people had rather similar problems. Instead of re-inventing the wheel and writing their own scripts each time they picked something (I) which already existed and (II) solved parts of their problems, in this case my script. They extended it to fulfil their needs, and submitted the changes. Over time, this lead to a surprisingly sophisticated and powerful script which can be used by many others to solve an even broader range of similar problems. The process was not coordinated, unplanned, but created a worthwhile result from which all parties benefit.

This way of developing Open Source software is quite common – the Linux kernel is arguably the most prominent example, but a broad range of other projects are developed that way as well: Ansible, PostgreSQL, Apache, Kubernetes, etc. But as shown above, this development model does not only benefit the really large, well known projects, but works for small, specialized solutions as well.

To me, this is one of the most preferred ways to show and explain the benefits of Open Source to others: different parties working together – not even necessarily at the same time – on the same source to solve similar problems, extending the quality and capabilities of the solution over time, creating worth for all parties involved and even everyone else who just wants to use the solution.

[Howto] Using Ansible to manage RHEL 5 systems

Ansible Logo

With the release of Ansible 2.4, we now require that managed nodes have a Python version of at least 2.6. Most notable, this leaves RHEL 5 users asking how to manage RHEL 5 systems in the future – since it only provides Python 2.4.

(I published this post originally at ansible.com/blog/ .)

Background

With the release of Ansible 2.4 in September 2017, we have moved to support Python 2.6 or higher on the managed nodes. This means previous support for Python-2.4 or Python-2.5 is no longer available:

Support for Python-2.4 and Python-2.5 on the managed system’s side was dropped. If you need to manage a system that ships with Python-2.4 or Python-2.5, you’ll need to install Python-2.6 or better on the managed system.

This was bound to happen at some point in time because Python 2.6 was released almost 10 years ago, and most systems in production these days are based upon 2.6 or newer version. Furthermore, Python 3 is getting more and more traction, and in the long term we need to be able to support it. However, as the official Python documentation shows, code that runs on both Python 2.x and Python 3.x requires at least Python 2.6:

If you are able to skip Python 2.5 and older, then the required changes to your code should continue to look and feel like idiomatic Python code.

Thus the Ansible project had to make the change.

As a result, older Linux and UNIX releases only providing Python 2.4 are now faced with a challenge: How do I automate the management of my older environments that only provide Python-2.4?

We know organizations want to run their business critical applications for as long as possible, and this means running on older versions of Linux. We have seen this with Red Hat Enterprise Linux (RHEL) 5 customers who opted for the extended life cycle support until 2020 once the version reaches its end of life. However, RHEL 5 ships with Python-2.4, and thus users will see an error message even with the simplest Ansible 2.4 module:

$ ansible rhel5.qxyz.de -m ping  
rhel5.qxyz.de | FAILED! => {
    "changed": false, 
    "module_stderr": "Shared connection to rhel5.qxyz.de closed.\r\n", 
    "module_stdout": "Traceback (most recent call last):\r\n  File\"/home/rwolters/.ansible/tmp/ansible-tmp-1517240216.46-158969762588665/ping.py\", line 133, in ?\r\n    exitcode = invoke_module(module, zipped_mod, ANSIBALLZ_PARAMS)\r\n  File \"/home/rwolters/.ansible/tmp/ansible-tmp-1517240216.46-158969762588665/ping.py\", line 38, in invoke_module\r\n    (stdout, stderr) = p.communicate(json_params)\r\n  File \"/usr/lib64/python2.4/subprocess.py\", line 1050, in communicate\r\n    stdout, stderr = self._communicate_with_poll(input)\r\n  File \"/usr/lib64/python2.4/subprocess.py\", line 1113, in _communicate_with_poll\r\n    input_offset += os.write(fd, chunk)\r\nOSError: [Errno 32] Broken pipe\r\n", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}

This post will show three different ways to solve these errors and ease the migration until your servers can be upgraded.

Solutions

1. Use Ansible 2.3

It is perfectly fine to use an older Ansible version. Some features and modules might be missing, but if you have to manage older Linux distributions and cannot install a newer Python version, using a slightly older Ansible version is a way to go. All old releases can be found at release.ansible.com/ansible.

As long as you are able to make the downgrade, and are only running the outdated Ansible version for a limited time, it is probably the easiest way to still automate your RHEL 5 machines.

2. Upgrade to a newer Python version

If you cannot use Ansible 2.3 but need to use a newer Ansible version with Red Hat Enterprise Linux 5 – upgrade the Python version on the managed nodes! However, as shown in the example below, especially with Python an updated version is usually installed to an alternative path to not break system tools. And Ansible needs to know where this is.

For example, the EPEL project provides Python 2.6 packages for Red Hat Enterprise Linux in their archives and we will show how to install and use them as an example. Note though that Python 2.6 as well as the EPEL packages for Red Hat Enterprise Linux 5 both reached end of life already, so you are on your own when it comes to support.

To install the packages on a managed node, the appropriate key and EPEL release package need to be installed. Then the package python26 is available for installation. 

$ wget  
https://archives.fedoraproject.org/pub/archive/epel/5/x86_64/epel-release-5-4.noarch.rpm
$ wget  
https://archives.fedoraproject.org/pub/archive/epel/RPM-GPG-KEY-EPEL-5
$ sudo rpm --import RPM-GPG-KEY-EPEL-5
$ sudo yum install epel-release-5-4.noarch.rpm
$ sudo yum install python26

The package does not overwrite the Python binary, but installs the binary at an alternative path, /usr/bin/python2.6. So we need to tell Ansible to look at a different place for the Python library, which can be done with the flag ansible_python_interpreter for example directly in the inventory:

[old]  
rhel5.qxyz.de ansible_python_interpreter=/usr/bin/python2.6 

That way, the commands work again:

$ ansible rhel5.qxyz.de  -m ping      
rhel5.qxyz.de | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

3. Use the power of RAW

Last but not least, there is another way to deal with old Python 2.4-only systems – or systems with no Python it all. Due to the way Ansible is built, almost all modules (for Linux/Unix systems, anyway) require Python to run. However, there are two notable exceptions: the raw module, and the script module. They both only send basic commands via the SSH connection, without invoking the underlying module subsystem.

That way, at least basic actions can be performed and managed via Ansible on legacy systems. Additionally, you can use the template function on the local control machine to create scripts on the fly dynamically before you execute them on the spot:

---
- name: control remote legacy system
  hosts: legacy
  gather_facts: no
  vars_files:
  - script_vars.yml

  tasks:
  - name: create script on the fly to manage system
    template:
      src: manage_script.sh.j2
      dest: "/manage_script.sh”
    delegate_to: localhost
  - name: execute management script on target system
    script:
      manage_script.sh
  - name: execute raw command to upgrade system
    raw:
      yum upgrade -y
    become: yes 

One thing to note here: since Python is not working on the target system, we cannot collect facts and thus we must work with gather_facts: no.

As you see, that way we can include legacy systems in the automation, despite the fact that we are limited to few modules to work with.

Conclusion

For many customers, they want to support their business critical applications for as long as possible, this often includes running it on an older version of Red Hat Enterprise Linux. “If it ain’t broke, don’t fix it,” is a common mantra within IT departments. But automating these older, traditional systems which do not provide standard libraries can be challenging. here are multiple ways to deal with that – deciding which way is best depends on the overall situation and possibilities of the automation setup. Until organizations are faced with the pressing business need to modernize these older systems, Ansible is there to help.

[HowTo] Combine Python methods with Jinja filters in Ansible

Ansible Logo

Ansible has a lot of ways to manipulate variables and their content. We shed some light on the different possibilities – and how to combine them.

Ansible inbuilt filters

One way to manipulate variables in Ansible is to use filters. Filters are connected to variables via pipes, |, and the result is the modified variable. Ansible offers a set of inbuilt filters. For example the ipaddr filter can be used to find IP addresses with certain properties in a list of given strings:

# Example list of values
test_list = ['192.24.2.1', 'host.fqdn', '::1', '192.168.32.0/24', 'fe80::100/10', True, '', '42540766412265424405338506004571095040/64']

# {{ test_list | ipaddr }}
['192.24.2.1', '::1', '192.168.32.0/24', 'fe80::100/10', '2001:db8:32c:faad::/64']

Jinja2 filters

Another set of filters which can be utilized in Ansible are the Jinja2 filters of the template engine Jinja2, which is the default templating engine in Ansible.

For example the map filter can be used to pick certain values from a given dictionary. Note the following code snippet where from a list of names only the first names are given out as a list due to the mapping filter (and the list filter for the output).

vars:
  names:
    - first: Foo
      last: Bar
    - first: John
      last: Doe
 
 tasks:
 - debug:
     msg: "{{ names | map(attribute='first') |list }}"

Python methods

Besides filters, variables can also be modified by the Python string methods: Python is the scripting language Ansible is written in, and and provides string manipulation methods Ansible can just use. In contrast to filters, methods are not attached to variables with a pipe, but with dot notation:

vars:
  - mystring: foobar something

- name: endswith method
  debug:
    msg: "{{ mystring.endswith('thing') }}"

...

TASK [endswith method] *****************************************************************
ok: [localhost] => {
 "msg": true
}

Due to the close relation between Python and Jinja2 many of the above mentioned Jinja2 filters are quite similar to the string methods in Python and as a result, some capabilities like capitalize are available as a filter as well as a method:

vars:
  - mystring: foobar something

tasks:
- name: capitalize filter
  debug:
    msg: "{{ mystring|capitalize() }}"

- name: capitalize method
  debug:
    msg: "{{ mystring.capitalize() }}"

Connecting filters and methods

Due to the different ways of invoking filters and methods, it is sometimes difficult to bring both together. Caution needs to be applied if filters and methods are to be mixed.

For example, if a list of IP addresses is given and we want the last element of the included address of the range 10.0.0.0/8, we first can use the ipaddr filter to only output the IP within the appropriate range, and afterwards use the split method to break up the address in a list with four elements:

vars:
 - myaddresses: ['192.24.2.1', '10.0.3.5', '171.17.32.1']

tasks:
- name: get last element of 10* IP
  debug:
    msg: "{{ (myaddresses|ipaddr('10.0.0.0/8'))[0].split('.')[-1] }}"

...

TASK [get last element of 10* IP] **************************************************************
ok: [localhost] => {
 "msg": "5"
}

As can be seen above, to attach a method to a filtered object, another set of brackets – ( ) – is needed. Also, since the result of this filter is a list, we need to take the list element – in this case this is easy since we only have one result, so we take the element 0. Afterwards, the split method is called upon the result, gives back a list of elements, and we take the last element (-1, but element 3 would have worked here as well).

 

Conclusion

There are many ways in Ansible to manipulate strings, however since they are coming from various sources it is sometimes a little bit tricky to find what is actually needed.

Insights into Ansible: environments of executed playbooks

Ansible LogoUsually when Ansible Tower executes a playbook everything works just as on the command line. However, in some corner cases the behavior might be different: Ansible Tower runs its playbooks in a specific environment.

Different playbook results in Tower vs CLI

Ansible is a great tool for automation, and Ansible Tower enhances these capabilities by adding centralization, a UI, role based access control and a REST API. To take advantage of Tower, just import your playbooks and press start – it just works.

At least most of the time: lately I was playing around with the Google Cloud Engine, GCE. Ansible provides several GCE modules thus writing playbooks to control the setup was pretty easy. But while GCE related playbooks worked on the plain command line, they failed in Tower:

PLAY [create node on GCE] ******************************************************

TASK [launch instance] *********************************************************
task path: /var/lib/awx/projects/_43__gitolite_gce_node_tower_pem_file/gce-node.yml:13
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 2573, in <module>
    main()
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 506, in main
    module, gce, inames)
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 359, in create_instances
    external_ip=external_ip, ex_disk_auto_delete=disk_auto_delete, ex_service_accounts=ex_sa_perms)
TypeError: create_node() got an unexpected keyword argument 'ex_can_ip_forward'

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "gce"}, "parsed": false}

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @gce-node.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1 

To me that didn’t make sense at all: the exact same playbook was running on command line. How could that fail in Tower when Tower is only a UI to Ansible itself?

Environment variables during playbook runs

The answer is that playbooks are run by Tower within specific environment variables. For example, the GCE login credentials are provided to the playbook and thus to the modules via environment variables:

GCE_EMAIL
GCE_PROJECT
GCE_PEM_FILE_PATH

That means, if you want to debug a playbook and want to provide the login credentials just the way Tower does, the shell command has to be:

GCE_EMAIL=myuser@myproject.iam.gserviceaccount.com GCE_PROJECT=myproject GCE_PEM_FILE_PATH=/tmp/mykey.pem ansible-playbook myplaybook.yml

The error at hand was also caused by an environment variable, though: PYTHONPATH. Tower comes along with a set of Python libraries needed for Ansible. Among them some which are required by specific modules. In this case, the GCE modules require the Apache libcloud, and that one is installed with the Ansible Tower bundle. The libraries are installed at /usr/lib/python2.7/site-packages/awx/lib/site-packages – which is not a typical Python path.

For that reason, each playbook is run from within Tower with the environment variable PYTHONPATH="/usr/lib/python2.7/site-packages/awx/lib/site-packages:". Thus, to run a playbook just the same way it is run from within Tower, the shell command needs to be:

PYTHONPATH="/usr/lib/python2.7/site-packages/awx/lib/site-packages:" ansible-playbook myplaybook.yml

This way the GCE error shown above could be reproduced on the command line. So the environment provided by Tower was a problem, while the environment of plain Ansible (and thus plain Python) caused no errors. Tower does bundle the library because you cannot expect the library for example in the RHEL default repositories.

The root cause is that right now Tower still ships with an older version of the libcloud library which is not fully compatible with GCE anymore (GCE is a fast moving target). If you run Ansible on the command line you most likely install libcloud via pip or RPM which in most cases provides a pretty current version.

Workaround for Tower

While upgrading the library makes sense in the mid term, a short term workaround is needed as well. The best way is to first install a recent version of libcloud and second identify the actual task which fails and point that exact task to the new library.

In case of RHEL, enable the EPEL repository, install python-libcloud and then add the environment path PYTHONPATH: "/usr/lib/python2.7/site-packages" to the task via the environment option.

- name: launch instance
  gce:
    name: "{{ node_name }}"
    zone: europe-west1-c
    machine_type: "{{ machine_type }}"
    image: "{{ image }}"
  environment:
    PYTHONPATH: "/usr/lib/python2.7/site-packages"