[Howto] ara – making Ansible runs easier to read and understand

Ara is a simple web server showing detailed information about Ansible runs. It is helpful in understanding and troubleshooting Ansible runs.

Background

Ansible runs, especially on the command line, do only provide limited information. Details about used variables, the timing of each task or other information are only available using additional plugins, but the details provided by them are usually narrowed to a use case.

A better way to provide information about Ansible runs is to collect the data and provide them in a web framework. That is what Ansible Tower (or AWX, the upstream project to Tower) does for example: collecting detailed data and providing them in the jobs overview.

But there are situations where a fully fledged Tower is too much, or where a comparing overview of the various runs is needed. This is where ara comes in:

ARA Records Ansible playbook runs and makes the recorded data available and intuitive for users and systems.
It makes your Ansible playbooks easier to understand and troubleshoot.

https://ara.recordsansible.org/

ara was originally developed by people of the OpenStack community, and still today has strong ties with it. It does not replace Ansible Tower at all, since it does not manage the execution at all. It complements the information and overview part, and in a way more competes with the logging solutions which can be connected to Ansible Tower.

How to install

The installation of ara is pretty straight forward and described in the documentation: the software is basically installed via pip, afterwards the server can be started as a local running instance. The connection between Ansible and ara is done via action and callback plugins.

The installation of the ara package is quickly done. Note that on systems with both Python 2 and 3 you need to pick the right pip version:

$ pip3 install --user ara
...
$ python3 -m ara.setup.action_plugins                                                                                                   /home/liquidat/.local/lib/python3.7/site-packages/ara/plugins/actions
$ python3  -m ara.setup.callback_plugins                                                                                                /home/liquidat/.local/lib/python3.7/site-packages/ara/plugins/callbacks

Notice that the binaries end up in ~/.local/bin. If that is not part of the $PATH variable, the server executable to start ara needs to be addressed directly, like ~/.local/bin/ara-manage runserver:

$ ~/.local/bin/ara-manage runserver                                                                                                      * Serving Flask app "ara" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
2019-05-06 02:45:49,156 INFO werkzeug:  * Running on http://127.0.0.1:9191/ (Press CTRL+C to quit)
2019-05-06 02:45:55,915 INFO werkzeug: 127.0.0.1 - - [06/May/2019 02:45:55] "GET / HTTP/1.1" 302 -

The web page can be accessed by pointing a web browser towards http://127.0.0.1:9191/. Since Ansible is not connected yet to ara no data are shown:

As mentioned, to connect ara to Ansible a callback plugin is used. There are different ways available to tell Ansible to use a callback plugin, the easiest is to set up a ansible.cfg with the appropriate data:

$ python3 -m ara.setup.ansible | tee -a ansible.cfg                                                                                        
[defaults]
callback_plugins=/home/liquidat/.local/lib/python3.7/site-packages/ara/plugins/callbacks
action_plugins=/home/liquidat/.local/lib/python3.7/site-packages/ara/plugins/actions

Note here that this creates a new section named [defaults]. Check if your ansible.cfg already has a section called [defaults] and if so merge the entries manually. Now call a few playbooks and check the results:

• 

• 

• 

• 

• 

• 

ara provides easy access to all existing runs, making it possible to easily compare different runs with each other. At the same time detailed information are provided for individual runs, making it easy to figure out what actually happened.

Summary

ara is an interesting attempt at better displaying the information from Ansible runs. It helps analyzing what is happening in each run, where problems might be hidden and so on.

If you use Ansible Tower already the information are available to you anyway. If you like the way how it is presented in ara you can even use both at the same time.

Useful command line options for ansible-playbook

Ansible provides quite some useful command line options. Most of them are especially interesting during debugging.

Background

There are three major ways to work with Ansible:

• launching single tasks with the ansible command
• executing playbooks viaansible-playbook
• using Tower to manage and run playbooks

While Tower might be the better option to run Ansible in the day-to-day business, and the ansible CLI itself is most likely only in one-time runs used, the executing of playbooks on the command line often happens during the development of playbooks, when no Tower is available – or during debugging. In such cases, there are quite some useful command line options which might not even be known to the seasoned Ansible user.

Do I say this right? – Syntax checking

Playbooks are written in YAML, and in YAML syntax is crucial – especially indentation:

Data structure hierarchy is maintained by outline indentation.

To check if a playbook is correctly formatted, the option --syntax-check looks at all involved playbooks and verifies the correct syntax. During a syntax check, no playbooks are actually executed.

$ ansible-playbook --syntax-check oraclejdk-destroy.yml
ERROR! Syntax Error while loading YAML.

The error appears to have been in '/home/liquidat/Gits/github/ansible-demo-oraclejdk/oracle-windows-destroy.yml': line 10, column 11,
but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  win_template: src=data/remove-program.j2 dest=C:\\temp\\remove-program.ps1
    - name: remove application
          ^ here

The syntax check helps if a playbook fails for no apparent reason – or if a playbook was edited a lot and it is simply not sure if everything was moved around correctly.

Whom am I talking to? – Listing affected hosts

With complex playbooks and dynamic inventories it sometimes is hard to say against which hosts a playbook will actually be executed. In such cases, the option --list-hosts will output a list of affected hosts, including the name of the actual play and the pattern with which the hosts were chosen:

$ ansible-playbook --list-hosts oraclejdk-destroy.yml

playbook: oraclejdk-destroy.yml

  play #1 (windows): remove OracleJDK on Windows	TAGS: []
    pattern: [u'windows']
    hosts (1):
      radon

  play #2 (rhel): remove OracleJDK on RHEL	TAGS: []
    pattern: [u'rhel']
    hosts (2):
      neon
      helium
...

This works also together with the -l option and might help debugging your inventory.

Again, no tasks are actually execute when the list of hosts is queried.

What’s going on here? – List tasks

Another thing which can get pretty complicated is the list of tasks actually executed: think of complex playbooks including other complex playbooks. That can get pretty complex and difficult to understand – here the option --list-tasks comes in handy. It lists what will be done, showing the names of the tasks but not executing any of them on the target nodes:

$ ansible-playbook --list-tasks oraclejdk-destroy.yml

playbook: oraclejdk-destroy.yml

  play #1 (windows): remove OracleJDK on Windows	TAGS: []
    tasks:
      copy Java remove script to temp	TAGS: []
      remove application	TAGS: []
      remove temp dir in Windows	TAGS: []

  play #2 (rhel): remove OracleJDK on RHEL	TAGS: []
    tasks:
      remove java dir	TAGS: []
...

What’s that thing? – List all tags

Besides all tasks, the used tags can be listed as well.

$ ansible-playbook --list-tags setup-control.yml

playbook: setup-control.yml

  play #1 (tuzak): 	TAGS: []
      TASK TAGS: [base_setup, db, imap, ldap, mail, oc, smtp]
...

Again, this option helps providing an overview what a playbook has to offer, how to use it. And again this option does not execute any task on the target node.

Are you sure? – Running in test mode

Ansible provides a so called check mode, also called dry run mode (in Tower for example). Invoked via --check the check mode does not alter the target nodes, but tries to output what would change and what not. Note however that this needs to be supported by the used modules, and not all modules support this.

For example, the following listing shows several tasks not supporting the dry run, which is indicated by the “skipping” line.

$ ansible-playbook --check oraclejdk-setup.yml

PLAY [set up OracleJDK on Windows] *********************************************

TASK [setup] *******************************************************************
ok: [radon]

TASK [set up temp dir in Windows] **********************************************
skipping: [radon]

TASK [copy JDK to Windows client] **********************************************
skipping: [radon]

TASK [run exe installer] *******************************************************
skipping: [radon]
...
PLAY [set up OracleJDK on RHEL] ************************************************

TASK [setup] *******************************************************************
ok: [helium]
ok: [neon]

TASK [copy JDK to RHEL client] *************************************************
skipping: [helium]
skipping: [neon]
....

This is quite useful to get an idea what impact the run of a playbook might have on target nodes. The lack of support in several modules dampens the positive effect a bit, though.

But since the --diff option (see below) supports it, it can be quite handy in certain situations.

Let me have a look at that… – Going through tasks step by step

Imagine that a playbook runs without errors, but somehow the result is not what exactly what was expected. In such cases one way to debug everything is to go through each task at a time, step by step, checking the state of all involved components after each task. This can be done with the option --step.

$ ansible-playbook --step oraclejdk-setup.yml

PLAY [set up OracleJDK on Windows] *********************************************
Perform task: TASK: setup (y/n/c): y

Perform task: TASK: setup (y/n/c): *********************************************

TASK [setup] *******************************************************************
ok: [radon]
Perform task: TASK: set up temp dir in Windows (y/n/c): y

Perform task: TASK: set up temp dir in Windows (y/n/c): ************************

TASK [set up temp dir in Windows] **********************************************
changed: [radon]
Perform task: TASK: copy JDK to Windows client (y/n/c): 
...

This is incredibly helpful on complex setups involving multiple nodes.

And yes, this time the tasks are actually executed on the target node!

Get me right there! – Starting playbooks in the middle

During debugging and development it might make sense to start playbooks not at the beginning, but somewhere in between. For example, because a playbook failed at task 14, and you don’t want to go through the first 13 tasks again. Starting at a given task requires the appropriate name of the task – and the option --start-at-task:

$ ansible-playbook --start-at-task="run exe installer" oraclejdk-setup.yml

PLAY [set up OracleJDK on Windows] *********************************************

TASK [setup] *******************************************************************
ok: [radon]

TASK [run exe installer] *******************************************************
ok: [radon]
...

In this example, the two tasks “set up temp dir in Windows” and “copy JDK to Windows client” are skipped, and the playbook starts directly at “run exe installer”. Note that skipped tasks are not shown or listed at all, and that the setup is run nevertheless.

As shown above, the proper name of each task is listed with the --list-tasks option.

Get down to business! – Showing diffs

Ansible is often used to deploy files, especially using templates. Usually, when a file is changed, Ansible just highlights that a change occurred – but not what was actually changed. In such cases, the option --diff comes in handy: it shows the diff in typical patch form:

$ ansible-playbook --diff examples/template.yml

PLAY [template example] ********************************************************

TASK [setup] *******************************************************************
ok: [helium]

TASK [copy template] ***********************************************************
changed: [helium]
--- before: /tmp/template.conf
+++ after: dynamically generated
@@ -1,2 +1,3 @@
 hostname: ansible-demo-helium
-bumble: bee
+foo: bar
+MX: 10 mx2.redhat.com.,5 mx1.redhat.com.

PLAY RECAP *********************************************************************
helium                     : ok=2    changed=1    unreachable=0    failed=0

This can be even combined with the option --check: in such cases, the diff is printed, but the change is not performed on the target node. That’s pretty handy indeed

That was interesting! – Summary

To summarize, ansible-playbook has quite some options to help debugging playbooks. The fact that many do not alter the target nodes makes it possible to use them on productive systems as well (but with care, as always). They also help a lot when it comes to understanding unknown playbooks, for example from other departments or coworkers.