[Howto] Writing an Ansible module for a REST API

Ansible LogoAnsible comes along with a great set of modules. But maybe your favorite tool is not covered yet and you need to develop your own module. This guide shows you how to write an Ansible module – when you have a REST API to speak to.

Background: Ansible modules

Ansible is a great tool to automate almost everything in an IT environment. One of the huge benefits of Ansible are the so called modules: they provide a way to address automation tasks in the native language of the problem. For example, given a user needs to be created: this is usually done by calling certain commandos on the shell. In that case the automation developer has to think about which command line tool needs to be used, which parameters and options need to be provided, and the result is most likely not idempotent. And its hard t run tests (“checks”) with such an approach.

Enter Ansible user modules: with them the automation developer only has to provide the data needed for the actual problem like the user name, group name, etc. There is no need to remember the user management tool of the target platform or to look up parameters:

$ ansible server -m user -a "name=abc group=wheel" -b

Ansible comes along with hundreds of modules. But what is if your favorite task or tool is not supported by any module? You have to write your own Ansible module. If your tools support REST API, there are a few things to know which makes it much easier to get your module running fine with Ansible. These few things are outlined below.

REST APIs and Python libraries in Ansible modules

According to Wikipedia, REST is:

… the software architectural style of the World Wide Web.

In short, its a way to write, provide and access an API via usual HTTP tools and libraries (Apache web server, Curl, you name it), and it is very common in everything related to the WWW.

To access a REST API via an Ansible module, there are a few things to note. Ansible modules are usually written in Python. The library of choice to access URLs and thus REST APIs in Python is usually urllib. However, the library is not the easiest to use and there are some security topics to keep in mind when these are used. Out of these reasons alternative libraries like Python requests came up in the past and are pretty common.

However, using an external library in an Ansible module would add an extra dependency, thus the Ansible developers added their own library inside Ansible to access URLs: ansible.module_utils.urls. This one is already shipped with Ansible – the code can be found at lib/ansible/module_utils/urls.py – and it covers the shortcomings and security concerns of urllib. If you submit a module to Ansible calling REST APIs the Ansible developers usually require that you use the inbuilt library.

Unfortunately, currently the documentation on the Ansible url library is sparse at best. If you need information about it, look at other modules like the Github, Kubernetes or a10 modules. To cover that documentation gap I will try to cover the most important basics in the following lines – at least as far as I know.

Creating REST calls in an Ansible module

To access the Ansible urls library right in your modules, it needs to be imported in the same way as the basic library is imported in the module:

from ansible.module_utils.basic import *
from ansible.module_utils.urls import *

The main function call to access a URL via this library is open_url. It can take multiple parameters:

def open_url(url, data=None, headers=None, method=None, use_proxy=True,
        force=False, last_mod_time=None, timeout=10, validate_certs=True,
        url_username=None, url_password=None, http_agent=None,
force_basic_auth=False, follow_redirects='urllib2'):

The parameters in detail are:

  • url: the actual URL, the communication endpoint of your REST API
  • data: the payload for the URL request, for example a JSON structure
  • headers: additional headers, often this includes the content-type of the data stream
  • method: a URL call can be of various methods: GET, DELETE, PUT, etc.
  • use_proxy: if a proxy is to be used or not
  • force: force an update even if a 304 indicates that nothing has changed (I think…)
  • last_mod_time: the time stamp to add to the header in case we get a 304
  • timeout: set a timeout
  • validate_certs: if certificates should be validated or not; important for test setups where you have self signed certificates
  • url_username: the user name to authenticate
  • url_password: the password for the above listed username
  • http_agent: if you wnat to set the http agent
  • force_basic_auth: for ce the usage of the basic authentication
  • follow_redirects: determine how redirects are handled

For example, to fire a simple GET to a given source like Google most parameters are not needed and it would look like:

open_url('https://www.google.com',method="GET")

A more sophisticated example is to push actual information to a REST API. For example, if you want to search for the domain example on a Satellite server you need to change the method to PUT, add a data structure to set the actual search string ({"search":"example"}) and add a corresponding content type as header information ({'Content-Type':'application/json'}). Also, a username and password must be provided. Given we access a test system here the certification validation needs to be turned off also. The resulting string looks like this:

open_url('https://satellite-server.example.com/api/v2/domains',method="PUT",url_username="admin",url_password="abcd",data=json.dumps({"search":"example"}),force_basic_auth=True,validate_certs=False,headers={'Content-Type':'application/json'})

Beware that the data json structure needs to be processed by json.dumps. The result of the query can be formatted as json and further used as a json structure:

resp = open_url(...)
resp_json = json.loads(resp.read())

Full example

In the following example, we query a Satellite server to find a so called environment ID for two given parameters, an organization ID and an environment name. To create a REST call for this task in a module multiple, separate steps have to be done: first, create the actual URL endpoint. This usually consists of the server name as a variable and the API endpoint as the flexible part which is different in each REST call.

server_name = 'https://satellite.example.com'
api_endpoint = '/katello/api/v2/environments/'
my_url = server_name + api_endpoint

Besides the actual URL, the payload must be pieced together and the headers need to be set according to the content type of the payload – here json:

headers = {'Content-Type':'application/json'}
payload = {"organization_id":orga_id,"name":env_name}

Other content types depends on the REST API itself and on what the developer prefers. JSON is widely accepted as a good way to go for REST calls.

Next, we set the user and password and launch the call. The return data from the call are saved in a variable to analyze later on.

user = 'abc'
pwd = 'def'
resp = open_url(url_action,method="GET",headers=headers,url_username=module.params.get('user'),url_password=module.params.get('pwd'),force_basic_auth=True,data=json.dumps(payload))

Last but not least we transform the return value into a json construct, and analyze it: if the return value does not contain any data – that means the value for the key total is zero – we want the module to exit with an error. Something went wrong, and the automation administrator needs to know that. The module calls the built-in error functionmodule.fail_json. But if the total is not zero, we get out the actual environment ID we were looking for with this REST call from the beginning – it is deeply hidden in the json structure, btw.

resp_json = json.loads(resp.read())
if resp_json["total"] == 0:
    module.fail_json(msg="Environment %s not found." % env_name)
env_id = resp_json["results"][0]["id"]

Summary

It is fairly easy to write Ansible modules to access REST APIs. The most important part to know is that an internal, Ansible provided library should be used, instead of the better known urllib or requests library. Also, the actual library documentation is still pretty limited, but that gap is partially filled by the above post.

[Short Tip] Call Ansible Tower REST URI – with Ansible

Ansible Logo

It might sound strange to call the Ansible Tower API right from within Ansible itself. However, if you want to connect several playbooks with each other, or if you user Ansible Tower mainly as an API this indeed makes sense. To me this use case is interesting since it is a way to document how to access, how to use the Ansible Tower API.

The following playbook is an example to launch a job in Ansible Tower. The body payload contains an extra variable needed by the job itself. In this example it is a name of a to-be launched VM.

---
- name: POST to Tower API
  hosts: localhost
  vars:
    job_template_id: 44
    vmname: myvmname

  tasks:
    - name: create additional node in GCE
      uri:
        url: https://tower.example.com/api/v1/job_templates/{{ job_template_id }}/launch/
        method: POST
        user: admin
        password: $PASSWORD
        status_code: 202
        body: 
          extra_vars:
            node_name: "{{ vmname }}"
        body_format: json

Note the status code (202) – the URI module needs to know that a non-200 status code is used to show the proper acceptance of the API call. Also, the job is identified by its ID. But since Tower shows the ID in the web interface it is no problem to get the correct id.

Ways to provide body payload in Ansible’s URI module

Ansible LogoTalin to a REST API requires to provide some information, usually in the form of JSON payload. Ansible offers various ways to do that in the URI module in playbooks.

In modern applications REST APIs are often the main API to integrate the given APP with the existing infrastructure. REST often requires posting JSON structures as payload.

Ansible offers the URI module to talk to REST APIs, and there are multiple ways add JSON payload to a playbook task that are shown below.

For example, given that the following arbitrary JSON payload needs to be provided to a REST API via POST:

{
  "mainlevel": {
    "subkey": "finalvalue"
  }
}

The first and for me preferred way to provide JSON payloads is to write down the structure in plain YAML (if possible) and afterwards tell the module to format it as JSON:

HEADER_Content-Type: application/json
status_code: 202
body: 
  mainlevel:
    subkey: finalvalue
body_format: json

Among various reasons this works well because variables can be easily used.

Another way is to define a variable and then use jinja to format it:

vars:
  mainlevel:
    "subkey": finalvalue
...
    body: ' {{mainlevel|to_json}}'

Caution: not the empty space here in the body line. It avoids type detection which tries to check if a string begins with { or [.

A quicker, shorter way is to use folded style:

body: >
  {"mainlevel":{"subkey":"finalvalue"}}

However, it might be difficult to add variables here.

Last, and honestly something I would try to avoid is the plain one-liner:

body: "{\"mainlevel\":{\"subkey\":\"finalvalue\"}}

As it can be seen, all quotation marks need to be escaped which makes it hard to read, hard to maintain and easy to introduce errors.

As shown Ansible is powerful and simple. Thus there are always multiple different ways to reach the goal you are aiming for – and it depends on the requirements what solution is the best one.

[Howto] Workaround failing MongoDB on RHEL/CentOS 7

Ansible LogoMongoDB is often installed right from upstream provided repositories. In such cases with recent updates the service might fail to start via systemctl. A workaround requires some SELinux work.

Ansible Tower collects system data inside a MongoDB. Since MongoDB is not part of RHEL/CentOS, it is installed directly form the upstream MongoDB repositories. However, with recent versions of MongoDB the database might not come up via systemctl:

[root@ansible-demo-tower init.d]# systemctl start mongod
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[root@ansible-demo-tower init.d]# journalctl -xe
May 03 08:26:00 ansible-demo-tower systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....
-- Subject: Unit mongod.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mongod.service has begun starting up.
May 03 08:26:00 ansible-demo-tower runuser[7266]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
May 03 08:26:00 ansible-demo-tower runuser[7266]: pam_unix(runuser:session): session closed for user mongod
May 03 08:26:00 ansible-demo-tower mongod[7259]: Starting mongod: [FAILED]
May 03 08:26:00 ansible-demo-tower systemd[1]: mongod.service: control process exited, code=exited status=1
May 03 08:26:00 ansible-demo-tower systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mongod.service has failed.
-- 
-- The result is failed.
May 03 08:26:00 ansible-demo-tower systemd[1]: Unit mongod.service entered failed state.
May 03 08:26:00 ansible-demo-tower systemd[1]: mongod.service failed.
May 03 08:26:00 ansible-demo-tower polkitd[11436]: Unregistered Authentication Agent for unix-process:7254:1405622 (system bus name :1.184, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_

The root cause of the problem is that the MongoDB developers do not provide a proper SELinux</a configuration with their packages, see the corresponding bug report.

A short workaround is to create a proper (more or less) SELinux rule and install it to the system:

[root@ansible-demo-tower ~]# grep mongod /var/log/audit/audit.log | audit2allow -m mongod > mongod.te
[root@ansible-demo-tower ~]# cat mongod.te 

module mongod 1.0;

require {
	type locale_t;
	type mongod_t;
	type ld_so_cache_t;
	class file execute;
}

#============= mongod_t ==============
allow mongod_t ld_so_cache_t:file execute;
allow mongod_t locale_t:file execute;
[root@ansible-demo-tower ~]# grep mongod /var/log/audit/audit.log | audit2allow -M mongod
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i mongod.pp

[root@ansible-demo-tower ~]# semodule -i mongod.pp 
[root@ansible-demo-tower ~]# sudo service mongod start
                                                           [  OK  ]

Keep in mind that audit2allow generated rule sets are not to be used on production systems. The generated SELinux rules need to be analyzed manually to verify that it covers nothing but the problematic use case.

Insights into Ansible: environments of executed playbooks

Ansible LogoUsually when Ansible Tower executes a playbook everything works just as on the command line. However, in some corner cases the behavior might be different: Ansible Tower runs its playbooks in a specific environment.

Different playbook results in Tower vs CLI

Ansible is a great tool for automation, and Ansible Tower enhances these capabilities by adding centralization, a UI, role based access control and a REST API. To take advantage of Tower, just import your playbooks and press start – it just works.

At least most of the time: lately I was playing around with the Google Cloud Engine, GCE. Ansible provides several GCE modules thus writing playbooks to control the setup was pretty easy. But while GCE related playbooks worked on the plain command line, they failed in Tower:

PLAY [create node on GCE] ******************************************************

TASK [launch instance] *********************************************************
task path: /var/lib/awx/projects/_43__gitolite_gce_node_tower_pem_file/gce-node.yml:13
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 2573, in <module>
    main()
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 506, in main
    module, gce, inames)
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 359, in create_instances
    external_ip=external_ip, ex_disk_auto_delete=disk_auto_delete, ex_service_accounts=ex_sa_perms)
TypeError: create_node() got an unexpected keyword argument 'ex_can_ip_forward'

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "gce"}, "parsed": false}

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @gce-node.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1 

To me that didn’t make sense at all: the exact same playbook was running on command line. How could that fail in Tower when Tower is only a UI to Ansible itself?

Environment variables during playbook runs

The answer is that playbooks are run by Tower within specific environment variables. For example, the GCE login credentials are provided to the playbook and thus to the modules via environment variables:

GCE_EMAIL
GCE_PROJECT
GCE_PEM_FILE_PATH

That means, if you want to debug a playbook and want to provide the login credentials just the way Tower does, the shell command has to be:

GCE_EMAIL=myuser@myproject.iam.gserviceaccount.com GCE_PROJECT=myproject GCE_PEM_FILE_PATH=/tmp/mykey.pem ansible-playbook myplaybook.yml

The error at hand was also caused by an environment variable, though: PYTHONPATH. Tower comes along with a set of Python libraries needed for Ansible. Among them some which are required by specific modules. In this case, the GCE modules require the Apache libcloud, and that one is installed with the Ansible Tower bundle. The libraries are installed at /usr/lib/python2.7/site-packages/awx/lib/site-packages – which is not a typical Python path.

For that reason, each playbook is run from within Tower with the environment variable PYTHONPATH="/usr/lib/python2.7/site-packages/awx/lib/site-packages:". Thus, to run a playbook just the same way it is run from within Tower, the shell command needs to be:

PYTHONPATH="/usr/lib/python2.7/site-packages/awx/lib/site-packages:" ansible-playbook myplaybook.yml

This way the GCE error shown above could be reproduced on the command line. So the environment provided by Tower was a problem, while the environment of plain Ansible (and thus plain Python) caused no errors. Tower does bundle the library because you cannot expect the library for example in the RHEL default repositories.

The root cause is that right now Tower still ships with an older version of the libcloud library which is not fully compatible with GCE anymore (GCE is a fast moving target). If you run Ansible on the command line you most likely install libcloud via pip or RPM which in most cases provides a pretty current version.

Workaround for Tower

While upgrading the library makes sense in the mid term, a short term workaround is needed as well. The best way is to first install a recent version of libcloud and second identify the actual task which fails and point that exact task to the new library.

In case of RHEL, enable the EPEL repository, install python-libcloud and then add the environment path PYTHONPATH: "/usr/lib/python2.7/site-packages" to the task via the environment option.

- name: launch instance
  gce:
    name: "{{ node_name }}"
    zone: europe-west1-c
    machine_type: "{{ machine_type }}"
    image: "{{ image }}"
  environment:
    PYTHONPATH: "/usr/lib/python2.7/site-packages"

[Short Tip] Query all registered repositories in Red Hat Satellite

redhat

The idea of RESTful APIs is pretty appealing: using the basic components of the WWW as APIs to bring together services. Operations like HTTP GET and POST, base URIs and media types like JSON are supported almost everywhere simply because the web is supported almost everywhere, it is pretty easy to provide REST enabled servers, services and clients with a few clicks and calls. For this reason the API of Red Hat Satellite – and most of the other Red Hat products – is built as REST API.

I’ve already written an article about how to access the Satellite REST API via Ansible. Today I came across a rather handy example: sometimes you need to know the URLs of the Satellite provided repos. This can of course be queried via the API. But in contrast to my old article, we do not query the Foreman part of the api ($SATELLITE_URL/api/) but the Katello part: /katello/api/.

All repositories can be shown via the URL /katello/api/repositories?organization_id=1. To query URLs on the command line I recommend Ansible:

$ ansible localhost -m uri -a "method=GET user=admin password=$PASSWORD force_basic_auth=yes validate_certs=no url=https://satellite-server.example.com/katello/api/repositories?organization_id=1&full_results=true"
localhost | SUCCESS => {
    "apipie_checksum": "7cd3aad709af2f1ae18a3daa0915d712", 
    "cache_control": "must-revalidate, private, max-age=0", 
    "changed": false,
...
    "id": 45, 
    "label": "EPEL_7_-_x86_64", 
...
    "product": {
      "cp_id": "1452001252604", 
      "id": 127, 
      "name": "EPEL", 
      "sync_plan": [
        "name", 
        "description", 
        "sync_date", 
        "interval", 
        "next_sync"
      ]
    }, 
    "relative_path": "Platin/Library/custom/EPEL/EPEL_7_-_x86_64", 
    "url": "http://dl.fedoraproject.org/pub/epel/7/x86_64/"
...

The option full_results just ensures that the entire result is shown even if it is pretty long. Note that the product I can be used to query the entire product information:

$ ansible localhost -m uri -a "method=GET user=admin password=$PASSWORD force_basic_auth=yes validate_certs=no url=https://satellite-server.example.com/katello/api/products/127"
localhost | SUCCESS => {
...
  "id": 127, 
  "label": "EPEL", 
  "last_sync": "2016-01-05 13:43:38 UTC", 
  "last_sync_words": "about 1 month", 
  "name": "EPEL", 
  "organization": {
...

The id of the repository can be used to query the full repository information, including a full repo path:

$ ansible localhost -m uri -a "method=GET user=admin password=$PASSWORD force_basic_auth=yes validate_certs=no url=https://satellite-server.example.com/katello/api/repositories/45"      
localhost | SUCCESS => {
...
  "content_type": "yum", 
  "full_path": "http://satellite-server.example.com/pulp/repos/Platin/Library/custom/EPEL/EPEL_7_-_x86_64",
...

If you want to skip the part figuring out the IDs manually but have a name you could search for, it is possible to filter the results. The search URL for this case would be: /katello/api/repositories?organization_id=1&full_results=true&search=*EPEL*" as shown in the following example:

$ ansible localhost -m uri -a "method=GET user=admin password=$PASSWORD force_basic_auth=yes validate_certs=no url=https://satellite-server.example.com/katello/api/repositories?organization_id=1&full_results=true&search=*EPEL*"
localhost | SUCCESS => {
...
  "relative_path": "Platin/Library/custom/EPEL/EPEL_7_-_x86_64", 
...

Useful command line options for ansible-playbook

Ansible LogoAnsible provides quite some useful command line options. Most of them are especially interesting during debugging.

Background

There are three major ways to work with Ansible:

  • launching single tasks with the ansible command
  • executing playbooks viaansible-playbook
  • using Tower to manage and run playbooks

While Tower might be the better option to run Ansible in the day-to-day business, and the ansible CLI itself is most likely only in one-time runs used, the executing of playbooks on the command line often happens during the development of playbooks, when no Tower is available – or during debugging. In such cases, there are quite some useful command line options which might not even be known to the seasoned Ansible user.

Do I say this right? – Syntax checking

Playbooks are written in YAML, and in YAML syntax is crucial – especially indentation:

Data structure hierarchy is maintained by outline indentation.

To check if a playbook is correctly formatted, the option --syntax-check looks at all involved playbooks and verifies the correct syntax. During a syntax check, no playbooks are actually executed.

$ ansible-playbook --syntax-check oraclejdk-destroy.yml
ERROR! Syntax Error while loading YAML.

The error appears to have been in '/home/liquidat/Gits/github/ansible-demo-oraclejdk/oracle-windows-destroy.yml': line 10, column 11,
but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  win_template: src=data/remove-program.j2 dest=C:\\temp\\remove-program.ps1
    - name: remove application
          ^ here

The syntax check helps if a playbook fails for no apparent reason – or if a playbook was edited a lot and it is simply not sure if everything was moved around correctly.

Whom am I talking to? – Listing affected hosts

With complex playbooks and dynamic inventories it sometimes is hard to say against which hosts a playbook will actually be executed. In such cases, the option --list-hosts will output a list of affected hosts, including the name of the actual play and the pattern with which the hosts were chosen:

$ ansible-playbook --list-hosts oraclejdk-destroy.yml

playbook: oraclejdk-destroy.yml

  play #1 (windows): remove OracleJDK on Windows	TAGS: []
    pattern: [u'windows']
    hosts (1):
      radon

  play #2 (rhel): remove OracleJDK on RHEL	TAGS: []
    pattern: [u'rhel']
    hosts (2):
      neon
      helium
...

This works also together with the -l option and might help debugging your inventory.

Again, no tasks are actually execute when the list of hosts is queried.

What’s going on here? – List tasks

Another thing which can get pretty complicated is the list of tasks actually executed: think of complex playbooks including other complex playbooks. That can get pretty complex and difficult to understand – here the option --list-tasks comes in handy. It lists what will be done, showing the names of the tasks but not executing any of them on the target nodes:

$ ansible-playbook --list-tasks oraclejdk-destroy.yml

playbook: oraclejdk-destroy.yml

  play #1 (windows): remove OracleJDK on Windows	TAGS: []
    tasks:
      copy Java remove script to temp	TAGS: []
      remove application	TAGS: []
      remove temp dir in Windows	TAGS: []

  play #2 (rhel): remove OracleJDK on RHEL	TAGS: []
    tasks:
      remove java dir	TAGS: []
...

What’s that thing? – List all tags

Besides all tasks, the used tags can be listed as well.

$ ansible-playbook --list-tags setup-control.yml

playbook: setup-control.yml

  play #1 (tuzak): 	TAGS: []
      TASK TAGS: [base_setup, db, imap, ldap, mail, oc, smtp]
...

Again, this option helps providing an overview what a playbook has to offer, how to use it. And again this option does not execute any task on the target node.

Are you sure? – Running in test mode

Ansible provides a so called check mode, also called dry run mode (in Tower for example). Invoked via --check the check mode does not alter the target nodes, but tries to output what would change and what not. Note however that this needs to be supported by the used modules, and not all modules support this.

For example, the following listing shows several tasks not supporting the dry run, which is indicated by the “skipping” line.

$ ansible-playbook --check oraclejdk-setup.yml

PLAY [set up OracleJDK on Windows] *********************************************

TASK [setup] *******************************************************************
ok: [radon]

TASK [set up temp dir in Windows] **********************************************
skipping: [radon]

TASK [copy JDK to Windows client] **********************************************
skipping: [radon]

TASK [run exe installer] *******************************************************
skipping: [radon]
...
PLAY [set up OracleJDK on RHEL] ************************************************

TASK [setup] *******************************************************************
ok: [helium]
ok: [neon]

TASK [copy JDK to RHEL client] *************************************************
skipping: [helium]
skipping: [neon]
....

This is quite useful to get an idea what impact the run of a playbook might have on target nodes. The lack of support in several modules dampens the positive effect a bit, though.

But since the --diff option (see below) supports it, it can be quite handy in certain situations.

Let me have a look at that… – Going through tasks step by step

Imagine that a playbook runs without errors, but somehow the result is not what exactly what was expected. In such cases one way to debug everything is to go through each task at a time, step by step, checking the state of all involved components after each task. This can be done with the option --step.

$ ansible-playbook --step oraclejdk-setup.yml

PLAY [set up OracleJDK on Windows] *********************************************
Perform task: TASK: setup (y/n/c): y

Perform task: TASK: setup (y/n/c): *********************************************

TASK [setup] *******************************************************************
ok: [radon]
Perform task: TASK: set up temp dir in Windows (y/n/c): y

Perform task: TASK: set up temp dir in Windows (y/n/c): ************************

TASK [set up temp dir in Windows] **********************************************
changed: [radon]
Perform task: TASK: copy JDK to Windows client (y/n/c): 
...

This is incredibly helpful on complex setups involving multiple nodes.

And yes, this time the tasks are actually executed on the target node!

Get me right there! – Starting playbooks in the middle

During debugging and development it might make sense to start playbooks not at the beginning, but somewhere in between. For example, because a playbook failed at task 14, and you don’t want to go through the first 13 tasks again. Starting at a given task requires the appropriate name of the task – and the option --start-at-task:

$ ansible-playbook --start-at-task="run exe installer" oraclejdk-setup.yml

PLAY [set up OracleJDK on Windows] *********************************************

TASK [setup] *******************************************************************
ok: [radon]

TASK [run exe installer] *******************************************************
ok: [radon]
...

In this example, the two tasks “set up temp dir in Windows” and “copy JDK to Windows client” are skipped, and the playbook starts directly at “run exe installer”. Note that skipped tasks are not shown or listed at all, and that the setup is run nevertheless.

As shown above, the proper name of each task is listed with the --list-tasks option.

Get down to business! – Showing diffs

Ansible is often used to deploy files, especially using templates. Usually, when a file is changed, Ansible just highlights that a change occurred – but not what was actually changed. In such cases, the option --diff comes in handy: it shows the diff in typical patch form:

$ ansible-playbook --diff examples/template.yml

PLAY [template example] ********************************************************

TASK [setup] *******************************************************************
ok: [helium]

TASK [copy template] ***********************************************************
changed: [helium]
--- before: /tmp/template.conf
+++ after: dynamically generated
@@ -1,2 +1,3 @@
 hostname: ansible-demo-helium
-bumble: bee
+foo: bar
+MX: 10 mx2.redhat.com.,5 mx1.redhat.com.

PLAY RECAP *********************************************************************
helium                     : ok=2    changed=1    unreachable=0    failed=0

This can be even combined with the option --check: in such cases, the diff is printed, but the change is not performed on the target node. That’s pretty handy indeed

That was interesting! – Summary

To summarize, ansible-playbook has quite some options to help debugging playbooks. The fact that many do not alter the target nodes makes it possible to use them on productive systems as well (but with care, as always). They also help a lot when it comes to understanding unknown playbooks, for example from other departments or coworkers.