Insights into Ansible: environments of executed playbooks

Ansible LogoUsually when Ansible Tower executes a playbook everything works just as on the command line. However, in some corner cases the behavior might be different: Ansible Tower runs its playbooks in a specific environment.

Different playbook results in Tower vs CLI

Ansible is a great tool for automation, and Ansible Tower enhances these capabilities by adding centralization, a UI, role based access control and a REST API. To take advantage of Tower, just import your playbooks and press start – it just works.

At least most of the time: lately I was playing around with the Google Cloud Engine, GCE. Ansible provides several GCE modules thus writing playbooks to control the setup was pretty easy. But while GCE related playbooks worked on the plain command line, they failed in Tower:

PLAY [create node on GCE] ******************************************************

TASK [launch instance] *********************************************************
task path: /var/lib/awx/projects/_43__gitolite_gce_node_tower_pem_file/gce-node.yml:13
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 2573, in <module>
    main()
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 506, in main
    module, gce, inames)
  File "/var/lib/awx/.ansible/tmp/ansible-tmp-1461919385.95-6521356859698/gce", line 359, in create_instances
    external_ip=external_ip, ex_disk_auto_delete=disk_auto_delete, ex_service_accounts=ex_sa_perms)
TypeError: create_node() got an unexpected keyword argument 'ex_can_ip_forward'

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "gce"}, "parsed": false}

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @gce-node.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1 

To me that didn’t make sense at all: the exact same playbook was running on command line. How could that fail in Tower when Tower is only a UI to Ansible itself?

Environment variables during playbook runs

The answer is that playbooks are run by Tower within specific environment variables. For example, the GCE login credentials are provided to the playbook and thus to the modules via environment variables:

GCE_EMAIL
GCE_PROJECT
GCE_PEM_FILE_PATH

That means, if you want to debug a playbook and want to provide the login credentials just the way Tower does, the shell command has to be:

GCE_EMAIL=myuser@myproject.iam.gserviceaccount.com GCE_PROJECT=myproject GCE_PEM_FILE_PATH=/tmp/mykey.pem ansible-playbook myplaybook.yml

The error at hand was also caused by an environment variable, though: PYTHONPATH. Tower comes along with a set of Python libraries needed for Ansible. Among them some which are required by specific modules. In this case, the GCE modules require the Apache libcloud, and that one is installed with the Ansible Tower bundle. The libraries are installed at /usr/lib/python2.7/site-packages/awx/lib/site-packages – which is not a typical Python path.

For that reason, each playbook is run from within Tower with the environment variable PYTHONPATH="/usr/lib/python2.7/site-packages/awx/lib/site-packages:". Thus, to run a playbook just the same way it is run from within Tower, the shell command needs to be:

PYTHONPATH="/usr/lib/python2.7/site-packages/awx/lib/site-packages:" ansible-playbook myplaybook.yml

This way the GCE error shown above could be reproduced on the command line. So the environment provided by Tower was a problem, while the environment of plain Ansible (and thus plain Python) caused no errors. Tower does bundle the library because you cannot expect the library for example in the RHEL default repositories.

The root cause is that right now Tower still ships with an older version of the libcloud library which is not fully compatible with GCE anymore (GCE is a fast moving target). If you run Ansible on the command line you most likely install libcloud via pip or RPM which in most cases provides a pretty current version.

Workaround for Tower

While upgrading the library makes sense in the mid term, a short term workaround is needed as well. The best way is to first install a recent version of libcloud and second identify the actual task which fails and point that exact task to the new library.

In case of RHEL, enable the EPEL repository, install python-libcloud and then add the environment path PYTHONPATH: "/usr/lib/python2.7/site-packages" to the task via the environment option.

- name: launch instance
  gce:
    name: "{{ node_name }}"
    zone: europe-west1-c
    machine_type: "{{ machine_type }}"
    image: "{{ image }}"
  environment:
    PYTHONPATH: "/usr/lib/python2.7/site-packages"
Advertisements

[Howto] Access Red Hat Satellite REST API via Ansible [Update]

Ansible LogoAs with all tools, Red Hat Satellite offers a REST API. Ansible offers a simple way to access the API.

Background

Most of the programs and functions developed these days offer a REST API. Red Hat for example usually follows the “API first” methodology with most of the products these days, thus all functions of all programs can be accessed via REST API calls. For example I covered how to access the CloudForms REST API via Python.

While exploring a REST API via Python teaches a lot about the API and how to deal with all the basic tasks around REST communication, in daily enterprise business API calls should be automated – say hello to Ansible.

Ansible offers the URI module to work with generic HTTP requests. It offers various authentication modules, can pass general headers and provides ways to deal with different return codes and has a generic body field. Together with Ansible’s extensive variable features this makes the ideal combination for automated REST queries.

Setup

The setup is fairly simple: a Red Hat Satellite Server in a newer version (6.1 or newer), Ansible, and that’s it. The URI module in Satellite comes pre-installed.

Since the URI module accesses the target hosts via http, the actual host executing the http commands is the host on which the playbooks run. As a result, the host definition in the playbook needs to be localhost. In such case it doesn’t make sense to gather facts, either, so gather_facts: no can be set to save time.

In the module definition itself, it might make sense for test environments to ignore certification errors if the Satellite server certificate is not properly signed: validate_certs: no. Also, sometimes the Python library stumbles upon the status code 401 to initiate authentication. In that case, the option force_basic_auth: yes might help.

Last but not least, the API itself must be understood. The appropriate documentation is pretty helpful here: Red Hat Satellite API Guide. Especially the numerious examples at the end are a good start to build own REST calls in Ansible.

Getting values

Getting values via the REST API is pretty easy – the usual URL needs to be queried, the result is provided as JSON (in this case). The following example playbook asks the Satellite for the information about a given host. The output is reduced to the puppet modules, the number of modules is counted and the result is printed out.

$ cat api-get.yml
---
- name: call API from Satellite
  hosts: localhost
  gather_facts: no
  vars:
    satelliteurl: satellite-server.example.com
    client: helium.example.com

  tasks:
    - name: get modules for given host from satellite 
      uri:
        url: https://{{ satelliteurl }}/api/v2/hosts/{{ client }}
        method: GET 
        user: admin
        password: password
        force_basic_auth: yes 
        validate_certs: no
      register: restdata
    - name: output rest data
      debug: msg="{{ restdata.json.all_puppetclasses | count }}" 

The execution of the playbook show the number of the installed Puppet modules:

$ ansible-playbook api-get.yml

PLAY [call API from Satellite] ************************************************ 

TASK: [get ip and name from satellite] **************************************** 
ok: [localhost]

TASK: [output rest data] ****************************************************** 
ok: [localhost] => {
    "msg": "8"
}

PLAY RECAP ******************************************************************** 
localhost                  : ok=2    changed=0    unreachable=0    failed=0

If the Jinja filter string | count is removed, the actual Puppet classes are listed.

Performing searches

Performing searches is simply another URL, and thus works the exact same way. The following playbook shows a search for all servers which are part of a given Puppet environment:

---
- name: call API from Satellite
  hosts: localhost
  gather_facts: no
  vars:
    satelliteurl: satellite-server.example.com
    clientenvironment: production

  tasks:
    - name: get Puppet environment from Satellite 
      uri:
        url: https://{{ satelliteurl }}/api/v2/hosts/?search=environment={{ clientenvironment }}
        method: GET 
        user: admin
        password: password
        force_basic_auth: yes 
        validate_certs: no
      register: restdata
    - name: output rest data
      debug: msg="{{ restdata.json }}"

Changing configuration: POST

While querying the REST API can already be pretty interesting, automation requires the ability to change values as well. This can be done by changing the method: in the playbook to POST. Also, additional headers are necessary, and a body defining what data will be posted to Satellite.

The following example implements the example CURL call from the API guide mentioned above to add another architecture to Satellite:

$ cat api-post.yml
---
- name: call API from Satellite
  hosts: localhost
  gather_facts: no
  vars:
    satelliteurl: satellite-server.example.com

  tasks:
    - name: set additional architecture in Satellite 
      uri:
        url: https://{{ satelliteurl }}/api/architectures
        method: POST
        user: admin
        password: password
        force_basic_auth: yes 
        validate_certs: no
        HEADER_Content-Type: application/json
        HEADER_Accept: :application/json,version=2
        body: >
          {"architecture":{"name":"i686"}}
      register: restdata
    - name: output rest data
      debug: msg="{{ restdata }}"

The result can be looked up in the web interface: an architecture of the type i686 can now be found.

Update
Note that the body: > notation, folded scalars, makes it much easier to paste payload. If you are providing the payload without the closing bracket but on the same line, all the quotation marks need to be escaped:

body: "{\"architecture\":{\"name\":\"i686\"}}"

Conclusion

Ansible can easily access, query and control the Red Hat Satellite REST API and thus other REST APIs out there as well.

Ansible offers the possibility to automate almost any tool which expose a REST API. Together with the dynamic variable system results from one tool can easily be re-used to perform actions in another tool. That way even complex setups can be integrated with each other via Ansible rather easy.

[Howto] Look up of external sources in Ansible

Ansible Logo Part of Ansible’s power comes from an easy integration with other systems. In this post I will cover how to look up data from external sources like DNS or Redis.

Background

A tool for automation is only as good as it is capable to integrate it with the already existing environment – thus with other tools. Among various ways Ansible offers the possibility to look up Ansible variables from external stores like DNS, Redis, etcd or even generic INI or CSV files. This enables Ansible to easily access data which are stored – and changed, managed – outside of Ansible.

Setup

Ansible’s lookup feature is already installed by default.

Queries are executed on the host where the playbook is executed – in case of Tower this would be the Tower host itself. Thus the node needs access to the resources which needs to be queried.

Some lookup functions for example for DNS or Redis servers require additional python libraries – on the host actually executing the queries! On Fedora, the python-dns package is necessary for DNS queries and the package python-redis for Redis queries.

Generic usage

The lookup function can be used the exact same way variables are used: curly brackets surround the lookup function, the result is placed where the variable would be. That means lookup functions can be used in the head of a playbook, inside the tasks, even in templates.

The lookup command itself has to list the plugin as well as the arguments for the plugin:

{{ lookup('plugin','arguments') }}

Examples

Files

Entire files can be used as content of a variable. This is simply done via:

vars:
  content: "{{ lookup('file','lorem.txt') }}"

As a result, the variable has the entire content of the file. Note that the lookup of files always searches the files relative to the path of the actual playbook, not relative to the path where the command is executed.

Also, the lookup might fail when the file itself contains quote characters.

CVS

While the file lookup is pretty simple and generic, the CVS lookup module gives the ability to access values of given keys in a CVS file. An optional parameter can identify the appropriate column. For example, if the following CSV file is given:

$ cat gamma.csv
daytime,time,meal
breakfast,7,soup
lunch,12,rice
tea,15,cake
dinner,18,noodles

Now the lookup function for CVS files can access the lines identified by keys which are compared to the values of the first column. The following example looks up the key dinner and gives back the entry of the third column: {{ lookup('csvfile','dinner file=gamma.csv delimiter=, col=2') }}.

Inserted in a playbook, this looks like:

ansible-playbook examples/lookup.yml

PLAY [demo lookups] *********************************************************** 

GATHERING FACTS *************************************************************** 
ok: [neon]

TASK: [lookup of a cvs file] ************************************************** 
ok: [neon] => {
    "msg": "noodles"
}

PLAY RECAP ******************************************************************** 
neon                       : ok=2    changed=0    unreachable=0    failed=0

The corresponding playbook gives out the variable via the debug module:

---
- name: demo lookups
  hosts: neon

  tasks:
    - name: lookup of a cvs file
      debug: msg="{{ lookup('csvfile','dinner file=gamma.csv delimiter=, col=2') }}"

DNS

The DNS lookup is particularly interesting in cases where the local DNS provides a lot of information like SSH fingerprints or the MX record.

The DNS lookup plugin is called dig – like the command line client dig. As arguments, the plugin takes a domain name and the DNS type: {{ lookup('dig', 'redhat.com. qtype=MX') }}. Another way to hand over the type argument is via slash: {{ lookup('dig', 'redhat.com./MX') }}

The result for this example is:

TASK: [lookup of dns dig entries] ********************************************* 
ok: [neon] => {
    "msg": "10 int-mx.corp.redhat.com."
}

Redis

It gets even more interesting when existing databases are queried. Ansible lookup supports for example Redis databases. The plugin takes as argument the entire URL: redis://$URL:$PORT,$KEY.

For example, to query a local Redis server for the key dinner:

---
tasks:
  - name: lookup of redis entries
    debug: msg="{{ lookup('redis_kv', 'redis://localhost:6379,dinner') }}" 

The result is:

TASK: [lookup of redis entries] *********************************************** 
ok: [neon] => {
    "msg": "noodles"
}

Template

As already mentioned, lookups can not only be used in Playbooks, but also directly in templates. For example, given the template code:

$ cat templatej2
...
Red Hat MX: {{ lookup('dig', 'redhat.com./MX') }}
$ cat template.conf
...
Red Hat MX: 10 mx2.redhat.com.,5 mx1.redhat.com.

Conclusion

As shown the lookup plugin of Ansible provides many possibilities to integrate Ansible with existing tools and environments which already contain valuable data about the systems. It is easy to use, integrates well with the existing Ansible concepts and can quickly be integrated. Just drop it where a variable would be dropped, and it already works.

I am looking forward to more lookup modules support in the future – I’d love to see a generic “http” and a generic “SQL” plugin, even with the ability to provide credentials, although these features can be somewhat realized with already existing modules.

[Howto] Introduction to Ansible variables

Ansible Logo To become more flexible, Ansible offers the possibility to use variables in loops, but also to use information the target system provides.

Background

Ansible uses variables to enable more flexibility in playbooks and roles. They can be used to loop through a set of given values, access various information like the hostname of a system and replace certain strings in templates by system specific values. Variables are provided through the inventory, by variable files, overwritten on the command line and set in Tower.

But note that variable names have some restrictions in Ansible:

Variable names should be letters, numbers, and underscores. Variables should always start with a letter.

If the variable in question is a dictionary, a key/value pair, it can be referenced both by bracket and dot notation:

foo.bar
foo['bar']

The question is however: how to use variables? And how to get them in the first place?

Variables and loops

Loops are arguably among the most common use cases of variables in general – the same is true for Ansible. While they are not using variables provided externally, I’d like to start with loops to give a first idea how to use them. To copy a set of files it is either possible to write a task for each file or to just loop through them:

tasks:
  - name: copy files
    copy: src={{ item }} dest=/tmp/{{ item }}
    with_items:
      - alha
      - beta

This already shows simple basics: variables can be used in module arguments, and they are referenced via curly brackets {{ }}.

Variables and templates

More importantly, variables can be used to substitute keys in configuration files with system or run-time specific values. Imagine a file which containing the actual host name simply copied over from a central source to the clients. In case of one hundred machines there would be one hundred copies of the configuration file on the server, all almost identical except for the corresponding host name. That is a waste of time and space.

It’s better to have one copy of the configuration file, with a variable as place-maker instead of the host names – this is where templates come into play:

$ cat template.j2
My host name is {{ ansible_hostname }}.

The Ansible module to use templates and activate variable substitution is the template module:

tasks:
  - name: copy template
    template: src=template.j2 dest="/tmp/abcapp.conf"

When this task is run against a set of systems, the all get a file called abcapp.conf containing the individual host name of the given system.

This is again a simple example, but shows the basics: in templates variables are also referenced by {{ }}. Templates can become much more sophisticated, but I will try to cover that in another post.

Using variables in conditions

Variables can be used as conditions, thus ensuring that certain tasks are only run when for example on a given host the requested variable is set to a certain value:

tasks:
  - name: install Apache on Solaris
    pkg5: name=web/server/apache-24
    when: ansible_os_family == "Solaris"

  - name: install Apache on RHEL
    yum:  name=httpd
    when: ansible_os_family == "RedHat"

In this case, the first task is only applied on Red Hat Enterprise Linux servers, while the second is only run on Solaris machines.

Getting variables from the system

Using variables is one thing, but they need to be defined first. I have covered so far some use cases of variables, but not where to get them.

Ansible already defines a rich set of variables, individual for each system: whenever Ansible is run on a system all facts and information about the system are gathered and set as a variable. The available variables can be output via the setup module:

$ ansible neon -m setup
neon | success >> {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "192.168.122.203"
        ], 
        "ansible_all_ipv6_addresses": [
            "fe80::5054:ff:feba:9db3"
        ], 
        "ansible_architecture": "x86_64", 
        "ansible_bios_date": "04/01/2014", 
...

All these variables can be used in templates, but also in the playbooks themselves as mentioned above in the conditions example. Roles can take advantage of these variables as well, of course.

Getting variables from the command line

Another way to define variables is to call Ansible playbooks with the option --extra-vars:

$ ansible-playbook --extra-vars "cli_var=production" system-setup.yml

The reference is again via the {{ }} brackets:

$ cat template.j2
environment: {{ cli_var }}.

Setting variables in playbooks

A more direct way is to define variables in playbooks directly, with the key vars:

...
hosts: all 
vars:
  play_var: bar 

tasks:
...

This can be extended by including an actual yaml file containing variables:

...
hosts: all
include_vars: setupvariables.yml

tasks:
...

Including further files with more variables comes in extremly handy when the variables for each system are kept in a system specific file, like $HOSTNAME.yml, because the included variable file can again be a variable:

...
hosts: all
include_vars: "{{ ansible_hostname }}.yml"

tasks:
...

In this case, the playbook reads in specific variables for each corresponding host – this way it is possible to set for example different virtual host names for Apache servers, or different backup systems in case of separated data centers.

Setting variables in the inventory

Sometimes it might make more sense to define specific variables in the already set up inventory:

[clients]
helium intevent_var=helium_123
neon invent_var=bar

The inventory more or less treats any argument which is not Ansible specific as a host variable.

It is also possible to set variables for entire host groups:

[clients]
helium
neon

[clients:vars]
invent_var=group-foo

Setting variables in the inventory makes sense when different teams use the same set of Ansible roles and playbooks but have different machine setups which need specific treatment at each location.

Setting variables on a system: local facts

Variables can also be set by dumping specialized files onto the system: when Ansible accesses a remote system it checks for the directory /etc/ansible/facts.d and all files ending in .fact are read. The files can be of various formats – INI, JSON, or even executables which return JSON code. This offers the possibility to add even generic fact providers via scripting.

For example:

$ cat /etc/ansible/facts.d/variables.fact 
[system]
foo=bar
dim=dum

The variables can be displayed via the setup module:

$ ansible neon -m setup
...
      "type": "loopback"
  }, 
  "ansible_local": {
      "variables": {
          "system": {
              "dim": "dum", 
              "foo": "bar"
          }
     }
  }, 
  "ansible_machine": "x86_64", 
...

Using the results of tasks: registered variables

The results of a task during a playbook run can also be stored inside a variable – together with conditionals this enables a playbook to react to the results of the given task with other tasks.

For example, given that an httpd service needs to run. If the service does not run, the entire server should be powered down immediately to ensure that no data corruption takes place. The corresponding playbook checks the httpd service, ignores errors but instead analyses the result and powers down the machine if the service cannot be started:

---
- name: register example
  hosts: all 
  sudo: yes

  tasks:
    - name: start service
      service: name=httpd state=started
      ignore_errors: True
      register: service_result

    - name: shutdown
      command: "shutdown -h +1m"
      when: service_result | failed

Another example would be to trigger a function to remove the host from the loadbalancer. Or to check if the database is available and would otherwise immediately close down the front firewall.

Accessing variables of other hosts

Last but not least it might be interesting to access the variables (read: facts) of other hosts. This can of course only be done if the facts are actually available.

If this is given, the data can be accessed via the hostvars key. The value for the key is the name given in the inventory.

Groups from within the inventory can also be accessed. That makes it possible to loop through a list of hosts in a group and for example gather all IP addresses or host names or other data.

The following template first accesses the host name of the machine “tower” and afterwards collects all the epoch data of all machines:

Tower name is: {{ hostvars['tower']['ansible_hostname'] }}
The epochs of the clients are:
{% for host in groups['clients'] %}
- {{ hostvars[host]['ansible_date_time']['epoch'] }}
{% endfor %}

The code for the loop is actual line statement of the Jinja2 template engine. The engine also offers if statements and other common operations.

This comes in handy in simple cases like filling the /etc/hosts, but can also empower admins to for example fill in the data of a loadbalancer or a firewall configuration.

Conclusion

Varialbes are a very powerful feature within Ansible to enrich the functionality it provides. Together with templates and the Jinja2 template and language engine the possibilities are almost endless. And sooner or later every admin will leave the simple Ansible calls and playbooks behind and start diving into variables, templates, and loops through variables of other hosts. To make system automation even easier.

[Short Tip] verify YAML in Shell via Python one-liner [Update]

python logo

Today the question came up how to verify YAML files easily. Of course, there are many very good online parser. But I was wondering if it is possible to do it simply in Bash/ZSH, using a Python one-liner. Here is the code:

$ python -c 'import yaml,sys;yaml.safe_load(sys.stdin)' < yamltest.txt

It throws an exception if the file is not a proper (aka importable) YAML file. Otherwise it just returns with a 0 exit code.

Please note that I am not sure how tolerant yaml.safe_load is. And note that PyYAML needs to be installed.

Update:
Updated to avoid cat abuse – safe the kittens! Thanks to ichor!