[Howto] Using the new Podman API

Podman is a daemonless container engine to develop, run and manage OCI containers. In a recent version the API was rewritten and now offers a REST interface as well as a docker compatible endpoint.

In case you never heard of Podman before, it is certainly worth a look. Besides offering a more secure drop-in-replacement for many docker functions, it can also manage pods and thus provides a container experience more aligned with what Kubernetes uses. It even can understand Kubernetes yaml (see podman-play-kube), easing the transition from single host container development over to fully fledged container management environments. Last but not least it is among the tools supporting newest features in the container space like cgroups v2.

Background: Podman API

Of course Podman is not perfect – due to the focus on Kubernetes yaml there is no support for docker-compose files (though alternatives exist), networking and routing based on names is not as simple as on Docker (read more about Podman container networking) and last but not least, the API was different – making it hard to migrate solutions dependent on the docker API.

This changed: recently, a new API was merged:

The new API is a simpler implementation based on HTTP/REST. We provide two basic groups of endpoints. The first one is for libpod; the second is for Docker compatibility, to ease adoption. 

New API coming for Podman

So how can I access the new API and fool around with it?

If you are familiar with Podman, or read carefully, the first question is: where is this API running if Podman is daemonless? And in fact, an API service needs to be started explicitly:

$ podman system service --timeout 5000

This starts the API on a UNIX socket. Other options, like a TCP socket or to run this without a timeout are also possible, the documentation provides examples.

How to use the Docker API endpoint

Let’s use the Docker API endpoint. To talk to a UNIX socket based REST API a recent curl (version >= 7.40) is quite helpful:

$ curl --unix-socket /$XDG_RUNTIME_DIR/podman/podman.sock http://localhost/images/json
[{"Containers":1,"Created":1583300892,"Id":"8c2e0da7c436e45be5ebf2adf26b41d13939190bd186214a4d45c30485071f9f","Labels":{"license":"MIT","name":"fedora","vendor":"Fedora Project","version":"31"},"ParentId":...

Note that here we are speaking to the rootless container, thus the unix domain socket is in the user runtime directory. Also, localhost has to be provided in the URL for very recent curl versions, otherwise it does not output anything!

The answer is a JSON listing, which is not easily readable. Simplify it with the help of Python (and silence curl info with the silent flag):

$ curl -s --unix-socket /$XDG_RUNTIME_DIR/podman/podman.sock http://localhost/containers/json|python -m json.tool
[
    {
        "Id": "4829e030ab1beb83db07dbc5e51481cb66562f57b79dd9eb3069dfcde91019ed",
        "Names": [
            "/87faf76aea6a-infra"
...

So what can you do with the API? Podman tries to recreate most of the docker API, so you can basically use the docker API documentation to see what should be possible. Note though that not all API endpoints are supported since Podman does not provide all functions Docker offers.

How to use the Podman API endpoint

As mentioned the API does provide two endpoints: the Docker endpoint, and a Podman specific endpoint. This second API is necessary for multiple reasons: first, Podman has functions which are alien to Docker and thus not part of the Docker API. The pod function is the most notable here. Another reason is that an independent API enables the Podman developers to further innovate in their own way and velocity, and to change the API when needed or wanted.

The API for Podman can be reached via curl as mentioned above. However, there are two notable differences: first, the Podman endpoint is marked via an additional “podman” string in the API URI, and second the Podman API is always versioned. To list the images as shown above, but via podman’s own API, the following call is necessary:

$ curl -s --unix-socket /$XDG_RUNTIME_DIR/podman/podman.sock http://localhost/v1.24/libpod/images/json
[{"Id":"8c2e0da7c436e45be5ebf2adf26b41d13939190bd186214a4d45c30485071f9f","RepoTags":["registry.fedoraproject.org/fedora:latest"],"Created":1583300892,"Size":199632198,"Labels":{"license":"MIT","name":"fedora","vendor":"Fedora ...

For pods, the endpoint is for example /pods instead of /images:

$ curl -s --unix-socket /$XDG_RUNTIME_DIR/podman/podman.sock http://localhost/v1.24/libpod/pods/json|python -m json.tool
[
    {
        "Cgroup": "user.slice",
        "Containers": [
            {
                "Id": "1510dca23d2d15ae8be1eeadcdbfb660cbf818a69d5780705cd6535d97a4a578",
                "Names": "wonderful_ardinghelli",
                "Status": "running"
            },
            {
                "Id": "6c05c20a42e6987ac9f78b277a9d9152ab37dd05e3bfd5ec9e675979eb93bf0e",
                "Names": "eff81a37b4b8-infra",
                "Status": "running"
            }
        ],
        "Created": "2020-04-19T21:45:17.838549003+02:00",
        "Id": "eff81a37b4b85e92916613239001cddc2ba42f3595236586f7462492be0ac5fc",
        "InfraId": "6c05c20a42e6987ac9f78b277a9d9152ab37dd05e3bfd5ec9e675979eb93bf0e",
        "Name": "testme",
        "Namespace": "",
        "Status": "Running"
    }
]

Currently there is no documentation of the API available – or at least none of the level of the current Docker API documentation. But hopefully that will change soon.

Takeaways

Podman providing a Docker API is a great step for people who are dependent on the Docker API but nevertheless want switch to Podman. But providing a unique, but simple to consume REST API for Podman itself is equally great because it makes it easy to integrate Podman processes into existing tools and frameworks.

Just don’t forget that the API is still in development!

Featured image by Magnascan from Pixabay

[Short Tip] Identify supported platforms of Ansible Galaxy

Ansible Galaxy recently got a fresh update and now has much more features worth a look. Among those are automatic quality scorings.

In a recent role upload my scoring was only 4.5. One of the problems was a “invalid platform”. I wondered which platforms are supported, and how the strings for those are, but the documentation is sparse in this regard.

However, Ansible Galaxy does feature an API to query those things. And in fact galaxy.ansible.com/api/v1/platforms/ shows the appropriate Fedora versions:

    {
        "id": 143,
        "url": "/api/v1/platforms/143/",
        "related": {},
        "summary_fields": {},
        "created": "2018-01-15T11:54:54.212531Z",
        "modified": "2018-01-15T11:54:54.212560Z",
        "name": "Fedora",
        "release": "27",
        "active": true
    },
    {
        "id": 162,
        "url": "/api/v1/platforms/162/",
        "related": {},
        "summary_fields": {},
        "created": "2018-04-30T16:35:24.066120Z",
        "modified": "2018-04-30T16:35:24.066153Z",
        "name": "Fedora",
        "release": "28",
        "active": true
    },
    {
        "id": 61,
        "url": "/api/v1/platforms/61/",
        "related": {},
        "summary_fields": {},
        "created": "2016-02-04T06:29:41.226911Z",
        "modified": "2016-02-04T06:29:41.226980Z",
        "name": "FreeBSD",
        "release": "10.0",
        "active": true
    }

So Fedora 29 is not supported right now, but there is even a bug report already.

[Howto] Writing an Ansible module for a REST API

Ansible comes along with a great set of modules. But maybe your favorite tool is not covered yet and you need to develop your own module. This guide shows you how to write an Ansible module – when you have a REST API to speak to.

Background: Ansible modules

Ansible is a great tool to automate almost everything in an IT environment. One of the huge benefits of Ansible are the so called modules: they provide a way to address automation tasks in the native language of the problem. For example, given a user needs to be created: this is usually done by calling certain commandos on the shell. In that case the automation developer has to think about which command line tool needs to be used, which parameters and options need to be provided, and the result is most likely not idempotent. And its hard t run tests (“checks”) with such an approach.

Enter Ansible user modules: with them the automation developer only has to provide the data needed for the actual problem like the user name, group name, etc. There is no need to remember the user management tool of the target platform or to look up parameters:

$ ansible server -m user -a "name=abc group=wheel" -b

Ansible comes along with hundreds of modules. But what is if your favorite task or tool is not supported by any module? You have to write your own Ansible module. If your tools support REST API, there are a few things to know which makes it much easier to get your module running fine with Ansible. These few things are outlined below.

REST APIs and Python libraries in Ansible modules

According to Wikipedia, REST is:

… the software architectural style of the World Wide Web.

In short, its a way to write, provide and access an API via usual HTTP tools and libraries (Apache web server, Curl, you name it), and it is very common in everything related to the WWW.

To access a REST API via an Ansible module, there are a few things to note. Ansible modules are usually written in Python. The library of choice to access URLs and thus REST APIs in Python is usually urllib. However, the library is not the easiest to use and there are some security topics to keep in mind when these are used. Out of these reasons alternative libraries like Python requests came up in the past and are pretty common.

However, using an external library in an Ansible module would add an extra dependency, thus the Ansible developers added their own library inside Ansible to access URLs: ansible.module_utils.urls. This one is already shipped with Ansible – the code can be found at lib/ansible/module_utils/urls.py – and it covers the shortcomings and security concerns of urllib. If you submit a module to Ansible calling REST APIs the Ansible developers usually require that you use the inbuilt library.

Unfortunately, currently the documentation on the Ansible url library is sparse at best. If you need information about it, look at other modules like the Github, Kubernetes or a10 modules. To cover that documentation gap I will try to cover the most important basics in the following lines – at least as far as I know.

Creating REST calls in an Ansible module

To access the Ansible urls library right in your modules, it needs to be imported in the same way as the basic library is imported in the module:

from ansible.module_utils.basic import *
from ansible.module_utils.urls import *

The main function call to access a URL via this library is open_url. It can take multiple parameters:

def open_url(url, data=None, headers=None, method=None, use_proxy=True,
        force=False, last_mod_time=None, timeout=10, validate_certs=True,
        url_username=None, url_password=None, http_agent=None,
force_basic_auth=False, follow_redirects='urllib2'):

The parameters in detail are:

• url: the actual URL, the communication endpoint of your REST API
• data: the payload for the URL request, for example a JSON structure
• headers: additional headers, often this includes the content-type of the data stream
• method: a URL call can be of various methods: GET, DELETE, PUT, etc.
• use_proxy: if a proxy is to be used or not
• force: force an update even if a 304 indicates that nothing has changed (I think…)
• last_mod_time: the time stamp to add to the header in case we get a 304
• timeout: set a timeout
• validate_certs: if certificates should be validated or not; important for test setups where you have self signed certificates
• url_username: the user name to authenticate
• url_password: the password for the above listed username
• http_agent: if you wnat to set the http agent
• force_basic_auth: for ce the usage of the basic authentication
• follow_redirects: determine how redirects are handled

For example, to fire a simple GET to a given source like Google most parameters are not needed and it would look like:

open_url('https://www.google.com',method="GET")

A more sophisticated example is to push actual information to a REST API. For example, if you want to search for the domain example on a Satellite server you need to change the method to PUT, add a data structure to set the actual search string ({"search":"example"}) and add a corresponding content type as header information ({'Content-Type':'application/json'}). Also, a username and password must be provided. Given we access a test system here the certification validation needs to be turned off also. The resulting string looks like this:

open_url('https://satellite-server.example.com/api/v2/domains',method="PUT",url_username="admin",url_password="abcd",data=json.dumps({"search":"example"}),force_basic_auth=True,validate_certs=False,headers={'Content-Type':'application/json'})

Beware that the data json structure needs to be processed by json.dumps. The result of the query can be formatted as json and further used as a json structure:

resp = open_url(...)
resp_json = json.loads(resp.read())

Full example

In the following example, we query a Satellite server to find a so called environment ID for two given parameters, an organization ID and an environment name. To create a REST call for this task in a module multiple, separate steps have to be done: first, create the actual URL endpoint. This usually consists of the server name as a variable and the API endpoint as the flexible part which is different in each REST call.

server_name = 'https://satellite.example.com'
api_endpoint = '/katello/api/v2/environments/'
my_url = server_name + api_endpoint

Besides the actual URL, the payload must be pieced together and the headers need to be set according to the content type of the payload – here json:

headers = {'Content-Type':'application/json'}
payload = {"organization_id":orga_id,"name":env_name}

Other content types depends on the REST API itself and on what the developer prefers. JSON is widely accepted as a good way to go for REST calls.

Next, we set the user and password and launch the call. The return data from the call are saved in a variable to analyze later on.

user = 'abc'
pwd = 'def'
resp = open_url(url_action,method="GET",headers=headers,url_username=module.params.get('user'),url_password=module.params.get('pwd'),force_basic_auth=True,data=json.dumps(payload))

Last but not least we transform the return value into a json construct, and analyze it: if the return value does not contain any data – that means the value for the key total is zero – we want the module to exit with an error. Something went wrong, and the automation administrator needs to know that. The module calls the built-in error functionmodule.fail_json. But if the total is not zero, we get out the actual environment ID we were looking for with this REST call from the beginning – it is deeply hidden in the json structure, btw.

resp_json = json.loads(resp.read())
if resp_json["total"] == 0:
    module.fail_json(msg="Environment %s not found." % env_name)
env_id = resp_json["results"][0]["id"]

Summary

It is fairly easy to write Ansible modules to access REST APIs. The most important part to know is that an internal, Ansible provided library should be used, instead of the better known urllib or requests library. Also, the actual library documentation is still pretty limited, but that gap is partially filled by the above post.

[Short Tip] Call Ansible Tower REST URI – with Ansible

It might sound strange to call the Ansible Tower API right from within Ansible itself. However, if you want to connect several playbooks with each other, or if you user Ansible Tower mainly as an API this indeed makes sense. To me this use case is interesting since it is a way to document how to access, how to use the Ansible Tower API.

The following playbook is an example to launch a job in Ansible Tower. The body payload contains an extra variable needed by the job itself. In this example it is a name of a to-be launched VM.

---
- name: POST to Tower API
  hosts: localhost
  vars:
    job_template_id: 44
    vmname: myvmname

  tasks:
    - name: create additional node in GCE
      uri:
        url: https://tower.example.com/api/v1/job_templates/{{ job_template_id }}/launch/
        method: POST
        user: admin
        password: $PASSWORD
        status_code: 202
        body: 
          extra_vars:
            node_name: "{{ vmname }}"
        body_format: json

Note the status code (202) – the URI module needs to know that a non-200 status code is used to show the proper acceptance of the API call. Also, the job is identified by its ID. But since Tower shows the ID in the web interface it is no problem to get the correct id.

Ways to provide body payload in Ansible’s URI module [2. Update]

Talin to a REST API requires to provide some information, usually in the form of JSON payload. Ansible offers various ways to do that in the URI module in playbooks.

In modern applications REST APIs are often the main API to integrate the given APP with the existing infrastructure. REST often requires posting JSON structures as payload.

Ansible offers the URI module to talk to REST APIs, and there are multiple ways add JSON payload to a playbook task that are shown below.

For example, given that the following arbitrary JSON payload needs to be provided to a REST API via POST:

{
  "mainlevel": {
    "subkey": "finalvalue"
  }
}

The first and for me preferred way to provide JSON payloads is to write down the structure in plain YAML (if possible) and afterwards tell the module to format it as JSON:

HEADER_Content-Type: application/json
status_code: 202
body: 
  mainlevel:
    subkey: finalvalue
body_format: json

Among various reasons this works well because variables can be easily used.

Another way is to define a variable and then use jinja to format it:

vars:
  mainlevel:
    "subkey": finalvalue
...
    body: ' {{mainlevel|to_json}}'

Caution: not the empty space here in the body line. It avoids type detection which tries to check if a string begins with { or [.

Sometimes the payload is the content of a file generated somewhere else. In these cases the best way is to use the lookup plugin to read the file:

body: "{{ lookup('file','myvalues.json') }}"

Of course the lookup plugin can access data from other places as well – for example from a database or a config store, which is a nice way of integrating existing infrastructure with each other via Ansible.

A quicker, shorter way is to use folded style:

body: >
  {"mainlevel":{"subkey":"finalvalue"}}

Note that folded style pastes things as they are – but ignores single new lines. So it might be difficult to add variables here.

Therefore, a little bit better suited is the literal style, indicated by the pipe:

body: |
{
  "mainlevel": {
    "subkey": "finalvalue"
  }
}

This is probably the easiest way to deal with in many debugging situations where you need to be able to quickly change thins in your code.

Last, and honestly something I would try to avoid is the plain one-liner:

body: "{\"mainlevel\":{\"subkey\":\"finalvalue\"}}

Note, all quotation marks need to be escaped which makes it hard to read, hard to maintain and easy to introduce errors.

As shown Ansible is powerful and simple. Thus there are always multiple different ways to reach the goal you are aiming for – and it depends on the requirements what solution is the best one.

For more details I can only recommend Understanding multi line strings in YAML and Ansible (Part I & Part II) from adminswerk.de.

Update:
Added how to add body payload from existing files.

2. Update:
Added details about literal style.