[Short Tip] Handling “can’t concat str to bytes” error in Ansible’s uri module

When working with web services, especially REST APIs, Ansible can be of surprising help when you need to automate those, our want to integrate them into your automation.

However, today I run into a strange Python bug while I tried to use the uri module:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: can't concat str to bytes
fatal: [localhost]: FAILED! => {"changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [200]: An unknown error occurred: can't concat str to bytes", "redirected": false, "status": -1, "url": "https://www.ansible.com"}

This drove me almost nuts because it happened on all kinds of machines I tested, even with Ansible’s devel upstream version. It was even independent of the service I targeted – the error happened way earlier. And a playbook to showcase this was suspiciously short and simple:

---                                 
- name: Show concat str byte error  
  hosts: localhost                  
  connection: local                 
  gather_facts: no                  
                                    
  tasks:                            
    - name: call problematic URL call
      uri:                          
        url: "https://www.ansible.com"
        method: POST                
        body:                       
          name: "myngfw"

When I was about to fill an issue at Ansible’s Github page I thought again and wondered that this is too simple: I couldn’t imagine that I was the only one hitting this problem. I realized that the error had to be on my side. And thus meant that something was missing.

And indeed: the body_format option was not explicitly stated, so Ansible assumed “raw”, while my body data were provided in json format. A simple

        body_format: json

solved my problems.

Never dare to ask me how long it took me to figure this one out. And that from the person who write and entire how to about how to provide payload with the Ansible URI module….

Ways to provide body payload in Ansible’s URI module [2. Update]

Talin to a REST API requires to provide some information, usually in the form of JSON payload. Ansible offers various ways to do that in the URI module in playbooks.

In modern applications REST APIs are often the main API to integrate the given APP with the existing infrastructure. REST often requires posting JSON structures as payload.

Ansible offers the URI module to talk to REST APIs, and there are multiple ways add JSON payload to a playbook task that are shown below.

For example, given that the following arbitrary JSON payload needs to be provided to a REST API via POST:

{
  "mainlevel": {
    "subkey": "finalvalue"
  }
}

The first and for me preferred way to provide JSON payloads is to write down the structure in plain YAML (if possible) and afterwards tell the module to format it as JSON:

HEADER_Content-Type: application/json
status_code: 202
body: 
  mainlevel:
    subkey: finalvalue
body_format: json

Among various reasons this works well because variables can be easily used.

Another way is to define a variable and then use jinja to format it:

vars:
  mainlevel:
    "subkey": finalvalue
...
    body: ' {{mainlevel|to_json}}'

Caution: not the empty space here in the body line. It avoids type detection which tries to check if a string begins with { or [.

Sometimes the payload is the content of a file generated somewhere else. In these cases the best way is to use the lookup plugin to read the file:

body: "{{ lookup('file','myvalues.json') }}"

Of course the lookup plugin can access data from other places as well – for example from a database or a config store, which is a nice way of integrating existing infrastructure with each other via Ansible.

A quicker, shorter way is to use folded style:

body: >
  {"mainlevel":{"subkey":"finalvalue"}}

Note that folded style pastes things as they are – but ignores single new lines. So it might be difficult to add variables here.

Therefore, a little bit better suited is the literal style, indicated by the pipe:

body: |
{
  "mainlevel": {
    "subkey": "finalvalue"
  }
}

This is probably the easiest way to deal with in many debugging situations where you need to be able to quickly change thins in your code.

Last, and honestly something I would try to avoid is the plain one-liner:

body: "{\"mainlevel\":{\"subkey\":\"finalvalue\"}}

Note, all quotation marks need to be escaped which makes it hard to read, hard to maintain and easy to introduce errors.

As shown Ansible is powerful and simple. Thus there are always multiple different ways to reach the goal you are aiming for – and it depends on the requirements what solution is the best one.

For more details I can only recommend Understanding multi line strings in YAML and Ansible (Part I & Part II) from adminswerk.de.

Update:
Added how to add body payload from existing files.

2. Update:
Added details about literal style.