[Short Tip] Flatten nested dict/list structures in Ansible with json_query

A few days ago I was asked how to best deal with structures in Ansible which are mixing dictionaries and lists. json_query can help here!

Ansible Logo

A few days ago I was asked how to best deal with structures in Ansible which are mixing dictionaries and lists. Basically, the following example was provided and the questioned remained how to deal with this – for example how to flatten it:

    myhash:
      cloud1:
        region1:
          - name: "city1"
          - size: "large"
          - param: "alpha"
        region2:
          - name: "city2"
          - size: "small"
          - param: "beta"
      cloud2:
        region1:
          - name: "city1"
          - size: "large"
          - param: "gamma"

I was wondering a lot how to deal with this – after all dict2items only deals with dicts and fails when it reaches the lists in there. I also fooled around with the map filter, but most of my results also required some previous knowledge about the data structure, were only acting by providing “cloud1.region1” or similar.

The solution was the json_query filter: it is based on jmespath and can deal with the above mentioned structure by list and object projections:

  tasks:
  - name: Projections using json_query
    debug:
      msg: "Item value is: {{ item }}"
    loop: "{{ myhash|json_query(projection_query)|list }}"
    vars:
      projection_query: "*.*[]"

And indeed, the loop does create a simplified output of all the elements in this nested structure:

TASK [Projections using json_query] **********************************************************
ok: [localhost] => (item=[{'name': 'city1'}, {'size': 'large'}, {'param': 'alpha'}]) => {
    "msg": "Item value is: [{'name': 'city1'}, {'size': 'large'}, {'param': 'alpha'}]"
}
ok: [localhost] => (item=[{'name': 'city2'}, {'size': 'small'}, {'param': 'beta'}]) => {
    "msg": "Item value is: [{'name': 'city2'}, {'size': 'small'}, {'param': 'beta'}]"
}
ok: [localhost] => (item=[{'name': 'city1'}, {'size': 'large'}, {'param': 'gamma'}]) => {
    "msg": "Item value is: [{'name': 'city1'}, {'size': 'large'}, {'param': 'gamma'}]"
}

Of course, some knowledge is still needed to make this work: you need to know if you are projecting on a list or on a dictionary. So if your data structure changes on that level between executions, you might need something else.

Image by Andrew Martin from Pixabay

[Howto] Sort spam mails to the junk folder in Zarafa

Tux
Groupware setups like Zarafa often leave the spam scanning to external tools – but still has to sort the tagged mails. This article shows two ways of sorting and filtering tagged spam mails in Zarafa.

Imagine the pretty common setup where the actual spam filtering is done on a separate MTA. In such cases the spam is scanned and tagged, but the groupware system, here Zarafa, has to sort the mail to the appropriate folders. For example, an e-mail header spam tag created by Spamassassin looks like:

X-Spam-Flag: YES
X-Spam-Score: 64.448
X-Spam-Level: ****************************************************************
X-Spam-Status: Yes, score=64.448 tagged_above=-999 required=6.2
	tests=[ADVANCE_FEE_2_NEW_MONEY=0.992, ADVANCE_FEE_3_NEW=2.734,
	ADVANCE_FEE_3_NEW_MONEY=0.001, ADVANCE_FEE_4_NEW=0.001,
	ADVANCE_FEE_4_NEW [...]

The question is how the groupware can auto-sort such mails – Zarafa knows (at least) two ways to accomplish that aim.

The first is to use Procmail, Maildrop or similar tools to check the mails for such tags: all mails are delivered from the separate MTA to Procmail, which forwards all mails to the zarafa-dagent. However, while normal mails are just forwarded to the normal zarafa-dagent without any options, spam mails are forwarded with the option zarafa-dagent -j. That way, zarafa-dagent knows that the mails are spam, and sorts them into the Junk folders. The Procmail configuration for that is for example:

:0 w
* ^X-Spam-Flag: YES
| /usr/bin/zarafa-dagent -j $1
EXITCODE=$?

:0 w
| /usr/bin/zarafa-dagent $1
EXITCODE=$?

The second way – which I prefer – is to tell zarafa-dagent to look for the spam tag itself. The Dagent can be configured to search for a spam tag flag sorting these mails into the Junk folder automatically. The configuration is done in /etc/zarafa/dagent.cfg:

spam_header_name = X-Spam-Flag
spam_header_value = yes

Please note that the values are case-insensitive.

Which method you use highly depends on your actual groupware setup: in case you already have a Procmail/Maildrop/Whatnot setup you might want to use the first method since it can easily be integrated with the existing setup. In case you do not have or do not want to introduce another layer like Procmail, you should go with the latter method. Also, the latter method is independent of the rest of the setup – so even with a working Procmail setup you might want to use the latter method to have it out of your mind.