Over 6 years ago I joined Red Hat. For me it was a huge step – going from a small SI to a large software vendor, moving to a dedicated pre-sales role (“Solution Architect”) in a larger team, and so on.
And I learned a lot. Like, A LOT. About how such a large enterprise is run, how a huge software vendor operates, but with the changing landscape in customers also how customers of certain sizes and in certain industries work. At the same time I learned a lot about how the sales process of a software vendor works, what a role a pre-sales engineer can take (and what not), and how good sales teams work. I was not without success in that role.
After my few years as a solution architect I had the chance to join the Ansible product team. This meant another big change since I suddenly stopped talking to customers on a daily base, but instead talked to the larger sales organization within Red Hat. Also, moving from a German team to a mainly US team meant a lot of changes in how my daily schedules were set up – but that worked well with the kids who arrived at the same time. The new job brought had a lot of new components for me as well: Technical Marketing Manager means to shape the product message into consumable bites for people with a technical taste. Suddenly I had to wonder how I can enable other solution architects to present this to a technical savy audience – especially if these solution architects are not product experts and do have to sell multiple products anyway. I had a steep learning curve, but again the feedback was not bad, and the constantly growing team was just awesome.
But nothing is forever: over the recent months I realized that I have growth aspirations which simply don’t fit with my position anymore. Thus I had to make the hard decision to look for something else – and I found this something else outside Red Hat. My future is still within the Open Source ecosystem, deeply connected to Linux – no surprises there. But more about that in another post.
Right now I’d just like to thank Red Hat for an awesome time. And I especially would like to thank the various teams I worked in over the time with all the people in there:
The public sector Germany team, with the simply best sales person I ever met and the best middleware solution architect I worked with.
The “Ansible workshop” crew with the greatest mind in writing workshops at the top. We rocked so many conferences and summits.
The Ansible product team when I joined – being right there when a just acquired company settles into the arms of a new owner is a very interesting experience, thanks for all the support!
The BTE which formed later on – I never worked in a team like that, but I loved almost every day. It was great to grow and thrive in this team, through all transitions and re-organisations.
Without you, I wouldn’t be the person I am today – thanks for that, thanks for all the support! All the best for the future =)
A few days ago I was asked how to best deal with structures in Ansible which are mixing dictionaries and lists. json_query can help here!
A few days ago I was asked how to best deal with structures in Ansible which are mixing dictionaries and lists. Basically, the following example was provided and the questioned remained how to deal with this – for example how to flatten it:
I was wondering a lot how to deal with this – after all dict2items only deals with dicts and fails when it reaches the lists in there. I also fooled around with the map filter, but most of my results also required some previous knowledge about the data structure, were only acting by providing “cloud1.region1” or similar.
The solution was the json_query filter: it is based on jmespath and can deal with the above mentioned structure by list and object projections:
tasks:
- name: Projections using json_query
debug:
msg: "Item value is: {{ item }}"
loop: "{{ myhash|json_query(projection_query)|list }}"
vars:
projection_query: "*.*[]"
And indeed, the loop does create a simplified output of all the elements in this nested structure:
Of course, some knowledge is still needed to make this work: you need to know if you are projecting on a list or on a dictionary. So if your data structure changes on that level between executions, you might need something else.
Last November we introduced Ansible security automation as our answer to the lack of integration across the IT security industry. Let’s have a closer look at one of the scenarios where Ansible can facilitate typical operational challenges of security practitioners.
Last November we introduced Ansible security automation as our answer to the lack of integration across the IT security industry. Let’s have a closer look at one of the scenarios where Ansible can facilitate typical operational challenges of security practitioners.
A big portion of security practitioners’ daily activity is dedicated to investigative tasks. Enrichment is one of those tasks, and could be both repetitive and time-consuming, making it a perfect candidate for automation. Streamlining these processes can free up their analysts to focus on more strategic tasks, accelerate the response in time-sensitive situations and reduce human errors. However, in many large organizations , the multiple security solutions aspect of these activities are not integrated with each other. Hence, different teams may be in charge of different aspects of IT security, sometimes with no processes in common.
That often leads to manual work and interaction between people of different teams which can be error-prone and above all, slow. So when something suspicious happens and further attention is needed, security teams spend a lot of valuable time operating on many different security solutions and coordinating work with other teams, instead of focusing on the suspicious activity directly.
In this blog post we have a closer look at how Ansible can help to overcome these challenges and support investigation enrichment activities. In the following example we’ll see how Ansible can be used to enable programmatic access to information like logs coming from technologies that may not be integrated into a SIEM. As an example we’ll use enterprise firewalls and intrusion detection and protection systems (IDPS).
Simple Demo Setup
To showcase the aforementioned scenario we created a simplified, very basic demo setup to showcase the interactions. This setup includes two security solutions providing information about suspicious traffic, as well as a SIEM: we use a Check Point Next Generation Firewall (NGFW) and a Snort IDPS as security solutions providing information. The SIEM to gather and analyze those data is IBM QRadar.
Also, from a machine called “attacker” we will simulate a potential attack pattern on the target machine on which the IDPS is running.
This is just a basic demo setup, a real world setup of an Ansible security automation integration would look different, and can feature other vendors and technologies.
Logs: crucial, but distributed
Now imagine you are a security analyst in an enterprise. You were just informed of an anomaly in an application, showing suspicious log activities. For example, we have a little demo where we curl a certain endpoint of the web server which we conveniently called “web_attack_simulation”:
As a security analyst you know that anomalies can be the sign of a potential threat. You have to determine if this is a false positive, that can be simply dismissed or an actual threat which requires a series of remediation activities to be stopped. Thus you need to collect more data points – like from the firewall and the IDS. Going through the logs of the firewall and IDPS manually takes a lot of time. In large organizations, the security analyst might not even have the necessary access rights and needs to contact the teams that each are responsible for both the enterprise firewall and the IDPS, asking them to manually go through the respective logs and directly check for anomalies on their own and then reply with the results. This could imply a phone call, a ticket, long explanations, necessary exports or other actions consuming valuable time.
It is common in large organisations to centralise event management on a SIEM and use it as the primary dashboard for investigations. In our demo example the SIEM is QRadar, but the steps shown here are valid for any SIEM. To properly analyze security-related events there are multiple steps necessary: the security technologies in question – here the firewall and the IDPS – need to be configured to stream their logs to the SIEM in the first place. But the SIEM also needs to be configured to help ensure that those logs are parsed in the correct way and meaningful events are generated. Doing this manually is time-intensive and requires in-depth domain knowledge. Additionally it might require privileges a security analyst does not have.
But Ansible allows security organizations to create pre-approved automation workflows in the form of playbooks. Those can even be maintained centrally and shared across different teams to enable security workflows at the press of a button.
Why don’t we add those logs to QRadar permanently? This could create alert fatigue, where too much data in the system generates too many events, and analysts might miss the crucial events. Additionally, sending all logs from all systems easily consumes a huge amount of cloud resources and network bandwidth.
So let’s write such a playbook to first configure the log sources to send their logs to the SIEM. We start the playbook with Snort and configure it to send all logs to the IP address of the SIEM instance:
Note that here we only have one task, which imports an existing role. Roles are an essential part of Ansible, and help in structuring your automation content. Roles usually encapsulate the tasks and other data necessary for a clearly defined purpose. In the case of the above shown playbook, we use the role ids_config, which manages the configuration of various IDPS. It is provided as an example by the ansible-security team. This role, like others mentioned in this blog post, are provided as a guidance to help customers that may not be accustomed to Ansible to become productive faster. They are not necessarily meant as a best practise or a reference implementation.
Using this role we only have to note a few parameters, the domain knowledge of how to configure Snort itself is hidden away. Next, we do the very same thing with the Check Point firewall. Again an existing role is re-used, log_manager:
- name: Configure Check Point to send logs to QRadar
hosts: checkpoint
tasks:
- include_role:
name: ansible_security.log_manager
tasks_from: forward_logs_to_syslog
vars:
syslog_server: "192.168.3.4"
checkpoint_server_name: "gw-2d3c54"
firewall_provider: checkpoint
With these two snippets we are already able to reach out to two security solutions in an automated way and reconfigure them to send their logs to a central SIEM.
We can also automatically configure the SIEM to accept those logs and sort them into corresponding streams in QRadar:
- name: Add Snort log source to QRadar
hosts: qradar
collections:
- ibm.qradar
tasks:
- name: Add snort remote logging to QRadar
qradar_log_source_management:
name: "Snort rsyslog source - 192.168.14.15"
type_name: "Snort Open Source IDS"
state: present
description: "Snort rsyslog source"
identifier: "ip-192-168-14-15"
- name: Add Check Point log source to QRadar
hosts: qradar
collections:
- ibm.qradar
tasks:
- name: Add Check Point remote logging to QRadar
qradar_log_source_management:
name: "Check Point source - 192.168.23.24"
type_name: "Check Point FireWall-1"
state: present
description: "Check Point log source"
identifier: "192.168.23.24"
Here we do use Ansible Content Collections: the new method of distributing, maintaining and consuming automation content. Collections can contain roles, but also modules and other code necessary to enable automation of certain environments. In our case the collection for example contains a role, but also the necessary modules and connection plugins to interact with QRadar.
Without any further intervention by the security analyst, Check Point logs start to appear in the QRadar log overview. Note that so far no logs are sent from Snort to QRadar: Snort does not know yet that this traffic is noteworthy! We will come to this in a few moments.
Remember, taking the perspective of a security analyst: now we have more data at our disposal. We have a better understanding of what could be the cause of the anomaly in the application behaviour. Logs from the firewall are shown, who is sending traffic to whom. But this is still not enough data to fully qualify what is going on.
Fine-tuning the investigation
Given the data at your disposal you decide to implement a custom signature on the IDPS to get alert logs if a specific pattern is detected.
In a typical situation, implementing a new rule would require another interaction with the security operators in charge of Snort who would likely have to manually configure multiple instances. But luckily we can again use an Ansible Playbook to achieve the same goal without the need for time consuming manual steps or interactions with other team members.
There is also the option to have a set of playbooks for customer specific situations pre-create. Since the language of Ansible is YAML, even team members with little knowledge can contribute to the playbooks, making it possible to have agreed upon playbooks ready to be used by the analysts.
Again we reuse a role, ids_rule. Note that this time some understanding of Snort rules is required to make the playbook work. Still, the actual knowledge of how to manage Snort as a service across various target systems is shielded away by the role.
---
- name: Add Snort rule
hosts: snort
become: yes
vars:
ids_provider: snort
tasks:
- name: Add snort web attack rule
include_role:
name: "ansible_security.ids_rule"
vars:
ids_rule: 'alert tcp any any -> any any (msg:"Attempted Web Attack"; uricontent:"/web_attack_simulation"; classtype:web-application-attack; sid:99000020; priority:1; rev:1;)'
ids_rules_file: '/etc/snort/rules/local.rules'
ids_rule_state: present
Finish the offense
Moments after the playbook is executed, we can check in QRadar if we see alerts. And indeed, in our demo setup this is the case:
With this information on hand, we can now finally check all offenses of this type, and verify that they are all coming only from one single host – here the attacker.
From here we can move on with the investigation. For our demo we assume that the behavior is intentional, and thus close the offense as false positive.
Rollback!
Last but not least, there is one step which is often overlooked, but is crucial: rolling back all the changes! After all, as discussed earlier, sending all logs into the SIEM all the time is resource-intensive.
With Ansible the rollback is quite easy: basically the playbooks from above can be reused, they just need to be slightly altered to not create log streams, but remove them again. That way, the entire process can be fully automated and at the same time made as resource friendly as possible.
Takeaways and where to go next
It happens that the job of a CISO and her team is difficult even if they have in place all necessary tools, because the tools don’t integrate with each other. When there is a security threat, an analyst has to perform an investigation, chasing all relevant pieces of information across the entire infrastructure, consuming valuable time to understand what’s going on and ultimately perform any sort of remediation.
Ansible security automation is designed to help enable integration and interoperability of security technologies to support security analysts’ ability to investigate and remediate security incidents faster.
As next steps there are plenty of resources to follow up on the topic:
Running CLI tools like ansible often requires a specific environment with dependencies on the core operating system libraries. That makes it hard to run different versions in parallel – or test the newest updates. And it might clutter the OS. Toolbox offers simple container management to avoid these shortcomings.
Running CLI tools like ansible often requires a specific environment with dependencies on the core operating system libraries. That makes it hard to run different versions in parallel – or test the newest updates. And it might clutter the OS. Toolbox offers simple container management to avoid these shortcomings.
The recent development of Linux distributions has seen a shift away from all-purpose distributions towards stable core distributions with limited packages and additional sand-boxed tooling running on top to enable management of applications. One of the most advanced distributions here is for sure Fedora Silverblue, but even the enterprise distribution Red Hat Enterprise Linux 8 brings a lot of changes which aim into the right direction. Technologies in this context are for example rpm-ostree for the management of immutable OS images and Flatpak for the management of GUI applications. Additionally, RHEL 8 comes along with so called app-streams – and of course there is always the option of using containers with for example podman.
In this blog post I want to focus on the last one: using containers to manage your CLI tools, thus keeping them independent of your operating system packaging and libraries. With Fedora and RHEL, there is tooling provided which makes this even easier: Toolbox.
Less OS clutter through isolated installation of dependencies
Easy to recreate environments (think of “works on my machine”)
Immutable environments possible
Think of it that way: with complex applications, behavior sometimes depends on certain versions of some libraries. When those are managed by the OS packaging system, it is hard to keep them up2date or just in the same version across multiple machines, not to speak about multiple distributions. Also, I don’t want my OS to be cluttered with weird dependencies which I might not even trust just to justify a weird application’s requirements. And I might want to install different versions of a tool to test them, – with different libraries as well, which is often impossible with OS package management.
Toolbox
In comes Toolbox:
Toolbox is a tool that offers a familiar package based environment for developing and debugging software that runs fully unprivileged using Podman.
The toolbox container is a fully mutable container; when you see yum install ansible for example, that’s something you can do inside your toolbox container, without affecting the base operating system.
While Toolbox is particularly interesting for immutable systems like Fedora Silverblue, it even makes sense to run it on other distributions. I started using it on my regular Fedora for example just to have certain tools available in certain versions for tests.
And why use Toolbox, and not just the usual container tools? Toolbox takes care of volume mounting and all the other necessary bits of container management, and enables you to just use a very basic set of commands to create – and reuse – your tool containers. It is simpler and easier than always typing in fully fledged podman or docker commands all the time.
It is very easy to get started with Toolbox. First, it needs to be installed on the system. For example, on Fedora 31, this can be done via:
$ sudo dnf install toolbox
After that, you are good to go. Since the idea is to have re-usable containers, let’s create the first. In my example I want to have a container with the newest Ansible version to run some automation. So we just create a new container called ansible:
$ toolbox create --container ansible
Image required to create toolbox container.
Download registry.fedoraproject.org/f31/fedora-toolbox:31 (500MB)? [y/N]: y
Created container: ansible
As you see, a base image for my distribution was downloaded, and the container created. Next, let’s access it and look around:
$ toolbox enter --container ansible
Welcome to the Toolbox; a container where you can install and run
all your tools.
- Use DNF in the usual manner to install command line tools.
- To create a new tools container, run 'toolbox create'.
For more information, see the documentation.
⬢[liquidat@toolbox ~]$
We are greeted with a short message and then dropped to a shell. Note the bubble at the start of the command prompt – a nice touch to differentiate if you are inside a toolbox or not. Next, let’s look at our environment:
⬢[liquidat@toolbox ~]$ pwd
/home/liquidat
⬢[liquidat@toolbox ~]$ ls
bin development documents downloads ...
⬢[liquidat@toolbox ~]$ ls /
README.md bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var
⬢[liquidat@toolbox ~]$ cat /README.md
# Toolbox — Unprivileged development environment
[Toolbox](https://github.com/debarshiray/toolbox) is a tool that offers a
[...]
As you see, the toolbox has actual access to the file system. That way we can use the tools just like normal shell tools, interact with things we have in our environment. However, at the same time we have limited access to the root system since we see the container root system (as identified by the readme), not the host root system.
Getting my first tool ready
As mentioned I’d like to have a container with the newest Ansible. Let’s install it:
As you see, Ansible was properly installed. And with this we are already done – we have our first tool ready, name “ansible”.
Using our tool
Now let’s assume I use the container for some things, exit it – and want to reuse it later on. This is no problem at all, since that is exactly what Toolbox was built for. And we have a name, which makes it fairly easy to remember how to access it. But even if we do not remember the name, we can easily list all available tools:
$ toolbox list
IMAGE ID IMAGE NAME CREATED
64e68e194389 registry.fedoraproject.org/f31/fedora-toolbox:31 2 weeks ago
CONTAINER ID CONTAINER NAME CREATED STATUS IMAGE NAME
8ec117845e06 ansible 47 minutes ago Up 47 minutes ago registry.fedoraproject.org/f31/fedora-toolbox:31
$ toolbox enter -c ansible
⬢[liquidat@toolbox ~]$ ansible --version
ansible 2.9.6
config file = /home/liquidat/.ansible.cfg
configured module search path = ['/home/liquidat/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/liquidat/.local/lib/python3.7/site-packages/ansible
executable location = /home/liquidat/.local/bin/ansible
python version = 3.7.6 (default, Jan 30 2020, 09:44:41) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]
As you see the container is in the same state as we left it: Ansible is still installed in the proper way, and ready to be used. And we can do this now with all kinds of other tools: be it another version of Ansible, or even some daemon we want to experiment with. It can all be easily installed and run and re-used, without worrying of cluttering the OS, or having the wrong library versions installed, or not being able to update some library because of a system dependency.
Summary
Toolbox is an interesting approach to simplify container management to fool around with CLI based tools. If you have an immutable environment like Fedora Silverblue, it might become a crucial piece in your daily operations since it is a pain to install additional packages on top of Silverblue’s ostree infrastructure. But even for “normal” distributions it is worth a try!
RHEL 8 has a new way how Python is installed and handled. How do you use it properly then, especially when multiple versions are installed? Read on to learn how to properly set up a virtual environment nevertheless.
RHEL 8 has a new way how Python is installed and handled. How do you use it properly then, especially when multiple versions are installed? Read on to learn how to properly set up a virtual environment nevertheless.
Red Hat Enterprise Linux 8 was released in May this year – and comes with a lot of changes. Think of a really modern OS here. Among those changes is also that Python is, well different: it is included, for sure. But at the same time, it isn’t.
The important piece is anyway that, when you work with Python in development environments or for example when you are dealing with Ansible, it makes sense to run everything in a Python virtual environment.
When in the future different versions of Python are offered via appstreams, make sure to pick the right selinux bindings when you link them into your virtual environment.
Another way to work with selinux libs is to create the virtual environment by using system packages:
![[Short Tip] Flatten nested dict/list structures in Ansible with json_query](https://liquidat.files.wordpress.com/2020/05/telephone-1822040_1920.jpg?w=1200)
![[Howto] Using toolbox in Fedora / RHEL 8 for easy management of CLI tools](https://liquidat.files.wordpress.com/2020/03/tool-3493486_1920.jpg?w=1200)
![[Howto] Get a Python virtual environment running on RHEL 8](https://liquidat.files.wordpress.com/2019/08/snake-653639_1920.jpg?w=1200)
/home/liquidat 