Hello Isovalent!

As mentioned in my last post I left Red Hat – and today is my first day at Isovalent!

In my new position I will be a technical marketing manager and thus working on technical content, messaging and enablement. With Cilium Enterprise Isovalent offers an eBPF based solution for Kubernetes networking, observability, and security – and since I am rather new to Kubernetes, I expect a steep learning curve.

I am looking forward to the challenges ahead of me, and will drop a blog post about it once in a while =)

Good bye Red Hat

Over 6 years ago I joined Red Hat. For me it was a huge step – going from a small SI to a large software vendor, moving to a dedicated pre-sales role (“Solution Architect”) in a larger team, and so on.

And I learned a lot. Like, A LOT. About how such a large enterprise is run, how a huge software vendor operates, but with the changing landscape in customers also how customers of certain sizes and in certain industries work. At the same time I learned a lot about how the sales process of a software vendor works, what a role a pre-sales engineer can take (and what not), and how good sales teams work. I was not without success in that role.

After my few years as a solution architect I had the chance to join the Ansible product team. This meant another big change since I suddenly stopped talking to customers on a daily base, but instead talked to the larger sales organization within Red Hat. Also, moving from a German team to a mainly US team meant a lot of changes in how my daily schedules were set up – but that worked well with the kids who arrived at the same time. The new job brought had a lot of new components for me as well: Technical Marketing Manager means to shape the product message into consumable bites for people with a technical taste. Suddenly I had to wonder how I can enable other solution architects to present this to a technical savy audience – especially if these solution architects are not product experts and do have to sell multiple products anyway. I had a steep learning curve, but again the feedback was not bad, and the constantly growing team was just awesome.

But nothing is forever: over the recent months I realized that I have growth aspirations which simply don’t fit with my position anymore. Thus I had to make the hard decision to look for something else – and I found this something else outside Red Hat. My future is still within the Open Source ecosystem, deeply connected to Linux – no surprises there. But more about that in another post.

Right now I’d just like to thank Red Hat for an awesome time. And I especially would like to thank the various teams I worked in over the time with all the people in there:

• The public sector Germany team, with the simply best sales person I ever met and the best middleware solution architect I worked with.
• The “Ansible workshop” crew with the greatest mind in writing workshops at the top. We rocked so many conferences and summits.
• The Ansible product team when I joined – being right there when a just acquired company settles into the arms of a new owner is a very interesting experience, thanks for all the support!
• The BTE which formed later on – I never worked in a team like that, but I loved almost every day. It was great to grow and thrive in this team, through all transitions and re-organisations.

Without you, I wouldn’t be the person I am today – thanks for that, thanks for all the support! All the best for the future =)

[Howto] Using systemd timers instead of /etc/cron entries

Executing certain commands at given intervals or times is a very typical task for system administrators. In the past it was common to use cron or some variation for this, in some way or the other.

Background

Cron does the job it was written for. But this was years ago, and these days Kernels offer neat things like CPU quotas and memory limits. Cron has no means to use those – but other tools have.

Additionally, newer tools provide dependencies, a proper configuration language (instead of hard-to-maintain bash lines), multiple triggers, randomized delays and real logging.

Especially the last bit, real logging, is essential: Cron can forward log messages it thinks needs to be forwarded. But without real kernel backed process management (cgroups) there is no real way for Cron to see if a job is running or has finished, and what log lines belong to it.

Systemd has all this – and thus it makes sense to create new recurrent jobs in Systemd and even migrate old ones sometimes.

Setting up the timer

What is needed are two things: a service file describing WHAT should be done, and a timer file describing WHEN to do it.

Let’s start with the WHAT: we create a typical service file named backupjob.service executing a backup bash script:

[Unit]
Description=Backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Note that we do not enable the service here! We can start it for quick and easy debugging – which is also way easier than with Cron.

Keep in mind that this is a typical systemd service. You can also add requirements, dependencies, performance options and so on in a standardized fashion. With Cron this is not possible, would have to be done in the bash script itself and thus would be messy, hard to maintain, and most likely duplicate work between multiple Cron jobs. And btw.: that way not at all following the UNIX philosophy!

Next, the WHEN. We need a timer file, something which describes when to execute the service, backupjob.timer.

[Unit]
Description=Run backup jobs regularly

[Timer]
OnCalendar=daily
AccuracySec=1h
Unit=backupjob.service

[Install]
WantedBy=timers.target

As you can see, this job is scheduled to run daily, with an accuracy of 1 hour. The accuracy is an interesting bit: systemd tries to avoid starting all services at the exact same time, to avoid massive system load at :00. Just recently a technician at a large cloud provider mentioned that data centers could be designed way more efficient if not everyone would put their Cron jobs to the full minute.

Speaking about, besides OnCalendar there is also a way to start jobs relative to the boot up time, or relative to when the timer was run last. For example, to run something every 15 minutes, set OnActiveSec=15min. More information can be found in the timer documentation.

Starting and stopping the timer

As mentioned above, the service unit files are not activated, instead the timers are: sudo systemctl enable --now backupjob.timer

Stopping a timer is equally simple: systemctl stop backupjob.timer . If you want to avoid that it is started again during the next boot, also disable it: systemctl disable backupjob.timer.

Additional tooling

One of the great things of using the systemd ecosystem is that it is very easy to work with timers: with systemctl list-timers a nice and clear overview of the current state and time of next execution is given:

❯ sudo systemctl list-timers
[sudo] password for liquidat: 
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES                     
Thu 2021-04-15 18:06:34 CEST 1h 14min left Thu 2021-04-15 16:09:32 CEST 42min ago    dnf-makecache.timer          dnf-makecache.service         
Fri 2021-04-16 00:00:00 CEST 7h left       Thu 2021-04-15 00:01:01 CEST 16h ago      logrotate.timer              logrotate.service             
Fri 2021-04-16 00:00:00 CEST 7h left       Thu 2021-04-15 00:01:01 CEST 16h ago      mlocate-updatedb.timer       mlocate-updatedb.service      
Fri 2021-04-16 00:00:00 CEST 7h left       Thu 2021-04-15 00:01:01 CEST 16h ago      unbound-anchor.timer         unbound-anchor.service        
Fri 2021-04-16 12:32:32 CEST 19h left      Thu 2021-04-15 12:32:32 CEST 4h 19min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2021-04-19 01:34:34 CEST 3 days left   Mon 2021-04-12 15:46:39 CEST 3 days ago   fstrim.timer                 fstrim.service                

6 timers listed.
Pass --all to see loaded but inactive timers, too.

At the same time, you can get the detailed status with systemctl status logrotate.timer – or of all of them via systemctl status *timer.

Logs are simply available via journalctl -u logrotate.timer – and the logs for the executed service can be read via journalctl -u logrotate.service.

And if you just don’t want to deal with systemd files right now – but nevertheless want all the goods from it, you can also launch systemd services or general commands with a one-time execution:

systemd-run --on-active="10h 30m" --unit myonetimescript.service

Final words

Writing systemd timers instead of traditional Cron jobs makes operating, maintaining and even writing of recurrent jobs much easier.

The only thing you might be missing is easily sending out stuff via mail.

Image by Ryan McGuire from Pixabay