[Howto] Using systemd timers instead of /etc/cron entries

Executing certain commands at given intervals or times is a very typical task for system administrators. In the past it was common to use cron or some variation for this, in some way or the other.

Background

Cron does the job it was written for. But this was years ago, and these days Kernels offer neat things like CPU quotas and memory limits. Cron has no means to use those – but other tools have.

Additionally, newer tools provide dependencies, a proper configuration language (instead of hard-to-maintain bash lines), multiple triggers, randomized delays and real logging.

Especially the last bit, real logging, is essential: Cron can forward log messages it thinks needs to be forwarded. But without real kernel backed process management (cgroups) there is no real way for Cron to see if a job is running or has finished, and what log lines belong to it.

Systemd has all this – and thus it makes sense to create new recurrent jobs in Systemd and even migrate old ones sometimes.

Setting up the timer

What is needed are two things: a service file describing WHAT should be done, and a timer file describing WHEN to do it.

Let’s start with the WHAT: we create a typical service file named backupjob.service executing a backup bash script:

[Unit]
Description=Backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Note that we do not enable the service here! We can start it for quick and easy debugging – which is also way easier than with Cron.

Keep in mind that this is a typical systemd service. You can also add requirements, dependencies, performance options and so on in a standardized fashion. With Cron this is not possible, would have to be done in the bash script itself and thus would be messy, hard to maintain, and most likely duplicate work between multiple Cron jobs. And btw.: that way not at all following the UNIX philosophy!

Next, the WHEN. We need a timer file, something which describes when to execute the service, backupjob.timer.

[Unit]
Description=Run backup jobs regularly

[Timer]
OnCalendar=daily
AccuracySec=1h
Unit=backupjob.service

[Install]
WantedBy=timers.target

As you can see, this job is scheduled to run daily, with an accuracy of 1 hour. The accuracy is an interesting bit: systemd tries to avoid starting all services at the exact same time, to avoid massive system load at :00. Just recently a technician at a large cloud provider mentioned that data centers could be designed way more efficient if not everyone would put their Cron jobs to the full minute.

Speaking about, besides OnCalendar there is also a way to start jobs relative to the boot up time, or relative to when the timer was run last. For example, to run something every 15 minutes, set OnActiveSec=15min. More information can be found in the timer documentation.

Starting and stopping the timer

As mentioned above, the service unit files are not activated, instead the timers are: sudo systemctl enable --now backupjob.timer

Stopping a timer is equally simple: systemctl stop backupjob.timer . If you want to avoid that it is started again during the next boot, also disable it: systemctl disable backupjob.timer.

Additional tooling

One of the great things of using the systemd ecosystem is that it is very easy to work with timers: with systemctl list-timers a nice and clear overview of the current state and time of next execution is given:

❯ sudo systemctl list-timers
[sudo] password for liquidat: 
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES                     
Thu 2021-04-15 18:06:34 CEST 1h 14min left Thu 2021-04-15 16:09:32 CEST 42min ago    dnf-makecache.timer          dnf-makecache.service         
Fri 2021-04-16 00:00:00 CEST 7h left       Thu 2021-04-15 00:01:01 CEST 16h ago      logrotate.timer              logrotate.service             
Fri 2021-04-16 00:00:00 CEST 7h left       Thu 2021-04-15 00:01:01 CEST 16h ago      mlocate-updatedb.timer       mlocate-updatedb.service      
Fri 2021-04-16 00:00:00 CEST 7h left       Thu 2021-04-15 00:01:01 CEST 16h ago      unbound-anchor.timer         unbound-anchor.service        
Fri 2021-04-16 12:32:32 CEST 19h left      Thu 2021-04-15 12:32:32 CEST 4h 19min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2021-04-19 01:34:34 CEST 3 days left   Mon 2021-04-12 15:46:39 CEST 3 days ago   fstrim.timer                 fstrim.service                

6 timers listed.
Pass --all to see loaded but inactive timers, too.

At the same time, you can get the detailed status with systemctl status logrotate.timer – or of all of them via systemctl status *timer.

Logs are simply available via journalctl -u logrotate.timer – and the logs for the executed service can be read via journalctl -u logrotate.service.

And if you just don’t want to deal with systemd files right now – but nevertheless want all the goods from it, you can also launch systemd services or general commands with a one-time execution:

systemd-run --on-active="10h 30m" --unit myonetimescript.service

Final words

Writing systemd timers instead of traditional Cron jobs makes operating, maintaining and even writing of recurrent jobs much easier.

The only thing you might be missing is easily sending out stuff via mail.

Image by Ryan McGuire from Pixabay

[Howto] Run programs as non-root user on privileged ports via Systemd

TuxRunning programs as a non-root user is must in security sensitive environments. However, these programs sometimes need to publish their service on privileged ports like port 80 – which cannot be used by local users. Systemd offers a simple way to solve this problem.

Background

Running services as non-root users is a quite obvious: if it is attacked and a malicious user gets control of the service, the rest of the system should still be secure in terms of access rights.

At the same time plenty programs need to publish their service at ports like 80 or 443 since these are the default ports for http communication and thus for interfaces like REST. But these ports are not available to non-root users.

Problem shown at the example gitea

To show how to solve this with systemd, we take the self hosted git service gitea as an example. Currently there are hardly any available packages, so most people end up installing it from source, for example as the user git. A proper sysmted unit file for such an installation in a local path, running the service as a local user, is:

$ cat /etc/systemd/system/gitea.service
[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
After=postgresql.service

[Service]
RestartSec=2s
Type=simple
User=git
Group=git
WorkingDirectory=/home/git/go/src/code.gitea.io/gitea
ExecStart=/home/git/go/src/code.gitea.io/gitea/gitea web
Restart=always
Environment=USER=git HOME=/home/git

[Install]
WantedBy=multi-user.target

If this service is started, and the application configuration is set to port 80, it fails during the startup with a bind error:

Jan 04 09:12:47 gitea.qxyz.de gitea[8216]: 2018/01/04 09:12:47 [I] Listen: http://0.0.0.0:80
Jan 04 09:12:47 gitea.qxyz.de gitea[8216]: 2018/01/04 09:12:47 [....io/gitea/cmd/web.go:179 runWeb()] [E] Failed to start server: listen tcp 0.0.0.0:80: bind: permission denied

Solution

One way to tackle this would be a reverse proxy, running on port 80 and forwarding traffic to a non-privileged port like 8080. However, it is much more simple to add an additional systemd socket which listens on port 80:

$ cat /etc/systemd/system/gitea.socket
[Unit]
Description=Gitea socket

[Socket]
ListenStream=80
NoDelay=true

As shown above, the definition of a socket is straight forward, and hardly needs any special configuration. We use NoDelay here since this is a default for Go on sockets it opens, and we want to imitate that.

Given this socket definition, we add the socket as requirement to the service definition:

[Unit]
Description=Gitea (Git with a cup of tea)
Requires=gitea.socket
After=syslog.target
After=network.target
After=postgresql.service

[Service]
RestartSec=2s
Type=simple
User=git
Group=git
WorkingDirectory=/home/git/go/src/code.gitea.io/gitea
ExecStart=/home/git/go/src/code.gitea.io/gitea/gitea web
Restart=always
Environment=USER=git HOME=/home/git
NonBlocking=true

[Install]
WantedBy=multi-user.target

As seen above, the unit definition hardly changes, only the requirement for the socket is added – and NonBlocking as well, to imitate Go behavior.

That’s it! Now the service starts up properly and everything is fine:

[...]
Jan 04 09:21:02 gitea.qxyz.de gitea[8327]: 2018/01/04 09:21:02 Listening on init activated [::]:80
Jan 04 09:21:02 gitea.qxyz.de gitea[8327]: 2018/01/04 09:21:02 [I] Listen: http://0.0.0.0:80
Jan 04 09:21:08 gitea.qxyz.de gitea[8327]: [Macaron] 2018-01-04 09:21:08: Started GET / for 192.168.122.1
[...]

Sources, further reading

[Howto] Workaround failing MongoDB on RHEL/CentOS 7

Ansible LogoMongoDB is often installed right from upstream provided repositories. In such cases with recent updates the service might fail to start via systemctl. A workaround requires some SELinux work.

Ansible Tower collects system data inside a MongoDB. Since MongoDB is not part of RHEL/CentOS, it is installed directly form the upstream MongoDB repositories. However, with recent versions of MongoDB the database might not come up via systemctl:

[root@ansible-demo-tower init.d]# systemctl start mongod
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[root@ansible-demo-tower init.d]# journalctl -xe
May 03 08:26:00 ansible-demo-tower systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....
-- Subject: Unit mongod.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mongod.service has begun starting up.
May 03 08:26:00 ansible-demo-tower runuser[7266]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
May 03 08:26:00 ansible-demo-tower runuser[7266]: pam_unix(runuser:session): session closed for user mongod
May 03 08:26:00 ansible-demo-tower mongod[7259]: Starting mongod: [FAILED]
May 03 08:26:00 ansible-demo-tower systemd[1]: mongod.service: control process exited, code=exited status=1
May 03 08:26:00 ansible-demo-tower systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mongod.service has failed.
-- 
-- The result is failed.
May 03 08:26:00 ansible-demo-tower systemd[1]: Unit mongod.service entered failed state.
May 03 08:26:00 ansible-demo-tower systemd[1]: mongod.service failed.
May 03 08:26:00 ansible-demo-tower polkitd[11436]: Unregistered Authentication Agent for unix-process:7254:1405622 (system bus name :1.184, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_

The root cause of the problem is that the MongoDB developers do not provide a proper SELinux</a configuration with their packages, see the corresponding bug report.

A short workaround is to create a proper (more or less) SELinux rule and install it to the system:

[root@ansible-demo-tower ~]# grep mongod /var/log/audit/audit.log | audit2allow -m mongod > mongod.te
[root@ansible-demo-tower ~]# cat mongod.te 

module mongod 1.0;

require {
	type locale_t;
	type mongod_t;
	type ld_so_cache_t;
	class file execute;
}

#============= mongod_t ==============
allow mongod_t ld_so_cache_t:file execute;
allow mongod_t locale_t:file execute;
[root@ansible-demo-tower ~]# grep mongod /var/log/audit/audit.log | audit2allow -M mongod
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i mongod.pp

[root@ansible-demo-tower ~]# semodule -i mongod.pp 
[root@ansible-demo-tower ~]# sudo service mongod start
                                                           [  OK  ]

Keep in mind that audit2allow generated rule sets are not to be used on production systems. The generated SELinux rules need to be analyzed manually to verify that it covers nothing but the problematic use case.