[Howto] My own mail & groupware server, part 2: initial mail server setup

Running your own mail and groupware server can be challenging. I recently had to re-create my own setup from the ground and describe the steps in a blog post series. This blog post is #2 of the series and covers the initial mail server setup.

This post is all about setting up an initial mail server. Read about the background to this setup and the decisions I took in the first post, My own mail & groupware server, part 1: what, why, how?

Getting a server

The first thing of hosting your own mail server is to answer a simple question: where? Do you have an internet connection at home with fast uploads and maybe even a fixed IPv4? Or is cloud the only option? And if you take the cloud, will it be a virtual server or a root server?

My home connection doesn’t really allow for a larger server setup, so cloud is the only option. Cloud always means someone else’s computer, never forget that! But if you pick a hardware machine it is at least harder to access/copy that machine without your knowledge compared to a virtual instance.

My mail setup always run on root servers, and for the last years I picked Hetzner as the hoster. Their server auction often has appealing things on sale, so that you can get something “decent enough” for around $30 per months.

So, for my new server I got one from the server auction again:

Server provisioning

The server was up quickly. The next step was to get it provisioned properly. Mailu requires docker compose. And since Docker is not properly supported on CentOS 8 I decided to got with CentOS 7. This I rebooted the server into security mode and started Hetzner’s custom installer to install centos77 minimal. RAID 1 was already configured, I just altered the partition sizes and moved most of the storage to the custom mount point /data.

DNS

Besides the basic provisioning I added rDNS entries for IPv4 and IPv6 – don’t forget those, they are important for many spam filters!

Speaking of DNS, the domain someone wants to use for mail needs to be set up in DNS as well. At least the following things should be done:

• create an A entry named @ for your server IPv4
• create an AAAA entry named @ for your IPv6
• create an A entry with your server’s host name for the server IPv4
• create an AAAA entry with your server’s host name the for server IPv6
• create a MX entry pointing to A entry for server host name (not a CNAME!)
• add a CAA entry for letsencrpyt: 0 issue "letsencrypt.org"

Many of those entries were still there from my previous setup, but I had to adjust the IP addresses to the new server and add the new host – host name “lisa”, named after the Simpsons.

Basic user and SSH

After setting up a new server, the next step usually is to add a new user: adduser liquidat creates it, usermod -aG wheel liquidat adds my user to the sudo group, and additionally I set the group wheel to NOPASSWD via visudo.

Also, copy the authorized keys from the root user to the new user – cp -r /root/.ssh /home/liquidat/ – and correct their ownership: chown -R liquidat:liquidat /home/liquidat/.ssh

Just to be sure the “right” ssh keys are there, copy your usual set over: ssh-copy-id liquidat@lisa.bayz.de . Personally, afterwards I removed all besides the currently most trusted (ssh-ed25519…).

Last but not least deactivate root login to ssh:

• Set PermitRootLogin no in /etc/ssh/sshd_config
• Also, in /etc/ssh/sshd_config set Port 2222 (we want to use port 22 for the git server later on)
• Restart sshd: systemctl restart sshd

Encrypted partition

One of the most important steps for me (YMMV) is an encrypted hard drive. In my personal risk assessment this impedes many possible hardware attacks from certain actors. Of course certain risks remain.

As mentioned, most of the storage is setup in a partition on /data. So it has to be encrypted properly, formatted and made available again. Note that this requires you to log into the machine after every reboot and actively decrypt the partition. If your server ever goes down, all services on it are down until you decrypted the partition!

• Umount existing mount: sudo umount /data/
• Remove /data entry from /etc/fstab
• Set up crypted device: sudo cryptsetup luksFormat /dev/md3
• Get passphrase via pwgen -y 50
• Decrypt the device: sudo cryptsetup luksOpen /dev/md3 verysecret
• Create a file system on it: sudo mkfs.ext4 /dev/mapper/verysecret
• Mount it: sudo mount /dev/mapper/verysecret /data

I can only recommend to verify the decryption afterwards:

• Reboot server
• Decrypt storage: sudo cryptsetup luksOpen /dev/md3 verysecret
• Mount storage: sudo mount /dev/mapper/verysecret /data

Docker Compose

Next I had to install Docker. Personally not my first choice to run containers, I would rather use Podman or even get my hands dirty with Kubernetes. But alas, time was short and pressure was high.

To get Docker onto the machine:

• Remove CentOS’ Docker packages: sudo yum remove -y docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
• Install tooling to easier add third party repos: sudo yum install -y yum-utils
• Add Docker’s third party repo for CentOS: sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
• Install Docker: sudo yum install -y docker-ce docker-ce-cli containerd.io
• We don’t want autostarts of Docker since the device is still encrypted: sudo systemctl disable docker
• Get Docker up: sudo systemctl start docker
• Create Docker group: sudo groupadd docker
• Add user to it: sudo usermod -aG docker $USER
• Load new group immediately: newgrp docker
• Get compose: sudo curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
• Make compose executable: sudo chmod +x /usr/local/bin/docker-compose

I will always wonder why they never managed to get compose out there as a package. Not that hard, I’d say?!

But anyhow, the stage was set now: the server was up and running, a user was ready to do work, the device was properly secured, I was ready to set up the mail server!

Installing Mailu

Getting Mailu up and running is really a matter of minutes. I must admit I was impressed – especially since I knew how much time my own setup ate over the years. Basically you use a config file generator from them which will generate a docker-compose.yml, and then start it. That’s really all!

• sudo mkdir /data/mailu
• create config file via https://setup.mailu.io/master/
  • make sure to add all kinds of subdomains you will be using
  • in my case: don’t activate webmail or caldav, I will be using Nextcloud for that
• download config files via wget as instructed​: one docker-compose.yml and a mailu.env containing all the entered variables
• verify that ANTIVIRUS is indeed defined and not commented out in mailu.env (thanks to dhoppe for that)
• docker-compose -p mailu up -d

And that’s it, really! My mail server setup was already running, after minutes. Next I added an admin user:

• create a password for the admin user via pwgen 20
• create admin password: docker-compose -p mailu exec admin flask mailu admin postmaster bayz.de $PASSWORD
• log in to admin interface: https://lisa.bayz.de/admin/, login is postmaster@$DOMAIN

The Mailu admin interface is nothing spectacular, but does it’s job:

• 

• 

• 

• 

Most basic settings can be done here, including setting up users and alias. A link to the rspamd web interface is also provided.

After this, I did some housework: not strictly necessary, but helpful:

• Add abuse alias to postmaster
• Add admin alias to postmaster (needed for RUA, dmarc aggregated reports)
• Generate DNS entries for SPF, DKIM, etc and add them to your DNS domain entries
• Add other users or even domains at will; all domains entered must be present in the mailu.env config file!

And that’s it! I was able to send myself mail via some freemail accounts. And with a classic mail client (Thunderbird, or something on the phone) I could also send mails. It all just worked!

Get word out there

However, I still had to get word out there that there is a new mail server and that it will be sending valid mails.

For example, I registered the new mail server at the DNS whitelist, DSWL: many spam filters check against that.

Next, I let Microsoft know of the new machine and registered it at postmater.live.

Last but not least I checked in with Google’s postmaster service.

Verifying and testing the setup

Now it was time for serious testing. I already said mail is hard, right? You better not do mistakes in your configuration, otherwise your mail is marked as spam quickly. So how about some online tests to check how good my new server scored against various spam filters? Here is a list of online checks of all kinds, including services to which someone can send mails to get them analyzed:

• Send a mail and get it tested against Spamassassin, SPF, DKIM, DMARC, rDNS, HELO: https://www.mail-tester.com/
• Send a mail and get it tested against SPF, DKIM, Spamassassin, including full message log: http://isnotspam.com/
• Live connection details of sending mails to server: https://www.wormly.com/test-smtp-server
• Various dedicated test tools (DMARC, DKIM, SPF, DNS, SMTP TLS, open relay): https://mxtoolbox.com
• Another set of dedicated test tools for SMTP Banner, Mail test, Blacklist lookup, abuse, open relay, imap banner, etc.: https://tools.dnsstuff.com/
• Check your DNS setup – this was the only one which correctly marked as red that I had my DNS MX entry pointed at a CNAME: https://intodns.com/

All these tests were green. And should always be! As a small private mail server I cannot afford it to have even the tiniest error. If you decide to setup something like this: do not proceed in your mail setup if some test shows something like “9/10” or other inferior results. Fix them all! I cannot stress this enough.

Having said that, you will realize that indeed this setup is not perfect: first and foremost, we will not be accepting mails via IPv6. Thus services testing delivery in IPv6 will report problems. Second, DANE is not working out of the box with Mailu. In the long term I hope that I will be able to update this guide to include both functions properly.

What’s next?

So the mail server was up and running. I was already able to use it with IMAP clients. And given my story leading to this setup you cannot believe how relieved I was once everything worked again and mails were coming in.

I knew that there was still a lot to do – and I will post more posts about the other steps in other blog posts – but the most important task was accomplished.

I’d like to thank the Mailu team for their awesome work on this piece of code – it is really great and I highly appreciate the ease of use and the simple admin capabilities.

Featured image by Felix Lichtenfeld from Pixabay

[Howto] Automated DNS resolution for KVM/libvirt guests with a local domain [Update]

I often run demos on my laptop with the help of libvirt. Managing 20+ machines that way is annoying when you have no DNS resolution for those. Luckily, with libvirt and NetworkManager, that can be easily solved.

The problem

Imagine you want to test something in a demo setup with 5 machines. You create the necessary VMs in your local KVM/libvirt environment – but you cannot address them properly by name. With 5 machines you also need to write down the appropriate IP addresses – that’s hardly practical.

It is possible to create static entries in the libvirt network configuration – however, that is still very inflexible, difficult to automate and only works for name resolution inside the libvirt environment. When you want to ssh into a running VM from the host, you again have to look up the IP.

Name resolution in  the host network would be possible by adding each entry to /etc/hosts additionally. But that would require the management of two lists at the same time. Not automated, far from dynamic, and very ponderous.

The solution

Luckily, there is an elegant solution: libvirt comes with its own in-build DNS server, dnsmasq. Configured properly, that can be used to serve DHCP and DNS to servers respecting a previous defined domain. Additionally, NetworkManager can be configured to use its own dnsmasq instance to resolve DNS entries – forwarding requests to the libvirt instance if needed.

That way, the only thing which has to be done is setting a proper host name inside the VMs. Everything else just works out of the box (with a recently Linux, see below).

The solution presented here is based on great post from Dominic Cleal.

Configuring libvirt

First of all, libvirt needs to be configured. Given that the network “default” is assigned to the relevant VMs, the configuration should look like this:

$ sudo virsh net-dumpxml default
<network connections='1'>
  <name>default</name>
  <uuid>158880c3-9adb-4a44-ab51-d0bc1c18cddc</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:fa:cb:e5'/>
  <domain name='qxyz.de' localOnly='yes'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.128' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

You can modify the network for example with the command virsh net-edit default. The interesting part is below the mac address: a local domain is defined and marked as localOnly. That domain will be the authoritative domain for the relevant VMs, and libvirt will configure dnsmasq to act as a resolver for that domain. The attribute makes sure that DNS requests regarding that domain will never be forwarded upstream. This is important to avoid loop holes.

Note, however: as mentioned in the comment by taurus, your domain should not be named “local” because this might cause trouble in relation to mDNS.

Configuring the VM guests

When the domain is set, the guests inside the VMs need to be defined. With recent Linux releases this is as simple as setting the host name:

$ sudo hostnamectl set-hostname neon.qxyz.de

There is no need to enter the host name anywhere else: the command above takes care of that. And the default configuration of DHCP clients of recent Linux releases sends this hostname together with the DHCP request – dnsmasq picks the host name automatically  up if the domain matches.

If you are on a Linux where the hostnamectl command does not work, or where the DHCP client does not send the host name with the request – switch to a recent version of Fedora or RHEL 😉

Because with such systems the host name must be set manually. To do so follow the documentation of your OS. Just ensure that the resolution of the name works locally. Additionally, besides the hostname itself the DHCP configuration must be altered to send along the hostname. For example, in older RHEL and Fedora versions the option

DHCP_HOSTNAME=neon.qxyz.de

has to be added to /etc/sysconfig/network-scripts/ifcfg-eth0.

At this point automatic name resolution between VMs should already work after a restart of libvirt.

Configuring NetworkManager

The last missing piece is the configuration of the actual KVM/libvirt host, so that the local domain, here qxyz.de, is properly resolved. Adding another name server to /etc/resolv.conf might work for a workstation with a fixed network connection, but certainly does not work for laptops which have changing network connections and DNS servers all the time. In such cases, the NetworkManager is often used anyway so we take advantage of its capabilities.

First of all, NetworkManager needs to start its own version of dnsmasq. That can be achieved with a simple configuration option:

$ cat /etc/NetworkManager/conf.d/localdns.conf 
[main]
dns=dnsmasq

This second dnsmasq instance just works out of the box. All DNS requests will automatically be forwarded to DNS servers acquired by NetworkManager via DHCP, for example. The only notable difference is that the entry in /etc/resolv.conf is different:

# Generated by NetworkManager
search whatever
nameserver 127.0.0.1

Now as a second step the second dnsmasq instance needs to know that for all requests regarding qxyz.de the libvirt dnsmasq instance has to be queried. This can be achieved with another rather simple configuration option, given the domain and the IP from the libvirt network configuration at the top of this blog post:

$ cat /etc/NetworkManager/dnsmasq.d/libvirt_dnsmasq.conf 
server=/qxyz.de/192.168.122.1

And that’s it, already. Restart NetworkManager and everything should be working fine.

As a side node: if the attribute localOnly would not have been set in the libvirt network configuration, queries for unknown qxyz.de entries would be forwarded from the libvirt dnsmasq to the NetworkManager dnsmasq – which would again forward them to the libvirt dnsmasq, and so on. That would quickly overload your dnsmasq servers, resulting in error messages:

dnsmasq[15426]: Maximum number of concurrent DNS queries reached (max: 150)

Summary

With these rather few and simple changes a local domain is established for both guest and host, making it easy to resolve their names everywhere. There is no need to maintain one or even two lists of static IP entries, everything is done automatically.

For me this is a huge relief, making it much easier in the future to set up demo and test environments. Also, it looks much nicer during a demo if you have FQDNs and not IP addresses. I can only recommend this setup to everyone who often uses libvirt/KVM on a local machine for test/demo environments.

[Howto] Look up of external sources in Ansible

Part of Ansible’s power comes from an easy integration with other systems. In this post I will cover how to look up data from external sources like DNS or Redis.

Background

A tool for automation is only as good as it is capable to integrate it with the already existing environment – thus with other tools. Among various ways Ansible offers the possibility to look up Ansible variables from external stores like DNS, Redis, etcd or even generic INI or CSV files. This enables Ansible to easily access data which are stored – and changed, managed – outside of Ansible.

Setup

Ansible’s lookup feature is already installed by default.

Queries are executed on the host where the playbook is executed – in case of Tower this would be the Tower host itself. Thus the node needs access to the resources which needs to be queried.

Some lookup functions for example for DNS or Redis servers require additional python libraries – on the host actually executing the queries! On Fedora, the python-dns package is necessary for DNS queries and the package python-redis for Redis queries.

Generic usage

The lookup function can be used the exact same way variables are used: curly brackets surround the lookup function, the result is placed where the variable would be. That means lookup functions can be used in the head of a playbook, inside the tasks, even in templates.

The lookup command itself has to list the plugin as well as the arguments for the plugin:

{{ lookup('plugin','arguments') }}

Examples

Files

Entire files can be used as content of a variable. This is simply done via:

vars:
  content: "{{ lookup('file','lorem.txt') }}"

As a result, the variable has the entire content of the file. Note that the lookup of files always searches the files relative to the path of the actual playbook, not relative to the path where the command is executed.

Also, the lookup might fail when the file itself contains quote characters.

CSV

While the file lookup is pretty simple and generic, the CSV lookup module gives the ability to access values of given keys in a CSV file. An optional parameter can identify the appropriate column. For example, if the following CSV file is given:

$ cat gamma.csv
daytime,time,meal
breakfast,7,soup
lunch,12,rice
tea,15,cake
dinner,18,noodles

Now the lookup function for CSV files can access the lines identified by keys which are compared to the values of the first column. The following example looks up the key dinner and gives back the entry of the third column: {{ lookup('csvfile','dinner file=gamma.csv delimiter=, col=2') }}.

Inserted in a playbook, this looks like:

ansible-playbook examples/lookup.yml

PLAY [demo lookups] *********************************************************** 

GATHERING FACTS ***************************************************************
ok: [neon]

TASK: [lookup of a csv file] **************************************************
ok: [neon] => {
    "msg": "noodles"
}

PLAY RECAP ********************************************************************
neon                       : ok=2    changed=0    unreachable=0    failed=0

The corresponding playbook gives out the variable via the debug module:

---
- name: demo lookups
  hosts: neon

  tasks:
    - name: lookup of a csv file
      debug: msg="{{ lookup('csvfile','dinner file=gamma.csv delimiter=, col=2') }}"

DNS

The DNS lookup is particularly interesting in cases where the local DNS provides a lot of information like SSH fingerprints or the MX record.

The DNS lookup plugin is called dig – like the command line client dig. As arguments, the plugin takes a domain name and the DNS type: {{ lookup('dig', 'redhat.com. qtype=MX') }}. Another way to hand over the type argument is via slash: {{ lookup('dig', 'redhat.com./MX') }}

The result for this example is:

TASK: [lookup of dns dig entries] *********************************************
ok: [neon] => {
    "msg": "10 int-mx.corp.redhat.com."
}

Redis

It gets even more interesting when existing databases are queried. Ansible lookup supports for example Redis databases. The plugin takes as argument the entire URL: redis://$URL:$PORT,$KEY.

For example, to query a local Redis server for the key dinner:

---
tasks:
  - name: lookup of redis entries
    debug: msg="{{ lookup('redis_kv', 'redis://localhost:6379,dinner') }}"

The result is:

TASK: [lookup of redis entries] ***********************************************
ok: [neon] => {
    "msg": "noodles"
}

Template

As already mentioned, lookups can not only be used in Playbooks, but also directly in templates. For example, given the template code:

$ cat templatej2
...
Red Hat MX: {{ lookup('dig', 'redhat.com./MX') }}
$ cat template.conf
...
Red Hat MX: 10 mx2.redhat.com.,5 mx1.redhat.com.

Conclusion

As shown the lookup plugin of Ansible provides many possibilities to integrate Ansible with existing tools and environments which already contain valuable data about the systems. It is easy to use, integrates well with the existing Ansible concepts and can quickly be integrated. Just drop it where a variable would be dropped, and it already works.

I am looking forward to more lookup modules support in the future – I’d love to see a generic “http” and a generic “SQL” plugin, even with the ability to provide credentials, although these features can be somewhat realized with already existing modules.

[Short Tip] dnsmasq and /etc/hosts

In case you do simple network tests with KVM virtual machines on your host, you might want to add some host names and IPs to /etc/hosts. However, that may not work: KVM, or better said libvirt ignores the entries in /etc/hosts. That is due to the fact that dnsmasq reads the entries of the file once: at startup. So you need to restart dnsmasq, or simply send it the SIGHUP signal:

killall -HUP dnsmasq

[Howto] Share Ethernet via Wifi with NetworkManager in KDE

I recently was taking part in a training at a place which had poor cellular reception, no wifi and only one single ethernet connection. Thus we had to the ethernet via wifi. I tried to do just that with my laptop via NetworkManager – and it worked out of the box.

The basic situation is rather common: you have one single network connection, and want to share it to other people or devices via wifi. If you want to do that manually you have to set up the wifi network on your own, including encryption, need to bring up a dhcp server, configure the routing and NATing, and so on. That can take quite some time, and is nothing you want to do during some precious training hours.

Thus I simply tried to bring up a shared wifi via the NetworkManager applet in KDE:

After providing a SSID name and configuring some security credentials the process was already done, and I was notified that the network was set up and ready. It was also shown in the plasma applet besides the ethernet connection:

And that was it already: everyone was able to connect to the network without any problems – and it didn’t even took me a minute to bring it all up. Since I know how much trouble it can be to bring such a connection up manually I was really impressed!

In case you want to give it a try, make sure that your wifi hardware and thus the appropriate driver; do support Access Point (AP) mode which is needed to bring up the wifi. If it says “for some devices only” you have no choice but give it a try.

By the way, in case you wonder about DNS and DHCP: both is done via dnsmasq as a local server, offering both. The DNS queries are forwarded to the DNS servers you got via DHCP from the ethernet connection (or, presumably the one you configured in NetworkManager).
However, I was not able to find any temporary configuration in /run or /var/ which showed the actual DNS servers – I had to call nm-tool to figure that one out:

$ nm-tool
- Device: eth0  [Standard-Ethernet] --------------------------------------------
[...]

  IPv4 Settings:
    Address:         192.168.3.27
    Prefix:          24 (255.255.255.0)
    Gateway:         192.168.3.1

    DNS:             192.168.2.4
    DNS:             192.168.2.3

If you know of any other way to find out these information, or even better simply the entire configuration of dnsmasq for that purpose please let me know =)

Besides, while the Plasma applet gave me the option to shut down the shared wifi network, I wasn’t able to bring it up again. There simply is no option in the network overview to fire up again such a network, thus I filled a bug report.

But, besides these two smaller issues, the plasma-nm applet and thus NetworkManager did a great job making sharing networks very easy.