[Howto] My own mail & groupware server, part 1: what, why, how?

Running your own mail and groupware server can be challenging. I recently had to re-create my own setup from the ground, and this blog post is the first in a series describing all the steps.

Running your own mail and groupware server can be challenging. I recently had to re-create my own setup from the ground, and this blog post is the first in a series describing all the steps.

Running mail servers on your own

Running your own mail server sounds tempting: having control over a central piece of your own communication does sound good, right?

But that can be quite challenging: The mail standards are not written for today’s way to operate technology. Many of them are vague, too generic to be really helpful, or just not deployed widely in reality. Other things are not standardized at all so you have to guess and test (max message sizes, anyone?). Also, even the biggest providers have their own interpretations of the standards and sometimes aggressively ignore them which you are forced to accept.

Also there is the spam problem: there is still a lot of spam out there. And since this is an ongoing fight mail server admins have to constantly adjust their systems to newest tricks and requirements. Think of SPF, DKIM, DMARC and DANE here.

Last but not least the market is more and more dominated by large corporations. If your email is tagged as spam by one of those, you often have no way to figure out what the problem is – or how to fix it. They simply will not talk to you if you are not of equal size (or otherwise important). In fact, if I have a pessimistic look into the future of email, it might happen that all small mail service providers die and we all have to use the big services.

Thus the question is if anyone should run their own mail server at all. Frankly, I would not recommend it if you are not really motivated to do so. So be warned.

However, if you do decide to do that on your own, you will learn a lot about the underlying technology, about how a core technology of “the internet” works, how companies work and behave, and you will have huge control about a central piece of today’s communication: mail is still a corner stone of today’s communication, even if we all hate it.

My background

To better understand my motivation it helps to know where I come from: In my past job at credativ I was project manager for a team dealing with large mail clusters. Like, really large. The people in the team were and are awesome folks who *really* understand mail servers. If you ever need help running your own open source mail infrastructure, get them on board, I would vouch for them anytime.

And while I never reached and never will reach the level of understanding the people in my team had, I got my fair share of knowledge. About the the technological components, the developments in the field, the challenges and so on. Out of this I decided at some point that it would be fun to run my own mail server (yeah, not the brightest day of my life, in hindsight…).

Thus at some point I set up my own domain and mail server. And right from the start I wanted more than a mail server: I wanted a groupware server. Calendars, address books, such a like. I do not recall how it all started, and how the first setup looked like, but I know that there was a Zarafa instance once, in 2013. Also I used OpenLDAP for a while, munin was in there as well, even a trac service to host a git repository. Certificates were shipped via StartSSL. Yeah, good times.

In summer 2017 this changed: I moved Zarafa out of the picture, in came SOGo. Also, trac was replaced by Gitlab and that again by Gitea. The mail server was completely based on Postfix, Dovecot and the likes (Amavisd, Spamassassin, ClamAV). OpenLDAP was replaced by FreeIPA, StartSSL by letsencrypt. All this was setup via docker containers, for easier separation of services and for simpler management. Nginx was the reverse proxy. Besides the groupware components and the git server there was also a OwnCloud (later Nextcloud) instance. Some of the container images were upstream, some I built myself. There was even a secondary mail server for emergencies, though that one was always somewhat out of date in terms of configuration.

This all served me well for years. Well, more or less. It was never perfect and missed a lot of features. But most mail got through.

Why the restart?

If it all served me well, why did I have to re-create the setup? Well, a few days ago I had to run an update of the certificates (still manually at that time). Since I had to bring down the reverse proxy for it, I decided run a full update of the underlying OS and also of the docker images and to reboot the machine.

It went fine, came back up – but something was wrong. Postfix had problems accepting mails. The more I dug down, the deeper the rabbit hole got. Postfix simply didn’t answer after the “DATA” part in the SMTP communication anymore. Somehow I got that fixed – but then Dovecot didn’t accept the mails for unknown reasons, and bounced were created!

I debugged for hours. But every time I thought I had figured it out, another problem came up. At one point I realized that the underlying FreeIPA service had erratic restarts and I had no idea why.

After three or four days I still had no idea what was going on, why my system was behaving that bad. Even with a verified working configuration from backup things went randomly broken. I was not able to receive or send mails reliably. My three major suspects were:

  • FreeIPA had a habit in the past to introduce new problems in new images – maybe this image was broken as well? But I wasn’t able to find overly obvious issues or reports.
  • Docker was updated from an outdated version to something newer – and Docker never was a friend of CentOS firewall rules. Maybe the recent update screwed up my delicate network setup?
  • Faulty RAM? Weird, hard to reproduce and changing errors of known-to-be-working setups can be the sign of faulty RAM. Maybe the hardware was done for.

I realized I had to make a decision: abandon my own mail hosting approaches (the more sensible option) – or get a new setup running fast.

Well – guess what I did?

Running your own mail server: there is a project for that!

I decided to re-create my setup. And this time I decided to not do it all by myself: Over the years I noticed that I was not the only person with the crazy idea to run their own mail server in containers. Others started entire projects around this with many contributors and additional tooling. I realized that I would loose little by using code from such existing projects, but would gain a lot: better tested code, more people to ask and discuss if problems arise, more features added by others, etc.

Two projects caught my interest over time, I followed them on Github for quite a while already: Mailu and mailcow. Indeed, my original plan was to migrate to one of them in the long term, like in 2021 or something, and maybe even hosted on Kubernetes or at least Podman. However, with the recent outage of my mail server I had to act quickly, and decided to go with a Docker based setup again.

Both projects mentioned above are basically built around Docker COmpose, Postfix, Dovecot, RSpamd and some custom admin tooling to make things easier. If you look closer they both have their advantages and special features, so if you think to run your own mail server I suggest you look into them yourself.

For me the final decision was to go with mailu: mailu does support Kubernetes and I wanted be prepared for a kube based future.

What’s next?

So with all this background you already know what to expect from the next posts: how to bring up mailu as a mail server, how to add Nextcloud and Gitea to the picture, and a few other gimmicks.

This will all be tailored to my needs – but I will try to keep it all as close to the defaults as possible. First to keep it simple but also to make this content reusable for others. I do hope that this will help others to start using their own setups or fine tuning what they already have.

Image by Gerhard Gellinger from Pixabay

[Howto] LDAP schema for Postfix

Postfix LogoThe official Postfix documentation to use LDAP for user and alias lookup mentions certain LDAP attributes which are not part of the default OpenLDAP. In this article I will shortly explain a basic theme providing these attributes and the corresponding object class.

Postfix can easily be connected to LDAP to lookup addresses and aliases. The Postfix LDAP documentation covers all the details. As mentioned there the default configuration of Postfix expects two LDAP attributes in the LDAP schema: mailacceptinggeneralid and maildrop. This also shows in the code in src/global/dict_ldap.c:

dict_ldap->query =
    cfg_get_str(dict_ldap->parser, "query_filter",
        "(mailacceptinggeneralid=%s)", 0, 0);

However, these attributes are not part of the default OpenLDAP installation, and the Postfix documentation does not mention how exactly that has to look like and where to get it. For that reason we at my employer credativ provide such a schema at Github: github.com/credativ/postfix-ldap-schema. The github repository contains the schema, the corresponding licence and a short documentation. A German introduction to the schema can also be found at credativ’s blog: LDAP-Schema für Postfix-Abfragen

The provided schema defines the necessary attribute types mailacceptinggeneralid and maildrop as well as the object class postfixUser. Please note that in this schema the used OIDs are of the type Experimental OpenLDAP, see also the OID database.

To use the schema it must be used by OpenLDAP, for example by including in in slapd.conf. A corresponding LDAP entry could look like:

dn: uid=mmu,ou=accounts,dc=example,dc=net
objectclass: top
objectclass: person
objectclass: posixAccount
objectclass: postfixUser
cn: Max Mustermann
sn: Mustermann
uid: mmu
uidNumber: 5001
gidNumber: 5000
homeDirectory: /home/vmail
mailacceptinggeneralid: mmu
mailacceptinggeneralid: max.mustermann
mailacceptinggeneralid: m.mustermann
mailacceptinggeneralid: bugs
maildrop: mmu

As you see the example covers multiple aliases. Also, the final mailbox is a domain less entity: maildrop: mmu does not mention any domain name. This only works if your mail boxes actually do not require (or even allow) domain names – in this case this was true since the mail is finally transported to a Dovecot server which does not know about the various domains.

Please note that this schema can only be the foundation for a more sophisticated, more complex schema which need to be tailored to fit the individual needs of the corresponding setup.

Postfix Architecture Overview

Postfix LogoPostfix consists of numerous services, commandos and queues. The documentation describes them in detail, but a general overview was missing – so I gave it a try.

The Postfix documentation is very detailed and of high quality. One example is postfix.org/OVERVIEW.html which explains the various services, queues and commandos and how they interact with each other. However, despite the included ASCII diagrams a “big picture” is missing. Since we do have quite some customers using Postfix at credativ my company gave me some time to spend at creating such an overview. Here you go:

Postfix Architecture Overview
Postfix Architecture Overview

The image links to the company website where you can download the PDF, but there is also a Postfix Architecture Overview Github repository including the odg, a pdf and a png.

Of course Postfix is far too complex to highlight all services and all commandos and all ways of communication, so the picture had to be simplified. For example, the process of flushing mails does only cover the deferred and incoming queue and not the processes which actually trigger the flushing. Also, policyd’s are missing, and I would actually like to include where Milter filters can be attached. Maybe I’ll update the graphic in the future. The license will be CC-BY-SA, so if you have ideas to extend the picture further you will be able to send me pull requests. Speaking about ideas, thanks to Patrick Ben Koetter who gave me the idea to create such an image.

[Howto] Sending test mails with Swaks

920839987_135ba34fff
Setting up e-mail servers can become a time consuming and complex task. Test mails can help verifying the functionality of the system – and here Swaks comes into play, the “swiss army knife for smtp”.

Swaks can be used to send test mails of all kinds. The advantage of Swaks compared to sending mails with your normal e-mail client is that you are able to alter almost any part of the e-mail: to, from, header, attachments, which server to speak to, etc.

Here are some common Swaks usage examples:

The first, basic example is sending a mail to your own server (here “bayz.de”):

$ swaks -f someone@example.net -t liquidat@example.com

If you need more recipients, add them via comma:

$ swaks -f someone@example.net -t liquidat@example.com,testme@example.com

It gets more interesting if you change the “TO” to another domain, but send the mail via the server for “bayz.de” nevertheless. If that works for arbitrary domains, and if the mails are forwarded to these you have big problem: an open relay.

$ swaks -f someone@example.net -t liquidat@example.com --server mail.example.com

Or do you need to know if a certain recipient is actually available?

$ swaks -f someone@example.net -t liquidat@example.com --quit-after RCPT

But you can also use Swaks to test a spam filter: If any of the bigger spam filters out there identifies the GTube sting in an e-mail, it will mark it as spam:

$ swaks -f someone@example.net -t liquidat@example.com --body /path/to/gtube/file

The same is true for anti virus programs and the Eicar file:

$ swaks -f someone@example.net -t liquidat@example.com --body /path/to/eicar/file

But Swaks can also be used to test user authentication and tls:

$ swaks -tls --server example.com -f liquidat@example.com -t someone@example.net  -ao --auth-user=liquidat

And this can of course be used to test authentication between servers:

$ swaks -tls -s example.com -f someone@example.net -t liquidat@example.com --ehlo $(host $(wget http://automation.whatismyip.com/n09230945.asp -O - -q))

The last bit makes sure your local test machine does provide a correct fqdn.

But in case your MTA setup does rely or use custom headers, how about adding some of these?

$ swaks -f someone@example.net -t liquidat@example.com --add-header "X-Custom-Header: Swaks-Tested"

If you have other interesting examples, don’t hesitate to drop them in the comments, I am happy to add them here.