[Howto] Launch traefik as a docker container in a secure way

Traefik is a great reverse proxy solution, and a perfect tool to direct traffic in container environments. However, to do that, it needs access to docker – and that is very dangerous and must be tightly secured!

The problem: access to the docker socket

Containers offer countless opportunities to improve the deployment and management of services. However, having multiple containers on one system, often re-deploying them on the fly, requires a dynamic way of routing traffic to them. Additionally, there might be reasons to have a front end reverse proxy to sort the traffic properly anyway.

In comes traefik – “the cloud native edge router”. Among many supported backends it knows how to listen to docker and create dynamic routes on the fly when new containers come up.

To do so traefik needs access to the docker socket. Many people decide to just provide that as a volume to traefik. This usually does not work because SELinux prevents it for a reason. The apparent workaround for many is to run traefik in a privileged container. But that is a really bad idea:

Docker currently does not have any Authorization controls. If you can talk to the docker socket or if docker is listening on a network port and you can talk to it, you are allowed to execute all docker commands. […]
At which point you, or any user that has these permissions, have total control on your system.

http://www.projectatomic.io/blog/2014/09/granting-rights-to-users-to-use-docker-in-fedora/

The solution: a docker socket proxy

But there are ways to securely provide traefik the access it needs – without exposing too much permissions. One way is to provide limited access to the docker socket via tcp via another container which cannot be reached from the outside that easily.

Meet Tecnativa’s docker-socket-proxy:

What?
This is a security-enhaced proxy for the Docker Socket.
Why?
Giving access to your Docker socket could mean giving root access to your host, or even to your whole swarm, but some services require hooking into that socket to react to events, etc. Using this proxy lets you block anything you consider those services should not do.

https://github.com/Tecnativa/docker-socket-proxy/blob/master/README.md

It is a container which connects to the docker socket and exports the API features in a secured and configurable way via TCP. At container startup it is configured with booleans to which API sections access is granted.

So basically you set up a docker proxy to support your proxy for docker containers. Well…

How to use it

The docker socket proxy is a container itself. Thus it needs to be launched as a privileged container with access to the docker socket. Also, it must not publish any ports to the outside. Instead it should run on a dedicated docker network shared with the traefik container. The Ansible code to launch the container that way is for example:

- name: ensure privileged docker socket container
  docker_container:
    name: dockersocket4traefik
    image: tecnativa/docker-socket-proxy
    log_driver: journald
    env:
      CONTAINERS: 1
    state: started
    privileged: yes
    exposed_ports:
      - 2375
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:z"
    networks:
      - name: dockersocket4traefik_nw

Note the env right in the middle: that is where the exported permissions are configured. CONTAINERS: 1  provides access to container relevant information. There are also SERVICES: 1 and SWARM: 1 to manage access to docker services and swarm.

Traefik needs to have access to the same network. Also, the traefik configuration needs to point to the docker container via tcp:

[docker]
endpoint = "tcp://dockersocket4traefik:2375"

Conclusion

This setup works surprisingly easy. And it allows traefik to access the docker socket for the things it needs without exposing critical permissions to take over the system. At the same time, full access to the docker socket is restricted to a non-public container, which makes it harder for attackers to exploit it.

If you have a simple container setup and use Ansible to start and stop the containers, I’ve written a role to get the above mentioned setup running.

[Howto] Workaround failing MongoDB on RHEL/CentOS 7

Ansible LogoMongoDB is often installed right from upstream provided repositories. In such cases with recent updates the service might fail to start via systemctl. A workaround requires some SELinux work.

Ansible Tower collects system data inside a MongoDB. Since MongoDB is not part of RHEL/CentOS, it is installed directly form the upstream MongoDB repositories. However, with recent versions of MongoDB the database might not come up via systemctl:

[root@ansible-demo-tower init.d]# systemctl start mongod
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[root@ansible-demo-tower init.d]# journalctl -xe
May 03 08:26:00 ansible-demo-tower systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....
-- Subject: Unit mongod.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mongod.service has begun starting up.
May 03 08:26:00 ansible-demo-tower runuser[7266]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
May 03 08:26:00 ansible-demo-tower runuser[7266]: pam_unix(runuser:session): session closed for user mongod
May 03 08:26:00 ansible-demo-tower mongod[7259]: Starting mongod: [FAILED]
May 03 08:26:00 ansible-demo-tower systemd[1]: mongod.service: control process exited, code=exited status=1
May 03 08:26:00 ansible-demo-tower systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mongod.service has failed.
-- 
-- The result is failed.
May 03 08:26:00 ansible-demo-tower systemd[1]: Unit mongod.service entered failed state.
May 03 08:26:00 ansible-demo-tower systemd[1]: mongod.service failed.
May 03 08:26:00 ansible-demo-tower polkitd[11436]: Unregistered Authentication Agent for unix-process:7254:1405622 (system bus name :1.184, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_

The root cause of the problem is that the MongoDB developers do not provide a proper SELinux</a configuration with their packages, see the corresponding bug report.

A short workaround is to create a proper (more or less) SELinux rule and install it to the system:

[root@ansible-demo-tower ~]# grep mongod /var/log/audit/audit.log | audit2allow -m mongod > mongod.te
[root@ansible-demo-tower ~]# cat mongod.te 

module mongod 1.0;

require {
	type locale_t;
	type mongod_t;
	type ld_so_cache_t;
	class file execute;
}

#============= mongod_t ==============
allow mongod_t ld_so_cache_t:file execute;
allow mongod_t locale_t:file execute;
[root@ansible-demo-tower ~]# grep mongod /var/log/audit/audit.log | audit2allow -M mongod
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i mongod.pp

[root@ansible-demo-tower ~]# semodule -i mongod.pp 
[root@ansible-demo-tower ~]# sudo service mongod start
                                                           [  OK  ]

Keep in mind that audit2allow generated rule sets are not to be used on production systems. The generated SELinux rules need to be analyzed manually to verify that it covers nothing but the problematic use case.