Versions used

  • Docker 1.8
  • VirtualBox 5
  • iptables 1.4.21

Integration testing of microservices

We develop car sharing services. They are composed of multiple libraries and interdependent systems (microservices) that communicate together: finance, payment and accounting; customer relationship; equipment monitoring; issue tracking; insurance claims; etc.

The pros and cons of microservices are well-known. Each library/system is easier to maintain and has a narrow, well-defined scope. On the other hand, communication between all parts increase complexity by introducing mutual dependencies. When it comes to ensuring that each system plays along with others nicely, we enter in the realm of integration testing.

We decided to automate integration tests. Our goal is to reproduce a production environment into a controlled environment that contains all our systems and their external dependencies such as an SMTP server, an Active Directory, etc. This article aims to explain how we filter traffic for our applications packaged with Docker so that only legitimate communications can get to the outside world. For example, pulling Docker images from a registry has to be allowed, but calls to pay-per-use services must be prevented.

The controlled environment is created using VirtualBox and iptables rules. Later, we decided to add an HTTP proxy to whitelist HTTP and HTTPS traffic based on hostname and not IP addresses, privoxy.

The VirtualBox setup is pretty basic, it must have the Docker daemon running so that our applications can be pulled and run as in a production environment.

Filter IP traffic

We want to set up our applications with the same configuration as on production. However, we do not want them to communicate with the outside world. It might call a production endpoint by mistake or consume pay-per-use service. Nobody wants this.

One way to do it is to override connection settings in our applications, but in this case additional settings are required to ensure no communication is made. We chose to use a whitelist on the firewall side. With this alternative, we need additional settings to allow an external communication, which is preferable.

Docker daemon configuration

Take back the iptables control from the Docker daemon. Add --iptables=false to the Docker daemon unit, located at /lib/systemd/system/docker.service on ArchLinux.

Now, reload the unit file and restart the service:

$ systemctl daemon-reload
$ systemctl restart docker.service

Setup iptables

Here is the final iptables rules file, may it serve you well.

*nat
-A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE
COMMIT

*filter
# Default policy: DROP all
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP

# Answer to established and related connections
-A INPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow anything on localhost
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

########################
# Docker configuration #
########################

# Allow dockers to communicate with other dockers
-A FORWARD -i docker0 -o docker0 -j ACCEPT

# Allow dockers to perform DNS lookups
-A FORWARD -p udp -i docker0 --dport 53 -j ACCEPT
-A FORWARD -p tcp -i docker0 --dport 53 -j ACCEPT

# Allow HTTP and HTTPS traffic
-A FORWARD -p tcp -i docker0 -m multiport --dports 80,443 -j ACCEPT

######################
# Host configuration #
######################

# Allow DNS lookup
-A OUTPUT -p udp --dport 53 -j ACCEPT
-A OUTPUT -p tcp --dport 53 -j ACCEPT

# Allow HTTP/HTTPS traffic through the proxy (if you decide to setup the HTTP proxy)
# -A OUTPUT -p tcp -m multiport --dports 80,443 -m owner --uid-owner privoxy -j ACCEPT

# Allow SSH connections on port 22
-A INPUT -p tcp --dport 22 -j ACCEPT

# Allow connection to the database from host
-A OUTPUT -p tcp --dport 5432 -j ACCEPT

# Allow incoming HTTPS connections
-A INPUT -p tcp --dport 443 -j ACCEPT

# Reject all other traffic
-A INPUT -j REJECT
-A FORWARD -j REJECT
-A OUTPUT -o eth0 -j REJECT

COMMIT

This file may be daunting if you are not used to iptables. Let's build it from scratch, step by step.

Tip

When working on your iptables rules file, use iptables-apply to apply it, instead of iptables-restore. It ensures your connection to the machine is still up (asking you to confirm so) before applying changes for good. This way, you won't be locked out.

A good way to start is to disable all traffic on the virtual machine. Create an iptables rules file (e.g. /etc/iptables/rules) and set the default policy to DROP for all the traffic. Do not apply those rules if you are using SSH to connect to the virtual machine, it would disconnect you and prevent any future SSH connection. If you use SSH, wait until the end of the iptables section before applying the rules.

*filter
# Incoming IP packets
-P INPUT DROP
# Packets from someone else (another IP address) destinated to another IP address
-P FORWARD DROP
# Outgoing IP packets
-P OUTPUT DROP
COMMIT

Now, every attempt to communicate should be dropped. Most applications will then fail on timeouts, which is not very convenient. It would be better to have a communication refused message when a packet is discarded by the firewall.

To reject the packets instead of simply dropping them, add the following rules right before the COMMIT instruction:

-A INPUT -j REJECT
-A FORWARD -j REJECT
-A OUTPUT -j REJECT

REJECT traffic

Rejecting communications ease later debugging, since you'll see an error message saying "your packet got discarded" instead of a timeout that could come from several causes.

It's time to allow some traffic. All the rules added from now on should be placed between the policy rules (line starting with -P) and the reject rules -A ...  REJECT as the iptables rules file is processed from top to bottom.

First, allow dockers to communicate with each other:

-A FORWARD -i docker0 -o docker0 -j ACCEPT

If you want them to connect to a network outside of the virtual machine, say the Internet, you'll also have to masquerade traffic from the dockers, so that the outside world can reply to the docker that performed the request. Add the following section in the rules file (e.g. before the *filter section):

*nat
-A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE
COMMIT

Warning

The IP subnet chosen by the Docker bridge might vary from one machine to another. Thus, you might have to change the IP subnet provided to POSTROUTING according to your setup.

You can find it using ip addr show docker0 on the machine running the Docker daemon.

For more details, read the summary of the Docker article about networking.

When a rule accepts traffic on a port for a TCP connection, it basically enables the TCP handshake to be handled correctly. Enabling established connections ensures subsequent traffic will be allowed. The related state is a bit more subtle, a good read is the official iptables description for connection state.

# TCP established and related connections
-A INPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT

# UDP established and related connections
-A INPUT -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT

From now on, when you allow a new connection, subsequent exchanges will be allowed. Pretty essential for SSH.

You can allow SSH incoming connections on default port (22).

-A INPUT -p tcp -dport 22 -j ACCEPT

Proceed the same way for all other traffic to allow. Probably DNS (53), the database, incoming HTTP and HTTPS requests, etc.

HTTP Proxy: when IP filtering is not enough

If you need to authorize outgoing HTTP traffic (PyPI [1], Docker registry, etc.), IP filtering is not enough: you need an HTTP proxy.

If you try to filter HTTP communications, you might end up with either rejected packets or accepted ones, for the same URL. It has to do with DNS resolution when several web servers listen to the same hostname (which is pretty common, e.g. for load-balancing purposes).

iptables resolves the IP address associated with one hostname once and for all, when the table is loaded. Thus only communications with this IP address will ever be allowed for the hostname. If another server is used for the same hostname, the communication will be rejected.

Such a behaviour may be encountered with the official PyPI [1].

Hence, filtering on an IP level cannot apply for this use case. HTTP/HTTPS is the correct "level" for filtering communications. We use privoxy to do it. It has a very simple setup, a whitelist feature (currently experimental) and you can easily mimic a whitelist using user actions.

Privoxy setup

Install privoxy for your distribution.

Modify the user.action file (e.g. located at /etc/privoxy/user.action) to block all traffic:

{ +block }
/

Unblock HTTP traffic based on hostname or path, following the syntax given in action files pattern documentation.

{ -block }
.pypi.python.org

Docker daemon setup

Once again, you may have to play around with the Docker daemon so that it knows about the system proxy. You should add or modify the Environment line in the Docker daemon unit file:

Environment="http_proxy=http://localhost:8118" "https_proxy=https://localhost:8118"

If you changed privoxy's listen port, you should change it here as well.

What's next?

This is the first step getting our integration tests running. There is a lot more setup ahead and we will probably write again on related matters. For now, this filtering seems to fit our needs, it's time to undertake some webservices configuration and Selenium tests.


[1](1, 2) PyPI is the official third-party software repository for Python.