Design

UI tend to be built upon an API also used for machine to machine interactions. The API is mutualized, this reduces code size and cyclomatic complexity and hence the probability of bugs. This also helps on the testing side, as API tests will also cover a larger part of the UI execution paths.

POSIX

The POSIX standard, among other things, sets the Command Line Interface as the common protocol for UI and API and provides standard I/O commands to interface software together (I/O redirection, pipe).

Web Apps

Web UIs are often built over a SOAP or REST API, with the HTTP protocol as a common underlying layer.

SMTP

It makes sense to provide a more "machine friendly" API alongside the UI, for the same design reasons it makes sense for command line interfaces or Web apps. Even more so if some external partners want e-mails and others want to interface their own software with yours.

Output

We listed the following requirements for our e-mail interface:

• The mail content can be configured by the user; bleach this content to avoid injection.
• The user writes a template with placeholders for the data.
• The HTML version of the e-mail uses a common base (CSS, logo, etc.).
• The e-mail has a plain text alternative matching the HTML content.
• The e-mail has a JSON or YAML alternative convenient for machine interfaces.

The template could look like this:

Dear partner,

Please check vehicle {{vehicle.plate_number}} located {{vehicle.address}}.

Kind regards,

So we decided to use the possibility of mime/alternative content:

• An application/json alternative contains all the data
• A Markdown or reST template is used to build the text/plain alternative
• The HTML version is injected into an HTML/CSS template for the text/html alternative

UTF-8

We chose to always use UTF-8 for character encoding, as this is the standard default.

Custom identifier

If the e-mail is linked to an internal resource, we need an identifier for traceability. As advised in RFC6648, we chose to refrain from using custom headers. The message ID is not a good fit either, because we can have multiple e-mails for a single resource. As most people tend to do, we decided to include our identifier in the Reply-To address:

From: mail-api@polyconseil.fr
Reply-To: mail-api+a74e22bb-3313-4dc4-af28-6c2bd1351c0f@polyconseil.fr
To: third-party@example.com
Subject: [a74e22bb] An Example
Message-ID: <mail-api-e4fa4b17-424f-47e2-93a3-58454c60f869@polyconseil.fr>
Refers-To: <mail-api-e4fa4b17-424f-47e2-93a3-58454c60f869@polyconseil.fr>

Subject

We use the Refers-To header to let mail applications group messages refering to the same issue together. However, some mail applications will group conversations depending on the subject, date and sender, and will not use the Refers-To header. We include some distinctive text in the subject to avoid having different e-mails grouped together if they're not meant to. We use a short version of our custom identifier for this purpose.

Security

S/MIME is the way to go if we want to provide a security layer on our e-mail interface. This is the only standard out there; we could consider using GPG, but never a custom system.

Attachments

MIME offers an extensive typing system for attachments and lets us transfer documents, photos, videos or even geolocalizations.

This is how our mail generation code looks:

import bleach
import django.conf.settings
import django.core.mail
import django.template.engines
import markdown
import rest_framework.renderers


def mail(email_address, subject, serializer, mail_template):
    message = django.core.mail.EmailMultiAlternatives(
        from_email='{}@{}'.format(
            django.conf.settings.APP_NAME,
            django.conf.settings.EMAIL_HOSTNAME,
        ),
        # use a unique ID for your item - eg. the instance PK
        reply_to=[
            '{}+{}@{}'.format(
                django.conf.settings.APP_NAME,
                serializer.instance.pk,
                django.conf.settings.EMAIL_HOSTNAME,
            )
        ],
        to=email_address,
        subject=subject,
    )
    # order alternatives carefully: last is preferred
    message.attach_alternative(
        rest_framework.renderers.JSONRenderer().render(serializer.data),
        'application/json'
    )
    # here we use the "django" engine, but another one could do
    body = bleach.clean(
        django.template.engines['django']
        .from_string(mail_template)
        .render(serializer.data)
    )
    # alternatively, we could try the text/markdown MIME type (RFC 7763)
    message.attach_alternative(body, 'text/plain')
    message.attach_alternative(
        django.template.engines['django'].get_template('email_base.html').render({
            'reply_to': reply_to,
            # the mail_template is supposed to be in markdown format
            'body': markdown.markdown(body),
        }),
        'text/html'
    )
    message.send()

Input

An SMTP interface should be easy to use for a human being, as Jon Postel said: "Be liberal in what you accept, and conservative in what you send". After all the guy invented SMTP in the first place, so his advice is worth considering.

Case

We try to parse any text in a case insensitive way as people often fail to use upper or lower case as required.

Languages

As our software is available in multiple languages, we feel our e-mail interface should also accept any of these languages as input.

Punctuation and structure

We decided to consider all punctuations and spaces as a single separator, as this avoids encoding bugs and typos. When we need more structure, we use a light standard format like YAML or INI.

Alternatives

We always try to work it out from an HTML or plain text alternative, even if having a clear application/json part is the most reliable. Some partners use software or interfaces not able to produce a clean application/json part. Some people using the system manually send a single HTML part with no plain text alternative.

import email
import json
import re
import yaml
import bs4.BeautifulSoup

def process_message(data):
    message = email.message_from_bytes(data)
    attachments = []
    json_part = text_part = html_part = ''
    for part in message.walk():
        try:
            if part.is_multipart():
                continue  # sub-parts are iterated over in this walk
            payload = part.get_payload(decode=True)
            if part.get_content_type() == 'application/json':
                json_part = json.loads(payload.decode('utf-8'))
            if part.get_content_maintype() == 'text':
                payload = payload.decode(part.get_content_charset() or 'utf-8')
                # if multiple text/plain parts are given, concatenate
                if part.get_content_subtype() == 'plain':
                    text_part += payload
                # get plain text version from HTML using BeautifulSoup
                if part.get_content_subtype() == 'html':
                    soup = bs4.BeautifulSoup(payload, 'html.parser')
                    for item in soup(["script", "style"]):
                        item.extract()
                    html_part += '\n' + '\n'.join(
                        line.strip() for line in soup.get_text().split('\n')
                        if line
                    )
            # images, videos, PDFs and vendor-specific files are valid attachments
            if (
                    part.get_content_maintype() in ('image', 'video')
                    or part.get_content_type() == 'application/pdf'
                    or (
                        part.get_content_maintype() == 'application'
                        and part.get_content_subtype().startswith('vnd.')
                    )
                ):
                attachments.append((part.get_content_type(), payload))
        except Exception:
            continue  # parsing e-mail is hard; log and adapt incrementally

    # use HTML as plain text if none was found
    text_part = text_part or html_part
    # remove signature
    match = re.search(r'^--\s*$', text_part, re.MULTILINE)
    if match:
        text_part = text_part[:match.start()].strip()
    # use plain text as JSON if none was found
    if not json_part:
        try:
            # using a yaml stream allows the user to write some JSON or YAML, then
            # use the '---' document separator to provide more unformatted text
            json_part = next(yaml.safe_load_all(text_part))
        except Exception:
            pass

IMAP

Some inputs do fail because, you know, PEBCAK. We want our operators to be able to access the mailbox manually and fix the situation. Relying on an existing e-mail server and leaving the mails on it seemed the easiest way.

This means IMAP. Our software connects to the server, processes e-mails and moves them to a designated folder depending on the result (success or failure of processing).

Use UIDs

Message IDs are not stable on a mail server: they change as messages get moved, copied or deleted. We always use the UID version of the IMAP commands.

Close the connection after use

Some IMAP servers will reject new connections if we don't close old ones properly.

Copy & delete instead of move

Some interfaces don't implement the UID MOVE capability correctly: we tested it, but needed to copy and delete instead.

Using a proxy

At some point our organisation switched to an external IMAP server. Our infrastructure team set up a proxy for us to connect through. We devised an elegant way to use Python imaplib and http libraries for this (example below).

import http
import imaplib


class ImapServer(imaplib.IMAP4_SSL):
    def __init__(
        self, host='',
        port=imaplib.IMAP4_SSL_PORT,
        proxy_host='',
        proxy_port=imaplib.IMAP4_SSL_PORT,
    ):
        self.proxy_host = proxy_host
        self.proxy_port = proxy_port
        super().__init__(host, port)

    def _create_socket(self):
        if self.proxy_host:
            connection = http.client.HTTPSConnection(
                host=self.proxy_host,
                port=self.proxy_port,
            )
            connection.settunnel(self.host, self.port)
            connection.connect()
            return connection.sock
        return super()._create_socket()

    def check(self, result):
        retval, data = result
        if retval != 'OK':
            raise self.error('IMAP error: %s' % data[-1] if data else '')
        return data

    def get(self, result):
        return self.check(result)[0]

    def unlist(self, result):
        return self.get(result).decode('ascii').split()

def fetch_mails():
    """Fetch mails from IMAP server INBOX.
    Messages are moved to 'processed' on success, or 'failed' otherwise.
    """
    imap_server = imap_connect()
    try:
        imap_server.check(imap_server.select('INBOX'))
        messages = imap_server.unlist(imap_server.uid('search', None, 'ALL'))
        for message in messages:
            data = imap_server.get(imap_server.uid('fetch', message, 'BODY.PEEK[]'))
            try:
                process_message(data[1])
            except Exception:
                imap_server.uid('copy', message, 'failed')
                imap_server.uid('store', message, '+FLAGS', r'(\Deleted)')
            else:
                imap_server.uid('copy', message, 'processed')
                imap_server.uid('store', message, '+FLAGS', r'(\Deleted)')
    except Exception:
        pass  # log error
    finally:
        imap_server.close()
        imap_server.logout()

Conclusion

Parsing incoming e-mails is challenging. Multiple iterations are needed to achieve a stable code and, even then, we still monitor this interface and improve it over time, adapt to the most common mistakes human operators do and to the limitations of legacy softwares we want to interface with. This code could, in our opinion, benefit from open-source developments and we are now considering writing a more generic version of our code to publish a first version of a library for this purpose.