Versions used

  • Python 3.5
  • Django 1.10
  • Django Rest Framework 3.7
  • IMAP IMAP4rev1


For us, the need for an e-mail interface emerged while developing backoffice tools and interfacing with partners, to track service requests and interface business worflows and backoffice tools together.

In our experience this is a very common need in our trade, and too often have we battled to avoid such interfaces and finally implemented them anyway. The result has always been a buggy interface, and us developers blaming the idea of an e-mail interface in itself.

So this time, we decided to approach the issue with a more positive attitude. In fact an SMTP API is a fertile idea if you need your system to interact with a number of different external partners and some of them prefer a simple e-mail. This makes sense because an e-mail combines notification, communication and display, provides broadcast, transfer and reply mechanisms, and e-mail softwares include many organizational features (filters, flags, snoozes, …).

For example, all major tracking systems (JIRA, Mantis, Redmine, …) have an SMTP interface that sends e-mails, but also accept incoming e-mails and handle them automatically.

Finally, interfacing Web APIs with multiple partners can raise issues in terms of infrastructure and security. The good thing about e-mails is: everyone has them. There will be no firewall issues, no need for a VPN installation or other networking issues.


UI tend to be built upon an API also used for machine to machine interactions. The API is mutualized, this reduces code size and cyclomatic complexity and hence the probability of bugs. This also helps on the testing side, as API tests will also cover a larger part of the UI execution paths.

The POSIX standard, among other things, sets the Command Line Interface as the common protocol for UI and API and provides standard I/O commands to interface software together (I/O redirection, pipe).
Web Apps
Web UIs are often built over a SOAP or REST API, with the HTTP protocol as a common underlying layer.
It makes sense to provide a more "machine friendly" API alongside the UI, for the same design reasons it makes sense for command line interfaces or Web apps. Even more so if some external partners want e-mails and others want to interface their own software with yours.


The 70's ARPANET file and message protocols are the ancestors of FTP and SMTP.

  • 1974 : TCP standard
  • 1981: SMTP standard
  • 1983: FTP is adapted for TCP
  • 1996: HTTP standard

For a long time, Remote Procedure Calls were designed to work on a local, proprietary network and SMTP was the only way for distant communications or communications between different computer brands.

A guide to Python SMTP API

SMTP offers a really nice set of extensions known as MIME (RFC2045 & RFC2046). We try to build upon these norms to provide a nice API over SMTP.


We listed the following requirements for our e-mail interface:

  • The mail content can be configured by the user; bleach this content to avoid injection.
  • The user writes a template with placeholders for the data.
  • The HTML version of the e-mail uses a common base (CSS, logo, etc.).
  • The e-mail has a plain text alternative matching the HTML content.
  • The e-mail has a JSON or YAML alternative convenient for machine interfaces.

The template could look like this:

Dear partner,

Please check vehicle {{vehicle.plate_number}} located {{vehicle.address}}.

Kind regards,

So we decided to use the possibility of mime/alternative content:

  • An application/json alternative contains all the data
  • A Markdown or reST template is used to build the text/plain alternative
  • The HTML version is injected into an HTML/CSS template for the text/html alternative
Workflow of e-mail content construction


There are no JSON or YAML text subtypes, so we use the application/json type. Sadly, this makes the JSON appear as an attachment in some applications, eg. GMail

We chose to always use UTF-8 for character encoding, as this is the standard default.
Custom identifier
If the e-mail is linked to an internal resource, we need an identifier for traceability. As advised in RFC6648, we chose to refrain from using custom headers. The message ID is not a good fit either, because we can have multiple e-mails for a single resource. As most people tend to do, we decided to include our identifier in the Reply-To address:
Subject: [a74e22bb] An Example
Message-ID: <>
Refers-To: <>
We use the Refers-To header to let mail applications group messages refering to the same issue together. However, some mail applications will group conversations depending on the subject, date and sender, and will not use the Refers-To header. We include some distinctive text in the subject to avoid having different e-mails grouped together if they're not meant to. We use a short version of our custom identifier for this purpose.
S/MIME is the way to go if we want to provide a security layer on our e-mail interface. This is the only standard out there; we could consider using GPG, but never a custom system.
MIME offers an extensive typing system for attachments and lets us transfer documents, photos, videos or even geolocalizations.

This is how our mail generation code looks:

import bleach
import django.conf.settings
import django.core.mail
import django.template.engines
import markdown
import rest_framework.renderers

def mail(email_address, subject, serializer, mail_template):
    message = django.core.mail.EmailMultiAlternatives(
        # use a unique ID for your item - eg. the instance PK
    # order alternatives carefully: last is preferred
    # here we use the "django" engine, but another one could do
    body = bleach.clean(
    # alternatively, we could try the text/markdown MIME type (RFC 7763)
    message.attach_alternative(body, 'text/plain')
            'reply_to': reply_to,
            # the mail_template is supposed to be in markdown format
            'body': markdown.markdown(body),


An SMTP interface should be easy to use for a human being, as Jon Postel said: "Be liberal in what you accept, and conservative in what you send". After all the guy invented SMTP in the first place, so his advice is worth considering.

We try to parse any text in a case insensitive way as people often fail to use upper or lower case as required.
As our software is available in multiple languages, we feel our e-mail interface should also accept any of these languages as input.
Punctuation and structure
We decided to consider all punctuations and spaces as a single separator, as this avoids encoding bugs and typos. When we need more structure, we use a light standard format like YAML or INI.
We always try to work it out from an HTML or plain text alternative, even if having a clear application/json part is the most reliable. Some partners use software or interfaces not able to produce a clean application/json part. Some people using the system manually send a single HTML part with no plain text alternative.
import email
import json
import re
import yaml
import bs4.BeautifulSoup

def process_message(data):
    message = email.message_from_bytes(data)
    attachments = []
    json_part = text_part = html_part = ''
    for part in message.walk():
            if part.is_multipart():
                continue  # sub-parts are iterated over in this walk
            payload = part.get_payload(decode=True)
            if part.get_content_type() == 'application/json':
                json_part = json.loads(payload.decode('utf-8'))
            if part.get_content_maintype() == 'text':
                payload = payload.decode(part.get_content_charset() or 'utf-8')
                # if multiple text/plain parts are given, concatenate
                if part.get_content_subtype() == 'plain':
                    text_part += payload
                # get plain text version from HTML using BeautifulSoup
                if part.get_content_subtype() == 'html':
                    soup = bs4.BeautifulSoup(payload, 'html.parser')
                    for item in soup(["script", "style"]):
                    html_part += '\n' + '\n'.join(
                        line.strip() for line in soup.get_text().split('\n')
                        if line
            # images, videos, PDFs and vendor-specific files are valid attachments
            if (
                    part.get_content_maintype() in ('image', 'video')
                    or part.get_content_type() == 'application/pdf'
                    or (
                        part.get_content_maintype() == 'application'
                        and part.get_content_subtype().startswith('vnd.')
                attachments.append((part.get_content_type(), payload))
        except Exception:
            continue  # parsing e-mail is hard; log and adapt incrementally

    # use HTML as plain text if none was found
    text_part = text_part or html_part
    # remove signature
    match ='^--\s*$', text_part, re.MULTILINE)
    if match:
        text_part = text_part[:match.start()].strip()
    # use plain text as JSON if none was found
    if not json_part:
            # using a yaml stream allows the user to write some JSON or YAML, then
            # use the '---' document separator to provide more unformatted text
            json_part = next(yaml.safe_load_all(text_part))
        except Exception:


Some inputs do fail because, you know, PEBCAK. We want our operators to be able to access the mailbox manually and fix the situation. Relying on an existing e-mail server and leaving the mails on it seemed the easiest way.

This means IMAP. Our software connects to the server, processes e-mails and moves them to a designated folder depending on the result (success or failure of processing).

Use UIDs
Message IDs are not stable on a mail server: they change as messages get moved, copied or deleted. We always use the UID version of the IMAP commands.
Close the connection after use
Some IMAP servers will reject new connections if we don't close old ones properly.
Copy & delete instead of move
Some interfaces don't implement the UID MOVE capability correctly: we tested it, but needed to copy and delete instead.
Using a proxy
At some point our organisation switched to an external IMAP server. Our infrastructure team set up a proxy for us to connect through. We devised an elegant way to use Python imaplib and http libraries for this (example below).
import http
import imaplib

class ImapServer(imaplib.IMAP4_SSL):
    def __init__(
        self, host='',
        self.proxy_host = proxy_host
        self.proxy_port = proxy_port
        super().__init__(host, port)

    def _create_socket(self):
        if self.proxy_host:
            connection = http.client.HTTPSConnection(
            connection.settunnel(, self.port)
            return connection.sock
        return super()._create_socket()

    def check(self, result):
        retval, data = result
        if retval != 'OK':
            raise self.error('IMAP error: %s' % data[-1] if data else '')
        return data

    def get(self, result):
        return self.check(result)[0]

    def unlist(self, result):
        return self.get(result).decode('ascii').split()

def fetch_mails():
    """Fetch mails from IMAP server INBOX.
    Messages are moved to 'processed' on success, or 'failed' otherwise.
    imap_server = imap_connect()
        messages = imap_server.unlist(imap_server.uid('search', None, 'ALL'))
        for message in messages:
            data = imap_server.get(imap_server.uid('fetch', message, 'BODY.PEEK[]'))
            except Exception:
                imap_server.uid('copy', message, 'failed')
                imap_server.uid('store', message, '+FLAGS', r'(\Deleted)')
                imap_server.uid('copy', message, 'processed')
                imap_server.uid('store', message, '+FLAGS', r'(\Deleted)')
    except Exception:
        pass  # log error


Parsing incoming e-mails is challenging. Multiple iterations are needed to achieve a stable code and, even then, we still monitor this interface and improve it over time, adapt to the most common mistakes human operators do and to the limitations of legacy softwares we want to interface with. This code could, in our opinion, benefit from open-source developments and we are now considering writing a more generic version of our code to publish a first version of a library for this purpose.