Decimail ServerAs the figure shows Decimail Server has two components, an SMTP daemon and an IMAP daemon. Both of these communicate with the PostgreSQL database where the messages are stored. Email client applications communicate with the SMTP daemon for outgoing messages and the IMAP daemon for incoming messages. The SMTP daemon also communicates with the outside world, sending messages via a smart host and receiving them directly. Apart from using a database rather than a filesystem to store messages this is not a great deal different from any other mail server.
There are other possible configurations. For example you can use another MTA in parallel with Decimail, rather than a smarthost, for outgoing mail. Although this documentation focuses on how I use Decimail, it tries to be a component that you can use in a variety of ways.
Incoming messages are received by the SMTP daemon. They can be recieved directly from the internet, but so far I have not been brave enough to try this. A safer option is to use fetchmail to collect email from a "conventional" mail server on another machine and deliver it to the SMTP daemon. This has the advantage that if the SMTP daemon refuses to accept the message due to some misfeature it will remain safely on the external server, rather than being bounced. Currently I use Exim to receive messages from the outside world, and it forwards them to Decimail. Exim is better at communicating with misbehaving mailers (such as spambots) than Decimail is. The most likely reason for the Decimail SMTP daemon to dislike a message is illegal formatting of the message and in particular its Date: header.
The SMTP daemon first saves a backup of the message in a file. The backup file has two purposes. First it protects against Decimail or PostgreSQL going awry and trashing the copies of the messages stored in the database (something that it has never done to me, I must point out!). Second it makes keeping backups in case of hardware failure or computer theft easier: you backup the files, which is a straightforward task, and not the database, which would be more difficult (especially to do incrementally.)
Having saved the backup, the message is stored in the database. Only once this is complete is an acknowledgement sent to the SMTP client, once again minimising the possibility that something will go wrong.
Messages composed in a normal email client are sent by SMTP to the SMTP daemon. Messages that are addressed to local addresses are treated as incoming messages as above. Messages to non-local addresses are forwarded to an external "smart host" for onward routing.
In principal Decimail could be enhanced to deliver the messages to their final destinations rather than using a smart host. But this is a non-trivial task as, for example, you have to cope with temporarily unavailable servers and subsequent retries. I'd rather leave this problem to another program. I'm using Exim for this.
Users read email using any client that supports IMAP. The Decimail IMAP server retrieves messages from the database and sends them to the client in response to IMAP commands. Practically all of the IMAP spec is implemented and I have had success with most of the email clients that I have tried; see the Daemon Implementation document for details.
Essentially the database just stores each message in its own binary field - potentially a very large one. But to simplify queries some elements of the data, such as the subject, date and address headers, are extracted from the messages when they arrive and are recorded separately.
In addition to the messages the database stores a table of mailbox definitions. For each mailbox it stores a name and the SQL query that must be executed to retrieve the IDs of its messages. Here are some examples:
| mailbox | query |
|---|---|
| /Chronological/2004/May | select msg_id from messages where msgdate between '2004-5-1' and '2004-5-31' |
| /Lists/Postgresql SQL | select msg_id from recipients where addr='pgsql_sql@postgresql.org' |
| /Today | select msg_id from messages where msgdate > now()-'24 hours' |
When the client opens one of these mailboxes the IMAP daemon runs the corresponding query to get a list of matching messages. If new messages arrive a notification comes via the database and the list can be updated.
It is not necessary for this table to be populated by hand. More likely it will be a view formed from the union of a number of sub-tables, some created manually and others automatically. For example, the chronological mailboxes can clearly be generated automatically. Mailboxes can also be added to the table in response to IMAP commands using the "actions framework", with queries generated automatically.