Decimail ServerI use a Debian GNU/Linux system. You can probably use Decimail Server on other systems without any great difficulty, and I would appreciate feedback.
At the time of writing very few people have installed Decimail Server. Please excuse any oddities that you encounter and PLEASE TELL ME ABOUT THEM! Don't hesitate to ask if you think something is wrong. Although I try to keep this document up to date, it's quite possible that it does not reflect the exact state of the current code.
This software is not simple to install: this will take a while. Please read through these instructions before you begin.
Decimail depends on numerous other bits of software, either to build or at runtime. I'll assume that you have the basic stuff like g++ installed. Here are the Debian packages names for some of the ones you might not have - package names for other distributions would be appreciated. Let me know if you find anything I have missed out.
On Debian the Postgres client libraries are built with support for Kerberos and other authentication / encryption facilities. Although Decimail doesn't use this, the libraries are still needed. This may also be the case with other distributions.
In addition, the following non-Debian programs are needed. I had no difficulty building them in /usr/local.
You'll need to create a (unix) user called "decimail" that the Decimail daemon processes will run as. You need to be able to "su" to this user so it should have a shell.
Create the directory where Decimail will keep backup copies of incoming messages:
Create a Postgresql user who will own the Decimail database:
Create the Decimail database:
Set up database permissions. The default Debian scheme allows each unix user to connect to the Postgresql server on the same machine as the Postgresql user with the same name as their unix name without authentication. In addition to this, root needs to be able to connect [*], and you probably want to be able to connect during development as your normal unix user. To do this, add the following to /etc/postgresql/pg_hba.conf:
and the following to /etc/postgresql/pg_ident.conf:
(Add one line for each user who needs to be able to connect to the database.)
Postgresql re-reads these files on SIGHUP.
I suggest checking that all of the necessary users can connect using psql. There are various things that can go wrong, and it's easier to fix them at this point rather than discovering that an obscure failure later is due to a postgresql permissions problem.
[*] Configuration information is stored in the database. This includes port numbers that the daemons listen on. Since only root can bind to priveledged port numbers, the daemons have to connect to the database once as root when they start in order to read the configuration.
tsearch2 is a Postgresql contrib package for text searching. You should be able to load it into your database with something as simple as:
A difficulty is what user to do this as. On one hand it needs to be the postgres superuser, since it loads a C module; on the other, it needs to be the decimail user since decimail will want to update the text indexes. It worked for me by doing it as the postgres user and then granting priviledges to decimail:
First, add the "plpgsql" language to the database:
You also need to make it possible to cast from the bytea type to text; do this as the postgres user:
The sql/*.sql files create the empty tables in the database. You ought to understand what they are doing and customise them to suit your needs before going any further. This is documented separately in the Using the Database document. Once you are happy with them, you can run them as follows:
Do this for each of the files in the following order:
(Do not run the file schema.sql which is currently present in the sql directory.)
Note that in each case the SQL starts by deleting any existing tables, views or whatever and then creates new ones in their place. This will lead to errors if there are no existing tables, e.g. the first time they are run; these errors can be safely ignored. Any errors towards the end of execution may be real errors. The creation commands are run within a transaction so either they will all work or they will all fail.
Note that re-running these files once there is data in your tables may DELETE ALL YOUR EMAIL! (Of course there are still backups in the files in /var/local/decimail/messages.) In particular the "create_*" files create the basic tables containing the messages themselves and other core data, and you really shouldn't re-run them unless you're quite certain what you are doing.
You need to insert data into some of the tables to define users and some related information. Make a script based on sql/example.sql. (Hmm, be careful, this file may not be very up-to-date.) If you want you can start with something simpler, but it does no harm to define, for example, mailing lists that you aren't actually subscribed to. Run it using psql as above.
Aspects of the configuration of the daemons are stored in the database. These include the port numbers on which the daemons listen and the "smart host" to which non-local mail is forwarded. See the section "Remove / Reconfigure Other Mailers" below for details. Currently the suggested way to define your configuration is to edit configuration.sql to suit your requirements and run it using psql as above. A newer alternative is to use the dmconfig script from the scripts directory.
If you subsequently change any of these settings you need to restart the daemons. Note that scripts/dmconfig doesn't do the for you.
libpbe is a collection of utility functions that I have used in Decimail and other projects; it is in the libpbe subdirectory. To compile it:
libpbe includes some stuff not used by Decimail. This won't be a problem, unless you don't have some dependency that one of these unused bits needs. In particular, if you don't have ImageMagick installed, you may want to append DISABLE_IMAGEMAGICK=1 to the make command.
There are three executables, the SMTP daemon, the IMAP daemon and the delete daemon (hmm, the delete daemon hasn't been tested recently; I don't often use it). To build them:
If you get any errors you may need to edit the Makefiles. In particular, the names of the Boost libraries are not the same on all platforms, and if you're not using Debian (for example if you've installed Boost from source) you'll need to change "-lboost_thread" to something like "-lboost_thread-gcc-mt_1_33_1".
At this point you could try out the SMTP and IMAP daemons. If you just run them from the command line they should detect that they're running interactively and conduct an SMTP or IMAP session with the console; failing that, -i will force them to be interactive. Getting sensible behaviour requires that you set up the database suitably earlier on, but in any case you should see some sort of activity.
Both daemons take PostgreSQL connection options on the command line. These are used to specify the database name, PostgreSQL username to connect as, and hostname and port number to connect to. For many of them (e.g. database on localhost) the default may be appropriate, but you'll probably need to specify the database name and username as shown below.
Here's an example SMTP session:
This session illustrates one of the things that could go wrong - my unix user doesn't have permission to access /var/local/decimail/messages, which is probably what you want as otherwise anyone could read everyone else's email. Another pitfall is that you're probably sending unix-style LF end of lines, whereas some bits of the fussy Decimail code expect strict RFC2822 CRLF.
Here's a slightly more promising IMAP session (of course you won't have any messages if you've only just installed it):
If that looks promising, run them with -d to make them daemons and try to connect either by telnetting to port 25 or 143 or perhaps even using a mail client. (You probably already have something listening on those ports that you'll need to stop; see the note below about other mailers, or change the configuration to listen on different ports.) (You should be able to run them interactively as any user who has permissions to connect to the database, but daemon mode with privileged port numbers (below 1024) requires that they start as root.)
The conventional place to put them is /usr/local/sbin:
(Or "make install" in each subdirectory.)
Most of the decimail configuration is stored in the database. The only thing that can't be stored there is information about how to connect to the database. Edit install/initd/decimail.conf to set this up; it gets copied to /etc/decimail.conf in the next step. See the PostgreSQL documentation for how to specify connection parameters. If you've used all the default settings you can probably set it to an empty string.
This is so that Decimail is automatically started at boot and stopped at shutdown.
(This installs appropriate symlinks from /etc/rc*.d to /etc/init.d. I think this is Debian-specific; other distributions will have their own way to do this. See the Makefile.)
The daemons will now start if you reboot, but you can start them manually like this (but see the note below about other mailers):
(You should probably choose to not start the delete daemon, whose purpose is to delete the backup file copies of messages that have been expunged from the database, until you are happy that things are running smoothly.)
(Note: currently, /etc/init.d/dmdeleted does not use the database connection parameters from /etc/decimail.conf, amongst other brokenness. This will be fixed soon.)
There are several ways in which Decimail can be used in connection with other MTAs. The key point is that Decimail does not do real routing of outgoing messages by itself. My setup runs exim on the same host for sending outgoing messages; I have also used Decimail alone on a machine with a smarthost supplied by my ISP.
If there is another machine on your local network or provided by your ISP that will forward outgoing messages for you, i.e. a "smarthost", this is the easiest approach to use. To do this:
If you don't have a smarthost, you can run another MTA to do outgoing message routing on the same machine as Decimail. In this case one or other has to run on a non-standard SMTP port number. For example, you could use exim on port 25 and Decimail on port 2525:
(In this case, outgoing mail doesn't pass through Decimail at all unless you set up your mail client to send to port 2525. This might be what you want, but it doesn't do anti-spam address transformation. If you want that, you need to use the opposite setup with exim on the non-standard port and Decimail forwarding to exim. Currently there is no way to configure the SMTP port that Decimail sends to, which is necessary to implement this. Let me know if you need this feature.)
If you have another MTA installed (sendmail, exim, qmail, etc.) you'll want to remove it if you are using the smarthost configuration. A difficulty for Debian users is that some other important packages require an MTA to be installed, and Debian won't let you remove your MTA without also removing these dependent packages. To overcome this, you need to tell Debian that you have installed an MTA - Decimail. To do this you need to use the "equivs" package (apt-get install equivs).
This should remove your existing MTA (though leaving its configuration files, so you can change back OK) while leaving other MTA-dependent packages alone.
N.B. A difficulty that I found with exim4 was that the exim4 package does not actually provide the mailer itself, but depends on one of a number of sub-packages with different variants of the exim daemon. You need to first remove the exim4 package (apt-get remove exim4), which should go without taking anything else with it (but check first with apt-get -s). Then install decimail-null as above; this should remove the actual exim daemon.
Decimail doesn't provide a "/usr/lib/sendmail" or similar command-line mechanism for injecting messages. Some applications may try to send mail by invoking /usr/lib/sendmail or /usr/sbin/sendmail, so you should provide something. One choice is the Debian ssmtp package. This does not seem to have a web page of its own, but users of other distributions could probably get it to work starting with the Debain source code (http://packages.debian.org/stable/source/ssmtp). In its configuration file, set the mailhub to localhost. ssmtp provides a sendmail-compatible binary that just forwards the messages by SMTP to the specified mailhub.
Another possibility is msmtp. In this case you'll need a wrapper script; an example, which I used for a while until I discovered ssmtp, is in install/sendmail/sendmail.
Probably the best way to import old email is to inject it via the smtpd. Perhaps you could use fetchmail to get it from your existing mailbox and sent it to Decimail? If that's not possible, there is an alternative which is to manually write the mail into the backup directory /var/local/decimal/messages and run an import utility.
Note that I went through this procedure once, and have not kept the code up to date. This probably won't work without some hacking. If you're sure that you can't import using fetchmail and SMTP, read on cautiously.
You ought to do this before you have started the daemons. The import utility talks directly to the database. In fact I'm not sure it's a good idea even to have test messages in the system. There is a comment below about how to delete everything.
First get the email into one message per file. "formail" is a good way to do this (it is in the procmail package); use the -s option with a script like this (which you can find in import/savemsg):
Have a look at the formail man page - it can do various useful transformations.
Make sure that the messages have CRLF line endings, rather than just LF. If necessary, convert using recode:
(Note that recode operates on files in-place, not as a filter.)
Now you need to rename the files to match Decimail's directory organisation for backup files. It has three levels of directories with 100 files or subdirectories each. (Decimail does this because directory lookups in "traditional" Unix filesystems, including ext3 in some configurations, do not scale well with the size of the directory. The optimum maximum directory size is apparently somewhere between 100 and 1000 entries.) So if you got a file called "2" from formail, it needs to be renamed "00/00/02". mmv (apt-get install mmv) is helpful for this. I suggest first prepending the necessary 0s one at a time (note that you press Ctrl-D to terminate mmv input):
Carry on until "2" is called "000002". Then insert the /s:
Unfortunately, mmv will not create new subdirectories (no -p option), so you need to make the various subdirectories before doing this. (Good method anyone? Something like ls | cut | mkdir should do it.)
It's OK to have gaps in the message numbering.
Now put all this under /var/local/decimail/messages.
The utility to import it into Decimail called dmimport. First build it with "make" in the import directory, and then invoke it like this:
It may take a while, depending on how many messages you are importing. dmimport puts a transaction around everything, so if something goes wrong the database is unchanged - no need to worry about only half of the messages having been read. The most likely problems are not having CRLF line endings and invalid Date: lines. If you find an invalid date you can either edit the message to make it RFC2822-style, or change common/Message.cc [this location might change] to add another format for strptime(). (The worst contenders for date format errors are automatically-generated messages. Please, anyone who generates email messages from a script, use RFC2822 style dates!!!)
IMAP requires that messages are in strictly ascending date order. I don't know if anything really bad would happen if this were violated (I don't think decimail cares), but it's wise to check. Try something like this:
This will list pairs of messages where the second has a date earlier than the first. Possible reasons include: the date was not correctly parsed (check the actual Date: headers, and remember timezones); the messages arrived out-of-order (dmimport has to use the Date: header, which indicates time of sending, not time of delivery); or the messages got out-of-order during splitting / importing.
If you want to fix it, try:
"man psql" to see how you can copy this output into a text file. Then do some shell trickery (awk?) to create a list of mv or cp commands to correct the order of the messages. (Easiest done on a set saved from before the subdirectoryfication.) Delete everything:
and re-import.
sql/create_db.sql sets the counter for new message IDs to 20000. IDs for the messages that you've imported come from their filenames. If you've imported more than 20000 messages, you'll need to change the counter so that new ones don't overlap imported ones: