Index

Tutorial:
ISP-style Email Service with Postfix 1.x

If you are running Debian-Sarge instead of Woody (3.0) please use the newer tutorial.

This tutorial is currently not correct and being rewritten. There has been a mixup with alias_maps and alias_mailbox_maps. Please don't use this tutorial. The updates tutorial will be available hopefully today. Sorry for the inconvenience.

What is this?

This tutorial is meant to help system administrators who want to receive email for multiple domains. In Postfix slang these are called 'postfix virtual domains'. They allow you to forward emails to different recipients and also protect you from spam because email is only accepted for defined email recipients. The users will then be able to fetch their emails using POP3 or you can deliver the emails locally. I have decided to use the Courier POP3 daemon and the so called 'maildir' format for email storage.

Normally you would configure the domains in the different configuration files of Postfix. But when the number of users, domains and daily changes on your system increases you may become tired of manually editing the configuration files. In my opinion the best solution then is a MySQL database storing the domains and email aliases. This is the way most web hosters offer their self-configurable email service.

(Unfortunately I did not yet figure out how quotas work in this setup. If you expect to learn how to enforce quotas then you will most likely be disappointed. Help is greatly appreciated on this topic.)

This approach has a few advantages:

What do I need?

You need at least the following Debian packages installed:

You are also expected to know a little about SMTP, POP3, Postfix, SQL statements, PAM (pluggable authentication modules) and general system administration.

Step 1: Create the database

You need to create a database first which holds the tables. If you are experienced in using MySQL you can of course do this work on the command line. However I would like to recommend installing 'phpmyadmin' which greatly simplifies MySQL database management.

First create a database. I call it 'provider' because I intend to do program a full-featured web GUI like most web hosting providers use. Either do this in phpmyadmin or use the following SQL statement:

CREATE DATABASE provider

(If you want to save a little time you can use this SQL dump to create the database and all necessary tables at once. That saves a little cut'n'pasting. It contains some additional tables that will be used later in the phpmywebhosting project - you may safely ignore them for now. I recommend however that you read this tutorial step-by-step because there are a few pitfalls which are not self-explaining. So be patient.)

Step 2: Create an alias table

Now you want to create a table called 'alias' . It does the same as the well-known /etc/aliases and defines system aliases. These are not used for virtual domains but only for local delivery. You may use it to forward emails directed to root to a real-life person. The fields are quite self explanatory. The id field is just an index and the alias and destination fields are the same as in an aliases file. Here comes the SQL statement:

CREATE TABLE alias (
id int(11) NOT NULL auto_increment,
alias varchar(128) NOT NULL default '',
destination varchar(128) NOT NULL default '',
PRIMARY KEY (id)
) TYPE=MyISAM;

Step 3: Create a relocated table

You may also want to create a 'relocated' table. Imagine a user has moved to another email address or is no longer working for your company. Then you may enter his/her email address ('email') here and the MTAs sending email to him will instantly get informed about his/her new address ('destination').

CREATE TABLE relocated (
id int(11) NOT NULL auto_increment,
email varchar(128) NOT NULL default '',
destination varchar(128) NOT NULL default '',
PRIMARY KEY (id)
) TYPE=MyISAM;

Step 4: Create a transport table

You can optionally define a transport table to change the mail routing. This allows you to provide additional information on where/how to send emails for a domain. For Postfix to use this feature via MySQL you need to have a 'transport' table. See the man page of 'transport' for further information. If you just want to manage your transport entries in /etc/postfix/transport then change the transport_maps to some local file in your main.cf.

CREATE TABLE transport (
id int(11) NOT NULL auto_increment,
domain varchar(128) NOT NULL default '',
destination varchar(128) NOT NULL default '',
PRIMARY KEY (id),
UNIQUE KEY domain (domain)
) TYPE=MyISAM;

Step 5: Create a virtual table

Now comes the most interesting table for your needs. In order to tell Postfix where to send an email it will look up the entries in the 'virtual' table. The email field contains the complete email address (with domain) and the destination field tells Postfix who should get this email forwarded. If the email address specified in as a destination contains just a user name and no domain it will delivered to the particular user of the domain which is specified in 'mydomain' in the main.cf. If you just want to deliver an email locally then put the email address both in the email and destination fields. Okay, let us create thetable like this:

CREATE TABLE virtual (
id int(11) NOT NULL auto_increment,
email varchar(50) NOT NULL default '',
destination text NOT NULL,
PRIMARY KEY (id)
) TYPE=MyISAM;

Step 6: Create a user table

In my case I do not only want to manage the reception of email via MySQL but also allow the users to create POP3 mailboxes. This table does two things: first it tells Postfix where in the file system the local email mailboxes are located and second it contains the password which the user needs to access the POP3 service. (If you neither want to specifiy the delivery location nor allow POP3 users then you may as well omit this table.)

Finally this is how the table looks:

CREATE TABLE users (
id int(11) NOT NULL auto_increment,
email varchar(128) NOT NULL default '',
password varchar(128) NOT NULL default '',
uid int(11) NOT NULL default '1008',
gid int(11) NOT NULL default '1008',
homedir varchar(128) NOT NULL default '/home/vmail',
maildir varchar(128) NOT NULL,
quota varchar(128) NOT NULL,
postfix enum('Y','N') NOT NULL default 'Y',
PRIMARY KEY (id),
UNIQUE KEY email (email)
) TYPE=MyISAM;

(I must admit that I haven't yet been successful in enforcing quotas. It seems like Postfix 1.x does not support it.)

Let me briefly explain the meaning of the fields. id is just an index field. email is the complete email address (with domain) of the user. Please note that I use this email address also as a username which the user will need to authenticate. password must contain a user password in crypt format (which you can create using the MySQL 'ENCRYPT()' function). uid and gid default to a pseudo user which I call 'vmail' who owns all the user mailboxes. (If you do not have created a user for that purpose do that now! Remember to put the user's UID and GUID as default values into the table description.)

The homedir is the destination where Postfix will store this user's emails. I use "/home/vmail" which is the "home directory" of the "vmail" user. The maildir entry is the additional path to this home directory. If it is a file (no trailing slash) then the email will be saved in mailbox format (readable with mutt and processable with procmail). If it is a directory (with a trailing slash) then the email will be saved in maildir format which puts every message in a seperate file suitable for use with the Courier POP3 daemon. If I have a user called "user@domain.tld" I insert "user@domain.tld/" here. You are free to name this directory whatever you like. Seperating homedir and maildir may look unnecessary but the Courier POP3 daemon needs it that way.

The quota field is not yet considered (unless someone helps me enforcing quotas). And finally the postfix field allows you to create a user account that is or is not used to deliver email via Postfix. If set to 'N' Postfix will not treat this account as a local delivery user. So most of the time you will want to leave this flag to 'Y'.

Step 7: Create a vmail user

In the last paragraph I told you to create a user account called "vmail". Have you done that? Remember that the UID and GID must match the appropriate fields in the 'user' table. Simply enter these lines in a root shell:

groupadd -g 1008 vmail[ENTER]
useradd -g vmail -u 1008 vmail

Step 8: Tell Postfix to use MySQL

First of all an important note (thanks to Ulrich Hammer). Remember that Postfix runs in a chroot environment. Thus it would not be able to access MySQL's socket file at /var/run/mysqld/mysqld.sock. There are two ways to work around this:

Now make sure that you have Postfix set up properly. These directives should be set in the file main.cf. Please make sure to change the values that are printed in strong letters to your personal needs. You will find other parameters like daemon_directory already in your main.cf. Don't touch them. The following list of parameters does not claim to make a complete configuration file.

Important! If your virtual domain is the same as your local domain then don't list $mydomain in your mydestination line.

# basic settings[ENTER]
myorigin = [your local domain][ENTER]
mydomain = [your local domain][ENTER]
mydestination = $myhostname, localhost.$mydomain, $mydomain[ENTER]
mynetworks = [local IP address range that is allowed to relay mail][ENTER]
[ENTER]
# virtual domain and delivery settings[ENTER]
virtual_mailbox_limit = 10000000[ENTER]
virtual_mailbox_base = /home/vmail[ENTER]
virtual_uid_maps = static:1008[ENTER]
virtual_gid_maps = static:1008[ENTER]
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-maps.cf[ENTER]
alias_maps = mysql:/etc/postfix/mysql-aliases.cf[ENTER]
relocated_maps = mysql:/etc/postfix/mysql-relocated.cf[ENTER]
transport_maps = mysql:/etc/postfix/mysql-transport.cf[ENTER]
virtual_maps = mysql:/etc/postfix/mysql-virtual.cf[ENTER]
[ENTER]
# define which email is accepted for delivery only[ENTER]
# alias_maps means: addresses in the MySQL aliase table[ENTER]
# virtual_mailbox_maps: addresses in the MySQL virtual table[ENTER]
# unix:passwd.byname: addresses of local users in /etc/passwd (optional)[ENTER]
local_recipient_maps = $alias_maps unix:passwd.byname[ENTER]
[ENTER]
# settings for authenticated SMTP[ENTER]
smtpd_sasl_auth_enable = yes[ENTER]
smtpd_sasl_local_domain = $myhostname[ENTER]
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, check_relay_domains[ENTER]
smtpd_sasl_security_options = noanonymous[ENTER]
broken_sasl_auth_clients = yes

As you can see Postfix will be querying the MySQL database according to the definitions in /etc/postfix/mysql-*.cf. For this to work you need to create these files first. They define how Postfix accesses the database and how you named the fields. Just copy and paste them:

/etc/postfix/mysql-virtual-maps.cf

user = [database admin user][ENTER]
password = [database admin password][ENTER]
dbname = provider[ENTER]
table = users[ENTER]
select_field = maildir[ENTER]
where_field = email[ENTER]
additional_conditions = and postfix = 'y'[ENTER]
hosts = [database server hostname]

/etc/postfix/mysql-aliases.cf

user = [database admin user][ENTER]
password = [database admin password][ENTER]
dbname = provider[ENTER]
table = alias[ENTER]
select_field = destination[ENTER]
where_field = alias[ENTER]
hosts = [database server hostname]

/etc/postfix/mysql-relocated.cf

user = [database admin user][ENTER]
password = [database admin password][ENTER]
dbname = provider[ENTER]
table = relocated[ENTER]
select_field = destination[ENTER]
where_field = email[ENTER]
hosts = [database server hostname]

/etc/postfix/mysql-transport.cf

user = [database admin user][ENTER]
password = [database admin password][ENTER]
dbname = provider[ENTER]
table = transport[ENTER]
select_field = destination[ENTER]
where_field = domain[ENTER]
hosts = [database server hostname]

/etc/postfix/mysql-virtual.cf

user = [database admin user][ENTER]
password = [database admin password][ENTER]
dbname = provider[ENTER]
table = virtual[ENTER]
select_field = destination[ENTER]
where_field = email[ENTER]
hosts = [database server hostname]

Step 9: Make Postfix understand authenticated SMTP (Auth-SMTP)

Imagine your users fetch their email using POP3. Now they need a way to send mails via your mail server. For security reasons Postfix allows users in the local (=trusted) network to send emails. (You obviously want this security if you don't want to work as an open relay for spammers.) My suggestion is that your users use authenticated SMTP. This means that their email clients provide a username and password to the mail server to make them "trusted". Most email cilents have this feature included.

Setting authenticated SMTP isn't that complicated. The only pitfall is that Debian runs Postfix in a chroot'ed environment by default. ("chroot'ed" means that Debian pretends that the "/var/spool/postfix" directory is pretended to be "/" - so Postfix cannot move out of this secure subdirectory. Anyone who finds a security problem and messes with Postfix can only do harm to that very subdirectory and not screw your whole system)

There may be different ways to use authentication with SMTP. I decided to use PAM (pluggable authentication modules). The authentication configuration is usually stored in "/etc/pam.d". As Postfix runs chroot'ed the configuration is searched for in "/var/spool/postfix/etc/pam.d". Just go there and put the following lines in a file called "smtp" (the backslashes show that all declarations are to be written in a single line):

auth required pam_mysql.so user=[database admin user] passwd=[database admin password] host=[database server hostname] db=provider table=users usercolumn=email passwdcolumn=password crypt=1[ENTER]
account sufficient pam_mysql.so user=[database admin user] passwd=[database admin password] host=[database server hostname] db=provider table=users usercolumn=email passwdcolumn=password crypt=1

The second step is to make SASL use the PAM for authentication. This is easy. Just put the following line in /etc/postfix/sasl/smtpd.conf:

pwcheck_method: pam

Third and last step: copy the /lib/security/pam_mysql.so to /var/spool/postfix/lib/security to make it accessible for Postfix (chroot'ed, remember?):

cp /lib/security/pam_mysql.so /var/spool/postfix/lib/security/

Step 10: Configure the Courier POP3/IMAP daemon for use with MySQL

The final step is now to tell the Courier POP3 daemon to use MySQL to access the username and password for authentication. First edit the file /etc/courier/authdaemonrc and change the directive authmodulelist to "authmysql" like this:

authmodulelist="authmysql"

Then you need to define the fields of the MySQL database table in /etc/courier/authmysqlrc like this:

MYSQL_SERVER [database server hostname][ENTER]
MYSQL_USERNAME [database admin user][ENTER]
MYSQL_PASSWORD [database admin password][ENTER]
MYSQL_PORT 0[ENTER]
MYSQL_DATABASE provider[ENTER]
MYSQL_USER_TABLE users[ENTER]
MYSQL_CRYPT_PWFIELD password[ENTER]
MYSQL_UID_FIELD uid[ENTER]
MYSQL_GID_FIELD gid[ENTER]
MYSQL_LOGIN_FIELD email[ENTER]
MYSQL_HOME_FIELD homedir[ENTER]
MYSQL_MAILDIR_FIELD maildir[ENTER]
MYSQL_QUOTA_FIELD quota

Do not forget to restart the authdaemon process using "/etc/init.d/courier-authdaemon restart".

If you like IMAP better you may also install "courier-imap" or "courier-imap-ssl" to provide IMAP services.

Step 11: Fill the database

Now to allow Postfix to actually receive email you need to write records into the database tables.

For every new (virtual) domain

For every new POP3 user

For every new "catch all" POP3 user

For every new email forwarding

For every new "catch all" email forwarding

For every new local user

Step 12: Cross your fingers and test your configuration

Start easy! Add one domain. Add one user account. Send one email. Read /var/log/mail.*.

How the delivery process works

The delivery process isn't well documented. But by reading a few kilobytes of debug output (see "Troubleshooting") I researched what steps Postfix takes to deliver an email:

Action SQL Query Result
Who shall be the recipient of the email? select destination from virtual where email='user@domain.tld' Either 'username' (if email is delivered locally) or 'username@domain.tld' (if email is forwarded to another address).
Is a relocation defined for this email address? select destination from relocated where email='user@domain.tld' Either there is no entry (which means there is no relocation) or an email address to the new user's email address at another place.
What transport shall be used for this domain? select destination from transport where domain = 'domain.tld' This defined the transport mode. 'virtual:' means to use virtual domains. Postfix delivers locally if no entry is found.
Is a relocation defined for this user? select destination from relocated where email = 'user' Either there is no entry (which means there is no relocation) or an email address where all emails of this domain are forwarded to.
Is a relocation defined for this domain? select destination from relocated where email = '@domain.tld' Either there is no entry (which means there is no relocation) or an email address where all emails of this domain are forwarded to.
Where shall the email be delivered? select maildir from users where email='user@domain.tld' and postfix='y'
The file system location where the email is delivered. The result is added to the homedir entry. If the result has a trailing slash then the email will be delivered in this directory in maildir format. If it has no trailing slash then this is the file where the email will be appended in mailbox format.
Who will be the owner of this email file? select uid from users where email = 'user@domain.tld' and postfix = 'y' A numerical UID. This should default to the user id of the "vmail" user.
Who will be the group of this email file? select gid from users where email = 'user@domain.tld' and postfix = 'y'

A numerical GID. This should default to the group id of the "vmail" user.

Troubleshooting

If it looks to you like nothing works at all it may be just a simple little typo in a configuration file or MySQL permission problem. I strongly advise that you temporarily enable MySQL debugging. To do that edit the file /etc/mysql/my.cnf, go to the [mysqld] section and append a line like "log = /var/log/mysql.log". Restart the MySQL daemon and you will get all queries logged into /var/log/mysql.log. Cute, huh? Send an email to your system and check the logs that Postfix really does MySQL queries. It also gives you a good understanding of what steps Postfix takes to deliver the email. Execute these queries one by one and check if the results are what you would have expected.

There are dozens of things you also need to take care of. Set MX records in your DNS zone. Open the TCP ports in your firewall. Check the permissions of /home/vmail. Just read the logs. You need to be a little patient.

Let me however give you a few frequently uttered problems and their causes:

Error message Cause Solution
postfix/smtpd[7285]: warning: connect to mysql server localhost: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) Your postfix server runs in a chroot jail and tries to connect to your MySQL server via the socket at /var/run/mysqld/mysqld.sock which is well outside the jail.

If you want to use TCP networking:
Specify something different than "localhost" to make Postfix use TCP instead of UNIX sockets.

If you want to use UNIX file system sockets:
Make sure that you re-create a socket symlink each time you start postfix because after a reboot the link in the chroot jail will lead nowhere. See Step 8.

Invalid command 'Auth_MySQL_DB', perhaps mis-spelled or defined by a module not included in the server configuration You are running the admin GUI without having the appropriate module installed into Apache. apt-get install libapache-mod-auth-mysql

What about the promised web GUI?

Many people have asked for my CGI-based web GUI which allows you to easily manage your email domains via the web browser. Feel free to download and use it under the terms of the GNU GPL. I'm very interested in your feedback and suggestions. To make it fly you will need the following three files:

File Meaning
ispmailadmin The main CGI file. Put this into your CGI directory and call it from your web browser (e.g. http://myserver/cgi-bin/ispmailadmin). Make sure this file is executable for your web server. Please verify if the configuration I define in the first lines of the Perl script suits your needs.
sqlutil.pm This is a MySQL utility class in Perl which I use for most of my database applications. Put it into the same directory as the 'ispmailadmin' file. You will need to change the lines at the beginning to tell it the hostname, username and password for accessing your MySQL database server or it will not work. Make sure it is not readable by others for security reasons.
style.css A CSS (cascading style sheet) file which should make the GUI look nice. It is not needed but you may want to copy it into your DocumentRoot path on the web server (or change a line in 'ispmailadmin' and put it somewhere else).

Please note that the CGI expects to called in an authenticated directory. If you forget this you will get a "Nobody logged on" error message. Make sure that you installed the 'libapache-mod-auth-mysql' package. To make your Apache web server ask the user for a name and password before using the web GUI you can use the following configuration directives in your httpd.conf:

<Location /cgi-bin/ispmailadmin>[ENTER]
AuthType Basic[ENTER]
AuthName "Customer login"[ENTER]
Auth_MySQL_DB provider[ENTER]
Auth_MySQL_Password_Table customers[ENTER]
Auth_MySQL_Username_Field customer[ENTER]
Auth_MySQL_Password_Field password[ENTER]
Auth_MySQL_Empty_Passwords off[ENTER]
#Auth_MySQL_Scrambled_Passwords on[ENTER]
Auth_MySQL_Encryption_Types Crypt_DES[ENTER]
require valid-user[ENTER]
</Location>

This assumes that you also have a MySQL table 'customers':

CREATE TABLE customers (
customer varchar(8) NOT NULL default '',
realname varchar(30) NOT NULL default '',
password tinytext NOT NULL,
PRIMARY KEY (customer)
) TYPE=MyISAM

In this table the customer and password fields are equivalent to the username and password fields your customers needs for logging in. The passwords must be encrypted by the ENCRYPT() function. The realname is the customers full name only used for display.

To make sure that a customer only changes the email aliases for his own domain(s) you must also create the table 'domains':

CREATE TABLE domains (
customer varchar(8) NOT NULL default '',
domain varchar(60) NOT NULL default ''
) TYPE=MyISAM

The customer field holds the same name as the field in the 'customers' table. The domain field holds the name of a domain this customers should gain access to. Of course there can be multiple entries if a customers has more than one domain.

What? You don't like my Perl script? Well, then go try the PHP scripts from Udo Müller. We are currently working on http://phpmywebhosting.sourceforge.net/

What about virus filtering?

Now that you operate your own professional email server you may offer your users a little more service: scanning incoming emails for viruses. Good news: it is perfectly easy to install. Bad news: it will most likely mean you have to spend money because good antivirus software does not come for free (yet).

Let's get to work. First make sure you installed the amavis-postfix package. Amavis is a piece of software that gets all emails forwarded, scans them using an external virus scanner and re-injects them into your mail system. It is important to note that Amavis does not include a virus scanner - it is just an interface between Postfix and your favorite virus scanner. So make sure you have a virus scanner already installed and keep it up-to-date.

Now please take a look at the /etc/amavisd.conf file. All configuration you find here are well explained and should be easy to edit. Make sure that you configure at least one virus scanner in the first part of the file. Restart the amavisd (/etc/init.d/amavis-postfix restart).

The last thing to do is tell Postfix that you want to have all your emails scanned. First edit the /etc/postfix/main.cf and add the following line:

content_filter = vscan:

Now you have told Postfix that you want to use the "vscan:" transport method on all emails. You may also consider setting "soft_bounce = yes" temporarily to make sure emails do not get bounced when something goes wrong. Emails will stay in your mail queue if you set this to true.

Finally tell Postfix what you mean by "vscan:". Add this section to the /etc/postfix/master.cf:

vscan unix - n n - 10 pipe flags=q user=amavis argv=/usr/sbin/amavis ${sender} ${recipient}[ENTER]
localhost:10025 inet n - n - - smtpd -o content_filter=

Restart Postfix (postfix reload) and you are set. You may want to watch the log file (tail -f /var/log/mail.log) and send a test mail. You should find a line like "X-Virus-Scanned: by AMaViS snapshot-20020222" in the email header. To make sure that your system handles infected emails correctly you may want to attach 'eicar.com'. Eicar is not a virus but a well-known (to virus scanners) signature to test if your virus recognition works. When you send this "infected" email to yourself you will receive an alert from Amavis with subject "VIRUS IN YOUR MAIL".

A last word to explain how Amavis works. Postfix receives your mail and pipes the email as defined in your "vscan" transport method into the amavis tool. Amavis will then use the configured virus scanner to scan all attachments. If the attachments are archives it will even use the archive tools to unpack them. After scanning the files it will contact Postfix on localhost's port 10025 and re-inject the email into the delivery process. But this time it sets the "content_filter" option to nothing and thus bypasses the content scanning this time. You will also see this behaviour in your /var/log/mail.log file. First you will see a "relay=vscan" and then a "relay=local".

What about webmail?

Now that you know how your users can get and send their emails via POP3, IMAP and SMTP you may wonder if there is a comfy way to give your users access to their mailbox via a web GUI. This is very easy if you followed the above steps. The package you need is called "sqwebmail" - a web mail system that uses the Courier services and can be told (via PAMs) to use any authentication scheme. Install it using "apt-get install sqwebmail".

To make sqwebmail use your MySQL database you need to change the /etc/pam.d/webmail file like this:

auth required pam_mysql.so user=<dbuser> passwd=<dbpassword> host=<dbhost> \
db=provider table=users usercolumn=email passwdcolumn=password crypt=1[ENTER]
account sufficient pam_permit.so

You also need to make the webmail CGI, icons and the style sheet visible for your web users. Assumed that you use an Apache web server you just put these lines in the /etc/apache/httpd.conf and you are set:

Alias /sqwebmail /usr/share/sqwebmail[ENTER]
ScriptAlias /webmail /usr/lib/courier/courier/webmail

Now access your web server like http://<hostname>/webmail/webmail and you should be presented a login dialog.

See also

I read a couple of other tutorials like this and still needed a lot of experiments to get my setup working. You may take a look at them if you are stuck here.

If you are not sure what 'transport' or 'virtual' does then take a look at their appropriate man pages.

Valuable contributions and corrections by

Support?

If you are totally desperate you may join the #postfix IRC channel at irc.freenode.net and ask. If you are lucky you will catch me there as "ChrisH".

Okay, show's over, no refunds, you heard the robot...

Christoph Haas (last change: 20.06.2004 ) ( 192 hits since 24.7.2004 )