Postfix before-queue Milter support

Introduction

Postfix version 2.3 introduces support for the Sendmail version 8 Milter (mail filter) protocol. This protocol is used by applications that run outside the MTA to inspect SMTP events (CONNECT, DISCONNECT), SMTP commands (HELO, MAIL FROM, etc.) as well as mail content. All this happens before mail is queued.

The reason for adding Milter support to Postfix is that there exists a large collection of applications, not only to block unwanted mail, but also to verify authenticity (examples: Domain keys identified mail, SenderID+SPF and Domain keys) or to digitally sign mail (examples: Domain keys identified mail, Domain keys). Having yet another Postfix-specific version of all that software is a poor use of human and system resources.

Postfix version 2.4 implements all the requests of Sendmail version 8 Milter protocols up to version 4, including message body replacement (body replacement is not available with Postfix version 2.3). See, however, the workarounds and limitations sections at the end of this document.

This document provides information on the following topics:

How Milter applications plug into Postfix

The Postfix Milter implementation uses two different lists of mail filters: one list of filters that are used for SMTP mail only, and one list of filters that are used for non-SMTP mail. The two lists have different capabilities, which is unfortunate. Avoiding this would require major restructuring of Postfix.

For those who are familiar with the Postfix architecture, the figure below shows how Milter applications plug into Postfix. Names followed by a number are Postfix commands or server programs, while unnumbered names inside shaded areas represent Postfix queues. To avoid clutter, the path for local submission is simplified (the OVERVIEW document has a more complete description).

SMTP-only
filters 		non-SMTP
filters
^
| 		|
v
^
|
|
| 		|
|
|
v
Network -> smtpd(8)
\
Network -> qmqpd(8) -> cleanup(8) -> incoming
/
pickup(8)
:
Local -> sendmail(1)

Building Milter applications

Milter applications have been written in C, JAVA and Perl, but this document deals with C applications only. For these, you need an object library that implements the Sendmail 8 Milter protocol. Postfix currently does not provide such a library, but Sendmail does.

On some Linux and *BSD distributions, the Sendmail libmilter library is installed by default. With this, applications such as dkim-milter and sid-milter build out of the box without requiring any tinkering:

$ gzcat dkim-milter-x.y.z.tar.gz | tar xf -
$ cd dkim-milter-x.y.z
$ make
[...lots of output omitted...]

On other platforms you have two options:

Running Milter applications

To run a Milter application, see the documentation of the filter for options. A typical command looks like this:

# /some/where/dkim-filter -u userid -p inet:portnumber@localhost ...other options...

Please specify a userid value that isn't used for other applications (not "postfix", not "www", etc.).

Configuring Postfix

Like Sendmail, Postfix has a lot of configuration options that control how it talks to Milter applications. With the initial Postfix Milter protocol implementation, many options are global, that is, they apply to all Milter applications. Future Postfix versions may support per-Milter timeouts, per-Milter error handling, etc.

Information in this section:

SMTP-Only Milter applications

The SMTP-only Milter applications handle mail that arrives via the Postfix smtpd(8) server. They are typically used to filter unwanted mail, and to sign mail from authorized SMTP clients. Mail that arrives via the Postfix smtpd(8) server is not filtered by the non-SMTP filters that are described in the next section.

NOTE: Do not use the header_checks(5) IGNORE action to remove Postfix's own Received: message header. This causes problems with mail signing filters. Instead, keep Postfix's own Received: message header and use the header_checks(5) REPLACE action to sanitize information.

You specify SMTP-only Milter applications (there can be more than one) with the smtpd_milters parameter. Each Milter application is identified by the name of its listening socket; other Milter configuration options will be discussed in later sections. Milter applications are applied in the order as specified, and the first Milter application that rejects a command will override the responses from other Milter applications. 

/etc/postfix/main.cf:
    # Milters for mail that arrives via the smtpd(8) server.
    # See below for socket address syntax.
    smtpd_milters = inet:localhost:portnumber ...other filters...

The general syntax for listening sockets is as follows:

unix:pathname

Connect to the local UNIX-domain server that is bound to the specified pathname. If the smtpd(8) or cleanup(8) process runs chrooted, an absolute pathname is interpreted relative to the Postfix queue directory.

inet:host:port

Connect to the specified TCP port on the specified local or remote host. The host and port can be specified in numeric or symbolic form.

NOTE: Postfix syntax differs from Milter syntax which has the form inet:port@host.

Non-SMTP Milter applications

The non-SMTP Milter applications handle mail that arrives via the Postfix sendmail(1) command-line or via the Postfix qmqpd(8) server. They are typically used to digitally sign mail. Although non-SMTP filters can be used to filter unwanted mail, there are limitations as discussed later in this section. Mail that arrives via the Postfix smtpd(8) server is not filtered by the non-SMTP filters.

NOTE: Do not use the header_checks(5) IGNORE action to remove Postfix's own Received: message header. This causes problems with mail signing filters. Instead, keep Postfix's own Received: message header and use the header_checks(5) REPLACE action to sanitize information.

You specify non-SMTP Milter applications with the non_smtpd_milters parameter. This parameter uses the same syntax as the smtpd_milters parameter in the previous section. As with the SMTP-only filters, you can specify more than one Milter application; they are applied in the order as specified, and the first Milter application that rejects a command will override the responses from the other applications. 

/etc/postfix/main.cf:
    # Milters for non-SMTP mail.
    # See below for socket address syntax.
    non_smtpd_milters = inet:localhost:portnumber ...other filters...

There's one small complication when using Milter applications for non-SMTP mail: there is no SMTP session. To keep Milter applications happy, the Postfix cleanup(8) server actually has to simulate the SMTP client CONNECT and DISCONNECT events, and the SMTP client EHLO, MAIL FROM, RCPT TO and DATA commands.

This generally works as expected, with only one exception: non-SMTP filters must not REJECT or TEMPFAIL simulated RCPT TO commands. When a non_smtpd_milters application REJECTs or TEMPFAILs a recipient, Postfix will report a configuration error, and mail will stay in the queue.

None of this is a problem for mail filters that digitally sign mail.

Milter error handling

The milter_default_action parameter specifies how Postfix handles Milter application errors. The default action is to respond with a temporary error status, so that the client will try again later. Specify "accept" if you want to receive mail as if the filter does not exist, and "reject" to reject mail with a permanent status. 

    # What to do in case of errors? Specify accept, reject, or tempfail.
    milter_default_action = tempfail

Milter protocol version

As Postfix is not built with the Sendmail libmilter library, you may need to configure the Milter protocol version that Postfix should use. The default version is 2. 

milter_protocol = 2

If the Postfix milter_protocol setting specifies a too low version, the libmilter library will log an error message like this: 

application name: st_optionneg[xxxxx]: 0xyy does not fulfill action requirements 0xzz

The remedy is to increase the Postfix milter_protocol version number. See, however, the limitations section below for features that aren't supported by Postfix.

If the Postfix milter_protocol setting specifies a too high version, the libmilter library simply hangs up without logging a warning, and you see a Postfix warning message like one of the following: 

postfix/smtpd[21045]: warning: milter inet:host:port: can't read packet header: Unknown error : 0
postfix/cleanup[15190]: warning: milter inet:host:port: can't read packet header: Success

The remedy is to lower the Postfix milter_protocol version number.

Milter protocol timeouts

Postfix uses different time limits at different Milter protocol stages. The table shows wich timeouts are used and when (EOH = end of headers; EOM = end of message).

Parameter Time limit Protocol stage
milter_connect_timeout 30s CONNECT
milter_command_timeout 30s HELO, MAIL, RCPT, DATA, UNKNOWN
milter_content_timeout 300s HEADER, EOH, BODY, EOM

Beware: 30s is not a lot for applications that do a lot of DNS lookups. However, if you increase the above timeouts too much, remote SMTP clients may hang up and mail may be delivered multiple times. This is an inherent problem with before-queue filtering.

Sendmail macro emulation

Postfix emulates a limited number of Sendmail macros, as shown in the table. Different macros are available at different SMTP protocol stages (EOM = end-of-message); their availability is not always the same as in Sendmail. See the workarounds section below for solutions.

Name Availability Description
i DATA, EOM Queue ID
j Always value of myhostname
_ Always The validated client name and address
{auth_authen} MAIL, DATA, EOM SASL login name
{auth_author} MAIL, DATA, EOM SASL sender
{auth_type} MAIL, DATA, EOM SASL login method
{client_addr} Always Client IP address
{client_connections} CONNECT Connection concurrency for this client
{client_name} Always Client hostname, "unknown" when lookup or verification fails
{client_ptr} CONNECT, HELO, MAIL, DATA Client name from reverse lookup, "unknown" when lookup fails
{cert_issuer} HELO, MAIL, DATA, EOM TLS client certificate issuer
{cert_subject} HELO, MAIL, DATA, EOM TLS client certificate subject
{cipher_bits} HELO, MAIL, DATA, EOM TLS session key size
{cipher} HELO, MAIL, DATA, EOM TLS cipher
{daemon_name} Always value of milter_macro_daemon_name
{mail_addr} MAIL Sender address
{rcpt_addr} RCPT Recipient address
{tls_version} HELO, MAIL, DATA, EOM TLS protocol version
v Always value of milter_macro_v

Postfix sends specific sets of macros at different SMTP protocol stages. The sets are configured with the parameters as described in the table (EOM = end of message).

Parameter name Protocol version Protocol stage
milter_connect_macros 2 or higher CONNECT
milter_helo_macros 2 or higher HELO/EHLO
milter_mail_macros 2 or higher MAIL FROM
milter_rcpt_macros 2 or higher RCPT TO
milter_data_macros 4 or higher DATA
milter_end_of_data_macros 2 or higher EOM
milter_unknown_command_macros 3 or higher unknown command

Workarounds

Content filters may break domain key etc. signatures. If you use an SMTP-based content filter, then you should add a line to master.cf with "-o disable_mime_output_conversion=yes" (note: no spaces around the "="), as described in the advanced content filter example.

Sendmail Milter applications were originally developed for the Sendmail version 8 MTA, which has a different architecture than Postfix. The result is that some Milter applications make assumptions that aren't true in a Postfix environment.

If you experience the ugly header problem, see if a recent version of the Milter application fixes it. For example, current versions of dkim-filter and dk-filter already have code that looks up the Postfix queue ID at a later protocol stage.

To fix the ugly message header with sid-filter applications, we change the source code, so that it does the queue ID lookup after Postfix receives the end of the message. 

        assert(ctx != NULL);
#endif /* !DEBUG */

        ret = mlfi_eoh(ctx);
        if (ret != SMFIS_CONTINUE)
                return ret;

NOTES:

To fix the ugly message header with other Milter applications, you will need to do something like this: 

dfc = cc->cctx_msg;
assert(dfc != NULL);

/* Determine the job ID for logging. */
if (dfc->mctx_jobid == 0 || strcmp(dfc->mctx_jobid, JOBIDUNKNOWN) == 0) {
        char *jobid = smfi_getsymval(ctx, "i");
        if (jobid != 0)
                dfc->mctx_jobid = jobid;
}

/* get hostname; used in the X header and in new MIME boundaries */

NOTES:

Limitations

This section lists limitations of the Postfix Milter implementation. Some limitations will be removed as the implementation is extended over time. Of course the usual limitations of before-queue filtering will always apply. See the CONTENT_INSPECTION_README document for a discussion.