DCC Greylists

This page is mirrored at Rhyolite Software and dcc-servers.net.

Introduction

In 2003 Evan Harris announced a notion he called greylisting. Greylisting does not absolutely reject mail, but requires mail from unfamiliar senders to be retransmitted by their ISPs' SMTP clients. Mail from familiar senders is passed immediately.

The idea is to delay mail from unfamiliar senders for half an hour, but immediately deliver mail from regular correspondents. It is based on the observation that large amounts of spam is sent via open proxies, botnets, and other mechanisms that do not involve proper mail transfer agents (MTAs). A proper sending MTA will repeat a transmission after a temporary 4yz rejection. RFC 2821 says that the sending MTA should retransmit 30 minutes or later after a failure, but spam sent through an open proxy as well as some viruses and worms are not retransmitted.

Greylisting is extremely effective against spam that is not otherwise detected by DCC clients. If you cannot use greylisting, consider body URL blacklisting by adding something like -Bsbl-xbl.spamhaus.org,any to DCCM_ARGS or DCCIFD_ARGS in /var/dcc/dcc_conf.

In the DCC implementation of greylisting the sendmail milter interface, dccm, or the general MTA interface, dccifd, sends a request to a modified version of the DCC server, greylist dccd. The requests contains the simple DCC body checksum of the message as well as an MD5 checksum of the MD5 checksums of IP address of the SMTP client sending the mail message, the envelope sender or Mail From value of the message, and the recipient or envelope Rcpt To value of the message. If the combination IP address, sender, and recipient is familiar, the DCC client tells the MTA to accept the message. Otherwise the DCC client tells the MTA to embargo or temporarily reject the message.

If the sending MTA persists and retransmits the message after the embargo but within the wait time, the triple (sender, IP address, addressee) is added to the database.

Considerations, Caveats, and Differences

• Greylisting helps DCC clients detect bulk mail sooner. The first time a message is embargoed, DCC clients report its checksums and target counts to their DCC server. By the end of the embargo at the first targets, enough other instances of a bulk message often have been seen by the DCC network to identify it as bulk for the first target.

• Greylisting must be done in an SMTP server or mail receiving system. This is why dccm and dccifd support greylisting, but dccproc does not.

• Greylisting can help reject spam at MX secondaries. It is common for unsolicited bulk mail to be sent to MX secondaries instead of primaries because secondaries often lack the filtering of primaries. Greylisting can defend against this attack while preserving the usefulness of MX secondaries as backups for primaries.

• If a triple is ever associated with spam, it is deleted from the greylist database by DCC clients. This renews the temporary embargo for subsequent mail involving the triple

• Unless the retransmitted message is identical to the original or the DCC greylist server is using weak-body greylisting it will not end the greylist embargo.

• Because greylisting in DCC clients uses slightly modified DCC server code and the DCC client-server and server-to-server protocols, it is easy to have multiple greylist servers handling locally distributed database of triples and timestamps. Each DCC client can be told with cdcc add to pick the fastest, closest working greylist server from a list. See the DCC installation instructions.

• All or part of the IP address of the SMTP client can be optionally ignored by DCC clients as far as the greylist triple is concerned. This feature may be useful for legitimate mail systems that shuffle messages among SMTP clients between retransmissions. See the dccm and dccd man pages.

• False positives are possible if a legitimate SMTP client does not retransmit an embargoed message after the embargo expires. Failing to retransmit clearly violates section 4.5.4 of the RFC 282 SMTP standard, because it will result in lost mail should something other than greylisting cause a temporary failure. Greylisting can be considered as causing failures that standards compliant SMTP clients will recover.

  Such false positives seem to be rare, but they do happen. That is one reason why embargoed messages are put into the dccm and dccifd per-user logs. On the other hand, such log entries needless excite some end users, because legitimate mail is almost always quickly retransmitted and delivered. Greylist log entries can be turned off for some or all users with option greylist-off lines in the global or per-user whiteclnt file.

• False negatives are common. Greylisting can only detect bogus SMTP clients. Bulk mail advertisers that use standards compliant software pass greylisting.

• Some legitimate SMTP clients have ridiculous retransmission delays. Approximately ten seconds is distressingly common. Some SMTP clients delay only a single second! Delaying for such short times not only violates RFC 2821, but is an utterly stupid waste of both SMTP client and server bandwidth, disk space, and CPU cycles. It makes no sense to assume that a problem that caused the temporary rejection of mail will be solved within seconds. Common causes of temporary SMTP failures include overflowing file systems and overloaded computers.

  Such nonsense affects SMTP servers using greylisting by causing them to receive, log, and reject unnecessary copies of messages until the embargo expires.

  Delays before retransmissions of 1 minute, 5 minutes, and 15 minutes are almost as common as the 30 minute delay from RFC 2821. A greylist embargo of 4.5 minutes seems to be long enough to exclude "open proxy" spam while ending before the first retransmission of an SMTP client using 5 minute delays between retransmissions.

• DCC clients apply their local white- and blacklists before checking with a DCC greylist server. Mail that is whitelisted is not subject to greylist embargoes. This is important, because many mail user agents (MUAs) such as WWW browsers do not understand the temporary rejections of greylisting. Sendmail.cf can be modified with the misc/hackmc -T script in the DCC source to tell dccm to completely whitelist mail senders that have been authenticated with SMTP-TLS or SMTP-AUTH. Otherwise, it is usually necessary to add lines like

      submit  ip  10.2.3.0/24
  

  listing mail submission clients to the main whiteclnt file.

• DCC per-user whitelists can turn off greylisting for individual mailboxes. The log files that DCC clients normally generate for temporarily embargoed or greylisted messages also can be turned off for some or all users. See the overall description of the DCC.

• Blacklisted mail is rejected regardless of greylisting. Any determination that a message is spam deletes the greylist triple from the greylist servers' databases.

• DCC greylist servers can use whitelists of all of the usual DCC values except env_To values to whitelist triples against embargoes. (Env_To values can be used for whitelisting on DCC greylisting clients.)

  A DCC server can have whitelist entries in its /var/dcc/whitelist file, but that mechanism is rarely useful. However, essentially the same mechanism using a greylist server's /var/dcc/grey_whitelist file. The grey_whitelist files of all members of a group of cooperating servers must be the same.

• Because DCC clients do greylisting at the end of the SMTP DATA command, the entire mail message is available in the DCC global and per-user logs. Such logs extremely valuable, because they allow users to know not only the senders of filtered mail the contents.

• DCC greylisting normally requires not only that a triple of sender, recipient and SMTP client IP address be familiar, but also that a triple does not become familiar until the same message has been presented and temporarily embargoed or rejected. This significant difference from other implementations prevents a spammer from retransmitting a different message (e.g. with different random words) to get past greylist filtering. This feature can be turned off with dccd -G weak-body.

• DCC greylisting does not create whitelist entries based on outgoing mail. Many people advocate preemptively whitelisting recipients of mail based on the assumption that they will respond from the same IP address and SMTP envelope sender. That assumption is true only in relatively rare and minor cases. Many systems send mail from SMTP clients at IP address that differ from the IP addresses of their receiving SMTP servers.

• Some mail systems use many IP addresses for their SMTP clients. Each appears distinct to a greylisting system. Mail from their IP addresses must be embargoed as if it were unrelated, because the system cannot know that a new IP address is related. This causes additional embargoing that can surprise end users. To deal with this problem, DCC greylisting can ignore all or part of the IP address when checking the triple of sender, recipient and SMTP client IP address. dccm -G noIP, dccm -G IPmask/xx, dccifd -G noIP, or dccifd -G IPmask/xx control this behavior.

  Another mechanism to deal with this problem involves whitelisting IP addresses instead of triples. This mechanism is available as dccd -G weak-IP.

Installation and Testing

See the DCC installation instructions for installation instructions for DCC greylisting.

Greylisting can be tested by sending mail from unfamiliar IP addresses or senders. The beginnings, continuations, and endings of greylist embargoes are indicated in log files by the string "embargo." Note that a message can be marked in a DCC log file as both embargoed and rejected as spam.

Flooding of data among DCC greylist servers can be examined with commands such as

	   cdcc "grey on; flood list; flood stats all"

Contact Vernon Schryver at vjs@rhyolite.com or use the form.
The operator of this web site will not give, sell, or otherwise transfer addresses maintained by this web site to any other party for the purposes of initiating, or enabling others to initiate, electronic messages.