Preface ======= On the morning of June 7, 2007 I decided to migrate our production servers over to OpenBSD spamd from postgrey. The entire process took about 6-7 hours to do, with a lot of cursing and irritation. All I was interested in was greylisting and tarpitting/spamtrapping. I was not interested in hardcore blacklisting; that is, I was not interested in running spamd -b or spamd-setup -b. Setting up OpenBSD spamd on FreeBSD looks to be a fairly easy/simple process at first. "Wow, there's a lot on Google about it!" Yes, and in every case on Google, the individuals who documented their experiences were not documented in such a way that made sense. Most of the writings kept insisting you read the official OpenBSD spamd manpages. However, the OpenBSD spamd manpage differs *severely* between versions; the OpenBSD 3.5 version is **completely** different than the OpenBSD CURRENT version. Thus, as you compare the manpages to the stories you find on the web, you quickly become confused. There's a FAQ entry at the bottom of this document which contains many questions I myself had. It doesn't help that the official OpenBSD spamd manpages are written by someone of European decent (read: English as a second language), that the official OpenBSD CURRENT spamd manpage differs majorly from the OpenBSD 3.5 manpage, and (quite possibly most importantly!) that there isn't a single piece of documentation anywhere that explains how all of these different pieces of the puzzle work together (for example, the very first question I had was "How the heck does spamd get the mail to postfix?! Does it spawn sendmail?"). It wasn't easy finding answers to my own questions, because no one had asked them before. And I sure as hell wasn't going to make myself out to be a complete idiot on a mailing list (no one on lists bothers to read long mails anyways; anything longer than 3-4 paragraphs gets ignored). So after 6-7 hours of figuring it all out, I now have a much better under- standing of what's going on behind the scenes, how all the pieces play with one another, and even some drawbacks to using spamd over postgrey. It was an educational experience, and any good system administrator will tell you that they feel much more at ease once they understand how a service functions than if they just run it blindly and consider it black magic(tm). So why OpenBSD spamd over postgrey? =================================== One of the core reasons I decided to move our production servers to OpenBSD spamd and off of postgrey is because of an odd bug pertaining to whitelisting. On numerous occasions (and I do have the data if anyone wants to look into it), we would see the following behaviour from postgrey: * postgrey is started with --delay=180 and --max-age=35 * Internet server 1.2.3.4 connects via SMTP and tries to deliver mail * postgrey hasn't seen 1.2.3.4 before, so it becomes greylisted * 1.2.3.4 connects ~20 minutes later, and the mail is accepted * About ~13 hours later, 1.2.3.4 connects and attempts to deliver a new or different piece of mail * postgrey thinks it hasn't seen 1.2.3.4 before and the server is greylisted once more * Rinse lather repeat This problem wouldn't happen for all SMTP servers, but it happened often enough with a few of the mail servers on the Internet which our users commonly received mail from (work-related mail, i.e. very important!) that I decided to look for alternatives. I was able to find some mailing list posts about this problem, but most of the issues turned out to be with the individual's servers; postfix complaining it couldn't talk to postgrey (Operation timed out, etc.) was the most common. In our case, there were no such problems. Since there isn't an easy way to dump the postgrey database in a user-friendly format (db41_dump on FreeBSD will dump the postgrey.db file, but the output isn't human-readable), I wasn't able to determine what postgrey thought was going on internally. The lack of useful userland tools for postgrey is a big disappointment. postgreyreport? Nearly useless. For example, looking back at 3 weeks of logs (during which time the aforementioned 1.2.3.4 server had delivered to us 3 or 4 pieces of mail), postgreyreport would claim it saw absolutely no sign of that server. "Okay", I thought, "let's try looking for the domain the people sent Email from". Nope, nothing. All I can think is that the entries got deleted from the postgrey.db somehow, which also makes no sense since (I assume) that should only happen after max-age is reached. I'd also seen a couple incidents of mail servers re-delivering mail to us, but postgrey delaying the mail for something like 70,000 seconds. I have a hard time believing someone's MTA would be set to retry mail every 19 hours. It just doesn't seem likely. Finally, generally-speaking, postgrey was performing very well, but more and more spam seemed to be slipping through the cracks. The need for a better greylisting daemon was definitely there. Issues with OpenBSD spamd on FreeBSD ==================================== I have the following complaints with either OpenBSD spamd or OpenBSD spamd on FreeBSD specifically (e.g. port-specific issues): * OpenBSD spamd doesn't insert a X-Greylist header -- and it can't, because of how the entire implementation is done; it's not really spamd's fault, but it's one feature of postgrey which is useful * The rc.d scripts on FreeBSD are called obspamd and obspamlogd, not spamd and spamlogd * Keeping the above item in mind, the manpages are not named obspamd and obspamlogd; they retain their original names (spamd, spamlogd) * Calling anything "spamd" is retarded in general: - SpamAssassin has a daemon called spamd (it came first), - OpenBSD spamd is called spamd (good luck doing "man spamd"), - Don't forget "spamdb" (count how many times you'll typo this! ;) ) - If you have both of these daemons on the same system, it gets confusing, - syslog.conf program name matching via "!spamd" is ambiguous; if SpamAssassin ever uses the "daemon" facility for logging, you're going to end up with both OpenBSD spamd and SA spamd logs in the same file. Frequently Asked Questions (FAQ) ================================ Here are some questions (and my own answers, based upon my findings aand my experience) I found myself asking during my quest to migrate to OpenBSD spamd: Q. Why is this thing using pf(4)? Is it really necessary? A. pf(4) is needed for a lot of reasons, as the entire infrastructure relies upon it. Simply put, it's involved in the following ways: 1) spamd (the daemon) updates specific pf(4) tables named and with the IP addresses of servers which are dynamically whitelisted or blacklisted, respectively. 2) spamd-setup will populate the table with blacklisted entries taken from the "black" section in spamd.conf. 3) Usually used to redirect TCP port 25 (SMTP) to the spamd daemon or postfix) itself based upon IP matches in the aforementioned two tables. Q. What is this pf(4) table I keep reading about? Is it needed if all I'm doing is greylisting? A. The table is used by obspamd in the case that you have blacklisted entries in spamd.conf, OR, if you run spamd -b. Q. These sites mention and and a mystery table called . What is and why is needed if spamd-setup can manage whitelists itself? A. The table is maintained dynamically by obspamd. If you pre-populate when pf(4) loads, the instant you run obspamd the table will get cleared. I could not get spamd-setup to populate with whitelisted entries (read: the "white:" entry in spamd.conf). I'm not 100% certain, but from what I could tell, the whitelist entries in spamd.conf aren't really whitelists, but rather a list of IPs or netblocks which are never to be blacklisted; blacklist exclusions, if you will. Q. I pre-populated table in my pf.conf(4), but the instant I start obspamd, pfctl -t spamd-white -T show shows the table as empty! A. See above. Q. ... and then I tried adding the same CIDRs/IPs to the whitelist.txt file and referred to it in spamd.conf. spamd-setup -d shows the correct number of whitelist entries, but when I start the daemon the servers connecting still get greylisted! A. See above. Q. Can I use different table names for things? seems quite vague, and I'd rather it be named and the whitelist also named appropriately. Is this possible? A. Sadly, no. The and table names are hard-coded. Q. Why are all the example docs using "from any to any" in their pf(4) rulesets? This seems like a bad idea. How should I write my rules? A. Everyone seems to think that having daemons bind to INADDR_ANY is a good idea. I am not one of these people, however. In an environment where a person uses a lot of IP addresses, saying "any" in firewalling rules becomes a bigger problem than if you have a box with a single IP (where INADDR_ANY basically means lo0 and your NICs). That said, these are the rules I choose to use on our production servers which rely heavily on IP aliases: wan_if="bge0" mailip="mx01.sc1.parodius.com" # Used for ports/mail/spamd # NOTE: spamd-white-local is *not* manipulated by spamd/spamd-setup. This # allows us to have a static list of IPs/netblocks which are always # whitelisted and aren't removed by the daemon. table persist table persist file "/usr/local/etc/spamd/whitelist.local" # Used for ports/mail/spamd (spamd/postfix portion) rdr pass on $wan_if proto tcp from to $mailip port smtp -> 127.0.0.1 port smtp rdr pass on $wan_if proto tcp from to $mailip port smtp -> 127.0.0.1 port smtp rdr pass on $wan_if proto tcp from ! to $mailip port smtp -> 127.0.0.1 port spamd # Used for ports/mail/spamd (spamlogd portion) pass in log on $wan_if inet proto tcp from any to $mailip port smtp keep state pass out log on $wan_if inet proto tcp from $mailip to any port smtp keep state Some additional details which should help clear up the remaining confusion: The hostname listed as $mailip is an IP alias on the machine itself. postfix listens on 127.0.0.1 port smtp (25) obspamdb listens on 127.0.0.1 port spamd (8025) Q. The pf.conf(4) rules don't work for me; I'm running obspamd and postfix on the same box. I think the rdr rules are wrong? A. Many of the rules on other sites do things awkwardly, redirect things to a completely different machine, or forget to list port services/numbers. Using the example configuration posted above should work just fine. Q. I read on a BSD mailing list that "rdr pass" is wrong and it breaks other rules! So why do all the examples use it? A. The individual who claimed "rdr pass" didn't work for him was also using if_bridge (IP bridging). I didn't look at his pf.conf completely, but my guess is that he's got a rule order problem. I can confirm that "rdr pass" works just fine, but we don't use bridging anywhere on our network. Q. How come people keep adding pf.conf(4) rules for incoming and outgoing mail, when greylisting and blacklisting only cares about incoming? A. I believe the "pass out log" pf.conf entries are optional. I too questioned the logic behind it, and concluded that it's a useful addition in the case that someone elses inbound SMTP server is the same as their outbound SMTP server. Basically, if you don't have the "pass out log" rule, any server you send mail to, which tries to send mail back to you, will get greylisted. Otherwise the greylisting wouldn't happen since it's been dynamically whitelisted when you sent mail there the first time. (I hope this makes sense... :) ) Q. Why does the OpenBSD CURRENT documentation for spamd show examples telling you to do "no rdr" while 3.5 says "rdr pass ... !"? A. I don't have an answer for this. Based on the documentation for pf.conf, I am left to believe the "no rdr" example in CURRENT is wrong. Q. Some guy on the web said to use spaces in spamd.conf and not tabs, or else the daemon doesn't work. Is this true? The examples have tabs! A. This is false; tabs and whitespace both work fine under OpenBSD spamd 4.1.2. Q. I deleted the /var/db/spamd file; how do I re-create it? cp /dev/null or touch just causes spamd-setup to complain about wrong file format. A. The easiest way to recreate it is to launch obspamd. Doing this will cause obspamd to create the file (with the correct owner/group). You may need to fix the file permissions though (0640 seems secure, not 0644). Q. I started obspamd and all I see in my syslog is obspamd saying "read_configline: fgetln (No buffer space available)". Help! A. You did mkdir /var/db/spamd didn't you? :-) It seems obspamd doesn't bother to verify the results from stat() actually refer to a file and not a directory. This portion of the problem is a bug/oversight in obspamd. Simply rmdir the directory you made and then see the above question. Q. I'm using spamdb to see all the greylisted/whitelisted entries, and I want/need to delete one of the IPs shown, but when I do spamdb -d ip I'm told "no entry for ip". How do I accomplish this? A. You can't easily. I've discussed this with the OpenBSD spamd maintainers, and was basically told "Why would you want to do that? It'll expire on its own." Apparently our opinions/mentalities on how to properly manage/administrate greylist services differs...