Make Postfix Trigger Blacklistd on Failed Authentication

The other day, I realized that from time to time, alpine, my console mail client for about 20+ years now, would close the connection to the IMAP server because of an “error”.
Digging in the logs, I realized my server was being bruteforced for months, if not years. NetBSD being the fantastic OS it is, it actually had nearly no effect on my server’s behaviour, only those annoying connections closing from time to time.
Here’s a /var/log/maillog extract of such attacks:

Aug  9 07:03:58 senate postfix/smtpd[29385]: lost connection after AUTH from unknown[141.98.80.67]
Aug  9 07:04:33 senate postfix/smtpd[7755]: lost connection after AUTH from unknown[141.98.80.67]
Aug  9 07:04:39 senate postfix/smtpd[29385]: lost connection after AUTH from unknown[141.98.80.67]
Aug  9 07:04:43 senate postfix/smtpd[7755]: lost connection after AUTH from unknown[141.98.80.67]
Aug  9 07:04:49 senate postfix/smtpd[29385]: lost connection after AUTH from unknown[141.98.80.67]
Aug  9 07:04:52 senate postfix/smtpd[7755]: lost connection after AUTH from unknown[141.98.80.67]

And their consequences from dovecot perspective:

Aug  5 08:50:40 senate dovecot: imap(nobody)<10273>: Fatal: master: service(imap): child 10273 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

I soon found that NetBSD has a very handy tool called blacklistd, which is similar to fail2ban except instead of parsing log files, it relies on actual live authentication failures, which is less CPU and I/O greedy. In NetBSD, there are a couple of system tools that been modified to support blacklistd, among them OpenSSH, bind and postfix.

There are very few articles explaining blacklistd setup, let alone tips and tricks, and for some reason, my setup was working for ssh and named, but postfix/smtpd would never trigger blacklistd.

Here are the links I found on blacklistd setup:

Basically, they all say more or less the same:

  • configure a blacklistd anchor in npf.conf, because yes, blacklistd only supports npf on NetBSD
  • add the services you’d like to be protected in /etc/blacklistd.conf

You’re set.

Here’s my configuration files:

$ cat /etc/npf.conf

$ext = vioif0

set bpf.jit on;
alg "icmp"

$tcp_allowed = {25, 465, 587, ssh, http, https}
$udp_allowed = {53}

table <blacklist> type ipset file "/etc/npf_blacklist"

group "external" on $ext {
        ruleset "blacklistd"

        block in final from <blacklist>
        block in family inet6 all

        pass stateful out final all
        pass proto ipv6-icmp all
        pass stateful in proto tcp to any port $tcp_allowed
        pass stateful in proto udp to any port $udp_allowed
}

group default {
        pass final on lo0 all
        pass all
}

Do not mind the IPv6 rules, this machine is behind a NAT but is reachable directly via IPv6 and I need it to reply only certain services. Nothing to do with the current article.

[local]
domain          dgram   *       *               *       3       24h
smtp            stream  *       *               *       3       24h
submissions     stream  *       *               *       3       24h
submission      stream  *       *               *       3       24h
imaps           stream  *       *               *       3       24h
ssh             stream  *       *               *       3       24h

Of course I added npf=YES and blacklistd=YES, and started them. Nevertheless, after a couple of minutes waiting and seeing a lot of failed authentication attempts, blacklistctl dump -ab didn’t show any blacklisted IP address.

After countless trials and errors, I finally decided to ask for help in the NetBSD “users” mailing list, and while I was not given a solution, an answer prompted me to have a look at NetBSD’s postfix source code, and effectively, the answer was here:

if (state->error_count >= var_smtpd_hard_erlim) {
	state->reason = REASON_ERROR_LIMIT;
	state->error_mask |= MAIL_ERROR_PROTOCOL;
	smtpd_chat_reply(state, "421 4.7.0 %s Error: too many errors",
			 var_myhostname);
	pfilter_notify(1, vstream_fileno(state->client));
	break;
}

pfilter_notify is the function that triggers the call to blacklist_r, but it only does when the number of failed tries is superior to postfix’s configuration variable smtpd_hard_error_limit which I didn’t set and then was using the default of 20 (postfix official documentation on the subject).

So I added the following to the /etc/postfix/main.cf file:

smtpd_client_connection_rate_limit = 20
smtpd_error_sleep_time = 10s
smtpd_soft_error_limit = 3
smtpd_hard_error_limit = 5

as suggested by this great post on FreeBSD’s forum and tadaa, one postfix reload later I was seeing IPs arriving on blacklistd’s bucket:

$ sudo blacklistctl dump -ab
  178.212.64.52/32:25           1/3     2020/08/08 18:40:15
   103.57.80.58/32:25           1/3     2020/08/09 01:31:51
   202.4.107.98/32:25           1/3     2020/08/08 08:42:36
  117.103.5.186/32:25           1/3     2020/08/08 15:52:26
217.115.213.186/32:25           1/3     2020/08/08 17:54:51
  95.181.45.234/32:25           2/3     2020/08/08 23:49:10

Hope this helps a NetBSD brother out.