Make Postfix Trigger Blacklistd on Failed Authentication
by Emile `iMil' Heitor - 2020-08-09
The other day, I realized that from time to time, alpine, my console mail client for about 20+ years now, would close the connection to the IMAP server because of an “error”.
Digging in the logs, I realized my server was being bruteforced for months, if not years. NetBSD being the fantastic OS it is, it actually had nearly no effect on my server’s behaviour, only those annoying connections closing from time to time.
Here’s a /var/log/maillog
extract of such attacks:
Aug 9 07:03:58 senate postfix/smtpd[29385]: lost connection after AUTH from unknown[141.98.80.67]
Aug 9 07:04:33 senate postfix/smtpd[7755]: lost connection after AUTH from unknown[141.98.80.67]
Aug 9 07:04:39 senate postfix/smtpd[29385]: lost connection after AUTH from unknown[141.98.80.67]
Aug 9 07:04:43 senate postfix/smtpd[7755]: lost connection after AUTH from unknown[141.98.80.67]
Aug 9 07:04:49 senate postfix/smtpd[29385]: lost connection after AUTH from unknown[141.98.80.67]
Aug 9 07:04:52 senate postfix/smtpd[7755]: lost connection after AUTH from unknown[141.98.80.67]
And their consequences from dovecot
perspective:
Aug 5 08:50:40 senate dovecot: imap(nobody)<10273>: Fatal: master: service(imap): child 10273 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
I soon found that NetBSD has a very handy tool called blacklistd, which is similar to fail2ban except instead of parsing log files, it relies on actual live authentication failures, which is less CPU and I/O greedy. In NetBSD, there are a couple of system tools that been modified to support blacklistd, among them OpenSSH, bind and postfix.
There are very few articles explaining blacklistd setup, let alone tips and tricks, and for some reason, my setup was working for ssh
and named
, but postfix/smtpd
would never trigger blacklistd
.
Here are the links I found on blacklistd
setup:
- https://netbsd.gw.com/cgi-bin/man-cgi?blacklistd.conf+5+NetBSD-9.0
- https://www.unitedbsd.com/d/63-how-to-use-blacklistd8-with-npf-as-a-fail2ban-replacement
- http://daemonforums.org/showthread.php?p=67524
- https://pub.nethence.com/bsd/blacklistd
- https://mail-index.netbsd.org/netbsd-users/2018/05/22/msg020796.html
- https://github.com/paul-chambers/blacklistd
- https://www.freebsd.org/doc/handbook/firewalls-blacklistd.html (yes,
blacklistd
has been ported to FreeBSD)
Basically, they all say more or less the same:
- configure a
blacklistd
anchor innpf.conf
, because yes,blacklistd
only supports npf on NetBSD - add the services you’d like to be protected in
/etc/blacklistd.conf
You’re set.
Here’s my configuration files:
$ cat /etc/npf.conf
$ext = vioif0
set bpf.jit on;
alg "icmp"
$tcp_allowed = {25, 465, 587, ssh, http, https}
$udp_allowed = {53}
table <blacklist> type ipset file "/etc/npf_blacklist"
group "external" on $ext {
ruleset "blacklistd"
block in final from <blacklist>
block in family inet6 all
pass stateful out final all
pass proto ipv6-icmp all
pass stateful in proto tcp to any port $tcp_allowed
pass stateful in proto udp to any port $udp_allowed
}
group default {
pass final on lo0 all
pass all
}
Do not mind the IPv6 rules, this machine is behind a NAT but is reachable directly via IPv6 and I need it to reply only certain services. Nothing to do with the current article.
[local]
domain dgram * * * 3 24h
smtp stream * * * 3 24h
submissions stream * * * 3 24h
submission stream * * * 3 24h
imaps stream * * * 3 24h
ssh stream * * * 3 24h
Of course I added npf=YES
and blacklistd=YES
, and started them. Nevertheless, after a couple of minutes waiting and seeing a lot of failed authentication attempts, blacklistctl dump -ab
didn’t show any blacklisted IP address.
After countless trials and errors, I finally decided to ask for help in the NetBSD “users” mailing list, and while I was not given a solution, an answer prompted me to have a look at NetBSD’s postfix
source code, and effectively, the answer was here:
if (state->error_count >= var_smtpd_hard_erlim) {
state->reason = REASON_ERROR_LIMIT;
state->error_mask |= MAIL_ERROR_PROTOCOL;
smtpd_chat_reply(state, "421 4.7.0 %s Error: too many errors",
var_myhostname);
pfilter_notify(1, vstream_fileno(state->client));
break;
}
pfilter_notify
is the function that triggers the call to blacklist_r
, but it only does when the number of failed tries is superior to postfix
’s configuration variable smtpd_hard_error_limit
which I didn’t set and then was using the default of 20
(postfix official documentation on the subject).
So I added the following to the /etc/postfix/main.cf
file:
smtpd_client_connection_rate_limit = 20
smtpd_error_sleep_time = 10s
smtpd_soft_error_limit = 3
smtpd_hard_error_limit = 5
as suggested by this great post on FreeBSD’s forum and tadaa, one postfix reload
later I was seeing IPs arriving on blacklistd
’s bucket:
$ sudo blacklistctl dump -ab
178.212.64.52/32:25 1/3 2020/08/08 18:40:15
103.57.80.58/32:25 1/3 2020/08/09 01:31:51
202.4.107.98/32:25 1/3 2020/08/08 08:42:36
117.103.5.186/32:25 1/3 2020/08/08 15:52:26
217.115.213.186/32:25 1/3 2020/08/08 17:54:51
95.181.45.234/32:25 2/3 2020/08/08 23:49:10
Hope this helps a NetBSD brother out.