Fail2ban

Install fail2ban
# yum install fail2ban -y

Create the filter definition files in filter.d
# cat >/etc/fail2ban/filter.d/qmail-smtp-authnotavail.conf << EOL
[Definition]
#Looks for failed auth outside TLS to SMTP
failregex = 503 auth not available \(\#5\.3\.3\) - <HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qmail-smtps-passfail.conf<< EOL
[Definition]
#Looks for failed password logins to SMTP
failregex = vchkpw-smtps: password fail ([^)]*) [^@]*@[^:]*:<HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qmail-smtps-usernotfound.conf<< EOL
[Definition]
failregex = vchkpw-smtps: vpopmail user not found .*:<HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qmail-submission-passfail.conf<< EOL
[Definition]
failregex = vchkpw-submission: password fail ([^)]*) [^@]*@[^:]*:<HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qmail-submission-usernotfound.conf<< EOL
[Definition]
failregex = vchkpw-submission: vpopmail user not found .*:<HOST>
ignoreregex =
EOL

Create jail.local
# cat >>/etc/fail2ban/jail.d/jail.local << EOL
[qmail-submission-passfail]
enabled = true
filter  = qmail-submission-passfail
action  = iptables[name=QMAIL-SUBMISSION, port=587, protocol=tcp]
logpath = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qmail-submission-usernotfound]
enabled = true
filter  = qmail-submission-usernotfound
action  = iptables[name=QMAIL-SUBMISSION, port=587, protocol=tcp]
logpath = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qmail-smtps-passfail]
enabled  = true
filter   = qmail-smtps-passfail
action   = iptables[name=QMAIL-SMTPS, port=465, protocol=tcp]
logpath  = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qmail-smtps-usernotfound]
enabled = true
filter = qmail-smtps-usernotfound
action = iptables[name=QMAIL-SMTPS, port=465, protocol=tcp]
logpath = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qmail-smtp-authnotavail]
enabled = true
filter = qmail-smtp-authnotavail
action = iptables[name=QMAIL-SMTP, port=25, protocol=tcp]
logpath = /var/log/qmail/smtptx/current(The smtp log run file must be created below before f2b will start)
maxretry = 3
bantime = 86400
findtime = 300
backend = auto

EOL

In order to log SMTP transactions do the following:
 1) # qmailctl stop
 2) Add 'SMTP_DEBUG="1"' to /etc/tcprules.d/tcp.smtp 
 3) Replace contents of '/var/qmail/supervise/smtp/log/run' script with below to log transactions to different file: 
     #!/bin/sh
     LOGSIZE=`cat /var/qmail/control/logsize`
     LOGCOUNT=`cat /var/qmail/control/logcount`
     exec /usr/bin/setuidgid qmaill \
       /usr/bin/multilog t s$LOGSIZE n$LOGCOUNT \
       '-*' '+@* server:[*' '+@* client:[*' /var/log/qmail/smtptx \
       '+*' '-@* server:[*' '-@* client:[*' /var/log/qmail/smtp 2>&1
 4) # qmailctl start && qmailctl cdb
 5) # tail -f /var/log/qmail/smtptx/current | tai64nlocal

Start fail2ban
# systemctl start fail2ban

Script to check blocking
# tee -a /usr/local/bin/f2bstat<<EOL
#!/bin/bash
for FILTER in qmail-submission-passfail \\
             qmail-submission-usernotfound \\
             qmail-smtps-passfail \\
             qmail-smtps-usernotfound \\
             qmail-smtp-authnotavail
do
   fail2ban-client status \$FILTER
   echo ""
done
EOL

Set permissions & run script (w/output sample)
# chmod 755 /usr/local/bin/f2bstat && f2bstat

qmail-submission-passfail:

Status for the jail: qmail-submission-passfail
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     1
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

qmail-submission-usernotfound:

Status for the jail: qmail-submission-usernotfound
|- Filter
|  |- Currently failed: 7
|  |- Total failed:     7
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

qmail-smtps-passfail:

Status for the jail: qmail-smtps-passfail
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

qmail-smtps-usernotfound:

Status for the jail: qmail-smtps-usernotfound
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 2
   |- Total banned:     2
   `- Banned IP list:   5.34.207.174 212.70.149.72

qmail-smtp-authnotavail:

Status for the jail: qmail-smtp-authnotavail
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/qmail/smtptx/current
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Basic commands

• Check banned IPs:

  Format: fail2ban-client get 'jail' banned
        # fail2ban-client get qmail-smtp-authnotavail banned
        ['xxx.xxx.xxx.xxx', 'yyy.yyy.yyy.yyy',...,]

• How to unblock an IP(s):

  Format: fail2ban-client set 'jail' unbanip xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy ...
        # fail2ban-client set qmail-smtp-authnotavail unbanip 192.168.1.105 192.168.1.112 192.168.1.119
        3
        #

• How to block an IP(s):

  Format: fail2ban-client set 'jail' unbanip xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy ...
        # fail2ban-client set qmail-smtp-authnotavail banip 192.168.9.105 192.168.1.112 192.168.1.119
        3
        #

References

[1] fail2ban homepage: http://www.fail2ban.org