Fail2ban

Install fail2ban
# yum install fail2ban -y

Create the filter definition files in filter.d
# cat >/etc/fail2ban/filter.d/qt-smtp-authnotavail.conf << EOL
[Definition]
#Looks for failed auth outside TLS to SMTP
failregex = 503 auth not available \(\#5\.3\.3\) - <HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qt-smtps-passfail.conf<< EOL
[Definition]
#Looks for failed password logins to SMTP
failregex = vchkpw-smtps: password fail ([^)]*) [^@]*@[^:]*:<HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qt-smtps-usernotfound.conf<< EOL
[Definition]
failregex = vchkpw-smtps: vpopmail user not found .*:<HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qt-sub-passfail.conf<< EOL
[Definition]
failregex = vchkpw-submission: password fail ([^)]*) [^@]*@[^:]*:<HOST>
ignoreregex =
EOL

# cat >/etc/fail2ban/filter.d/qt-sub-usernotfound.conf<< EOL
[Definition]
failregex = vchkpw-submission: vpopmail user not found .*:<HOST>
ignoreregex =
EOL

Create jail.local
# cat >>/etc/fail2ban/jail.d/jail.local << EOL
[qt-sub-passfail]
enabled = true
filter  = qt-sub-passfail
action  = iptables[name=QT-SUB-PASSFAIL, port=587, protocol=tcp]
logpath = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qt-sub-usernotfound]
enabled = true
filter  = qt-sub-usernotfound
action  = iptables[name=QT-SUB-USERNOTFOUND, port=587, protocol=tcp]
logpath = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qt-smtps-passfail]
enabled  = true
filter   = qt-smtps-passfail
action   = iptables[name=QT-SMTPS-PASSFAIL, port=465, protocol=tcp]
logpath  = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qt-smtps-usernotfound]
enabled = true
filter = qt-smtps-usernotfound
action = iptables[name=QT-SMTPS-USERNOTFOUND, port=465, protocol=tcp]
logpath = /var/log/maillog
maxretry = 3
bantime  = 86400
findtime = 3600
backend = auto

[qt-smtp-authnotavail]
enabled = true
filter = qt-smtp-authnotavail
action = iptables[name=QT-SMTP-AUTHNOTAVAIL, port=25, protocol=tcp]
logpath = /var/log/qmail/smtptx/current
maxretry = 3
bantime = 86400
findtime = 300
backend = auto
EOL

In order to log SMTPTX (transactions) do the following:
 1) # qmailctl stop
 2) Add 'SMTP_DEBUG="1"' to /etc/tcprules.d/tcp.smtp 
 3) Replace contents of '/var/qmail/supervise/smtp/log/run' script with below to log transactions to different file:
     #!/bin/sh
     LOGSIZE=`cat /var/qmail/control/logsize`
     LOGCOUNT=`cat /var/qmail/control/logcount`
     exec /usr/bin/setuidgid qmaill \
       /usr/bin/multilog t s$LOGSIZE n$LOGCOUNT \
       '-*' '+@* server:[*' '+@* client:[*' /var/log/qmail/smtptx \
       '+*' '-@* server:[*' '-@* client:[*' /var/log/qmail/smtp 2>&1
 4) # qmailctl start && qmailctl cdb
 5) # tail -f /var/log/qmail/smtptx/current | tai64nlocal

Start fail2ban
# systemctl start fail2ban

Script to check blocking
# tee -a /usr/local/bin/f2bstat<<EOL
#!/bin/bash
for JAIL in qt-sub-passfail \\
            qt-sub-usernotfound \\
            qt-smtps-passfail \\
            qt-smtps-usernotfound \\
            qt-smtp-authnotavail
do
   fail2ban-client status \$JAIL
   echo ""
done
EOL

Set permissions & run script (w/output sample)
# chmod 755 /usr/local/bin/f2bstat && f2bstat

qt-sub-passfail:

Status for the jail: qt-sub-passfail
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     1
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

qt-sub-usernotfound:

Status for the jail: qmail-submission-usernotfound
|- Filter
|  |- Currently failed: 7
|  |- Total failed:     7
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

qt-smtps-passfail:

Status for the jail: qmail-smtps-passfail
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

qt-smtps-usernotfound:

Status for the jail: qmail-smtps-usernotfound
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/maillog
`- Actions
   |- Currently banned: 2
   |- Total banned:     2
   `- Banned IP list:   5.34.207.174 212.70.149.72

qt-smtp-authnotavail:

Status for the jail: qmail-smtp-authnotavail
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/qmail/smtptx/current
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Basic commands

• Check banned IPs:

Format: fail2ban-client get 'jail' banned
  # fail2ban-client get qt-smtp-authnotavail banned
  ['xxx.xxx.xxx.xxx', 'yyy.yyy.yyy.yyy',...,]

• How to unblock an IP(s):

Format: fail2ban-client set 'jail' unbanip xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy ...
  # fail2ban-client set qt-smtp-authnotavail unbanip 192.168.1.105 192.168.1.112 192.168.1.119
  3

• How to block an IP(s):

Format: fail2ban-client set 'jail' unbanip xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy ...
  # fail2ban-client set qt-smtp-authnotavail banip 192.168.9.105 192.168.1.112 192.168.1.119
  3

• Help:

  # fail2ban-client -h

References

[1] fail2ban homepage: http://www.fail2ban.org