Mailscanner process mail too slow

Discussion:

Carlos R Laguna

2014-12-23 21:32:29 UTC

Hi everyone, i am having some issue here, but can't figure out what is
it, my current box is mailscanner 4.86.6-1 ubuntu 12.04.5.
The symptoms are:
mail processing too slow
in queue are over 15k mail on hold
almost no clamscan process

mailscanner log http://paste.desdelinux.net/5097

Any hint will be great*

MailScanner --lint output **(connecting to SpamAssassin cache db take up
to 15s)

Reading configuration file /opt/MailScanner/etc/MailScanner.conf
Reading configuration file /opt/MailScanner/etc/conf.d/baruwa.conf
Read 876 hostnames from the phishing whitelist
Read 5890 hostnames from the phishing blacklists
Config: calling custom init function BaruwaLowScore
Baruwa: Populating spam score settings
Baruwa: Read 20 spam score settings
Config: calling custom init function BaruwaShouldScan
Baruwa: Starting scanning settings
Baruwa: Read 21 settings
Config: calling custom init function BaruwaBlacklist
Baruwa: Starting blacklists
Baruwa: Read 0 blacklist items
Baruwa: Ip blocks blacklisted:
Config: calling custom init function BaruwaSQL
Baruwa: Starting SQL logger
Config: calling custom init function BaruwaHighScore
Baruwa: Populating high spam score settings
Baruwa: Read 20 high spam score settings
Config: calling custom init function BaruwaWhitelist
Baruwa: Starting whitelists
Baruwa: Read 1 whitelist items
Baruwa: Ip blocks whitelisted:

Checking version numbers...
Version number in MailScanner.conf (4.84.6) is correct.

Your envelope_sender_header in spam.assassin.prefs.conf is correct.
MailScanner setting GID to (105)
MailScanner setting UID to (104)

Checking for SpamAssassin errors (if you use it)...
Using SpamAssassin results cache
Connected to SpamAssassin cache database
SpamAssassin reported no errors.
Connected to Processing Attempts Database
Created Processing Attempts Database successfully
There are 390 messages in the Processing Attempts Database
Using locktype = flock
MailScanner.conf says "Virus Scanners = clamav"
Found these virus scanners installed: clamav
===========================================================================
Filename Checks: Windows/DOS Executable (1 eicar.com)
Other Checks: Found 1 problems
Virus and Content Scanning: Starting
1.message: Eicar-Test-Signature FOUND

./1/eicar.com: Eicar-Test-Signature FOUND

Virus Scanning: ClamAV found 2 infections
Infected message 1 came from 10.1.1.1
Virus Scanning: Found 2 viruses
===========================================================================
Virus Scanner test reports:
ClamAV said "eicar.com contains Eicar-Test-Signature"

If any of your virus scanners (clamav)
are not listed there, you should check that they are installed correctly
and that MailScanner is finding them correctly via its virus.scanners.conf.
Config: calling custom end function BaruwaLowScore
Baruwa: Shutting down spam score settings
Config: calling custom end function BaruwaShouldScan
Baruwa: Shutting down scanning settings
Config: calling custom end function BaruwaBlacklist
Baruwa: Shutting down blacklists
Config: calling custom end function BaruwaSQL
Baruwa: Shutting down SQL logger
Config: calling custom end function BaruwaHighScore
Baruwa: Shutting down high spam score settings
Config: calling custom end function BaruwaWhitelist
Baruwa: Shutting down whitelists

Thanks for your time and effort. regards
________________________________________________________________
XII Edicion del Evento Nacional de Informatica para Jovenes. INFOCLUB.
Abril. 2015. Ver www.jovenclub.cu
________________________________________________________________

--
MailScanner mailing list
***@lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner

Before posting, read http://wiki.mailscanner.info/posting

Support MailScanner development - buy the book off the website!

Steve Freegard

2014-12-23 22:07:27 UTC

Permalink

Post by Carlos R Laguna
Any hint will be great*
MailScanner.conf says "Virus Scanners = clamav"
Found these virus scanners installed: clamav

clamscan is *really* slow. You should be using clamd; it's massively
faster.

Regards,
Steve.

Carlos R Laguna

2014-12-23 23:47:12 UTC

Permalink

Post by Steve Freegard

Post by Carlos R Laguna
Any hint will be great*
MailScanner.conf says "Virus Scanners = clamav"
Found these virus scanners installed: clamav

clamscan is *really* slow. You should be using clamd; it's massively
faster.
Regards,
Steve.

Hi Steve

Thanks for the reply, i was working on it at the moment your email come
in. I will update any improvement. Thanks again and regards.
________________________________________________________________
XII Edicion del Evento Nacional de Informatica para Jovenes. INFOCLUB.
Abril. 2015. Ver www.jovenclub.cu
________________________________________________________________

--
Este mensaje ha sido analizado por MailScanner
en busca de virus y otros contenidos peligrosos,
y se considera que está limpio.

Kevin Miller

2014-12-23 22:21:32 UTC

Permalink

If you haven't already done so, one thing that can help, assuming you have the spare RAM is to add this to your /etc/fstab:

none /var/spool/MailScanner/incoming tmpfs defaults,size=1024m 0 0

That's the working directory where messages are scanned and evaluated. Doing it in RAM is much faster than on disk.

Adjust the size as appropriate if needed. I'm using a gig of memory - I could probably get by with a lot less as I don't have that busy of a mail server.

What is Max Children set to in MailScanner.conf?

...Kevin
--
Kevin Miller
Network/email Administrator, CBJ MIS Dept.
155 South Seward Street
Juneau, Alaska 99801
Phone: (907) 586-0242, Fax: (907) 586-4500
Registered Linux User No: 307357

-----Original Message-----
Sent: Tuesday, December 23, 2014 12:32 PM
Subject: Mailscanner process mail too slow
Hi everyone, i am having some issue here, but can't figure out what is
it, my current box is mailscanner 4.86.6-1 ubuntu 12.04.5.
mail processing too slow
in queue are over 15k mail on hold
almost no clamscan process
mailscanner log http://paste.desdelinux.net/5097
Any hint will be great*
MailScanner --lint output **(connecting to SpamAssassin cache db take up
to 15s)
Reading configuration file /opt/MailScanner/etc/MailScanner.conf
Reading configuration file /opt/MailScanner/etc/conf.d/baruwa.conf
Read 876 hostnames from the phishing whitelist
Read 5890 hostnames from the phishing blacklists
Config: calling custom init function BaruwaLowScore
Baruwa: Populating spam score settings
Baruwa: Read 20 spam score settings
Config: calling custom init function BaruwaShouldScan
Baruwa: Starting scanning settings
Baruwa: Read 21 settings
Config: calling custom init function BaruwaBlacklist
Baruwa: Starting blacklists
Baruwa: Read 0 blacklist items
Config: calling custom init function BaruwaSQL
Baruwa: Starting SQL logger
Config: calling custom init function BaruwaHighScore
Baruwa: Populating high spam score settings
Baruwa: Read 20 high spam score settings
Config: calling custom init function BaruwaWhitelist
Baruwa: Starting whitelists
Baruwa: Read 1 whitelist items
Checking version numbers...
Version number in MailScanner.conf (4.84.6) is correct.
Your envelope_sender_header in spam.assassin.prefs.conf is correct.
MailScanner setting GID to (105)
MailScanner setting UID to (104)
Checking for SpamAssassin errors (if you use it)...
Using SpamAssassin results cache
Connected to SpamAssassin cache database
SpamAssassin reported no errors.
Connected to Processing Attempts Database
Created Processing Attempts Database successfully
There are 390 messages in the Processing Attempts Database
Using locktype = flock
MailScanner.conf says "Virus Scanners = clamav"
Found these virus scanners installed: clamav
========================================================================
===
Filename Checks: Windows/DOS Executable (1 eicar.com)
Other Checks: Found 1 problems
Virus and Content Scanning: Starting
1.message: Eicar-Test-Signature FOUND
./1/eicar.com: Eicar-Test-Signature FOUND
Virus Scanning: ClamAV found 2 infections
Infected message 1 came from 10.1.1.1
Virus Scanning: Found 2 viruses
========================================================================
===
ClamAV said "eicar.com contains Eicar-Test-Signature"
If any of your virus scanners (clamav)
are not listed there, you should check that they are installed correctly
and that MailScanner is finding them correctly via its
virus.scanners.conf.
Config: calling custom end function BaruwaLowScore
Baruwa: Shutting down spam score settings
Config: calling custom end function BaruwaShouldScan
Baruwa: Shutting down scanning settings
Config: calling custom end function BaruwaBlacklist
Baruwa: Shutting down blacklists
Config: calling custom end function BaruwaSQL
Baruwa: Shutting down SQL logger
Config: calling custom end function BaruwaHighScore
Baruwa: Shutting down high spam score settings
Config: calling custom end function BaruwaWhitelist
Baruwa: Shutting down whitelists
Thanks for your time and effort. regards
________________________________________________________________
XII Edicion del Evento Nacional de Informatica para Jovenes. INFOCLUB.
Abril. 2015. Ver www.jovenclub.cu
________________________________________________________________
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

Carlos R Laguna

2014-12-24 00:57:33 UTC

Permalink

Post by Kevin Miller
none /var/spool/MailScanner/incoming tmpfs defaults,size=1024m 0 0
That's the working directory where messages are scanned and evaluated. Doing it in RAM is much faster than on disk.
Adjust the size as appropriate if needed. I'm using a gig of memory - I could probably get by with a lot less as I don't have that busy of a mail server.
What is Max Children set to in MailScanner.conf?
...Kevin
--
Kevin Miller
Network/email Administrator, CBJ MIS Dept.
155 South Seward Street
Juneau, Alaska 99801
Phone: (907) 586-0242, Fax: (907) 586-4500
Registered Linux User No: 307357

Hi Kevin

Actually ... (This is a container **openvz)

Filesystem Size Used Avail Use% Mounted on
/dev/simfs 60G 49G 11G 82% /
none 1.5G 4.0K 1.5G 1% /dev
none 308M 80K 308M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.5G 108K 1.5G 1% /run/shm
tmpfs 1.5G 52M 1.5G 4% /var/spool/MailScanner/incoming
tmpfs 1.5G 68K 1.5G 1% /tmp

the max of children is 15 in a Intel Xeon x 6 4Gb of ram, at this time
the mail is flowing again, but it always does at this specific hour so i
am not sure if the change to clamd work or not.Thanks for your time and
efforts. Regards
________________________________________________________________
XII Edicion del Evento Nacional de Informatica para Jovenes. INFOCLUB.
Abril. 2015. Ver www.jovenclub.cu
________________________________________________________________

Carlos R Laguna

2014-12-24 02:48:41 UTC

Permalink

Hi Kevin

Actually ... (This is a container **openvz)

Filesystem Size Used Avail Use% Mounted on
/dev/simfs 60G 49G 11G 82% /
none 1.5G 4.0K 1.5G 1% /dev
none 308M 80K 308M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.5G 108K 1.5G 1% /run/shm
tmpfs 1.5G 52M 1.5G 4% /var/spool/MailScanner/incoming
tmpfs 1.5G 68K 1.5G 1% /tmp

the max of children is 15 in a Intel Xeon x 6 4Gb of ram, at this time
the mail is flowing again, but it always does at this specific hour so i
am not sure if the change to clamd work or not.Thanks for your time and
efforts. Regards

________________________________________________________________
XII Edicion del Evento Nacional de Informatica para Jovenes. INFOCLUB.
Abril. 2015. Ver www.jovenclub.cu
________________________________________________________________
--
Este mensaje ha sido analizado por MailScanner
en busca de virus y otros contenidos peligrosos,
y se considera que está limpio.

Steve Freegard

2014-12-24 11:40:17 UTC

Permalink

i am not sure if the change to clamd work or not.

The speed difference between the two is huge:

[***@mail1-ec2 ~]# clamdscan -c /etc/clamd.d/scan.conf /tmp/test.eml
/tmp/test.eml: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.016 sec (0 m 0 s)

.vs.

[***@mail1-ec2 ~]# clamscan /tmp/test.eml
/tmp/test.eml: OK

----------- SCAN SUMMARY -----------
Known viruses: 4687566
Engine version: 0.98.4
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 18.914 sec (0 m 18 s)

So 18.9 seconds .vs. 0.016 seconds.

That would allow MailScanner to scan only 3000 messages per hour (with
15 children: (3600/18)*15 = 3000 that 200 messages per child) whereas
with clamd that would be 3,375,000 messages. However that doesn't take
into consideration the overhead of unpacking the messages, SpamAssassin
scans (usually 3 seconds each minimum) and all the other stuff that
MailScanner does, but you get the idea; the difference between the two
is massive.

MailScanner --lint should tell you if clamd is working or not as it
should report that it found 'clamd' and detect the Eicar test:

[***@mta41 ~]# MailScanner --lint
Trying to setlogsock(unix)
ConfigSQL configuration loaded with serial 594, next check in 900 seconds
Reading ruleset allowexternal for keyword allowexternal
<snip>
Checking version numbers...
Version number in MailScanner.conf (4.77.10) is correct.

Checking for SpamAssassin errors (if you use it)...
Using SpamAssassin results cache
Connected to SpamAssassin cache database
SpamAssassin reported no errors.
I have found clamd f-prot-6 scanners installed, and will use them all by
default.
Connected to Processing Attempts Database
Created Processing Attempts Database successfully
There are 0 messages in the Processing Attempts Database
Using locktype = posix
MailScanner.conf says "Virus Scanners = auto"
Found these virus scanners installed: clamd, f-prot-6
===========================================================================
Filename Checks: Blocked Filename Detected (1 eicar.com)
Other Checks: Found 1 problems
Virus and Content Scanning: Starting
Clamd::INFECTED:: Eicar-Test-Signature :: ./1/
Virus Scanning: Clamd found 1 infections
Scanning: -
at /usr/lib/MailScanner/MailScanner/SweepViruses.pm line 2338
Virus Scanning: F-Prot6 found 1 infections
Infected message 1 came from 10.1.1.1
Infected message 1.message->eicar.com came from
Virus Scanning: Found 2 viruses
===========================================================================

If any of your virus scanners (clamd,f-prot-6)
are not listed there, you should check that they are installed correctly
and that MailScanner is finding them correctly via its virus.scanners.conf.

Regards,
Steve.

Carlos R Laguna

2014-12-24 23:41:30 UTC

Permalink

Post by Steve Freegard

i am not sure if the change to clamd work or not.

/tmp/test.eml: OK
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.016 sec (0 m 0 s)
.vs.
/tmp/test.eml: OK
----------- SCAN SUMMARY -----------
Known viruses: 4687566
Engine version: 0.98.4
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 18.914 sec (0 m 18 s)
So 18.9 seconds .vs. 0.016 seconds.
That would allow MailScanner to scan only 3000 messages per hour (with
15 children: (3600/18)*15 = 3000 that 200 messages per child) whereas
with clamd that would be 3,375,000 messages. However that doesn't take
into consideration the overhead of unpacking the messages, SpamAssassin
scans (usually 3 seconds each minimum) and all the other stuff that
MailScanner does, but you get the idea; the difference between the two
is massive.
MailScanner --lint should tell you if clamd is working or not as it
Trying to setlogsock(unix)
ConfigSQL configuration loaded with serial 594, next check in 900 seconds
Reading ruleset allowexternal for keyword allowexternal
<snip>
Checking version numbers...
Version number in MailScanner.conf (4.77.10) is correct.
Checking for SpamAssassin errors (if you use it)...
Using SpamAssassin results cache
Connected to SpamAssassin cache database
SpamAssassin reported no errors.
I have found clamd f-prot-6 scanners installed, and will use them all by
default.
Connected to Processing Attempts Database
Created Processing Attempts Database successfully
There are 0 messages in the Processing Attempts Database
Using locktype = posix
MailScanner.conf says "Virus Scanners = auto"
Found these virus scanners installed: clamd, f-prot-6
===========================================================================
Filename Checks: Blocked Filename Detected (1 eicar.com)
Other Checks: Found 1 problems
Virus and Content Scanning: Starting
Clamd::INFECTED:: Eicar-Test-Signature :: ./1/
Virus Scanning: Clamd found 1 infections
Scanning: -
at /usr/lib/MailScanner/MailScanner/SweepViruses.pm line 2338
Virus Scanning: F-Prot6 found 1 infections
Infected message 1 came from 10.1.1.1
Infected message 1.message->eicar.com came from
Virus Scanning: Found 2 viruses
===========================================================================
If any of your virus scanners (clamd,f-prot-6)
are not listed there, you should check that they are installed correctly
and that MailScanner is finding them correctly via its virus.scanners.conf.
Regards,
Steve.

Hi, and thank for your reply

However the mail queue persist to over 13k, system load ~0.30 memory
consumption 60% of 4Gb almost no cpu over 25%. A litle more information,
this deploy has 5 server two mailscanner using baruwa in cluster, the
mysql server is a percona mariadb cluster 3 there is almost 2million
messages in the db and another 1million in archive the sa_bayes database
is also in this cluster. At this point no sure if there is a botleneck
impacting mailscanner process. Thanks for all yours answers. and Regards

MailScanner --lint
Trying to setlogsock(unix)

Reading configuration file /opt/MailScanner/etc/MailScanner.conf
Reading configuration file /opt/MailScanner/etc/conf.d/baruwa.conf
Read 876 hostnames from the phishing whitelist
Read 5890 hostnames from the phishing blacklists
Config: calling custom init function BaruwaLowScore
Baruwa: Populating spam score settings
Baruwa: Read 20 spam score settings
Config: calling custom init function BaruwaShouldScan
Baruwa: Starting scanning settings
Baruwa: Read 21 settings
Config: calling custom init function BaruwaBlacklist
Baruwa: Starting blacklists
Baruwa: Read 0 blacklist items
Baruwa: Ip blocks blacklisted:
Config: calling custom init function BaruwaSQL
Baruwa: Starting SQL logger
Config: calling custom init function BaruwaHighScore
Baruwa: Populating high spam score settings
Baruwa: Read 20 high spam score settings
Config: calling custom init function BaruwaWhitelist
Baruwa: Starting whitelists
Baruwa: Read 1 whitelist items
Baruwa: Ip blocks whitelisted:

Checking version numbers...
Version number in MailScanner.conf (4.84.6) is correct.

Your envelope_sender_header in spam.assassin.prefs.conf is correct.
MailScanner setting GID to (105)
MailScanner setting UID to (104)

Checking for SpamAssassin errors (if you use it)...
Using SpamAssassin results cache
Connected to SpamAssassin cache database
SpamAssassin reported no errors.
Connected to Processing Attempts Database
Created Processing Attempts Database successfully
There are 326 messages in the Processing Attempts Database
Using locktype = flock
MailScanner.conf says "Virus Scanners = clamd"
Found these virus scanners installed: clamd
===========================================================================
Filename Checks: Windows/DOS Executable (1 eicar.com)
Other Checks: Found 1 problems
Virus and Content Scanning: Starting
Clamd::INFECTED::Eicar-Test-Signature :: ./1/
Clamd::INFECTED:: Eicar-Test-Signature :: ./1/eicar.com
Virus Scanning: Clamd found 2 infections
Infected message 1 came from 10.1.1.1
Virus Scanning: Found 2 viruses
===========================================================================
Virus Scanner test reports:
Clamd said "eicar.com was infected: Eicar-Test-Signature"

If any of your virus scanners (clamd)
are not listed there, you should check that they are installed correctly
and that MailScanner is finding them correctly via its virus.scanners.conf.
Config: calling custom end function BaruwaLowScore
Baruwa: Shutting down spam score settings
Config: calling custom end function BaruwaShouldScan
Baruwa: Shutting down scanning settings
Config: calling custom end function BaruwaBlacklist
Baruwa: Shutting down blacklists
Config: calling custom end function BaruwaSQL
Baruwa: Shutting down SQL logger
Config: calling custom end function BaruwaHighScore
Baruwa: Shutting down high spam score settings
Config: calling custom end function BaruwaWhitelist
Baruwa: Shutting down whitelists

MailScanner.conf

http://paste.desdelinux.net/5098
________________________________________________________________
XII Edicion del Evento Nacional de Informatica para Jovenes. INFOCLUB.
Abril. 2015. Ver www.jovenclub.cu
________________________________________________________________
--
Este mensaje ha sido analizado por MailScanner
en busca de virus y otros contenidos peligrosos,
y se considera que está limpio.