ZIP file attachment not recognized and therefore no file check performed

[Tony Larco]

I apologize if this has been answered in another thread. I did spend
quite some time poking through the archived mailing list articles, the
MailScanner docs, and googling around, but we are just stumped and are
hoping a MailScanner guru could enlighten us about this situation.

First, we are running the following (from /usr/sbin/MailScanner -v) -
This is SUSE Linux Enterprise Server 10 (x86_64)
This is Perl version 5.008008 (5.8.8)
This is MailScanner version 4.78.17
Using F-Prot for AV scanning

High level overview - We use Barracuda's for our mail gateways that hand
off to MailScanner before getting routed to the appropriate mail server
for delivery. This solution has worked great for years, but last week
something strange happened that we cannot figure out.

On Friday we started receiving emails that contained some kind of 0-day
malware. The Barracudas were blocking some of these email, but based on
score and not on the emails containing a virus. Later in the day
Barracuda started recognizing the virus so the problem was mitigated at
the mail gateway, but some did slip by the first line of defense and
were passed to MailScanner.

The attachment was a zipped up EXE file, but something was unique about
these messages. We block ZIP and EXE files to most of our users, but
our MailScanner instance was not acknowledging these emails contained a
ZIP file and therefore not doing the "Filename Check". What is very
interesting is when MailScanner delivered the email to an invalid
recipient and it was bounced back to the sender, MailScanner detected
the existence of a ZIP file and blocked it on the way out! But not on
the way in! This is the heart of the issue... how can we determine why
these messages were not interrogated while other (legit) zip files were
being rejected at the same time?

We observed these emails were encoded with windows-1251 encoding
(http://en.wikipedia.org/wiki/Windows-1251) and the content type of the
attachment was simply "Content Type ;" Other than that, we did not see
anything out of the ordinary with these emails.

We tried to create a zip file of the same name as the malware and send
it from gmail and the ZIP file was detected immediately by MailScanner,
so we were not able to reproduce the problem strictly by name. Now that
F-prot is detecting this, its getting dropped for containing a virus,
and we can really cannot test further in our production environment. We
took this into our lab, but we were not testing with the exact same
version of MailScanner and we were not able to recreate the problem. In
our minds, whether MailScanner could detect the virus or not, it should
have detected the ZIP and/or EXE and rejected it for this reason alone.

Any information about this issue would be greatly appreciated.
Management is now questioning the usefulness of MailScanner versus some
commercial offering, but I believe in FOSS. Thank you in advance for
taking the time to read this post!

Regards,

Tony




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mailscanner.info/pipermail/mailscanner/attachments/20131016/3bfe2168/attachment.html

[Martin Hepworth]

umm well the mailscanner version you are using is 4 years old. i know the
releases arent coming as fast and furious we they used to but still..

blocking invalid recipients should be done up front anyway

I'd look at your rulesets in MailScanner to make sure you're not trusting
the Baracuda to some level and there for missing the checks. You should be
able to trace the messages causing problems in the mail logs .

Given you're already scanning with baracuda's and they still delivered the
malware how is their commercial offereing any better??

Martin
--
Martin Hepworth, CISSP
Oxford, UK

Post by Tony Larco
I apologize if this has been answered in another thread. I did spend
quite some time poking through the archived mailing list articles, the
MailScanner docs, and googling around, but we are just stumped and are
hoping a MailScanner guru could enlighten us about this situation.
First, we are running the following (from /usr/sbin/MailScanner -v) -
This is SUSE Linux Enterprise Server 10 (x86_64)
This is Perl version 5.008008 (5.8.8)****
This is MailScanner version 4.78.17
Using F-Prot for AV scanning
High level overview - We use Barracuda's for our mail gateways that hand
off to MailScanner before getting routed to the appropriate mail server for
delivery. This solution has worked great for years, but last week
something strange happened that we cannot figure out.
On Friday we started receiving emails that contained some kind of 0-day
malware. The Barracudas were blocking some of these email, but based on
score and not on the emails containing a virus. Later in the day Barracuda
started recognizing the virus so the problem was mitigated at the mail
gateway, but some did slip by the first line of defense and were passed to
MailScanner.
The attachment was a zipped up EXE file, but something was unique about
these messages. We block ZIP and EXE files to most of our users, but our
MailScanner instance was not acknowledging these emails contained a ZIP
file and therefore not doing the "Filename Check". What is very
interesting is when MailScanner delivered the email to an invalid recipient
and it was bounced back to the sender, MailScanner detected the existence
of a ZIP file and blocked it on the way out! But not on the way in! This
is the heart of the issue... how can we determine why these messages were
not interrogated while other (legit) zip files were being rejected at the
same time?
We observed these emails were encoded with windows-1251 encoding (
http://en.wikipedia.org/wiki/Windows-1251) and the content type of the
attachment was simply "Content Type ;" Other than that, we did not see
anything out of the ordinary with these emails.
We tried to create a zip file of the same name as the malware and send it
from gmail and the ZIP file was detected immediately by MailScanner, so we
were not able to reproduce the problem strictly by name. Now that F-prot
is detecting this, its getting dropped for containing a virus, and we can
really cannot test further in our production environment. We took this
into our lab, but we were not testing with the exact same version of
MailScanner and we were not able to recreate the problem. In our minds,
whether MailScanner could detect the virus or not, it should have detected
the ZIP and/or EXE and rejected it for this reason alone.
Any information about this issue would be greatly appreciated. Management
is now questioning the usefulness of MailScanner versus some commercial
offering, but I believe in FOSS. Thank you in advance for taking the time
to read this post!
Regards,
Tony
--
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mailscanner.info/pipermail/mailscanner/attachments/20131016/6e30d629/attachment.html

[Steve Freegard]

Tony,

Post by Tony Larco
The attachment was a zipped up EXE file, but something was unique about
these messages. We block ZIP and EXE files to most of our users, but
our MailScanner instance was not acknowledging these emails contained a
ZIP file and therefore not doing the "Filename Check". What is very
interesting is when MailScanner delivered the email to an invalid
recipient and it was bounced back to the sender, MailScanner detected
the existence of a ZIP file and blocked it on the way out! But not on
the way in! This is the heart of the issue... how can we determine why
these messages were not interrogated while other (legit) zip files were
being rejected at the same time?
We observed these emails were encoded with windows-1251 encoding
(http://en.wikipedia.org/wiki/Windows-1251) and the content type of the
attachment was simply "Content Type ;" Other than that, we did not see
anything out of the ordinary with these emails.
We tried to create a zip file of the same name as the malware and send
it from gmail and the ZIP file was detected immediately by MailScanner,
so we were not able to reproduce the problem strictly by name. Now that
F-prot is detecting this, its getting dropped for containing a virus,
and we can really cannot test further in our production environment. We
took this into our lab, but we were not testing with the exact same
version of MailScanner and we were not able to recreate the problem. In
our minds, whether MailScanner could detect the virus or not, it should
have detected the ZIP and/or EXE and rejected it for this reason alone.

Being as MailScanner caught it on the way out - I suspect you have a
ruleset somewhere that was bypassing the scans for the hosts that were
sending them.

Look for rulesets on:

Dangerous Content Scanning
Maximum Archive Depth (make sure this isn't set to 0)
Find Archives By Content
Archives: Allow Filenames
Archives: Deny Filesname
Archives: Filename Rules

If that doesn't yield anything - then I would set:

Log Permitted Filenames = yes

And then if one comes through you can simply look at the logs and see if
the file was even noticed by MailScanner (which it wouldn't be if
Maximum Archive Depth = 0 or if the file is hitting an allow rule).

The important thing when testing is that you emulate the client that
send the message exactly. Same IP, HELO/EHLO, MAIL FROM, RCPT TO and
message data. If you are using Postfix then you can use XCLIENT to
emulate the IP of the sender e.g. XCLIENT ADDR=1.2.3.4

Alternatively - you can use the ruleset tester, but then you're only
testing one rule by hand e.g.:

[root at mta41 ~]# MailScanner --value="Maximum Archive Depth"
--from=foo at bar.com --to=smf at fsl.com --ip 1.2.3.4
Looked up internal option name "maxzipdepth"
With sender = foo at bar.com
recipient = smf at fsl.com
Client IP = 1.2.3.4
Virus =
Result is "3"

Hope that points you in the right direction.

Kind regards,
Steve.

[Scott B. Anderson]

-----Original Message-----
From: mailscanner-bounces at lists.mailscanner.info [mailto:mailscanner-
bounces at lists.mailscanner.info] On Behalf Of Tony Larco
Sent: Wednesday, October 16, 2013 8:31 AM
To: mailscanner at lists.mailscanner.info
Subject: ZIP file attachment not recognized and therefore no file check
performed
I apologize if this has been answered in another thread. I did spend quite some
time poking through the archived mailing list articles, the MailScanner docs, and
googling around, but we are just stumped and are hoping a MailScanner guru
could enlighten us about this situation.
First, we are running the following (from /usr/sbin/MailScanner -v) - This is SUSE
Linux Enterprise Server 10 (x86_64) This is Perl version 5.008008 (5.8.8) This is
MailScanner version 4.78.17 Using F-Prot for AV scanning
High level overview - We use Barracuda's for our mail gateways that hand off to
MailScanner before getting routed to the appropriate mail server for delivery.
This solution has worked great for years, but last week something strange
happened that we cannot figure out.
On Friday we started receiving emails that contained some kind of 0-day
malware. The Barracudas were blocking some of these email, but based on score
and not on the emails containing a virus. Later in the day Barracuda started
recognizing the virus so the problem was mitigated at the mail gateway, but some
did slip by the first line of defense and were passed to MailScanner.
The attachment was a zipped up EXE file, but something was unique about these
messages. We block ZIP and EXE files to most of our users, but our MailScanner
instance was not acknowledging these emails contained a ZIP file and therefore
not doing the "Filename Check". What is very interesting is when MailScanner
delivered the email to an invalid recipient and it was bounced back to the sender,
MailScanner detected the existence of a ZIP file and blocked it on the way out!
But not on the way in! This is the heart of the issue... how can we determine why
these messages were not interrogated while other (legit) zip files were being
rejected at the same time?
We observed these emails were encoded with windows-1251 encoding
(http://en.wikipedia.org/wiki/Windows-1251) and the content type of the
attachment was simply "Content Type ;" Other than that, we did not see
anything out of the ordinary with these emails.
We tried to create a zip file of the same name as the malware and send it from
gmail and the ZIP file was detected immediately by MailScanner, so we were not
able to reproduce the problem strictly by name. Now that F-prot is detecting
this, its getting dropped for containing a virus, and we can really cannot test
further in our production environment. We took this into our lab, but we were
not testing with the exact same version of MailScanner and we were not able to
recreate the problem. In our minds, whether MailScanner could detect the virus
or not, it should have detected the ZIP and/or EXE and rejected it for this reason
alone.
Any information about this issue would be greatly appreciated.
Management is now questioning the usefulness of MailScanner versus some
commercial offering, but I believe in FOSS. Thank you in advance for taking the
time to read this post!
Regards,
Tony

I believe I have had the same behavior from MailScanner 4.84.3 and 4.84.5 recently. -- However, since I use ESET, it is unpacking and scanning the archives even if TNEF or other MS-related perl module is failing to do so. The other thing I set is to not deliver password protected archives. (I quarantine just in case someone needs one)
Can you extract the 0day executable to your Mailscanner server and run the 'file' command on it? I wonder if the magic detection in file is failing to see it as either an executable or an archive file.


Scott

...

[Alex Neuman van der Hans]

<! rant>
It's not the first time I've encountered problems with a similar setup.

Barracuda gateways mangle the e-mails in unpredictable, nonstandard ways - first of which being that all e-mail appears to come from the gateway, making IP-based blocklists using "fail2ban" difficult, just to give one example.

I'd personally rather depend on an open source system that *does* work, like MailScanner; I would question the usefulness of a Barracuda mail gateway that not only is useless against 0-day exploits, but also mangles e-mail in unpredictable ways "breaking" other lines of defense instead of working in tandem.

MailScanner does archiving, MCP, and a bunch of other things that Barracuda either doesn't do outright or charges through the nose to do.

</rant !>

Your lab may have a different configuration; it may be that you have a rule such as "accept e-mail from 192.168.x.y as is" and you're not really scanning the way you believe you are. Assume nothing.

You mention you've tried sending from GMail. Have you tried reproducing the actual, real environment the originals were sent in? GMail is probably "doing things right" and not sending "weird" e-mails. Perhaps you'd have to go as far as infecting a VM and seeing what it does.

Do you accept TNEF? It's also unpredictable enough to be used by some virus writers since only Microsoft understands it - and not 100% at that. Is the exploit TNEF-encoded?

Perhaps with some additional details regarding the nature of the "0-day" we can look further into it.

<! rant>
? and at least, with MailScanner, you get real help from real users, not boilerplate "it's not my problem" e-mails from a manufacturer that doesn't really care about your problems.
</rant !>

--

Alex Neuman van der Hans
Reliant Technologies / Vida Digital
http://vidadigital.com.pa/

[phone]
[phone]
[phone] (USA)

Follow @AlexNeuman on Twitter
http://facebook.com/vidadigital

I apologize if this has been answered in another thread. I did spend quite some time poking through the archived mailing list articles, the MailScanner docs, and googling around, but we are just stumped and are hoping a MailScanner guru could enlighten us about this situation.
First, we are running the following (from /usr/sbin/MailScanner -v) -
This is SUSE Linux Enterprise Server 10 (x86_64)
This is Perl version 5.008008 (5.8.8)
This is MailScanner version 4.78.17
Using F-Prot for AV scanning
High level overview - We use Barracuda's for our mail gateways that hand off to MailScanner before getting routed to the appropriate mail server for delivery. This solution has worked great for years, but last week something strange happened that we cannot figure out.
On Friday we started receiving emails that contained some kind of 0-day malware. The Barracudas were blocking some of these email, but based on score and not on the emails containing a virus. Later in the day Barracuda started recognizing the virus so the problem was mitigated at the mail gateway, but some did slip by the first line of defense and were passed to MailScanner.
The attachment was a zipped up EXE file, but something was unique about these messages. We block ZIP and EXE files to most of our users, but our MailScanner instance was not acknowledging these emails contained a ZIP file and therefore not doing the "Filename Check". What is very interesting is when MailScanner delivered the email to an invalid recipient and it was bounced back to the sender, MailScanner detected the existence of a ZIP file and blocked it on the way out! But not on the way in! This is the heart of the issue... how can we determine why these messages were not interrogated while other (legit) zip files were being rejected at the same time?
We observed these emails were encoded with windows-1251 encoding (http://en.wikipedia.org/wiki/Windows-1251) and the content type of the attachment was simply "Content Type ;" Other than that, we did not see anything out of the ordinary with these emails.
We tried to create a zip file of the same name as the malware and send it from gmail and the ZIP file was detected immediately by MailScanner, so we were not able to reproduce the problem strictly by name. Now that F-prot is detecting this, its getting dropped for containing a virus, and we can really cannot test further in our production environment. We took this into our lab, but we were not testing with the exact same version of MailScanner and we were not able to recreate the problem. In our minds, whether MailScanner could detect the virus or not, it should have detected the ZIP and/or EXE and rejected it for this reason alone.
Any information about this issue would be greatly appreciated. Management is now questioning the usefulness of MailScanner versus some commercial offering, but I believe in FOSS. Thank you in advance for taking the time to read this post!
Regards,
Tony
--
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

[Steve Basford]

Post by Tony Larco
On Friday we started receiving emails that contained some kind of 0-day
malware. The Barracudas were blocking some of these email, but based on
score and not on the emails containing a virus. Later in the day
Barracuda started recognizing the virus so the problem was mitigated at
the mail gateway, but some did slip by the first line of defense and
were passed to MailScanner.

No sure this is what you want to do but you could add-in ClamAV and then
add-on Sanesecurity signatures:

rogue.hdb is updated at least hourly with md5 of current emailed malware,
phish.hdb will block known and some simple guess-worked content of bad
stuff in zip/rar files.

If you want to go one stage further... add-in foxhole_generic.cdb to block
double extensions in zip/rar/7zip or foxhole_all.cdb which will block
anything bad in zip/rar/7zip... more info here:
http://sanesecurity.com/foxhole-databases/

More sig databases here:
http://sanesecurity.com/usage/signatures/

Download Scripts here:
http://sanesecurity.com/usage/linux-scripts/

If you have a full/header of the missed/mangled malware and you can give me
a download link for it (pastebin etc.) I'll take a look... see if any sigs
could be tweaked to detect it in the future...

Here's an example stat of stuff being detected:
Loading Image...

Sorry for the length of post... or it's it a little off-topic...

Cheers,

Steve
Sanesecurity.com