ScamNailer update STILL not working

[Mark Sapiro]

ScamNailer gets the information about current data by doing a DNS lookup
of a TXT record for emails.msupdate.greylist.bastionmail.com. For over 6
weeks, this has been returning "emails.2013-164.6", i.e. week 16, day 4
update 6. It is currently week 23, day 0.

I posted a patch at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100746.html>
that works around this by guessing the current update, and it seems to
work for me, but contrary to Matt Hampton's statement at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100694.html>,
the underlying issue of the TXT record for
emails.msupdate.greylist.bastionmail.com not being updated is NOT FIXED.

Does anyone care?

[Robert Lopez]

Post by Mark Sapiro
ScamNailer gets the information about current data by doing a DNS lookup
of a TXT record for emails.msupdate.greylist.bastionmail.com. For over 6
weeks, this has been returning "emails.2013-164.6", i.e. week 16, day 4
update 6. It is currently week 23, day 0.
I posted a patch at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100746.html>
that works around this by guessing the current update, and it seems to
work for me, but contrary to Matt Hampton's statement at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100694.html>,
the underlying issue of the TXT record for
emails.msupdate.greylist.bastionmail.com not being updated is NOT FIXED.
Does anyone care?
--
Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
--
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

Mark,
I also only have the 2013-164.6 as the most recent file and do care
about the problem of the file not being updated.
I confess to not trying your patch because I do not understand it.

--
Robert Lopez

[Mark Sapiro]

Post by Robert Lopez

Post by Mark Sapiro
ScamNailer gets the information about current data by doing a DNS lookup
of a TXT record for emails.msupdate.greylist.bastionmail.com. For over 6
weeks, this has been returning "emails.2013-164.6", i.e. week 16, day 4
update 6. It is currently week 23, day 0.
I posted a patch at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100746.html>
that works around this by guessing the current update, and it seems to
work for me ...

[...]

Post by Robert Lopez
I confess to not trying your patch because I do not understand it.

The string such as "emails.2013-164.6" identifies the most recent
ScamNailer update. It is very structured and predictable.

In this string, 2013 is the year, 16 is the week # within the year, 4 is
the day number within the week and 6 is the 6th update of the day.

The patch first figures for the current date (GMT) what the year, week
and day of week are. It's a bit tricky because the perl gmtime function
doesn't return a week number so we have to calculate it from day of
year, but it also depends on what day of the week Jan 1 fell on. Anyway,
the only thing it can't determine is the update number. For each day,
update 0 is the current full file at the beginning of the day and the
remaining updates are incremental changes added as needed.

So the first thing the patch does after computing the above is put
together the year, week and day into something like 2013-234 which is
the correct string for Thursday, 13 June, 2013. It then compares that to
the string it got from the TXT record, and if the one from the TXT
record is >=, it just accepts that and effectively does nothing, but if
it's <, the patched code replaces the string from the TXT record with
its own and arbitrarily says the update # is 99.

The patched ScamNailer then retrieves the base if necessary and the
sequential updates from the last cached one until it gets a 404 or some
other error (or retrieves update 99, but so far that hasn't happened.
Then it resets the update # to the last one retrieved and proceeds.

The rest of the patch just adds another condition around a piece of the
code to avoid the issue described in the comment.

I hope this explanation helps. The patched code has been working well
for me because the updates are actually being pushed to the update
server(s), it's just the information about the current update name that
isn't there.

[Richard Siddall]

Post by Mark Sapiro
ScamNailer gets the information about current data by doing a DNS lookup
of a TXT record for emails.msupdate.greylist.bastionmail.com. For over 6
weeks, this has been returning "emails.2013-164.6", i.e. week 16, day 4
update 6. It is currently week 23, day 0.
I posted a patch at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100746.html>
that works around this by guessing the current update, and it seems to
work for me, but contrary to Matt Hampton's statement at
<http://lists.mailscanner.info/pipermail/mailscanner/2013-May/100694.html>,
the underlying issue of the TXT record for
emails.msupdate.greylist.bastionmail.com not being updated is NOT FIXED.
Does anyone care?

It would be nice if it was fixed.

Richard.

[Mark Sapiro]

Post by Mark Sapiro
ScamNailer gets the information about current data by doing a DNS lookup
of a TXT record for emails.msupdate.greylist.bastionmail.com. For over 6
weeks, this has been returning "emails.2013-164.6", i.e. week 16, day 4
update 6. It is currently week 23, day 0.

It's now even worse. Currently the only TXT record for
emails.msupdate.greylist.bastionmail.com is an SPF record "v=spf1 a
-all". This causes ScamNailer to die with "Failed to retrieve valid
current details\n".

I have attached the latest version of my patch which works around this.
--
Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
-------------- next part --------------
--- ScamNailer-2.10 2012-03-05 03:04:14.000000000 -0800
+++ ScamNailer.new 2013-06-19 12:46:23.000000000 -0700
@@ -18,6 +18,7 @@
use LWP::UserAgent;
use FileHandle;
use DirHandle;
+use Time::Local;

# Output filename, goes into SpamAssassin. Can be over-ridden by just
# adding the output filename on the command-line when you run this script.
@@ -137,7 +138,8 @@
sub GetPhishingUpdate {
my $cache = $emailscurrent . 'cache/';
my $status = $emailscurrent . 'status';
- my $urlbase = "http://cdn.mailscanner.info/emails.";
+# my $urlbase = "http://cdn.mailscanner.info/emails.";
+ my $urlbase = "http://www.mailscanner.eu/emails.";
my $target= $emailscurrent . 'phishing.emails.list';
my $query="emails.msupdate.greylist.bastionmail.com";

@@ -212,9 +214,23 @@
last;
}
}
+ if ($currentbase == -1) {
+ $currentbase = 0;
+ $currentupdate = 0;
+ warn "No appropriate TXT found at $query.\n";
+ }
}

- die "Failed to retrieve valid current details\n" if $currentbase eq "-1";
+
+ my $day = (gmtime)[6];
+ my $year = (gmtime)[5] + 1900;
+ my $janone = (gmtime(timegm(0,0,0,1,0,$year-1900)))[6];
+ my $week = sprintf ("%02d", int (((gmtime)[7] + $janone) / 7));
+ my $mybase = "$year-$week$day";
+ if ($currentbase lt $mybase) {
+ $currentbase = $mybase;
+ $currentupdate = 99;
+ }

print "I am working with: Current: $currentbase - $currentupdate and Status: $status_base - $status_update\n" unless $quiet;

@@ -273,8 +289,10 @@
#print "Getting $urlbase . $currentbase.$i\n" unless $quiet;
my $req = HTTP::Request->new(GET => $urlbase.$currentbase.".".$i);
my $res = $ua->request($req);
- warn "Failed to retrieve $urlbase$currentbase.$i"
- unless $res->is_success;
+ unless ($res->is_success) {
+ warn "Failed to retrieve $urlbase$currentbase.$i";
+ $currentupdate = $i - 1;
+ }
my $line;
foreach $line (split("\n", $res->content)) {
# Is it an addition?
@@ -299,6 +317,12 @@
}
}
}
+ # Because of our guess and retrieve until error strategy, we could be
+ # here without having retrieved any new updates which will result in
+ # our cached $status_update being erased. This does no real harm, but
+ # it causes extra work on the next run. To avoid this we skip the next
+ # section in that case.
+ if (!($status_update eq $currentupdate)) {
# OK do we have a previous version to work from?
if ($status_update>0) {
# Yes - we open the most recent version
@@ -341,6 +365,7 @@
}
}
close (FILEOUT);
+ }
}

}

[Mark Sapiro]

Post by Mark Sapiro
I have attached the latest version of my patch which works around this.

That patch contained a bit that isn't part of this issue. It wouldn't
hurt, but here's a patch without that extra bit.
--
Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
-------------- next part --------------
--- ScamNailer-2.10 2012-03-05 03:04:14.000000000 -0800
+++ ScamNailer.new 2013-06-19 13:08:56.000000000 -0700
@@ -18,6 +18,7 @@
use LWP::UserAgent;
use FileHandle;
use DirHandle;
+use Time::Local;

# Output filename, goes into SpamAssassin. Can be over-ridden by just
# adding the output filename on the command-line when you run this script.
@@ -212,9 +213,23 @@
last;
}
}
+ if ($currentbase == -1) {
+ $currentbase = 0;
+ $currentupdate = 0;
+ warn "No appropriate TXT found at $query.\n";
+ }
}

- die "Failed to retrieve valid current details\n" if $currentbase eq "-1";
+
+ my $day = (gmtime)[6];
+ my $year = (gmtime)[5] + 1900;
+ my $janone = (gmtime(timegm(0,0,0,1,0,$year-1900)))[6];
+ my $week = sprintf ("%02d", int (((gmtime)[7] + $janone) / 7));
+ my $mybase = "$year-$week$day";
+ if ($currentbase lt $mybase) {
+ $currentbase = $mybase;
+ $currentupdate = 99;
+ }

print "I am working with: Current: $currentbase - $currentupdate and Status: $status_base - $status_update\n" unless $quiet;

@@ -273,8 +288,10 @@
#print "Getting $urlbase . $currentbase.$i\n" unless $quiet;
my $req = HTTP::Request->new(GET => $urlbase.$currentbase.".".$i);
my $res = $ua->request($req);
- warn "Failed to retrieve $urlbase$currentbase.$i"
- unless $res->is_success;
+ unless ($res->is_success) {
+ warn "Failed to retrieve $urlbase$currentbase.$i";
+ $currentupdate = $i - 1;
+ }
my $line;
foreach $line (split("\n", $res->content)) {
# Is it an addition?
@@ -299,6 +316,12 @@
}
}
}
+ # Because of our guess and retrieve until error strategy, we could be
+ # here without having retrieved any new updates which will result in
+ # our cached $status_update being erased. This does no real harm, but
+ # it causes extra work on the next run. To avoid this we skip the next
+ # section in that case.
+ if (!($status_update eq $currentupdate)) {
# OK do we have a previous version to work from?
if ($status_update>0) {
# Yes - we open the most recent version
@@ -341,6 +364,7 @@
}
}
close (FILEOUT);
+ }
}

}

[Jeff Earickson]

Gang,

I have still been getting constant "not working" out of ScamNailer lately:

Failed to retrieve http://mailscanner.eu/emails.2013-260.11 at
/etc/MailScanner/ScamNailer line 289.

or

Failed to retrieve http://cdn.mailscanner.info/emails.2013-260.11 at
/etc/MailScanner/ScamNailer line 289.

This is after applying Mr. Sapiro's patch. Are these two websites dead? I
can get to cdn.mailscanner.info via
web, but that is it. Any ideas?



-----------------------------------
Jeff A. Earickson, Ph.D
Senior Server System Administrator
Colby College,
4214 Mayflower Hill,
Waterville ME, 04901-8842
207-859-4214 (fax [phone])
Eastern Time Zone, USA
-----------------------------------

Post by Mark Sapiro

Post by Mark Sapiro
I have attached the latest version of my patch which works around this.

That patch contained a bit that isn't part of this issue. It wouldn't
hurt, but here's a patch without that extra bit.
--
Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
--
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mailscanner.info/pipermail/mailscanner/attachments/20130630/15428a73/attachment.html

[Mark Sapiro]

Post by Jeff Earickson
Failed to retrieve http://mailscanner.eu/emails.2013-260.11 at
/etc/MailScanner/ScamNailer line 289.
or
Failed to retrieve http://cdn.mailscanner.info/emails.2013-260.11 at
/etc/MailScanner/ScamNailer line 289.
This is after applying Mr. Sapiro's patch. Are these two websites dead? I
can get to cdn.mailscanner.info via
web, but that is it. Any ideas?

This is normal and expected behavior with my patch. Since the
information as to the number of the latest update is not available in
the TXT record at emails.msupdate.greylist.bastionmail.com, I guess that
the latest update is 99 and sequentially retrieve the updates until I
get a 404. In your case, the updates 260.1 through 260.10 were
successfully retrieved on this or a prior run and 260.11 didn't exist
yet and returned a 404. I then reset the current update to 10 and proceed.

If you don't want to see the "Failed to retrieve" message, change the
part of the patched Scamnailetr which reads

unless ($res->is_success) {
warn "Failed to retrieve $urlbase$currentbase.$i";
$currentupdate = $i - 1;
}

to

unless ($res->is_success) {
$currentupdate = $i - 1;
}