Cloud-based scanning

[Denis Beauchemin]

Hello,

We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.

The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.

I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?

Thanks.

Denis

[James Nelson]

Denis,

I also use office 365 for an edge filtering. Mine is synchronized with my internal active directory via the Microsoft Dirsync tool, but aside from the additional load you will place on your internal servers by not offloading directory based blocking at the office 365 level, there's not technically any harm in what you describe, as long as you configure your internal servers not to send NDRs, which in addition to creating additional traffic can be used by spammers for address validation

Post by Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.
The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.
I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?
Thanks.
Denis
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

[Denis Beauchemin]

James,

Thanks for the info. But if my servers don't send NDRs, people who make a typo in the email address of one of our users won't be notified of the error and will not be able to retry with the correct email address... This is not desirable but on the other hand you are right about address harvesting. And if my servers do send NDRs they could be used to joe-job some poor pal.

I don't really like both avenues...

Thanks.

Denis

-----Message d'origine-----
De : mailscanner-***@lists.mailscanner.info [mailto:mailscanner-***@lists.mailscanner.info] De la part de James Nelson
Envoyé : 3 mars 2015 09:41
À : MailScanner discussion
Objet : Re: Cloud-based scanning

Denis,

I also use office 365 for an edge filtering. Mine is synchronized with my internal active directory via the Microsoft Dirsync tool, but aside from the additional load you will place on your internal servers by not offloading directory based blocking at the office 365 level, there's not technically any harm in what you describe, as long as you configure your internal servers not to send NDRs, which in addition to creating additional traffic can be used by spammers for address validation

Post by Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.
The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.
I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?
Thanks.
Denis
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

--
MailScanner mailing list
***@lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner

Before posting, read http://wiki.mailscanner.info/posting

Support MailScanner development - buy the book off the website!

[Glenn Steen]

Well, one problem with doing most any rejections "after the fact" is
that you may become a "spam reflector", apart from the efficiecy
argument (rejections are a LOT cheaper than bounces;-).
ISTR there being a lot of discussion about this about... Ohh... 6?
years ago. There were some trends in how spammers crafted their filth
back then ("joe jobs" etc) that made rejections particularily
attractive. I suppose nothing really has changed on that account.

Cheers!
--
-- Glenn

On 3 March 2015 at 15:47, Denis Beauchemin

Post by Denis Beauchemin
James,
Thanks for the info. But if my servers don't send NDRs, people who make a typo in the email address of one of our users won't be notified of the error and will not be able to retry with the correct email address... This is not desirable but on the other hand you are right about address harvesting. And if my servers do send NDRs they could be used to joe-job some poor pal.
I don't really like both avenues...
Thanks.
Denis
-----Message d'origine-----
Envoyé : 3 mars 2015 09:41
À : MailScanner discussion
Objet : Re: Cloud-based scanning
Denis,
I also use office 365 for an edge filtering. Mine is synchronized with my internal active directory via the Microsoft Dirsync tool, but aside from the additional load you will place on your internal servers by not offloading directory based blocking at the office 365 level, there's not technically any harm in what you describe, as long as you configure your internal servers not to send NDRs, which in addition to creating additional traffic can be used by spammers for address validation

Post by Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.
The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.
I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?
Thanks.
Denis
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

--
-- Glenn
email: glenn < dot > steen < at > gmail < dot > com
work: glenn < dot > steen < at > ap1 < dot > se

[Jason Ede]

It's one argument that made greylisting so attractive for a long while (and I think the argument still holds)...

Steve Freegard of FSL published this... http://www.fortantispam.com/wp-content/uploads/2013/02/greylisting_whitepaper.pdf

Jason

-----Original Message-----
Sent: 03 March 2015 15:37
To: MailScanner discussion
Subject: Re: Cloud-based scanning
Well, one problem with doing most any rejections "after the fact" is that you
may become a "spam reflector", apart from the efficiecy argument
(rejections are a LOT cheaper than bounces;-).
ISTR there being a lot of discussion about this about... Ohh... 6?
years ago. There were some trends in how spammers crafted their filth back
then ("joe jobs" etc) that made rejections particularily attractive. I suppose
nothing really has changed on that account.
Cheers!
--
-- Glenn
On 3 March 2015 at 15:47, Denis Beauchemin

Post by Denis Beauchemin
James,
Thanks for the info. But if my servers don't send NDRs, people who make a

typo in the email address of one of our users won't be notified of the error
and will not be able to retry with the correct email address... This is not
desirable but on the other hand you are right about address harvesting. And
if my servers do send NDRs they could be used to joe-job some poor pal.

Post by Denis Beauchemin
I don't really like both avenues...
Thanks.
Denis
-----Message d'origine-----
James Nelson Envoyé : 3 mars 2015 09:41 À : MailScanner discussion
Objet : Re: Cloud-based scanning
Denis,
I also use office 365 for an edge filtering. Mine is synchronized with
my internal active directory via the Microsoft Dirsync tool, but aside
from the additional load you will place on your internal servers by
not offloading directory based blocking at the office 365 level,
there's not technically any harm in what you describe, as long as you
configure your internal servers not to send NDRs, which in addition to
creating additional traffic can be used by spammers for address
validation

On Mar 3, 2015, at 8:22 AM, Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it

would work would be to scan the emails there and then deliver them to our
servers.

Post by Denis Beauchemin

The problem is that we can't for the moment tell MS365 which email

addresses are valid and which are not (the data is in an LDAP server that is
not synchronized with our AD). Thus MS365 will be forwarding all harmless
emails to our internal servers who will reject invalid email addresses.

Post by Denis Beauchemin

I seem to remember this is really not a good idea but I can't remember

why. Can someone shed some light on this please?

Post by Denis Beauchemin

Thanks.
Denis
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

--
-- Glenn
email: glenn < dot > steen < at > gmail < dot > com
work: glenn < dot > steen < at > ap1 < dot > se
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

[Jerry Benton]

Or just enable recipient verification on your end and let 365 deal with the backscatter.

-
Jerry Benton
www.mailborder.com

Post by James Nelson
Denis,
I also use office 365 for an edge filtering. Mine is synchronized with my internal active directory via the Microsoft Dirsync tool, but aside from the additional load you will place on your internal servers by not offloading directory based blocking at the office 365 level, there's not technically any harm in what you describe, as long as you configure your internal servers not to send NDRs, which in addition to creating additional traffic can be used by spammers for address validation

Post by Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.
The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.
I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?
Thanks.
Denis
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!

[Steve Freegard]

Hi Denis,

Post by Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.
The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.
I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?
Thanks.
Denis

It depends if the MX is queue-and-forward or a proxy type.

For a proxy - this situation wouldn't be an issue, provided your backend
servers reject the recipients, it will reject them as a proxy would
simply man-in-the-middle the SMTP conversation filtering the bad stuff.
The issue with that method is that it's not particularly efficient.

If they're a queue-and-forward type, then they'll be accepting the
message (e.g. sending a '250 queued id=....' at the end of data) and
then delivering the message to your backends afterwards. This is more
common and where the problems are:

1) In SMTP - once you've accepted the message you have the
responsibility to either deliver the message to it's destination or to
bounce it back to the return-path. As we all know, the return-path is
trivial to forge and therefore this causes backscatter from the MX.

2) Wasted resources on the MX. Scanning mail for recipients that are
simply going to be rejected at delivery is a waste of I/O.

Both can be considerable issues considering that invalid recipients can
outweigh the valid on some domains.

As you're paying Microsoft to deal with this - you might not really care
about either (in my experience most people don't).

Their mitigation for 1) might be that they simply never send bounces in
this case - that's bad for your users because if someone genuinely
misaddresses a message then they don't get a bounce and never know that
it wasn't delivered (e.g. it goes down a black hole).

You'd have to check the Microsoft terms of service to see what they have
to say about both of these.

Kind regards,
Steve.

[Denis Beauchemin]

Thanks Steve and all the others.

My server will act as a proxy and will reject invalid addresses in the connect phase so I should be safe. I will be testing this RSN with an alternate domain name, in case something goes wrong...

Denis

-----Message d'origine-----
De : mailscanner-***@lists.mailscanner.info [mailto:mailscanner-***@lists.mailscanner.info] De la part de Steve Freegard
Envoyé : 3 mars 2015 13:02
À : MailScanner discussion
Objet : Re: Cloud-based scanning

Hi Denis,

Post by Denis Beauchemin
Hello,
We are about to move our MX to the cloud in Microsoft 365. The way it would work would be to scan the emails there and then deliver them to our servers.
The problem is that we can't for the moment tell MS365 which email addresses are valid and which are not (the data is in an LDAP server that is not synchronized with our AD). Thus MS365 will be forwarding all harmless emails to our internal servers who will reject invalid email addresses.
I seem to remember this is really not a good idea but I can't remember why. Can someone shed some light on this please?
Thanks.
Denis

It depends if the MX is queue-and-forward or a proxy type.

For a proxy - this situation wouldn't be an issue, provided your backend servers reject the recipients, it will reject them as a proxy would simply man-in-the-middle the SMTP conversation filtering the bad stuff.
The issue with that method is that it's not particularly efficient.

If they're a queue-and-forward type, then they'll be accepting the message (e.g. sending a '250 queued id=....' at the end of data) and
then delivering the message to your backends afterwards. This is more
common and where the problems are:

1) In SMTP - once you've accepted the message you have the responsibility to either deliver the message to it's destination or to
bounce it back to the return-path. As we all know, the return-path is
trivial to forge and therefore this causes backscatter from the MX.

2) Wasted resources on the MX. Scanning mail for recipients that are simply going to be rejected at delivery is a waste of I/O.

Both can be considerable issues considering that invalid recipients can outweigh the valid on some domains.

As you're paying Microsoft to deal with this - you might not really care about either (in my experience most people don't).

Their mitigation for 1) might be that they simply never send bounces in this case - that's bad for your users because if someone genuinely misaddresses a message then they don't get a bounce and never know that it wasn't delivered (e.g. it goes down a black hole).

You'd have to check the Microsoft terms of service to see what they have to say about both of these.

Kind regards,
Steve.
--
MailScanner mailing list
***@lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner

Before posting, read http://wiki.mailscanner.info/posting

Support MailScanner development - buy the book off the website!

[Steve Freegard]

Denis,

Post by Denis Beauchemin
Thanks Steve and all the others.

No problem - you're welcome.

Post by Denis Beauchemin
My server will act as a proxy and will reject invalid addresses in the connect phase so I should be safe. I will be testing this RSN with an alternate domain name, in case something goes wrong...

It will be interesting to see whether Microsoft will actually ask it
like this (e.g. in a call-ahead style).

What I suspect will happen is that Microsoft will actually accept mail
to unknown recipients and then bounce it when the delivery is attempted
to you later.

I'd be interested to hear your results and experiences once you've
switched over your initial test domain.

Kind regards,
Steve.

[Denis Beauchemin]

Here is my first impression of the Microsoft 365 email filtering service:

1- you may not know it but you are already getting email from it because it looks like all email coming from @hotmail.com, @outlook.com and others are sharing the same outbound servers as 365.
a) look here for the IP addresses used: https://technet.microsoft.com/en-us/library/dn163583(v=exchg.150).aspx
b) all server names seen so far contain this string: outbound.protection.outlook.com

2- it looks like emails to invalid addresses are handled correctly as you can see in the following email I received after sending from gmail:
smtpe2.usherbrooke.ca rejected your message to the following email addresses:

***@usherbrooke.quebec
The email address wasn't found at the destination domain. It might be misspelled or it might not exist any longer. Try retyping the address and resending the message.
If that doesn't work, contact the recipient (by phone or instant messaging, for example) to check that the address is correct. If the problem continues, forward this message to your email admin.

For Email Administrators
For more tips to help fix this issue, see DSN 5.1.1 Errors in Exchange Online and Office 365.

smtpe2.usherbrooke.ca gave this error:
<***@usherbrooke.quebec>... User unknown

3- MailScanner is still useful for fine-grained work such as:
a) detecting and neutralizing different tags: <A>, phishing, Form, Scripp, Web bugs, HTML
b) phishing attempts
c) whitelisted stuff (could probably be done by 365)
d) Sanesecurity and other Clam stuff
e) local SpamAssassin rules

4- I ended up archiving all emails that come from the list in #1 so I could get a look at them if I wanted to make sure MailScanner did the right thing when it decided to tag one of those emails as spam

5- During part of yesterday I managed to get these stats for the Microsoft 365 emails that went through MailScanner (I should get more in a few days):
=== Emails Rejected
Domain of sender address ***@student.fi2.be does not exist: 2
Domain of sender address no-reply-***@Support.inc.out.com does not exist: 2

HAM: 774
SPAM: 20
Actions Attach, Deliver, Header: 10
Actions Delete: 10

<A> tags: 577
Form tags: 1
Phishing tags: 23
Script tags: 2
Web Bug tags: 29
HTML Form tags: 1
HTML IMG tags: 354
HTML Script tags: 2

Expanding TNEF: 5
Removed TNEF: 5
Added TNEF: 4

Phishing Frauds: 52
Whitelisted: 27

=== Clamd::Infected::
Sanesecurity.ScamL.151.UNOFFICIAL: 1

=== Found spam-virus
Sanesecurity.ScamL.151.UNOFFICIAL: 2

=== SpamAssassin Rules
ADVANCE_FEE_4_NEW_FRM_MNY: 1
AXB_X_FF_SEZ_S: 18
BAYES_50: 3
BAYES_60: 2
BAYES_80: 3
BAYES_99: 12
BAYES_999: 9
BIGNUM_EMAILS: 1
BODY_URI_ONLY: 2
DCC_CHECK: 3
DEAR_WINNER: 1
EMPTY_MESSAGE: 9
FILL_THIS_FORM: 1
FILL_THIS_FORM_LONG: 1
FORM_FRAUD_5: 1
HAS_SHORT_URL: 2
HK_LOTTO: 1
HTML_FONT_LOW_CONTRAST: 1
HTML_MESSAGE: 8
LIST_PARTIAL: 3
LOTS_OF_MONEY: 1
MISSING_HEADERS: 3
MONEY_FRAUD_5: 1
RCVD_IN_BL_SPAMCOP_NET: 7
RCVD_IN_BRBL: 12
RCVD_IN_DNSWL_LOW: 2
RCVD_IN_DNSWL_NONE: 18
RCVD_IN_LASHBACK: 6
RCVD_IN_PSBL: 4
RCVD_IN_SORBS_WEB: 2
RCVD_IN_UCE_PFSM_1: 3
REPLYTO_WITHOUT_TO_CC: 1
TVD_SPACE_RATIO: 2
T_FSL_HELO_BARE_IP_2: 11
UDES_BUY15: 2
UDES_BUY17: 2
UDES_BUY99: 2
UDES_VIRUS01: 1
URIBL_AB_SURBL: 1
URIBL_DBL_ABUSE_BOTCC: 1
URIBL_DBL_ABUSE_REDIR: 2
US_DOLLARS_3: 1

6- For now I would not accept Microsoft 365-filtered emails without any local filtering; it does a good job of blocking with RBL because I didn't get any email that I blocked with Spamhaus or URIBL; I guess my MX servers will just go idle because they won't have to work as hard as before

Have a good week-end everybody!

Denis

-----Message d'origine-----
De : mailscanner-***@lists.mailscanner.info [mailto:mailscanner-***@lists.mailscanner.info] De la part de Steve Freegard
Envoyé : 3 mars 2015 18:15
À : MailScanner discussion
Objet : Re: Cloud-based scanning

Denis,

Post by Denis Beauchemin
Thanks Steve and all the others.

No problem - you're welcome.

Post by Denis Beauchemin
My server will act as a proxy and will reject invalid addresses in the connect phase so I should be safe. I will be testing this RSN with an alternate domain name, in case something goes wrong...

It will be interesting to see whether Microsoft will actually ask it like this (e.g. in a call-ahead style).

What I suspect will happen is that Microsoft will actually accept mail to unknown recipients and then bounce it when the delivery is attempted to you later.

I'd be interested to hear your results and experiences once you've switched over your initial test domain.

Kind regards,
Steve.
--
MailScanner mailing list
***@lists.mailscanner.info
http://lists.mailscanner.info/mailman/listinfo/mailscanner

Before posting, read http://wiki.mailscanner.info/posting

Support MailScanner development - buy the book off the website!

[Steve Freegard]

Post by Denis Beauchemin
The email address wasn't found at the destination domain. It might be misspelled or it might not exist any longer. Try retyping the address and resending the message.
If that doesn't work, contact the recipient (by phone or instant messaging, for example) to check that the address is correct. If the problem continues, forward this message to your email admin.
For Email Administrators
For more tips to help fix this issue, see DSN 5.1.1 Errors in Exchange Online and Office 365.

Unfortunately - that isn't really the 'proper' way. They're doing what
I expected they'd do - they're accepting the message and bouncing it
afterwards (instead of rejecting it outright at receipt and making the
originating hop bounce it without it leaving their system).

Basically - if a spammer decided to send mail to a bunch of old expired
(or simply invalid) userbrooke.queuec addresses as recipients with a
spoofed-but-valid return-path, then the poor owner of the spoofed
address would get a load of backscatter in return (from Microsoft).

Helpfully too - Microsoft attach the original mail to the bounce (I just
tried it myself), so it could potentially be used as a crude way to make
Microsoft send a load of spam (e.g. send the mail with a return-path of
the victim and intentionally make the recipient invalid, then Microsoft
will bounce the message to the victim with the spam payload attached).

Kind regards,
Steve.

[Glenn Steen]

As said, potential spam reflector... Yuk.
Good explanation Steve, thanks for that!

Cheers!
--
-- Glenn

Post by Denis Beauchemin

Post by Denis Beauchemin
2- it looks like emails to invalid addresses are handled correctly as
smtpe2.usherbrooke.ca rejected your message to the following email
The email address wasn't found at the destination domain. It might be

misspelled or it might not exist any longer. Try retyping the address and
resending the message.

Post by Denis Beauchemin
If that doesn't work, contact the recipient (by phone or instant

messaging, for example) to check that the address is correct. If the
problem continues, forward this message to your email admin.

Post by Denis Beauchemin
For Email Administrators
For more tips to help fix this issue, see DSN 5.1.1 Errors in Exchange

Online and Office 365.
Unfortunately - that isn't really the 'proper' way. They're doing what
I expected they'd do - they're accepting the message and bouncing it
afterwards (instead of rejecting it outright at receipt and making the
originating hop bounce it without it leaving their system).
Basically - if a spammer decided to send mail to a bunch of old expired
(or simply invalid) userbrooke.queuec addresses as recipients with a
spoofed-but-valid return-path, then the poor owner of the spoofed
address would get a load of backscatter in return (from Microsoft).
Helpfully too - Microsoft attach the original mail to the bounce (I just
tried it myself), so it could potentially be used as a crude way to make
Microsoft send a load of spam (e.g. send the mail with a return-path of
the victim and intentionally make the recipient invalid, then Microsoft
will bounce the message to the victim with the spam payload attached).
Kind regards,
Steve.
--
MailScanner mailing list
http://lists.mailscanner.info/mailman/listinfo/mailscanner
Before posting, read http://wiki.mailscanner.info/posting
Support MailScanner development - buy the book off the website!