All External Mail to Office 365 Fails SPF, Marked as Junk by EOP in a Hybrid Deployment

Hi folks,

In short, our legitimate emails are landing in Junk folders as EOP (Exchange Online Protection) stamps email messages as junk (SCL5) and SPF-failed. This happens with all external domains (e.g. gmail.com/hp.com/microsoft.com) to client’s domain (contoso.com).

Background info:

We are at the beginning of migrating mailboxes to Office 365 (Exchange Online). This is a Hybrid Deployment/Rich-Coexistence configuration, where:

On-Premise = Exchange 2003 (Legacy) & 2010 (Installed for Hybrid Deployment)
Off-Premise = Office 365 (Exchange Online)
EOP is configured for SPF checking.
MX records are pointing at the on-premises as we haven't completed migrating all mailboxes from on-premise to Exchange Online.

The problem is when external users sends emails to an Office 365 mailbox in the organization (mail flow: External -> Mail Gateway -> on-premise mail servers -> EOP -> Office 365), EOP performs an SPF lookup and hard/soft failing messages with the external facing IP address of the Mail Gateway from which it received the mail.

On-premises mailboxes do not have this problem; only mailboxes migrated to Office 365.

An illustration of the problem with mail flow and details:
 
Mail Flow from External to Office 365 with EOP
Note: 23.1.4.9 is the public IP address of the on-premise hybrid Exchange 2010 server connector to Exchange Online.

For example (Message headers):

Case 1. Email from microsoft.com to contoso.com

Authentication-Results: spf=fail (sender IP is 23.1.4.9)
smtp.mailfrom=microsoft.com; contoso.mail.onmicrosoft.com;
dkim=none (message not signed) header.d=none;
Received-SPF: Fail (protection.outlook.com: domain of microsoft.com does not
designate 23.1.4.9 as permitted sender) receiver=protection.outlook.com;
client-ip=23.1.4.9; helo=exchange2010.contoso.com;

X-MS-Exchange-Organization-SCL: 5
X-Forefront-Antispam-Report: CIP:23.1.4.9;CTRY:HK;IPV:CAL;IPV:NLI;EFV:NLI;SFV:SPM;
SFS:(2980300002)(339900001)(489007)(189002)(252514010)(199003)(3905003)(5003600100002)
(450100001)(33646002)(86362001)(69596002)(66066001)(84326002)(46102003)(81156007)
(19300405004)(6806004)(26826002)(67866002)(92566002)(17760045003)(85426001)(86146001)
(18206015028)(19580395003)(19625215002)(62966003)(64706001)(24736003)(10290500002)
(19580405001)(106116001)(99936001)(50986999)(107886002)(10400500002)(86612001)
(5005710100001)(77156002)(15843345004)(105606002)(19618635001)(108616004)(189998001)
(16236675004)(87936001)(16234385003)(5001960100002)(10090500001)(19627595001)(76176999)
(15975445007)(106466001)(4001540100001)(19617315012)(54356999)(66926002)(575784001)
(2900100001)(5001830100001)(5001860100001)(512914005)(2950100001)(102836002)(2656002)
(110136002)(82516004);DIR:INB;SFP:;SCL:5;SRVR:SG2PR0301MB1285;H:exchange2010.contoso.com;
FPR:;SPF:Fail;PTR:009.4.1.23.static.isp.com;MX:1;A:1;LANG:en;

Case 2. Email from hp.com to contoso.com

Authentication-Results: spf=none (sender IP is 23.1.4.9)
smtp.mailfrom=hp.com; contoso.mail.onmicrosoft.com;
dkim=none (message not signed) header.d=none;
Received-SPF: None (protection.outlook.com: hp.com does not designate permitted sender hosts)
X-MS-Exchange-Organization-SCL: 5
X-Forefront-Antispam-Report: CIP:23.1.4.9;CTRY:HK;IPV:NLI;EFV:NLI;SFV:SPM;SFS:(6009001)
(2980300002)(428002)(53754006)(377454003)(189002)(6602003)(199003)(66066001)(64706001)
(86362001)(2656002)(24736003)(5001860100001)(5250100002)(2900100001)(87936001)(5001960100002)
(81156007)(5001830100001)(4001540100001)(110136002)(189998001)(101416001)(69596002)(512954002)
(105586002)(106466001)(102836002)(15975445007)(50986999)(77156002)(19300405004)(450100001)
(54356999)(62966003)(84326002)(16236675004)(6806004)(108616004)(92566002)(5003600100002)
(19580395003)(46102003)(33646002)(19625215002)(19580405001)(82516004);DIR:INB;SFP:;SCL:5;
SRVR:TY1PR0301MB1120;H:exchange2010.contoso.com;FPR:;SPF:None;PTR:009.4.1.23.static.isp.com;MX:1;A:1;LANG:en;

Case 3. Email from gmail.com to contoso.com

Authentication-Results: spf=softfail (sender IP is 23.1.4.9)
smtp.mailfrom=gmail.com; contoso.mail.onmicrosoft.com;
dkim=fail (signature did not verify) header.d=gmail.com;
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
gmail.com discourages use of 23.1.4.9 as permitted sender)

X-MS-Exchange-Organization-SCL: 5
X-Forefront-Antispam-Report: CIP:23.1.4.9;CTRY:HK;IPV:CAL;IPV:NLI;EFV:NLI;SFV:SPM;
SFS:(6009001)(2980300002)(199003)(189002)(189998001)(73392002)(97736004)(4001540100001)
(64706001)(76482005)(105596002)(83322999)(106466001)(81156007)(5001860100001)(5001960100002)
(68736005)(5000100001)(62966003)(564344004)(107886002)(5001830100001)(42186005)(82202001)
(92566002)(450100001)(2351001)(55446002)(229853001)(54356999)(46102003)(63696999)(87572001)
(956001)(110136002)(50986999)(81442002)(87936001)(61266001)(86362001)(69596002)(77156002)
(73972006)(26826002)(512874002)(6806004)(84326002)(59536001)(82516004);DIR:INB;SFP:;SCL:5;
SRVR:SG2PR0301MB1287;H:exchange2010.contoso.com;FPR:;SPF:SoftFail;PTR:ErrorRetry;A:1;MX:1;LANG:en;

How to stop external emails from being marked as junk by EOP during coexistence stage of a Hybrid Deployment?

Thanks!
wandersickAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Amit KumarCommented:
We have same infra and same happens with us also. so sometimes we suggest to client that they work on their side to update their SPF records and sometimes we whitelist subjected e-mail address.

I think you can do one more thing, configure your SPF check rule to take action on Hardfail, so at least softfail mails will be in.
wandersickAuthor Commented:
Amit, thanks but if you mean enabling "SPF: Hard Fail" in Advanced Spam Filtering Options, I don't think it would help. Even if it would let soft-failed mails in without going to Junk folder, we needs hard-failed mails in too. For example, the case 1 in my question is a hard-fail case, but it is a legitimate email from Microsoft to our organization.
Amit KumarCommented:
Then better to disable SPF check as MS EOP is very restrictive in SPF check...
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

wandersickAuthor Commented:
Regarding disabling SPF check in EOP, I wish I could do that. I can't figure out how to do so in the Exchange admin center web interface.

The closest I could get to is setting up a mail flow rule:
Screenshot-2015-08-16-000224.png
However, it does not work at all (or I don't know the amount of time it requires to take effect).

Anyway, this is not the way I desire it to be, as it would render EOP completely useless for mails with SCL=5 that are really spam rather than legitimate ones.
Amit KumarCommented:
Go to protection in EOP console, then spam filter then see how many rules are there, open them and check advanced filter, there will be one option for SPF hardfail. if it is yes choose it to no and save that rule then monitor.
wandersickAuthor Commented:
Thanks, but we did not modify any of the EOP options; the "SPF: Hard Fail" advanced option is already Off by default. Screenshot-2015-08-16-080255.png
Amit KumarCommented:
Then log a SR in MS and ask them to check, this is weird situation.
wandersickAuthor Commented:
The issue has been solved by Microsoft Office 365 support and/or their backend team.

We did nothing. As far as I know, they found a problem with the TLS tunnel and the mail flow from Exchange 2003 to Office 365 where this line in message header "X-MS-Exchange-Organization-AuthAs: Anonymous" should be "Internal". However, we verified all our settings; nothing should be wrong with TLS. And StartTLS is displayed when running EHLO command on our Exchange 2010 Hybrid server by telnet.

Later, we were informed we had the IP from our Inbound Connector (i.e. public IP of Exchange 2010 Hybrid server) defined in our IP Allow List (It was also suggested by an Office 365 support as a troubleshooting step actually). They let us know we should not need to do this and in fact doing so can cause routing issues. They found that on initial pass the email were not getting marked as Spam so there is also a possible issue here. We then removed the IP from the IP Allow List.

After they fixed it, we found that for internal-originated mail from Exchange 2003 to Office 365:
- X-MS-Exchange-Organization-AuthAs: Internal  (It was "Anonymous")
- SCL=-1 (It was SCL=5)
- Received-SPF: SoftFail (It was the same)

And for external mails (e.g. gmail.com) to Office 365::
- X-MS-Exchange-Organization-AuthAs: Anonymous (It was the same)
- SCL=1 (It was SCL=5)
- Received-SPF: SoftFail (It was the same)

Although SPF check still soft-fails for gmail.com (external) to Office 365, I believe it should be OK as long as mails go to Inbox instead of Junk folder (I will double-confirm this).

Thanks everyone at Experts-Exchange.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
wandersickAuthor Commented:
Troubleshooting details gone through with Microsoft Support
TTAF4Commented:
Thanks, I need to do the same for my org
Gopi VCommented:
Hi Outlook Team,

The issue seems still arises,  when you are sending mails from anydomain.com to outlook.com ! It is because of validating the SPF record of the return-path domain with the relayed MTAs IP (multiple hops relayed by outlook). So it should be fixed at their outlook end.

For these mails, SCL is 5, and the mail moved to JUNK folder.

And moreover, this is not consistence across all of mails, but this do happened.

From my observation this happens,
1. whenever we are sending from new IP
2. without TLS.

Please look into this.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.