Exchange 2013 dropping emails from outside domain hosted by GoogleMail.

Currently running Exchange 2013 SP1 on Windows 2012 r2
Kaspersky security 9.0 for Microsoft Exchange
Fortinet 600C Firewall

I have a single domain which seems to be unable to deliver mail to us.  The mail is rejected with the error 503 5.5.1 Bad Sequence of Commands

All other mail appears to be flowing normally... it is just this one domain which is hosted on Googlemail.

This just started Wednesday and mail was flowing normally before then.   I have since whitelisted the domain since then on each the firewall, server and antivirus software.  The server has also been rebooted.

Our marketing people are blaming me for this and I have had zero luck resolving the issue.
Verbose logging is enabled on the ReceiveConnector.  The text of a conversation is as below (names and IP's changed to protect the innocent).

What am I missing?

EBC75AAF7CDF,47,MYLOCALIP:25,SENDERIP:37391,>,250-MAIL Hello [SENDERIP],
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,48,MYLOCALIP:25,SENDERIP:37391,>,250-SIZE 37748736,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,49,MYLOCALIP:25,SENDERIP:37391,>,250-PIPELINING,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,50,MYLOCALIP:25,SENDERIP:37391,>,250-DSN,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,51,MYLOCALIP:25,SENDERIP:37391,>,250-ENHANCEDSTATUSCODES,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,52,MYLOCALIP:25,SENDERIP:37391,>,250-AUTH NTLM LOGIN,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,53,MYLOCALIP:25,SENDERIP:37391,>,250-X-EXPS GSSAPI NTLM,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,54,MYLOCALIP:25,SENDERIP:37391,>,250-8BITMIME,
2015-11-13T16:45:14.666Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,55,MYLOCALIP:25,SENDERIP:37391,>,250 XRDST,
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,56,MYLOCALIP:25,SENDERIP:37391,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,57,MYLOCALIP:25,SENDERIP:37391,<,MAIL FROM:<SENDER EMAIL ADDRESS> SIZE=2654,
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,58,MYLOCALIP:25,SENDERIP:37391,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,59,MYLOCALIP:25,SENDERIP:37391,*,08D2EBC75AAF7CDF;2015-11-13T16:45:13.556Z;1,receiving message
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,60,MYLOCALIP:25,SENDERIP:37391,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,61,MYLOCALIP:25,SENDERIP:37391,<,RCPT TO:<RECIPIENT EMAIL ADDRESS>,
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,62,MYLOCALIP:25,SENDERIP:37391,<,DATA,
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,63,MYLOCALIP:25,SENDERIP:37391,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,64,MYLOCALIP:25,SENDERIP:37391,>,250 2.1.0 Sender OK,
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,65,MYLOCALIP:25,SENDERIP:37391,>,250 2.1.5 Recipient OK,
2015-11-13T16:45:14.838Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,66,MYLOCALIP:25,SENDERIP:37391,>,354 Start mail input; end with <CRLF>.<CRLF>,
2015-11-13T16:45:15.463Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,67,MYLOCALIP:25,SENDERIP:37391,*,,Proxy destination(s) obtained from OnProxyInboundMessage event
2015-11-13T16:45:15.463Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,68,MYLOCALIP:25,SENDERIP:37391,*,,NextHopFqdn property is null or whitespace when creating InboundProxyLayer
2015-11-13T16:45:15.759Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,69,MYLOCALIP:25,SENDERIP:37391,>,250 2.6.0 <CAL-gAAgDUkDRcvjdR0dJ=kRsnu4VCNGbKyy46cwkwJBgxsPYEA@mail.gmail.com> Queued mail for delivery,
2015-11-13T16:45:15.931Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,70,MYLOCALIP:25,SENDERIP:37391,<,QUIT,
2015-11-13T16:45:15.931Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,71,MYLOCALIP:25,SENDERIP:37391,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2015-11-13T16:45:15.931Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,72,MYLOCALIP:25,SENDERIP:37391,>,221 2.0.0 Service closing transmission channel,
2015-11-13T16:45:15.931Z,MAIL\Default Frontend MAIL,08D2EBC75AAF7CDF,73,MYLOCALIP:25,SENDERIP:37391,-,,Local

Please help.  While I know that this likely isn't directly my fault the ignorant person on marketing is verbally saying I am not doing my job and is threatening to go to the CEO if this isn't resolved soon.
LVL 8
Jeff RodgersNetworks & Communications Systems ManagerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

BembiCEOCommented:
If I look into you connector log, I cannot really see what is bad. So it seems to be that the connecter takes the mail. My may also have a look into the connection log.
The sequence her looks fine so far.
So, where do you see the error message, in the NDR?
What I see is a "NextHopFQDN is empty"...
Make sure your connectors have a helo name set.


So it looks like the mail is dropped inside the exchange.
Queued for delivery usually means, that the mail is inside the exchange queue.
You can goto the mail queue and stop all connectors. Then send (or wait for) a mail from googlemail and release on connector after another and stop it again if it Is gone to the next stage. This way you can see, if the mail reaches the queue and on which stage it fails.

Bad Sequence usually means, that the sequence HELO, MAIL FROM, RCPT TO, DATA, QUIT is out of order. As I said, the log above logs fine from the order. But maybe not all of the mails are wrong, only a few ones. The connector takes the sequence as it comes from the source, so if googlemail server is the sender, the sequence is provided by googlemail.

If the sequence comes fine though the connecter, I would investigate a virus scanner, which is installed on Exchange server, you may temporay switch if off, send a mail from google mail and see what happens. Sometimes virus scanner, which involve into the mail flow may be the reason.

I have seen some articles connected to Outlook, so investigate the mails (in the queue you can inspect the mailheader, if the queue is stopped. There you find the sender program, just to narrow int down. If outlook is the source, than it is possible that some of the senders uses an affected outlook version. Or even again virus scanner, which involve into the transport.
If you can confirm, this affect only a few mails, but not all, it is a possible reason, that the problems comes from the sender.
Some people reported, it was resolve by alone after a while.
Make sure your exchange is up to date, the virus scanner is up to date and your outlook clients too.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
HariomExchange ExpertsCommented:
"Queued mail for delivery" means your exchange server has already accepted the e-mail and sent internally for delivery

First check the message tracking logs to see where that e-mails goes

also check your Kaspersky security 9.0 for Microsoft Exchange to make sure it is not blocking anything
0
Jeff RodgersNetworks & Communications Systems ManagerAuthor Commented:
Sorry for the late response.  I'm going to award and even split on this one as I worked the problem myself and found the solution.  

The sender was listed on numerous DNSBL's which when it hit Kaspersky caused the email to bounce with a poorly defined response.   Checking their DNS settings revealed listings on SORBS, Spamhaus listings.

The email did arrive at the connector and was passed onwards only to be rejected for one of several reasons.

Of course, if the person who sent the email would now just believe me that our setup isn't the problem,  everyone would be happy.  Sigh...

Thanks for your help guys!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Email Protocols

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.