firewall question for packet inspection rules

Hello Team,

I have a customer who is experiencing some random email issues from different sender/s domains. At the begging, we suspected there was an issue with our Edge servers and some sort of filtering or spamming rules that were blocking those emails.

After further investigation with our cloud spam provider [Symantec Message Labs], they advised the error displayed below could be some sort of packet inspection at your firewalls and or IDS/IPS

Email flow below:

Email goes to external MX records managed by company X

Email flows from external world to Spam gateway provider [Message labs], from message labs flows to inbound edge 2013 servers.

Between Message labs and the edge servers are multiple firewall, IDS/IDP appliances with multiple rules

The problematic email was never delivered to our spam gateway because of error 451.4.4.2 internal connection closed by remote host.

To clarify the connection error, this is being closed by the recipient server.  It could be implying this is some form of packet inspect or firewall based blocks.  We are able to connect to the  primary route x.x.x.x and cannot replicate the issue.

Our spam gateway provider advised to investigate the reason for closing the connection when this particular email is attempted to be delivered on your server.

We disabled any content/attachment filtering at the Edge servers to remove Exchange inbound inspection from the equation, but issue persists.

Our infrastructure settings below

2 sites, each sites contains 3 Exchange 2013 MBX servers in a DAG, 2 CAS servers behind a F5 HLB, 2 Edge servers in a DMZ network for inbound email, then proxy all request to MBX servers

From a security point of view, we do have following equipment:

Firewalls checkpoint model 12000 CGL running GAIA version R77.20

NMS (network security manager) which is the device that manages the IPS/IDS

IPS/IDS Mcafee Instrushield 4050

We also have the intrushield 1400

On each site we have cluster with an active/standby configuration

2 gateways per site for high availability


Questions:


What needs to be done at the IDS/IPS to fix this issue? Please provide instruction step by step depending on the hardware model and information provided above?

At the IDS/IPS, how can we detect if an inspection packet rule is blocking emails from certain domains/senders/ or with some rules, such as subject, body, attachment, and any other criteria?

How can I guarantee that my VIP users will not get email blocked by any of this packet inspection without putting on risk the security of the company?

What are the main setting to check on the security appliances for email inspection and validation?
Jerry SeinfieldAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jerry SeinfieldAuthor Commented:
Any updates?

please, do not neglect this question
Jerry SeinfieldAuthor Commented:
Any updates?

Any IT security professionals on this site? is hard to believe  nobody can provide their input
bbaoIT ConsultantCommented:
> I have a customer who is experiencing some random email issues from different sender/s domains.

so the issue is for incoming messages only, correct?

> Email flows from external world to Spam gateway provider [Message labs], from message labs flows to inbound edge 2013 servers.
> Between Message labs and the edge servers are multiple firewall, IDS/IDP appliances with multiple rules

so the incoming path is: Internet -> spam gateway -> IDS/IDP -> Exchange 2013, correct?

> The problematic email was never delivered to our spam gateway because of error 451.4.4.2 internal connection closed by remote host.
> At the IDS/IPS, how can we detect ...

if all my understanding is correct, i actually got confused as the error happens on the spam gateway (before the IDS/IDP), how could you use IDS/IDP to detect something not reached to IDS/IDP?
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

Jerry SeinfieldAuthor Commented:
Message labs is the cloud provider, but all emails sent to our domain goes first to Message labs which is a cloud provider, then to External firewall, IDS/IPS, then F5 hardware load balancer, then to Exchange 2013
bbaoIT ConsultantCommented:
> "error 451.4.4.2 internal connection closed by remote host."

can you clarify which side gives the above error message and what "remote host" refers to?
Jerry SeinfieldAuthor Commented:
I've requested that this question be deleted for the following reason:

Lack of knowledge and expertise shown by the experts in regards to this question
bbaoIT ConsultantCommented:
the issue here is ambiguous problem description, not something "lack of knowledge and expertise".
Blue Street TechLast KnightCommented:
Hi Jerry,

451.4.4.2 internal connection closed by remote host can occur, unfortunately, for multiple reasons, which is why most throw up their hands and blindly point fingers like Message Labs did (BTW what they said makes no sense...read further & I'll explain).

Here is a list of the usual suspects for 451.4.4.2 internal connection closed by remote host errors:

IP Blacklisted - I'd check your Public IP to make sure its not been blacklisted at http://mxtoolbox.com/blacklists.aspx. If you don't have your Access Rules setup, MX, and Message Labs setup correctly your IP can still be compromised, hence blacklisted. In fact even if you do have it all setup correctly, I have still seen IPs blacklisted because your email server ultimately is not in the cloud - the only true way to protect your IP's integrity.
DNS Reverse Lookup Mismatch - Make sure you have a valid DNS PTR record - this only handled by your ISP & you should contact them to set it up - you can't add this to your external DNS. I have seen a number of companies filter on this alone (which is crazy IMO) and yield a 451 connection closed because this record did not match the IP address. You can check your PTR record to see if it exists or if its pointing in the correct location here: http://mxtoolbox.com/SuperTool.aspx?action=ptr
SPF Record Mismatch - Similar to the PTR record mismatch, other spam filters check this and hardfail if its incorrect. Make sure your SPF is allowing all the locations that are sending out email.

Basically the sender's email server/spam filter doesn't trust or like a configuration you have on your end.

I don't believe this would be hitting any firewall that you have control of based solely on what you have stated. "Email flows from external world to Spam gateway provider [Message labs], from message labs flows to inbound edge 2013 servers." "...After further investigation with our cloud spam provider [Symantec Message Labs], they advised the error displayed below could be some sort of packet inspection at your firewalls and or IDS/IPS." What I have bolded is where you/Message labs are saying the problem lies - this is completely inaccurate based solely on your mail flow and how the internet works. If Message Labs is NOT receiving the email how can your firewall block anything?!

Are you seeing this error message in the queue, if not where are you seeing this error (I'm only asking because I'm skeptical of ML's response)? Have you performed Message Tracking on a test email or any email that was trying to be delivered on their end or on yours?

Let me know how it goes!

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jerry SeinfieldAuthor Commented:
Hi Diverseit,

Thanks for your quick response.

To answer your question, yes, we performed message tracking at all levels [Message labs console, exchange edge servers, exchange mailbox servers], and the message was never delivered by message labs [cloud spam provider], when message labs tries to deliver the messages to next hop [Firewall, internal IDS/IPS, + F5], they got 451.4.4.2

Unfortunately, message tracking at the Exchange edge servers does not show the problematic message because never hit the servers, remember our mailflow is as per below

Message Labs ---> External Firewall -- F5-- Exchange Edge servers-  Mailbox server, we have Exchange 2013 multiple MBX in a 2 site cross DAG, and multiple CAS behind a F5 HLB

Your thoughts?
Blue Street TechLast KnightCommented:
So you went through the usual suspect list and all is clear? This error is typically indicative of a misconfig on your end to some degree whether by compromise (IP Blacklisting) or by SPF or PTR.

How are you replicating through S2S VPN? How have you configured outbound mail flow...is it sending from two IPs or have you unified?
Blue Street TechLast KnightCommented:
What did it end up being?

Thanks for the points...glad I could help!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Hardware Firewalls

From novice to tech pro — start learning today.