IDPS slows SMB traffic

Hello folks

we have an issue where our in-line IDPS system (McAfee M-8000) slows down smb traffic . here are some details :

- if we switch IDPS to layer 2 mode ; performance is fine . no cabling or duplex issue
- if we remove all IDPS policies - smb is still slow . so it's not the policy causing it
- There is no rate limiting or QoS profile on the device

any thoughts ?
LVL 10
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

btanExec ConsultantCommented:
Maybe some evidence gathering first for troubleshooting to isolate issue too - A latency issue can often be related to dropped packet issues. Often, when latency is seen on a Sensor, dropped packets also occur.

I supposed you already the above steps - wondering if the diagnostics trace from the Sensor, shows any errors in the interested interface and it is in the egress or ingress exchange that fails, or interface to ISP is slow or the interface port is having addition transceiver (for fibre to copper  vice versa) ... maybe best to even try the ping throughout as you make changes to see any dropped symptoms...
You could absolutely prove it is the IDPS causing the delay and not some weird way it's interacting with its environment with some network tracing.  Capture and analysis with Wireshark would be pretty straight forward.  To make like easier, run Wireshark on an old spare server with two network interfaces.  Have one interface capture on one side of the IPDS and the other interface on the other.  This way you get traces from both interfaces on a common clock so timestamps are sync'd.

To analyse, compare the Time from Request value for, say, a READ or WRITE Command and Response.  Or you could use the Wireshark plugin called TRANSUM to give you a more detailed analysis.  The TRANSUM User Guide explains the meanings of the response time values it gives.
btanExec ConsultantCommented:
if need to you can run some PRTG sensor type (free for PRTG 100 for 100 sensor) uses SNMP, packet sniffing, and NetFlow to track network traffic. But do check on McAfee Manager that talks to IDPS and poll for SNMP to see if IDPS is failing in any way from port bandwidth and also there can be fail open kit for its line deployment.
Fail-open kits can be deployed in production networks for the following reasons:
• Reduce the network downtime to seconds during any Sensor reboot or Sensor failure
• Protect your network during link failure on the Sensor
• Bypass the traffic when troubleshooting network issues. This will help you identify or eliminate the Sensor as the cause of network issues

In case of active fail-open kits, during normal Sensor in-line fail-open operation, the built-in monitoring sends a heartbeat signal (1 every second) to the Bypass Switch. If the Sensor does not receive 3 heart beat signals within its programmed interval, the Fail-Open Bypass Switch removes the Sensor from the data path, and moves it into the bypass mode, providing continuous data flow.
I understand there is IPS quarantine rules which its rule drops all traffic from a source IP, compare to an ACL rule can be more specific to the type of traffic that is dropped. so do make sure that ACL rules are configured explicitly not dropping certain segment or client unintentionally. The quarantine rules are processed independently from, and evaluated before, ACL rules.
When a particular alert is declared as a false positive, the next decision is whether to disable the corresponding attack altogether OR apply a particular exception object to that attack that will disable alerting for a particular IP address or range of IP addresses. In almost all cases, it is a best practice to implement the latter.
Also be aware of the time periods in which your scheduled processes (such as database backup or report generation) occur, and try not to attempt other tasks during that time period, as this can lead to process locking. This includes having many users logged into the system simultaneously.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Virus Depot: Cyber Crime Becomes Big Business

The rising threat of malware-as-a-service is not one to be overlooked. Malware-as-a-service is growing and easily purchased from a full-service cyber-criminal store in a “Virus Depot” fashion. View our webinar recording to learn how to best defend against these attacks!

akhalighiAuthor Commented:
all very nice suggestions. I'll try and get back .
btanExec ConsultantCommented:
Suggested troubleshooting approach given and the question should not be deleted.
btanExec ConsultantCommented:
Troubleshooting steps given.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.