CPU usage at 99% on a PIX 535

Ok smart people, here is a challenge for you.

Background:

I have a PIX 535 w/failover, 6 interfaces (inside, outside, syslog, state full, and 2 not used)  My configuration is 35 pages printed.  We are an enterprise entity running off a single firewall.  I am logging informational to a kiwi syslog over udp to filter the monster that the pix creates for a log.  The PIX has a gig of ram.  I am using turbo access lists.

Problem:

The cpu usage goes to 99% and stays there for 15-20 minutes if I make any changes to a object group.  This slows traffic but not to a halt.  If I make more than 5 changes in 20 minutes it will slow significantly and traffic will timeout through the PIX.  We have the w32.korgo.V virus here and when it was on 6 different machines it would bring the PIX to a halt.  It tries to make connections to other IP's on port 445 and we have 445 blocked on the firewall.  Even though those 6 machines were infected and getting denied by the PIX every time they attempted to connect outside it, this would cause the pix to slow down enough to time out all traffic and most attempts to connect to the PIX itself.  We blocked 445 on the router before the pix and the cpu immediately dropped to 6% usage and stays below 10.  

It is not just the virus but a number of things that will cause the pix to slow lately and I do not see any errors or unexplained syslog messages.  Cisco has sent us a new 535 w/failover and the same thing happened on the new boxes.  This same config has worked perfectly for 5 months with nothing like this happening.  I keep getting "escalated" to a different engineer in Cisco, but I am convinced that the smartest Cisco people do not work for Cisco.  Does anyone have any ideas without me sending the config?

Thanks,

Sunny
sunnyd24Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

lrmooreCommented:
Have you also blocked outbound ICMP on the router before the PIX? This helped us out tremendously with the late Welchia/MSBLast infestation..
0
lrmooreCommented:
Using the turbo acls, I can see a CPU spike when changes to object groups because the acl has to be recompiled, but it should only take seconds, not 15-20 minutes.. I think that could be what is compounding the problem if you try to make multiple changes in a short period of time.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sunnyd24Author Commented:
ICMP is blocked at the internal router yes.  I have never had a problem with the PIX performance in the past.  I am not making any more changes now than before, but for some reason the performance is shot.
0
Become an IT Security Management Expert

In today’s fast-paced, digitally transformed world of business, the need to protect network data and ensure cloud privacy has never been greater. With a B.S. in Network Operations and Security, you can get the credentials it takes to become an IT security management expert.

martapCommented:

Could'nt you try to selectively disable Turbo-ACL to see what the effect is on your CPU resources?
Did you upgrade your PIX OS lately?
0
sunnyd24Author Commented:
Finally! I found a solution.  For anyone else who may run into this problem:

Having problems with high CPU on the PIX when an access-list compiles using object-groups?
   Very large ACLs ( >200k) may not compile, or have very poor performace
   Solution:  Upgrade to 6.3(3.107) or higher, and disable turbo-acl and add the new CLI  
      access-list <acl_name> object-group-search
 
Try this by disabling turbo acl and adding the access-list <acl_name> object-group-search command.  
1. no access-lsit compiled
2. access-list <inside_out > object-group-search,access-list <outside_access_in > object-group-search, and one for each acl

My config is 935k.

Tell all your friends and neighbors.
0
lrmooreCommented:
Thanks for sharing that with us!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software Firewalls

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.