URGENT: high severity problem with the PIX 525 at Osaka conference

Dear all,

Bellow you will find the description my colleagues sent. A memory increasing problem risk to provoke a general outage at the plenary (Client coference we run) that is running in Japan today.
 

-      SEVERITY HIGH : PIX FW version 8.0.4 memory increasing problem
The upgrade of the IOS on the PIX from 6.3 to 8.0.4 has fixed the problem on the VPN Client which used IPSEC. However, this has introduced a new problem. The memory usage on the PIX is continuously increasing until the max. When it reaches the limit of the capacity, there are no Internet Access here.

It has happened around 10:00 this morning. We have decided to failover the Standby unit.
But, The standby unit has a copy of the memory content thus we were in the same situation.
Finally, We have shut down the Standby unit and start the Primary one.
After the start-up we had only 84MB used now it is increasing by 25MB every hours.
We suspect 2 processes on the PIX:
Dispatch Unit and Unicorn Admin Handler are 2 processes that used the biggest amount of memory and they are increasing.

Thanks in advance for anything you can do to help us to resolve this critical issue.

If you need more information, you can reach me at :

miguel.paton@etsi.org


Best regards

PIX-125MB-AFTER-2H.TXT
PIX-171MB-AFTER-6H.TXT
LVL 1
martineitAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

lrmooreCommented:
From Cisco web site
Here are some possible causes and resolutions for high memory utilization. It sounds like a memory leak and may have to contact Cisco TAC for an upgrade to the OS, or downgrade to 8.0(3)19 or something..

Event logging: Event logging can consume large amounts of memory. In order to resolve this issue, install and log all events to an external server, such as a syslog server.

Memory Leakage: A known issue in the security appliance software can lead to high memory consumption. In order to resolve this issue, upgrade the security appliance software.

Debugging Enabled: Debugging can consume large amounts of memory. In order to resolve this issue, disable debugging with the undebug all command.

0
martineitAuthor Commented:
Hello,

Seams to me the origin of the memory leak is on a bug quite well known : CSCsj84640 : Memory leak on CRYPTO_malloc.

There is no available higher version than  8.0.4. The version bellow which doesn't have this bug is the the Version 7.1(2).
Before instruct my colleagues to downgrade to the this version I'd like to know if there is any other solution. The PIX is in production at this moment.

Best regards

0
yuriskCommented:
Hard to believe someone has other options here - problem started clearly after upgrade, the 1st
solution that TAC would ask to try - upgrade/downgrade , the only question asked here is to what version.
I 've had one client with VPn clients problem - more than 5 connections were freezing the ASA 5510  on memory( it was FOS 7.0)  , after a search on cisco.com I found 7.2(4) to be least buggy , without using 8.x that would for sure introduce new bugs.So  It has already passed about 2 months - havent heard from this client yet.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

lrmooreCommented:
Open a TAC case with Cisco. They may have an unreleased bug fix for you.
0
RickCommented:
any progress on this subject/post?

i've wrote the other post you referenced to...

These last days we are having increased fallouts, i'm runnung os 8.0.(4) and adsm 6.1.(5)51 on our Pix 515E / 64MB. we have 1 site2site vpn and use the device for remote access for 5 / 10 employees..
Mostly it begins with the pix not being to able to accept incomming VPN connects and since yesterday it block all incomming connections for remote control when the memory is full. So i have to do hardware reboot. Earlier i rebooted the machine remotely by cli.
Since the PIX support is dead in june, should i even buy a ASA?? it uses the same software and probebly has the same errors...
Regards, Rick
0
lrmooreCommented:
>should i even buy a ASA??
Absolutely, without question. It does not use the same hardware and the software is not exactly the same. Better memory management on ASA. More development is going into the ASA because the PIX is dead.
You can also try upgrade to a bug fix release 8.0(4)16 or 8.0(4)23
 
0
RickCommented:

haha, this is funny :-S
I've listed 4 OS versions, check out the bugs.. listed in each (and solved or not...)
taken from cisco release notes for the asa 5500 series

ASA 5500 series version 8.0.(3)
CSCsj84640 - Memory leak on CRYPTO_malloc (Open Caveats)
CSCso64944 - (doesn't excist)
CSCsj25896 - (doesn't excist)

ASA  5500 series version 8.0.(4)
CSCsj84640 - (doesn't excist? / not solved or open!)
CSCso64944 - ASA memory leak due to IPSEC (Open Caveats) (huh?? new name?)
CSCsj25896 - ASA may reload with traceback in Thread name: CTM Message Handler (Resolved Caveats)

(i don't know if i would apply a 5580 to a 5500 series, but ok.. just to show u)
ASA  5580 version 8.1.(1)
non of the above ceveats! (not open or closed)

ASA  5580 version 8.1.(2)
CSCsj25896 Crypto Accelerator Memory Leak (re-opened under a new name??)

Very strange all!
A company who does some IT projects for us suggested that we buy a Zyxcel firewall/router, 1/3 cost of a new 5510
0
RickCommented:
Taken from the Cisco Bug toolkit. I suggest to close this issue

"This status of this bug (CSCsj84640 ) is terminated,

suggesting that a conscious decision to not fix this bug was made.
It is possible the engineers were not able to duplicate the issue in a lab environment or some other reason was made to terminate this bug.

This bug has a Severe severity level 2 designation. Important functions are unusable but the router's other functions and the rest of the network is operating normally."

Probely going for a 5505 or 5510 soon, gonna try 8.0(4)23 later on, hope it helps!
Thnx
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Cisco

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.