Link to home
Start Free TrialLog in
Avatar of pgreetis
pgreetis

asked on

No Internet Acess: PAT seems to be reaching max limit, PIX 501

Hi all,

Recently users have been complaining about not being able to access the internet. I checked our firewall, but nothing seemed to be out of the ordinary, and still no one could access the internet. Then I checked the cpu usage, and it fluctuated between 79 - 96%. I did a show xlate and found a LOT of translations taking place (numerous were from one or two pc's, which says virus, but AV hasnt found anything). However, when I did a clear xlate, internet connectivity seemed to be restored. The cpu usage is still pretty high, and I have to do a clear xlate every 15-20 minutes to keep connectivity up. I am not sure what's going on, but from what I have seen, it seems that the Pix might be reaching the limit of how many PATs it can do.  

Any help is appreciated on this.
Avatar of calvinetter
calvinetter
Flag of United States of America image

Yes, definitely smells like an infested PC or 2 - malware itself, or malware that's installed an SMTP engine, trying to spam the world.  Mini SMTP engines on PCs are popular w/ malware these days.

As a temporary measure only, you can lower the timeout for NAT translations (default is 3 hrs):
timeout xlate 0:15:00   <- sets it to 15 min limit
clear xlate

Capture the output of "sh xlate" to a file, & see what source ports & what source internal IPs are accounting for the most traffic.  Then track down those PCs & take them off the network while you scan w/ both AV tools & anti-spyware tools.

cheers
Avatar of pgreetis
pgreetis

ASKER

Hi, thank you for that...I did lower the timelimit to 15, and that seemed to hold up for a day, but this morning, some people were complaining again, so I checked the firewall again. And this time, there weren't that many PATs (maybe 50 or so) and people were still having problems. Additionally, i did a show cpu usage, and it was at 1% (everytime internet connectivity goes dows- so does the cpu usage). Also, I cannot telent into the pix anymore for some odd reason.

So, here are the symptoms of the firewall as soon as connecetivity goes down (these were there before as well, the only difference is that PATs werent this low):

CPU usage goes down to 1%
Cannot telnet at all
PATs USED to reach 6000+, currently there are only like 50 (from one pc).
All computers that try to open connections are on port 80- they're contacting advertising, spam websites, and some come up as page not found, or "nothing to see here" (we tracked down the websites).
Operation is normal only after I do a clear xlate (as in I can telnet back in, connectivity goes up).


So yes, this is definitely a virus and its spreading, however, what confuses me, is why the pix stops responding. I have currently re-routed all traffic to a different pix, however some pc's are still on the faulty pix (the ones with viruses), and they open numerous connections, but nowhere near the limit of PAT.

Thoughts?

Thank you for your response and help on this.
 Regarding the PIX, I'd recommend some preventative measures:
- If the PIX isn't running the latest 6.3(5) code, & you have current Cisco "SmartNet" support on it, I strongly suggest upgrading the firmware to 6.3(5).  Same goes for any other 501 or 506 PIX you may have there.
- If you're not using the PDM web interface to manage/configure your PIX, you should disable the web server:
  no http server enable

  Regarding the workstations, take the offending PCs off the local LAN, clean out all temp files (ccleaner is good for this: www.ccleaner.com), then do a full hd scan with antivirus & anti-spyware apps; Adaware & Spybot used together are quite good at finding spyware.  AVG Free is a good freeware antivirus app (free.grisoft.com), finds some malware that commercial ones might miss - use it in conjunction with whatever commercial AV app you've got.

cheers
Hello, once again, thank you for your continued help, and my apologies for not updating this sooner. We finally figured out what was going on. Seems like one of the machines was sending orders to the other ones, and the other ones at that time started opening connections to spam websites. The affected machine itself was trying to connect to two specific IPs. As soon as I blocked those IP's everything went back to normal.

The last question that was bothering me a bit was when this happens (too many xlates), when we try to telnet, now the pix does something different...it gives a screen that has:

Username:

Password:


EDIT, EXECUTE (I think) OTHER OPTIONS.

(All this is on one screen, at the same time).

Essentially if I pick edit, it lets me enter a username and password, but thats it...nothing else happens..Im not sure if this behaviour is normal, or why it is heppening. I can console in fine though.

So, thats the last of it, I think. We have the issue figured out, and also reduced the xlate to 15 mins(which helped a lot). Currently scanning machines and keep an eye on the ones that were opening, and know for a fact which one machine is responsible for this mess. I will try and get a support contarct with Cisco and upgrade the PIX as well.

Once again, thank you for all your help.
>EDIT, EXECUTE (I think) OTHER OPTIONS.
   That's definitely not normal, never seen that from a PIX.  Sounds like maybe one of the infected PCs were hijacking the telnet session?? Or just your telnet client barfing when it can't connect??

  I strongly suggest NOT using telnet to your PIX, since everything (including the enable password) are sent in the clear!
   Use SSH instead (if it has DES or 3DES licensed):
sh ver   <-- look for  "VPN-DES:    Enabled"   If so, procede below:
pix(config)# ca zero rsa
pix(config)# ca gen rsa key 1024   <-- wait, or go & get coffee
pix(config)# ca save all   <-- very important! without this it won't save the key after a reload
pix(config)# ssh 0.0.0.0 0.0.0.0 inside

  Then use an SSH client (putty is a great, free one - just Google it) to connect to the PIX.  Use the same username/password if using AAA, or "pix" as username, & telnet password.  ** Be sure to configure 'putty' or other SSH client to use SSH v1 - PIX 6.x doesn't support SSH v2 connections!

cheers
calvinetter....Im an idiot. We figured out what the problem was, and I should be ashamed of myself for making such a stupid mistake....we installed a new linksys switch, and without checking its default setting I plugged it into the core- turns out its default IP out of the box was exactly the same as our PIX...the interface we were getting was of the switch.

However, we did have a virus, and the above was actually happening with one of the servers opening all the connections. You suggestions helped a LOT, and did keep us up and running till we found the infected machine.

So, problem is solved, and I hang my head in shame for making such a stupid mistake. ^_^

Thanks for all your help!
Hey, don't worry about it - stuff happens!  Glad you got it going.

cheers
Avatar of Keith Alabaster
ASKER CERTIFIED SOLUTION
Avatar of Computer101
Computer101
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial