Workstations connect to network, but unable to connect to internet

lludden used Ask the Experts™
On my local network, I have several workstations that suddenly (overnight) lose the ability to connect to the internet.  Some of these were powered on, some were off.  Some are running XP, some are running various flavors of Windows 7.  I have a cable modem that is connects to a Cisco PIX 501 firewall/router, which then is connected to an access point and a switch.  Initially, none of the systems were able to connect to the internet (but all could connect to each other).  I reset the PIX and had my ISP clear arp caching for my connection.  This got my domain controller up and working.  The domain controller also hosts my DNS server.  I was able to get my primary Win7 workstation to connect after going through several diagnostics (netsh int ip reset, netsh winsock reset), and changing its IP address.  I had two laptops (WinXP and Win7) working, but after a reboot, they no longer connect either.

I checked the settings on my PIX with what they were six months ago, and it has not changed.  There was a very odd blinking pattern on the PIX this morning, but it hasn't repeated since I rebooted it.  When I ssh into the PIX, I can ping IP's on both interfaces.  My clients that cannot connect to the internet can resolve names, so I know they are connecting to the domain controller and getting a response back.

At this point, I am thinking that there is something flaky happening with the PIX hardware, but I am open to suggestions.

Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
How many workstations do you have on your networks, which are trying to get out via the PIX?

By default, the 501 comes with a 10-user license. Once that count is exceeded, users are unable to get out onto the 'net.

That being said: what serves the role of your DHCP server, is it your DC or is it the PIX?
I have fewer than ten total systems.  At this point, when I open the web interface for the PIX, it shows 2 of 10 connections used.  The DC is the DHCP server.  The PIX is only being used for firewall/routing.  I have several static IP's that get routed to different PCs/IIS sites.
Not saying this is a foolproof test but try doing these steps and report back. Do this from your working Win 7 box and your non working laptop. Sounds like a DNS issue to me.

1) ping 
2) ping  (google's IP address)
When I ping to the name, it resolves to an IP (on both machines), but times out.

When I ping to the IP, it just times out.

Tracert also just times out.

Both commands work fine from my workstation that is plugged into the same switch.
If your pings to IP are not working we can rule out DNS. What if for a test, you connect your switch directly to your cable modem. Your cable modem should be able to handle access to internet on its on. That way you can bypass the Pix
Do you see any difference with a workstation plugged directly into the 501 versus your switch?

Also - you mentioned an "odd" flashing pattern on the PIX - was it the POWER, VPN TUNNEL (shouldn't be), or one of the LAN port lights?

If you SSH into the PIX and check its log (show logging), do you see anything odd (warnings, critical errors)?

FYI: pings may not be a reliable test - I for one (as do many others) disable ICMP ECHO traffic going out onto the internet. You can enable this via:
pixfirewall# conf t
pixfirewall(config)# access-list ping_acl permit icmp any any
pixfirewall(config)# access-group ping_acl in interface outside
pixfirewall(config)# exit
pixfirewall# wr mem

Open in new window

I have enabled imcp on the PIX.  The pix has been rebooted since I was getting the weird flashing.  It was almost like a disco, with the lights for the power, link, and active connections all flashing in sequence.

If I disconnect the cable modem from the pix, I will need to have the ISP reset the arp cache and again when I put it back.  I'll do that, but I want to try any other approaches first.  

Plugging into the ethernet port on the PIX rather than the switch doesn't change anything.

Tomorrow I will be able to get another PIX, and plan on configuring it with the same settings I have now and swapping the hardware.   I'd still be interested in hear any other possible solutions.
Before I say that your PIX is damaged (the "disco PIX" is a little odd), I would first suspect the power supply. Perhaps if you get a chance to try a new one tomorrow, hopefully you will also have a new power pack - just to make sure!

All of the symptoms you describe certainly point towards the PIX, so I look forward to hearing what the outcome of the test will be.
Ernie BeekSenior infrastructure engineer
Top Expert 2012
There have been issues with 501's regarding the power supply/connection. If I remember correct it had something to do with the connector. If you picked it up or just moved the pix it would powercycle or start discoing.
That should be easy to test. If so, you'd be better of with a replacement because I don't think Cisco replaces them anymore (as they did before).
Really, really good suggestions from everyone.. What about the duplex settings matching between Cisco products. That would explain the blinking pattern and inability to communicate with the outside world.
Here is an odd thought which occurred to me: is there any chance that your clients are seeing an incorrect DHCP server (perhaps one running on your DC)?

Perhaps the DC's DHCP attempts to make the DC as the gateway to the internet - which is obviously not correct - but does allow DNS queries to work. It would exhibit itself as being able to resolve host names, but nothing would be reachable...

A couple of quick "ipconfig /all" checks and perhaps a "traceroute" should be able to help you rule out this possibility.
Everything worked when I got home that night.  I have a new PIX configured and ready to go if it happens again, so I will be able to pinpoint the problem as either the PIX or my ISP.

I really appreciate all the suggestions from everyone.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial