Network drop-outs on XP workstations connected to Cisco Layer-3 Switches
Posted on 2006-06-17
I have a network configuration of (qty 6) Cisco 2950 and (qty 1) 2960G all connected by fiber. On the 2950 switches I have 60+ IP cameras all running significant data rates (appr 300mbs total) back to (qty 2) Windows 2003 servers that are connected to the 2960G. There are (qty 6) XP workstations (WS) used for live viewing, (qty 3) at various locations on the 2950s and (qty 3) located at the 2960G switch. Everything is running on the same network subnet and VLAN.
The good news is all switches, IP Cameras and Windows 2003 servers work perfectly, running continuously for two months without any glitches or issues of any kind.
The problem is with the XP WS used for live viewing. Their network connectivity will drop-out intermittently at different times. I have observed these boxes for hours at a time and they will drop their live video feed intermittently during the day. The software is designed to reconnect to the Windows 2003 servers and on a couple of the boxes, this works very well most of the time. The problem that makes the client complain is when one or more of the XP WS does not recover and displays ‘no video’ until we fix it. Half of the XP boxes exhibit this network drop-out at least twice a day while a couple will run for a week or longer. 5 of the XP WS are Dell 206/207 and other one is a comparable white box.
This issue has been going on for several months now and we have tried many different things to fix this problem when it occurs. The one troubleshooting technique we have used for the last month is to simply run Ping tests to the two servers until the WS starts working again. Typically we will observe the first ping return ‘Request Times Out’ and the balance return normally and then the network connectivity is restored immediately and the video application starts running again. In some cases where the network drop-out occurred close to our testing we will see Ping tests return “Request Times Out’ for 5 to 10 minutes or longer and then, like magic everything will return to normal, until the next time.
Recently we setup FreePing on all machines each pinging all the other machines. The Servers will reliably ping each other 100% of the time. The XP WS will not ping anything reliably clocking in at 93% to 99%
Other troubleshooting items of interest are:
· All NIC cards are set to Auto1000 and have the latest drivers.
· Servers CPU loading is 50%, WS CPU loading is less than 10%
· IPSEC is off/disabled
· One gateway, no dns\wins servers, using IP addressing exclusively.
· No trace problems, goes from source->switch->destination.
· No errors or issues in the switch logs, as verified by a Cisco tech support
· Cisco had us verify/set all host ports to ‘Fastport’ with no change
· 4 of the XP WS have been tested with dumb 100mbs switches and they will run for weeks at a time with no issues under the same loading conditions.
I have reviewed all my Cisco books however the fact that the Windows 2003 Servers and the majority of the network (60+ IP Cameras) is operating perfectly has me quite baffled.