We are seeing a strange issue between some servers and network connectivity between them.
We have 4 cabinets, cabinets 1 & 2 have 4 Cisco 3750 switches, stacked (Stack1) cabinet 3 has 2 Cisco 3750 switches, stacked (Stack2) and cabinet 4 has 2 Cisco 3750.
We have started to see issues where servers (windows 2k8 based) suddenly can't see servers in cabinet 3. Physical or virtual. If, for example, I connect to Server1 based in cabinet 1 and ping Server2 based in cabinet 3, the ping will fail. If I set it to a constant ping, it never returns a response. If I then connect to Server2 in Cabinet 3 and ping back to Server1, that works and straight away the failing ping from Server1 - Server2 starts to work again.
Similar behavior can be seen with something like UNC.
There are also issues when connecting out from Cabinet3 to say Cabinet4.
We have looked at some error logs on the Cisco switches and couldn't see much. Two things of note the switch stack in Cabinet3 has a failed GBIC, but is using a redundant path. And the switches in Cabinet4 were recently restarted.
We are using some graphical Cisco tools and we have noticed that the ARP tables for switch stacks in Cabinet 3 & 4 are empty. Which would I assume explain the strange ping issues. But we cannot figure out why this would be happening!?
Does anyone have any idea what may be causing this?