Solved

Intermittent connectivity

Posted on 2009-07-07
6
383 Views
Last Modified: 2012-05-07
Recently we've been having some problems with intermittent connectivity.
The network at my new employers utilizes a large flat topology, with about nine switches daisy chained together and a single /24 subnet that is very near capacity. We're using mostly HP hardware, including several end of life chassis / module design switches. The majority of our servers reside on one switch A, the users on the remainders. I'll call the most prevalent problem child switch B.  Switch B is midway up the daisy chain, and switch A is on end. I can ping, ssh, rdp, etc into any server from any other server connected to switch A but some servers I cannot reach from switch B.

I tried running nmap's ping sweep to get a feel for what is going on since the switch logs are useless. The results are inconsistent. Two scans run simultaneously from switch B on different ports will return widely varying results, some times with as many as 20 hosts unaccounted for from one port to the other. Neither port on B matches up with a scan run from a host on switch A.

I remember seeing similar behavior around 5 years ago but I don't definitively remember the cause or the temporary solution we used. Long term we purchased a router, which I will do here as well. I think the problem turned out to be the MAC or connection table was getting full and the new connections trying to be established were simply dropping. Does that sound about right for the cause of this behavior? Is there anything I can do before getting my router installed a few weeks from now?
0
Comment
Question by:timbrigham
6 Comments
 
LVL 9

Accepted Solution

by:
jfer0x01 earned 200 total points
ID: 24794847
Hello,

the problem is that you have 9 switches daisy chained!

perhaps, it's time to invest in a larger switch, instead of many small ones, to consolidate your cabling centrally

if not, you said it yourself, replace switch b

most likely, you have a user, with a different pattern in traffic use than before, which is causing more packets to be dropped as they pass through the switches, which now results in sporadic service

try runnnig a network monitor tool, such as Wireshark, or NetMon (MS tool) to analyze the packets that are being dropped, to tie them to a source machine

Jfer

Jfer
0
 
LVL 2

Assisted Solution

by:regnighc
regnighc earned 200 total points
ID: 24795169
Definitly the 9 switches not helping the situation, that will cause propagation delays and will start causing errors.

I would agree with Jfer

0
 
LVL 1

Author Comment

by:timbrigham
ID: 24796053
I agree as well, hence installing a router. :)
I was hoping there was something I could do in the interim to resolve the problem before the router gets here.  

Considering the size of our organization, three of our switches - including B - are large HP units, 96 ports each. Going any larger really isn't an option.
None of my network taps are placed conveniently to monitor switch B. I've used port mirroring on routers in the past, but I'm a little leery to do so on switch that is already having problems. What kind of performance impact could I expect to receive by setting up a port mirror?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 16

Assisted Solution

by:SteveJ
SteveJ earned 100 total points
ID: 24798537
Agree with all . . . some poor switch is seeing a boat load of MAC addresses associated with one port and likely is puking when trying to allocate cut-through buffers for them.

Good luck,
SteveJ
0
 
LVL 1

Author Comment

by:timbrigham
ID: 24825728
I have the problem isolated.
Apparently at some point, my coworkers intentionally connected a switch A to a couple other switches in addition to B in an effort to increase speed. The network diagram didn't reflect the update so I took it on good faith the cabling was correct. Since spanning tree was also disabled on our switches we have a major layer 2 loop that needs to be broken. I'll work it into this weekend's maintenance window.  That should clear things up until I get the router installed.

Thanks all - without your direction I wouldn't have found this.
Points awarded shortly.

0
 
LVL 9

Expert Comment

by:jfer0x01
ID: 24826683
Good to know you found the source

Jfer
0

Featured Post

New My Cloud Pro Series - organize everything!

With space to keep virtually everything, the My Cloud Pro Series offers your team the network storage to edit, save and share production files from anywhere with an internet connection. Compatible with both Mac and PC, you're able to protect your content regardless of OS.

Join & Write a Comment

What’s a web proxy server? A proxy server is a server that goes between clients and web servers, used in corporate to enforce corporate browsing policy and ensure security. Proxy servers are commonly used in three modes. A)    Forward proxy …
Even if you have implemented a Mobile Device Management solution company wide, it is a good idea to make sure you are taking into account all of the major risks to your electronic protected health information (ePHI).
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now