[Webinar] Streamline your web hosting managementRegister Today

x
?
Solved

Intermittent connectivity

Posted on 2009-07-07
6
Medium Priority
?
457 Views
Last Modified: 2012-05-07
Recently we've been having some problems with intermittent connectivity.
The network at my new employers utilizes a large flat topology, with about nine switches daisy chained together and a single /24 subnet that is very near capacity. We're using mostly HP hardware, including several end of life chassis / module design switches. The majority of our servers reside on one switch A, the users on the remainders. I'll call the most prevalent problem child switch B.  Switch B is midway up the daisy chain, and switch A is on end. I can ping, ssh, rdp, etc into any server from any other server connected to switch A but some servers I cannot reach from switch B.

I tried running nmap's ping sweep to get a feel for what is going on since the switch logs are useless. The results are inconsistent. Two scans run simultaneously from switch B on different ports will return widely varying results, some times with as many as 20 hosts unaccounted for from one port to the other. Neither port on B matches up with a scan run from a host on switch A.

I remember seeing similar behavior around 5 years ago but I don't definitively remember the cause or the temporary solution we used. Long term we purchased a router, which I will do here as well. I think the problem turned out to be the MAC or connection table was getting full and the new connections trying to be established were simply dropping. Does that sound about right for the cause of this behavior? Is there anything I can do before getting my router installed a few weeks from now?
0
Comment
Question by:timbrigham
6 Comments
 
LVL 9

Accepted Solution

by:
jfer0x01 earned 800 total points
ID: 24794847
Hello,

the problem is that you have 9 switches daisy chained!

perhaps, it's time to invest in a larger switch, instead of many small ones, to consolidate your cabling centrally

if not, you said it yourself, replace switch b

most likely, you have a user, with a different pattern in traffic use than before, which is causing more packets to be dropped as they pass through the switches, which now results in sporadic service

try runnnig a network monitor tool, such as Wireshark, or NetMon (MS tool) to analyze the packets that are being dropped, to tie them to a source machine

Jfer

Jfer
0
 
LVL 2

Assisted Solution

by:regnighc
regnighc earned 800 total points
ID: 24795169
Definitly the 9 switches not helping the situation, that will cause propagation delays and will start causing errors.

I would agree with Jfer

0
 
LVL 1

Author Comment

by:timbrigham
ID: 24796053
I agree as well, hence installing a router. :)
I was hoping there was something I could do in the interim to resolve the problem before the router gets here.  

Considering the size of our organization, three of our switches - including B - are large HP units, 96 ports each. Going any larger really isn't an option.
None of my network taps are placed conveniently to monitor switch B. I've used port mirroring on routers in the past, but I'm a little leery to do so on switch that is already having problems. What kind of performance impact could I expect to receive by setting up a port mirror?
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
LVL 16

Assisted Solution

by:Steve Jennings
Steve Jennings earned 400 total points
ID: 24798537
Agree with all . . . some poor switch is seeing a boat load of MAC addresses associated with one port and likely is puking when trying to allocate cut-through buffers for them.

Good luck,
SteveJ
0
 
LVL 1

Author Comment

by:timbrigham
ID: 24825728
I have the problem isolated.
Apparently at some point, my coworkers intentionally connected a switch A to a couple other switches in addition to B in an effort to increase speed. The network diagram didn't reflect the update so I took it on good faith the cabling was correct. Since spanning tree was also disabled on our switches we have a major layer 2 loop that needs to be broken. I'll work it into this weekend's maintenance window.  That should clear things up until I get the router installed.

Thanks all - without your direction I wouldn't have found this.
Points awarded shortly.

0
 
LVL 9

Expert Comment

by:jfer0x01
ID: 24826683
Good to know you found the source

Jfer
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains the fundamentals of industrial networking which ultimately is the backbone network which is providing communications for process devices like robots and other not so interesting stuff.
In this article, the configuration steps in Zabbix to monitor devices via SNMP will be discussed with some real examples on Cisco Router/Switch, Catalyst Switch, NAS Synology device.
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…

612 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question