Solved

Intermittent connectivity

Posted on 2009-07-07
6
441 Views
Last Modified: 2012-05-07
Recently we've been having some problems with intermittent connectivity.
The network at my new employers utilizes a large flat topology, with about nine switches daisy chained together and a single /24 subnet that is very near capacity. We're using mostly HP hardware, including several end of life chassis / module design switches. The majority of our servers reside on one switch A, the users on the remainders. I'll call the most prevalent problem child switch B.  Switch B is midway up the daisy chain, and switch A is on end. I can ping, ssh, rdp, etc into any server from any other server connected to switch A but some servers I cannot reach from switch B.

I tried running nmap's ping sweep to get a feel for what is going on since the switch logs are useless. The results are inconsistent. Two scans run simultaneously from switch B on different ports will return widely varying results, some times with as many as 20 hosts unaccounted for from one port to the other. Neither port on B matches up with a scan run from a host on switch A.

I remember seeing similar behavior around 5 years ago but I don't definitively remember the cause or the temporary solution we used. Long term we purchased a router, which I will do here as well. I think the problem turned out to be the MAC or connection table was getting full and the new connections trying to be established were simply dropping. Does that sound about right for the cause of this behavior? Is there anything I can do before getting my router installed a few weeks from now?
0
Comment
Question by:timbrigham
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 9

Accepted Solution

by:
jfer0x01 earned 200 total points
ID: 24794847
Hello,

the problem is that you have 9 switches daisy chained!

perhaps, it's time to invest in a larger switch, instead of many small ones, to consolidate your cabling centrally

if not, you said it yourself, replace switch b

most likely, you have a user, with a different pattern in traffic use than before, which is causing more packets to be dropped as they pass through the switches, which now results in sporadic service

try runnnig a network monitor tool, such as Wireshark, or NetMon (MS tool) to analyze the packets that are being dropped, to tie them to a source machine

Jfer

Jfer
0
 
LVL 2

Assisted Solution

by:regnighc
regnighc earned 200 total points
ID: 24795169
Definitly the 9 switches not helping the situation, that will cause propagation delays and will start causing errors.

I would agree with Jfer

0
 
LVL 1

Author Comment

by:timbrigham
ID: 24796053
I agree as well, hence installing a router. :)
I was hoping there was something I could do in the interim to resolve the problem before the router gets here.  

Considering the size of our organization, three of our switches - including B - are large HP units, 96 ports each. Going any larger really isn't an option.
None of my network taps are placed conveniently to monitor switch B. I've used port mirroring on routers in the past, but I'm a little leery to do so on switch that is already having problems. What kind of performance impact could I expect to receive by setting up a port mirror?
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 16

Assisted Solution

by:SteveJ
SteveJ earned 100 total points
ID: 24798537
Agree with all . . . some poor switch is seeing a boat load of MAC addresses associated with one port and likely is puking when trying to allocate cut-through buffers for them.

Good luck,
SteveJ
0
 
LVL 1

Author Comment

by:timbrigham
ID: 24825728
I have the problem isolated.
Apparently at some point, my coworkers intentionally connected a switch A to a couple other switches in addition to B in an effort to increase speed. The network diagram didn't reflect the update so I took it on good faith the cabling was correct. Since spanning tree was also disabled on our switches we have a major layer 2 loop that needs to be broken. I'll work it into this weekend's maintenance window.  That should clear things up until I get the router installed.

Thanks all - without your direction I wouldn't have found this.
Points awarded shortly.

0
 
LVL 9

Expert Comment

by:jfer0x01
ID: 24826683
Good to know you found the source

Jfer
0

Featured Post

Optimum High-Definition Video Viewing and Control

The ATEN VM0404HA 4x4 4K HDMI Matrix Switch supports 4K resolutions of UHD (3840 x 2160) and DCI (4096 x 2160) with refresh rates of 30 Hz (4:4:4) and 60 Hz (4:2:0). It is ideal for applications where the routing of 4K digital signals is required.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article is a collection of issues that people face from time to time and possible solutions to those issues. I hope you enjoy reading it.
This article is in regards to the Cisco QSFP-4SFP10G-CU1M cables, which are designed to uplink/downlink 40GB ports to 10GB SFP ports. I recently experienced this and found very little configuration documentation on how these are supposed to be confi…
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Suggested Courses
Course of the Month3 days, 15 hours left to enroll

630 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question