Solved

Intermittent connectivity

Posted on 2009-07-07
6
431 Views
Last Modified: 2012-05-07
Recently we've been having some problems with intermittent connectivity.
The network at my new employers utilizes a large flat topology, with about nine switches daisy chained together and a single /24 subnet that is very near capacity. We're using mostly HP hardware, including several end of life chassis / module design switches. The majority of our servers reside on one switch A, the users on the remainders. I'll call the most prevalent problem child switch B.  Switch B is midway up the daisy chain, and switch A is on end. I can ping, ssh, rdp, etc into any server from any other server connected to switch A but some servers I cannot reach from switch B.

I tried running nmap's ping sweep to get a feel for what is going on since the switch logs are useless. The results are inconsistent. Two scans run simultaneously from switch B on different ports will return widely varying results, some times with as many as 20 hosts unaccounted for from one port to the other. Neither port on B matches up with a scan run from a host on switch A.

I remember seeing similar behavior around 5 years ago but I don't definitively remember the cause or the temporary solution we used. Long term we purchased a router, which I will do here as well. I think the problem turned out to be the MAC or connection table was getting full and the new connections trying to be established were simply dropping. Does that sound about right for the cause of this behavior? Is there anything I can do before getting my router installed a few weeks from now?
0
Comment
Question by:timbrigham
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 9

Accepted Solution

by:
jfer0x01 earned 200 total points
ID: 24794847
Hello,

the problem is that you have 9 switches daisy chained!

perhaps, it's time to invest in a larger switch, instead of many small ones, to consolidate your cabling centrally

if not, you said it yourself, replace switch b

most likely, you have a user, with a different pattern in traffic use than before, which is causing more packets to be dropped as they pass through the switches, which now results in sporadic service

try runnnig a network monitor tool, such as Wireshark, or NetMon (MS tool) to analyze the packets that are being dropped, to tie them to a source machine

Jfer

Jfer
0
 
LVL 2

Assisted Solution

by:regnighc
regnighc earned 200 total points
ID: 24795169
Definitly the 9 switches not helping the situation, that will cause propagation delays and will start causing errors.

I would agree with Jfer

0
 
LVL 1

Author Comment

by:timbrigham
ID: 24796053
I agree as well, hence installing a router. :)
I was hoping there was something I could do in the interim to resolve the problem before the router gets here.  

Considering the size of our organization, three of our switches - including B - are large HP units, 96 ports each. Going any larger really isn't an option.
None of my network taps are placed conveniently to monitor switch B. I've used port mirroring on routers in the past, but I'm a little leery to do so on switch that is already having problems. What kind of performance impact could I expect to receive by setting up a port mirror?
0
Forrester Webinar: xMatters Delivers 261% ROI

Guest speaker Dean Davison, Forrester Principal Consultant, explains how a Fortune 500 communication company using xMatters found these results: Achieved a 261% ROI, Experienced $753,280 in net present value benefits over 3 years and Reduced MTTR by 91% for tier 1 incidents.

 
LVL 16

Assisted Solution

by:SteveJ
SteveJ earned 100 total points
ID: 24798537
Agree with all . . . some poor switch is seeing a boat load of MAC addresses associated with one port and likely is puking when trying to allocate cut-through buffers for them.

Good luck,
SteveJ
0
 
LVL 1

Author Comment

by:timbrigham
ID: 24825728
I have the problem isolated.
Apparently at some point, my coworkers intentionally connected a switch A to a couple other switches in addition to B in an effort to increase speed. The network diagram didn't reflect the update so I took it on good faith the cabling was correct. Since spanning tree was also disabled on our switches we have a major layer 2 loop that needs to be broken. I'll work it into this weekend's maintenance window.  That should clear things up until I get the router installed.

Thanks all - without your direction I wouldn't have found this.
Points awarded shortly.

0
 
LVL 9

Expert Comment

by:jfer0x01
ID: 24826683
Good to know you found the source

Jfer
0

Featured Post

Why Off-Site Backups Are The Only Way To Go

You are probably backing up your data—but how and where? Ransomware is on the rise and there are variants that specifically target backups. Read on to discover why off-site is the way to go.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
This article is a collection of issues that people face from time to time and possible solutions to those issues. I hope you enjoy reading it.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question