Solved

Tracking Down Network Timeout Issues - Need some good tools/analyzers

Posted on 2013-02-01
17
826 Views
Last Modified: 2013-03-20
Can anyone recommend some good tools or even just a methodology for tracking down intermittent timeouts/delays on a network?  I have a client who complains of timeouts and slowness to web pages.  They'll wait a moment, refresh the page and then it will load just fine.  Seems to happen across all web pages and from most if not all of their machines.  I've tested their bandwidth and they're getting their contracted rate on d/l and u/l speeds which should be more than enough for their size.  

They also complain about losing connection to some internal network shares so I believe the issue is probably internal.  Their switches are not optimally cabled, and they have a larger-than-I-would-recommend broadcast domain (one flat network using /23 SM), so I'm thinking these issues could be caused by broadcast storms, excaberated by the less than optimal cabling (switch to switch to switch instead of hub and spoke).  

I'm having a hard time replicating the issue when I go in to test, due to the intermittent nature of it, but I have seen it firsthand.  I'm wondering if anyone can recommend some good software (preferably free) that I can use to analyze their network to see if they're getting an inordinate amount of broadcast traffic, or that could give me any other clues on how to proceed to track this issue down.  I've used Wireshark before and could probably capture using that and then filter just the broadcast traffic, but not sure how to tell how much is too much.  Any advice on any of this is highly appreciated.
0
Comment
Question by:hachemp
  • 8
  • 3
  • 3
  • +2
17 Comments
 
LVL 77

Assisted Solution

by:Rob Williams
Rob Williams earned 200 total points
ID: 38844275
80% of network performance issues and drop outs are caused by bad cabling.  This cannot be tested with software but rather requires a cabling technician equipped with a proper Fluke or HP certification tool ($8,-$10K) to test and certify the network.  They can locate split pairs, poor terminations, cross talk, EMI, and much more that can have drastic affects on network performance.

This is why major software vendors such as ADP insist on a current network certification prior to installing their software.

Also slow connections can also be caused by improperly configured DNS.  For example having your internal servers as the primary DNS server and the ISP's as an alternate can cause slow connections to local resources and internet.  Only internal DNS servers should be used and ISP's added as forwarders on the server, or use root hints.
0
 

Author Comment

by:hachemp
ID: 38845275
Thanks for the response.  I have not checked any of the physical cabling yet but I have a Fluke so I may give that a shot.  In my experience if there's a problem in the wiring it's either all or nothing (not all of the time but mostly), and I've conducted multiple ping tests through all of the major switches and getting consistent 1ms ping times all around.  I'm sure it could still be cabling regardless but just doesn't seem likely to me given the circumstances.

I had already checked DNS...their DHCP server is handing out two internal DNS servers.  However, those servers were set up with forwarders to external DNS servers from a different ISP than what they have currently.  I didn't like that, so I pulled those out and am just using root hints on both now...still waiting to see if that made any difference.
0
 
LVL 77

Expert Comment

by:Rob Williams
ID: 38845604
Which Fluke do you have?

The DNS changes sound good. They may help with browsing, but not with internal file access.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 38845908
What type of switches?  Are they managed?

Anything in the logs?

Can you setup a syslogd server and have the switches forward their logs to it?

If Cisco, do they have portfast enabled everyplace that does not need to carry multiple tagged VLANs?
0
 
LVL 77

Expert Comment

by:Rob Williams
ID: 38846675
To add to my previous comment/question about your Fluke meter.  The meter needs to be  a Fluke Cable certification meter such as a DSP or DTX series.  Other network tools such as the Fluke Nettool, CableIQ, MicroScanner, or even the LAN tools cannot do a cable certification (i.e. a full test).  Generally only quality cable installers and large campuses own these.
0
 
LVL 3

Expert Comment

by:fritz5150
ID: 38847956
The easiest way to accomplish this level of diagnostic would be to configure a span port on the central most switch and setup wireshark to do a capture to a local laptop or pc. Keep in mind that this will be a lot of information, so be sure you have a drive with sufficient storage space. You can then use the expert composite analysis to see any issues in the capture.
0
 

Author Comment

by:hachemp
ID: 38852769
Thanks for all of the comments guys.  Yeah RobWill unfortunately my Fluke is not that cool (NetTool Series II)...it will do wiremapping but not cable certification.

They have a few Enterasys switches (B5G124-48P2) and a Dell PowerConnect 5212 as their central switch.  Unfortunately they have absolutely no idea what the passwords are to any of them.  I have quite a bit of experience with the Dells but none with Enterasys.  At some point I'm probably going to have to reset them all to default so I can get into the management...not ideal but I don't seem to have much choice.

Fritz, thanks for the tip on Wireshark.  I have used it a few times but have never used the expert analysis.  I can't set up a span port without being able to get into the switches but I can at least run WS from another computer and see what kind of traffic is on the network.  I'll post back once I have done that.  I'll make sure to spread the points around as best I can for all who are contributing...and thanks again.
0
 
LVL 21

Accepted Solution

by:
Rick_O_Shay earned 300 total points
ID: 38863962
You can reset the password on the B5s with the button on the back.
Once in the CLI I first would check all of the ports for errors to see if there is anything amiss with that.

"show port counters" (for all ports) "show port counters ge.1.X" (for a specific port X) and  "show rmon stats" "show rmon stats ge.1.X"

There is also a quick cable test you can run which will knock down the port for a second:
"show port cablestatus ge.1.X"

If I remember correctly that works only with gig copper ports and requires a PC or other device to be connected at the remote end.

The Enterasys network management platform will give you all of this stuff at a glance if you have that installed.

The "show logging buffer" will show you the switch event logs but it is set pretty high by default so you probably won't get much from that.

That should give you a start with the Enterasys switches.

Then as mentioned by yourself and others get a Wireshark capture running. You can quickly look at the statistics to see what is going on by endpoint and conversation to get an idea of who is consuming bandwidth, broadcasting, etc. That works while the capture is running or stopped. This is all of course relative to the station running Wireshark. A port mirror to get a capture of the router or a server can be helpful.
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 

Author Comment

by:hachemp
ID: 38864038
Great stuff, Rick, and thanks much for the commands.  Do you happen to know if that button on the back will clear the config, or just reset the password?
0
 
LVL 21

Expert Comment

by:Rick_O_Shay
ID: 38864257
That just resets the password.
0
 

Author Comment

by:hachemp
ID: 38864278
Excellent, thank you sir.
0
 

Author Comment

by:hachemp
ID: 38894786
Hello, I was finally able to go onsite and touch those Enterasys switches.  They had an B5G124-24P2 that I hit the password reset button on, and it reset to 'admin' and blank password...had no problems logging in.  However, on the B5G124-48P2, they have two of these connected with a stacking cable.  On one of them, connecting to the console, all I get is the following prompt:

(Unit 1)>

I don't seem to be able to do anything at all from there.  Pushing and hold the password reset does nothing to the CLI or otherwise.  Guessing this is the slave switch in the stack?

When I connected the console cable to the other one, I did get prompted for username and password.  I tried the default, no go of course, so I reached back and pushed the password reset button.  I get this on the CLI:

<161>Feb 14 18:58:51      10.1.1.240-2 USER_MGR(1): 217 % Password Reset button has been pressed

All good, right?  So I tried again to log in with admin and no password and get this:

<165>Feb 14 18:59:29      10.1.1.240-2 USER_MGR(1): 218 % User:admin failed login from console

I tried holding this button in probably 20 different times, ranging from tapping it to holding it for probably 60 seconds.  Each time I get the password reset button message on the CLI, and each time I was unable to log in with what should be the default credentials.  I scoured the internet for any other default credentials for these switches and found none.  The only thing I can figure out is that someone must have disabled the admin user.  

So my question is...is there any way for me to get into this switch?  Any way to reenable the admin user or otherwise gain access?  Thanks in advance!
0
 
LVL 57

Expert Comment

by:giltjr
ID: 38894991
It is configured to use some type of remote authentication, like RADIUS or TACACS+?

If so you might have to disconnect the network connection so it can't get to the server and then it may fail over to local authentication.

At least that is how it works on Cisco when you do remote authentication.
0
 

Author Comment

by:hachemp
ID: 38895199
Thanks, that would make perfect sense.  However, I'm about 90% certain they don't have RADIUS or TACACS set up anywhere.  However, next time I'm there I'll certainly give that a shot.
0
 
LVL 21

Expert Comment

by:Rick_O_Shay
ID: 38895236
I don't think I have ever had an issue using the password reset but I don't use stacking cables on mine. Might try breaking the stack for a minute, if you can, and see if it lets you reset either one. There is a login lockout if you use the wrong password 3 times. That gets unlocked again in 15 minutes if it happens.
0
 

Author Comment

by:hachemp
ID: 38934703
Thanks for the tips...I'm gonna be onsite hopefully this week and I'll give that a shot.
0
 

Author Closing Comment

by:hachemp
ID: 39003672
Sorry, haven't been able to get back out there so closing the ticket and awarding points.  I appreciate the info.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Transparency shows that a company is the kind of business that it wants people to think it is.
Even if you have implemented a Mobile Device Management solution company wide, it is a good idea to make sure you are taking into account all of the major risks to your electronic protected health information (ePHI).
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now