ausadmin
asked on
Using Ethereal to pinpoint a network throughput problem
Hi All
Getting sent out tomorrow to do some network analysis which I've only done a very little of.
Some rental place gave me an ancient version of sniffer pro 3.0.5 on a dinosaur laptop.
The problem is that the client has a number of CCTV cameras mainly on 100 mb switches but there may be some 10s or 10 links.
When they up the cameras to 25 frames per second they get dropouts - in fact get dropouts at anything above 6. They've got about 25 cameras and they said they've done 18 camera installs before on 100mbs networks - no worries.
So a few Qs: Pointing me to the right bits of the manual would be peachy
(oh and I have to operate in a strictly windows environment)
- Does ethereal handle switched networks and if so do I have to set anything special up.
- Can ethereal generate traffic and do throughput testing if so - how? and if not do people know of any other free tools I can use. There used to be this thing called Raccoon speed tester that did timed transfers but I can't find it anymore on the web. I'm guessing if it gets down to some switch not handling the type of packets I'll need something that can throw different packet types.
- Reporting - the client wants the problem pin-pointed and proved on paper.
- General tips and gotchas
- Should I send the sniffer pro back - is it going to give me anything important in this scenario that ethereal won't.
Sorry for all the Qs but I've been chucked in the deep end on this one and one of our techs has already hit this and bounced so it's up to me to save the day :(
Much TIA
Getting sent out tomorrow to do some network analysis which I've only done a very little of.
Some rental place gave me an ancient version of sniffer pro 3.0.5 on a dinosaur laptop.
The problem is that the client has a number of CCTV cameras mainly on 100 mb switches but there may be some 10s or 10 links.
When they up the cameras to 25 frames per second they get dropouts - in fact get dropouts at anything above 6. They've got about 25 cameras and they said they've done 18 camera installs before on 100mbs networks - no worries.
So a few Qs: Pointing me to the right bits of the manual would be peachy
(oh and I have to operate in a strictly windows environment)
- Does ethereal handle switched networks and if so do I have to set anything special up.
- Can ethereal generate traffic and do throughput testing if so - how? and if not do people know of any other free tools I can use. There used to be this thing called Raccoon speed tester that did timed transfers but I can't find it anymore on the web. I'm guessing if it gets down to some switch not handling the type of packets I'll need something that can throw different packet types.
- Reporting - the client wants the problem pin-pointed and proved on paper.
- General tips and gotchas
- Should I send the sniffer pro back - is it going to give me anything important in this scenario that ethereal won't.
Sorry for all the Qs but I've been chucked in the deep end on this one and one of our techs has already hit this and bounced so it's up to me to save the day :(
Much TIA
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Heh, I'll bet £10 it's not duplex ;) Most switches are autosensing, esp. 10/100 switches.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
>- Should I send the sniffer pro back
I wouldn't. The Expert analysis module is worth its weight in gold... Gives Symptoms/diagnoses
I wouldn't. The Expert analysis module is worth its weight in gold... Gives Symptoms/diagnoses
Ok, my tenner is there on the table... :)
Documentation to backup my suggestion to first look for duplex mismatches http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00800a7af0.shtml
Assuming a mish-mash of older/newer cameras and old hubs and new switches, it never hurts to start looking at the most obvious places. Many uplinks become bottlenecks because of a port duplex mismatch between switches or between a switch and a hub, but I'll give purplepomegranite his due that this is also another obvious place to look.
Rule out issues using the 7 layer OSI model. Always start at layer 1 - a good physical inspection. Layout/topology, cable quality, loose connectors, environmental interference (cables running across florescent light fixtures), Then go to layer 2 looking at speed/duplex settings, etc..
The next place to look is port utilization. Of course old hubs can't give you that... but the uplink to the switchport can.
Do you know what type switches they have? Many have a web-based utility that can give you a good look at all port utilization as well as error counts.
Where are these cameras sending the video? To a server? Can the server handle the input? Is its CPU pegged out? Out of memory? Is it local, or across a layer 3 ip subnet boundary? What's in between the cameras and the server, if anything?
These are all just meant to get you thinking about what to look for..
I'll take that tenner bet... <8-}
ausadmin, I'm looking forward to your report on what you find!
Assuming a mish-mash of older/newer cameras and old hubs and new switches, it never hurts to start looking at the most obvious places. Many uplinks become bottlenecks because of a port duplex mismatch between switches or between a switch and a hub, but I'll give purplepomegranite his due that this is also another obvious place to look.
Rule out issues using the 7 layer OSI model. Always start at layer 1 - a good physical inspection. Layout/topology, cable quality, loose connectors, environmental interference (cables running across florescent light fixtures), Then go to layer 2 looking at speed/duplex settings, etc..
The next place to look is port utilization. Of course old hubs can't give you that... but the uplink to the switchport can.
Do you know what type switches they have? Many have a web-based utility that can give you a good look at all port utilization as well as error counts.
Where are these cameras sending the video? To a server? Can the server handle the input? Is its CPU pegged out? Out of memory? Is it local, or across a layer 3 ip subnet boundary? What's in between the cameras and the server, if anything?
These are all just meant to get you thinking about what to look for..
I'll take that tenner bet... <8-}
ausadmin, I'm looking forward to your report on what you find!
Good article that, lrmoore, I am also looking forward to the report back!! :)
ASKER
Wow this is great guys - I might have a chance.
Dumb question - how much is 'excessive' in a 100 mbps environment and a 10 mbps environment (all switched) - I spoke to the client again and he thinks some of the modules in the switches may be 10.
Will ethereal show which ones are 10 - or do I just figure that out with the numbers?
And packet generation - I was thinking that one of the switches maybe struggling with large or many small packets. Anything anyone know that can test generate this kind of traffic?
Dumb question - how much is 'excessive' in a 100 mbps environment and a 10 mbps environment (all switched) - I spoke to the client again and he thinks some of the modules in the switches may be 10.
Will ethereal show which ones are 10 - or do I just figure that out with the numbers?
And packet generation - I was thinking that one of the switches maybe struggling with large or many small packets. Anything anyone know that can test generate this kind of traffic?
ASKER
Last Q promise - I've got hold of 2 fluke devices - a fluke nettool and an Etherscope - how easy are they to drive - should I sink my time into those?
ASKER
Whoops I lied :-/
That link for the cisco switches is definately great and geting favoritised. Their switches are 3com baseline 2226 (3C16475C)
http://www.3com.com/products/en_US/detail.jsp?tab=support&pathtype=support&sku=3C16475A
I've only ever worked with cisco and netgear.
Anyone got sofware/links on these?
That link for the cisco switches is definately great and geting favoritised. Their switches are 3com baseline 2226 (3C16475C)
http://www.3com.com/products/en_US/detail.jsp?tab=support&pathtype=support&sku=3C16475A
I've only ever worked with cisco and netgear.
Anyone got sofware/links on these?
I think it would be a good idea to verify / audit all the network equipment first, check that 100 FDX capable equipment is enabled throughout, and that any old hubs or 10 FDX equipment are phased out.
In particular, pay attention to the CCTV cameras, and individually check that they are negotiating the best possible speed with the switches. If some CCTV cameras are old and can only run 10 HDX, then separate these out onto their own hub and uplink this hub to 100Mbs switch.
A Sniffer isn't going to help just yet - it's more a process of verifying network design and configuration at this stage (which is exactly what I would do if someone contracted me in to do this work).
In particular, pay attention to the CCTV cameras, and individually check that they are negotiating the best possible speed with the switches. If some CCTV cameras are old and can only run 10 HDX, then separate these out onto their own hub and uplink this hub to 100Mbs switch.
A Sniffer isn't going to help just yet - it's more a process of verifying network design and configuration at this stage (which is exactly what I would do if someone contracted me in to do this work).
It sounds to me like the client doesn't really know much about his network!! The first thing I'd be doing is charging him to give him a network diagram- any network that consists of more than ten or so network devices should really have one. It makes adding to the system much much easier in the future.
The Fluke Nettool could well be useful. It identifies duplex mismatching which lrmoore raised, and can analyse the traffic between points, both useful features in this case. In fact it should pretty much identify most problems, so once you have your diagram (yeah, you've guessed it, I'm a real stickler for that!!) testing various points on the network should isolate the problem. Don't know about the Etherscope, never used one.
In my view switches are switches. All managed switches have a telnet interface and some a web-interface. Once you're in that, it should be pretty straight-forward to find out what it is doing. The switches you have linked support the web interface in the latest firmware (available here http://www.3com.com/products/en_US/result.jsp?selected=6&sort=effdt&sku=3C16490&order=desc).
The Fluke Nettool could well be useful. It identifies duplex mismatching which lrmoore raised, and can analyse the traffic between points, both useful features in this case. In fact it should pretty much identify most problems, so once you have your diagram (yeah, you've guessed it, I'm a real stickler for that!!) testing various points on the network should isolate the problem. Don't know about the Etherscope, never used one.
In my view switches are switches. All managed switches have a telnet interface and some a web-interface. Once you're in that, it should be pretty straight-forward to find out what it is doing. The switches you have linked support the web interface in the latest firmware (available here http://www.3com.com/products/en_US/result.jsp?selected=6&sort=effdt&sku=3C16490&order=desc).
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
lol, oh yeah, forget that firmware download... looks like there was a PWR on the end of the product code which is for a different switch. My mistake, heh.
Don't need to be able to sniff on the switches though if you have the Fluke Nettool with you. I'd rely on that, you can use it without actually changing the network (which as stated, could change or remove the problem).
Don't need to be able to sniff on the switches though if you have the Fluke Nettool with you. I'd rely on that, you can use it without actually changing the network (which as stated, could change or remove the problem).
For an EXCELLENT white paper discussing "Fixing Ethernet and Fast Ethernet Link Problems" with the first section entitled, "Auto-negotiation - how it is supposed to work" go to:
http://www.flukenetworks.com/us/_Promotions/ESV/Autonegotiation+WP+Internal+Splash.htm
You will have to fill out a little survey and register, but it's well worth it.
At the risk of "posting marketing material" ... I'm going to paste in this quote only so that questioners and other Experts can sample the depth of material covered in this white paper. Once you read through this, you'll totally understand the duplex mismatch problem.
Paste:
Auto-negotiation – how it is supposed to work
Ethernet nodes connected together via a twisted pair cable negotiate their speed as well as duplex mode prior to establishing link. This process is called auto-negotiation and is performed by means of so-called Link Pulses. One of two types of link pulse signal is used to inform a “link partner” that a new station is on the line and that it is seeking to establish a connection.
These two versions of link pulse are Normal Link Pulse (NLP) and Fast Link Pulse (FLP).
The 10BASE-T link pulse (the NLP) consists simply of a half-wave pulse, transmitted eight times per second on the TX pair whenever data is not being sent. Data signals are from +1 to -1 volts.
When Fast Ethernet was developed, the standards body was very careful to ensure that backward compatibility be maintained. To accomplish this, they selected the simplest mechanism available for negotiating which physical signaling system and configuration to use. The FLP burst relies upon the presence or absence of NLPs in a certain order to communicate a binary word where each bit position represented a particular link ability. Fast Link Pulse bursts are implemented using NLPs, but as a burst of pulses that contains all of the information about the connecting device’s speed and duplex capabilities - this is the
FLP “link word,” which informs a link partner of what its abilities are. The presence of a “data” pulse in the FLP indicates a binary 1, while the absence indicates a binary 0. Data pulses appear between clocking pulses. There are 17 clocking pulses and the opportunity for 16 data pulses, so an FLP may have between 17 and 33 pulses.
-- end paste.
Hope this helps.
http://www.flukenetworks.com/us/_Promotions/ESV/Autonegotiation+WP+Internal+Splash.htm
You will have to fill out a little survey and register, but it's well worth it.
At the risk of "posting marketing material" ... I'm going to paste in this quote only so that questioners and other Experts can sample the depth of material covered in this white paper. Once you read through this, you'll totally understand the duplex mismatch problem.
Paste:
Auto-negotiation – how it is supposed to work
Ethernet nodes connected together via a twisted pair cable negotiate their speed as well as duplex mode prior to establishing link. This process is called auto-negotiation and is performed by means of so-called Link Pulses. One of two types of link pulse signal is used to inform a “link partner” that a new station is on the line and that it is seeking to establish a connection.
These two versions of link pulse are Normal Link Pulse (NLP) and Fast Link Pulse (FLP).
The 10BASE-T link pulse (the NLP) consists simply of a half-wave pulse, transmitted eight times per second on the TX pair whenever data is not being sent. Data signals are from +1 to -1 volts.
When Fast Ethernet was developed, the standards body was very careful to ensure that backward compatibility be maintained. To accomplish this, they selected the simplest mechanism available for negotiating which physical signaling system and configuration to use. The FLP burst relies upon the presence or absence of NLPs in a certain order to communicate a binary word where each bit position represented a particular link ability. Fast Link Pulse bursts are implemented using NLPs, but as a burst of pulses that contains all of the information about the connecting device’s speed and duplex capabilities - this is the
FLP “link word,” which informs a link partner of what its abilities are. The presence of a “data” pulse in the FLP indicates a binary 1, while the absence indicates a binary 0. Data pulses appear between clocking pulses. There are 17 clocking pulses and the opportunity for 16 data pulses, so an FLP may have between 17 and 33 pulses.
-- end paste.
Hope this helps.
I'm still waiting with anticipation to know whether I have won the tenner or lost it!!!
Purple, you've "lost it" ;D
lol, oh yeah... thanks for the clarification :)
Interested.
Definitely interested... is my tenner safe or not?!
Not!
I think the main thing you need to do is identify what percentage of bandwidth is camera and what is other things. Just because they've done 18 cameras elsewhere doesn't mean they can do 25 cameras here- it depends on the topology as purple said, and also on what else is using bandwidth. How much bandwidth is each camera generating? They should be able to tell you that. Or you can figure it out if you know image resolution and type of compression- there are tables out there that will give you bandwidth requirements.
25 cameras at 25 frames/sec could easily generate more than 100mb/sec traffic- and you can only expect a 100mb link to support about 60mb/sec without beginning to see a lot of congestion- indicated by dropped frames.