We help IT Professionals succeed at work.

How do I best troubleshoot packet loss problems on Cisco Catalyst 65xx

DanRaposo
DanRaposo asked
on
I am using Solar winds Orion to monitor some of the switches in our infrastructure and I am seeing packet loss nearly every morning at the same time.  What are the troubleshooting steps that I should take in order to solve where this packet loss is coming from?  I am not overly experienced with the Solar Winds monitoring tools, but I can find my way around the IOS on the switch itself.

Comment
Watch Question

The first step would be to log in to the switch in question and take a look at the output of sh logs. Ideally there will be something telling in the buffer. Next, I'd issue a "clear counters" so that I have a fresh slate with regard to interface errors and the like. After the packet loss occurs check the log again and then scroll through sh int. Specifically, you'd be looking for the following under each interface:

input errors, CRC errors, frame errors, overrun and ignore errors. Theyre all on the same line:

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

A bit further down also take not of any output, collision errors or interface resets:
   
    0 output errors, 0 collisions, 0 interface resets

If you find nothing there (Generally you will) if at all possible log into the switch while the loss is occuring. Flip through sh int and take a look at the rx/tx loads. Are any of them near 255? If so perhaps its the result of an oversubscribed port.
Top Expert 2010
Commented:
Depending on what you have installed in the 6500, troubleshooting can be quite complicated:

Know thy switch:
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd803e508c.html

http://www.cisco.com/en/US/prod/collateral/routers/ps368/product_data_sheet0900aecd8057f3b6.html

troubleshoot thy switch:
https://www.cisco.com/en/US/products/hw/switches/ps708/prod_tech_notes_list.html
https://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a0080094713.shtml
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801751d7.shtml

You could be running into cef and dcef issues; Issues where packets could software or hardware processed based on if they they were punted or needed gleaning. Are you maxing out the backplane, are you having issues with the Layer 2 engine or layer 3 engine (PFC card), maybe issues that are related to the MSFC (If you have one installed), depending on what supervisor you are using, NAT, PAT, GRE, amongst other protocols are handled in software and not in hardware, where as others are handled in hardware. As already stated, you could be saturated your links, or you also could be having issues with queuing delay or the CPU is processing other priority traffic. Honestly, it could be anything as you are dealing with a complex multilayer device.

"How do I best troubleshoot packet loss problems on Cisco Catalyst 65xx"

What has led you to think the 6500 is causing your issue. Do you have data to backup this claim, maybe if you can share the data we could better assist you. Just because solarwinds shows packet loss does not mean there is an issue. What protocol(s) are you using to gather this data? If you could  explain your environment more in detail will help us help you.What applications are having issues during the time of the packet loss (If any), what protocols are being used that are having issues (You did not state there was an issue, only that solarwinds detects packet loss).

Thanks
Billy
Hi,DanRaposo

I think you need the IP Accounting and IP SLA help. You use  NetBrain Workstation to monitor the devices, then use IP Accounting and IP SLA to drill down into the issue details. You can get more information about these features from:
http://www.netbraintech.com/netbrain-product/operator-edition/operator-edition.php

Hope it may help.
Susan Samder