Solved

Cisco 6509 - TroubleShooting Ports

Posted on 2011-09-07
14
1,518 Views
Last Modified: 2012-05-12
Dear Experts,

On our 6509, we have a x6516-GE-TX card with 2 ports that aren't working. I've reset the ports to their default values and matched the config to other 'good' ports. The only thing I've noticed that's different on the 'bad' ports are some 'pkts dropped from output queue' [OQD's] when I do a show int summary. Not sure if this is the problem or not.

The end client for this port is a standard PC with a gigabit nic, nothing special. Works fine if plugged into other switch ports.

Please let me know what other troubleshoot options I could perform.

TIA,

Mike

0
Comment
Question by:ENCL
  • 6
  • 5
  • 3
14 Comments
 
LVL 22

Expert Comment

by:eeRoot
ID: 36499801
Can you set the port so it is on a VLAN that has DHCP and then connect a known good laptop with a known good cable?  Do you get an IP address?  Also try the command "ping -T -L 10000 x.x.x.x" where x.x.x.x is the IP of the switch.  If the port is good, the pings will get through and the OQD count will not increase.  If the ports is bad, the the pings will fail and the OQD, or some other error counter, will increase.
0
 
LVL 2

Expert Comment

by:AbuAnaza
ID: 36501564
Are you plugging the laptop directly into the port or patching it through a data port on the floor/wall? sometimes it could be a bad termination on the floor/wall itself.

Moreover, Queue drops are usually associated to high traffic so you can check the interface utilization as well via the "sh int gi 0/1" command. May not be very likely tough :(

Did you clear counters on this interface and then try with a different cat6 cable? How about switchport speed/duplex settings, what is it currently set to?

Basically a layer 1 or 2 problem so have to look around this area ....
0
 
LVL 2

Expert Comment

by:AbuAnaza
ID: 36501622
From Cisco Website:

Output Queue Drops
Output drops are caused by a congested interface. For example, the traffic rate on the outgoing interface cannot accept all packets that should be sent out. The ultimate solution to resolve the problem is to increase the line speed. However, there are ways to prevent, decrease, or control output drops when you do not want to increase the line speed. You can prevent output drops only if output drops are a consequence of short bursts of data. If output drops are caused by a constant high-rate flow, you cannot prevent the drops. However, you can control them.

When packets are processed, they are sent to the output queue of the outgoing interface. Issue the show interfaces exec command to view the size of the queue, the current number of packets in the queue, and the number of drops. Based on the type of interface and the type of queueing configured, the number of output queue drops is not explicitly shown, because the output drops counter summarizes the output drops separately at the processing level and at the interrupt level:

router#show interfaces serial 0/0
  ...
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)  
  ...
router#show interfaces serial 0/0
...
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
...
However, it takes longer to process a packet than to sent the packet from the output queue to the wire. Therefore, it is highly unlikely that output queue drops (drops at processing level) can occur without drops at interrupt level. Output queue drops occur only if the interface is already congested at interrupt level, so that packets cannot be pulled out of the output queue before the queue becomes full. Therefore, output drops at processing level (output queue drops) and output drops at interrupt level always occur together, and there is practically no need to distinguish between these two counters.

Note:  However, there is one exception. If the output queue is constantly full and if no packets are sent out of the interface at all, you must check for a hardware failure on the interface.
0
 

Author Comment

by:ENCL
ID: 36503680
Hi Guys,

@eeRoot

Interface is on VLAN with DHCP, using known good notebook and known good cable. Notebook configured with static IP. OQD counter increases as I peform the ping, ping result from the notebook is a 'request timed out'.

@AbuAnaza

Laptop connected directly to switchport.
Using known good patch cable
I've tired changing the switchport speed between auto speed/duplex and forcing to 1000mbps & full duplex, no difference.
I've cleared the counters, connected using another 'good' patch cable and the OQD counter continues to increase.

Interface currently set to:

Duplex: Full
Speed: 1000
Vlan: Vlan1 [this has dhcp access], should get a valid IP, but doesn't.
Status: Connected

Starting to think the ports are fried :-(

Let me know if there's anything else to check.

Again, thanks for the help.

- Mike
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36507016
Can you at least get an IP address via DHCP or ping anything?  Does the laptop's MAC address show up in the switch's MAC address table?  If not, then it does sound like you have a bad port or two.
0
 
LVL 2

Expert Comment

by:AbuAnaza
ID: 36518136
ENCL,
If you are not getting an ip address then it means you have a layer 1 or layer 2 problem. What is the status of the port to which the laptop is connected i.e. output of "sh int gi x/x" command? Is the port UP/UP or does it say "not connected"? Did you try a shut/no shut on this port?

eeRoot mentioned a good point - you should check if the MAC of this laptop is showing up in the arp table of your switch. If not, this could very well be a hardware problem.

0
 
LVL 2

Expert Comment

by:AbuAnaza
ID: 36518140
If you are not getting an ip address then it means you have a layer 1 or layer 2 problem
err ... the above is only applicable to this partilcular scenario and its not a rule of thumb ... I feel better now :)
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 

Author Comment

by:ENCL
ID: 36531576
Hi Guys,

I cannot get an IP via DHCP, plus even using a static IP doesn't work.
Yes, the notebook's MAC address does show up in the switch's mac address table.

show int gi7/6, the port is up / up. I've tried the shut / no shut a few times.

If the switch is seeing the MAC address of the client, could this still be a config issue on the interface or a layer 1 / 2 issue?

Attached is the output of some of the show commands.

BTW, int gi7/5 works, this has been my baseline. If I plug the notebook into this port all is good.

Thanks again.

- Mike 6509-Port-Troubleshooting.txt
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36533160
it sounds like the port is bad.  Receiving, but not transmitting.
0
 
LVL 2

Expert Comment

by:AbuAnaza
ID: 36533991
output for:

sh run int gi 7/5
sh run int gi 7/6

Secondly, I think it is important at this point to open up your favourite packet capturing software in order to find out what is actually going on; that is if you are curious to find out. Else, if you have a SMARTNET with Cisco then you can open up a TAC and let them sort if out for you.

Commands to setup monitor:
monitor session 10 source interface Gi7/6
monitor session 10 destination interface Gi7/5 (port on which you will receive capture)

This should give you an idea of what is going on exactly.

Oh, and it would be ideal if you can clear the counters of this interface so you get an idea of the IN/OUT activities

clear counters gi 7/6

keep us posted ...
0
 

Author Comment

by:ENCL
ID: 36538533
Hi Guys,

Here's the output for the sh run int gi 7/5-6

6509#sho run int gi 7/5
Building configuration...

Current configuration : 66 bytes
!
interface GigabitEthernet7/5
 description OPEN
 switchport
end

6509#sho run int gi 7/6
Building configuration...

Current configuration : 80 bytes
!
interface GigabitEthernet7/6
 description OPEN - ### BAD ###
 switchport
end

@AbuAnaza

I setup the monitor session as you suggested and capture the traffic with WireShark. But I'm not trainied in network analysis to know what to look for here. If you're willing to take a look at the traffic outsite of this forum please let me know. I would love to know if there's anything of interest in the capture. But if you don't have the time I fully understand, as this is a little above and beyond typical tech forum help.

BTW, we don't have SmartNet on this switch, just to expensive :-(

Regards,

Mike

Mike
0
 
LVL 2

Accepted Solution

by:
AbuAnaza earned 500 total points
ID: 36540670
It seems this port is on the default vlan1, Do you have just 1 vlan on the switch?

Can you assign it to another vlan and see if you get the same result? We can create a test vlan and then see if we get a different response:


config t

vlan 87
 name test
 state active
 exit

int vlan 87
 ip address 192.168.254.5 255.255.255.0
 no shutdown
 exit

int gi 7/6
 switchport mode access
 switchport access vlan 87
 spanning-tree portfast
 exit

Then you have to assign a statis ip to a laptop (192.168.254.3/24) and see if you are able to ping the vlan interface that we created

ping 192.168.254.5

Once you have setup the monitor command on the switch and you connect the test laptop with the above config, you can start the packet capture. You can upload it here and we'll have a look at it, no issues

0
 

Author Comment

by:ENCL
ID: 36546154
Hi AbuAnza,

Port was on default vlan 1, we have one other vlan used as our guest network.

I followed your steps, attached are the results of the capture.

IP 192.168.254.3 was connected to gi 7/6
IP 192.168.254.4 was connected to gi 7/5

The ping to the test vlan [192.168.254.5] failed from the ntoebook on gi 7/6.

Regards,

Mike

9-15-11.txt
0
 

Author Closing Comment

by:ENCL
ID: 36996698
bad ports
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This is an article about my experiences with remote access to my clients (so that I may serve them) and eventually to my home office system via Radmin Remote Control. I have been using remote access for over 10 years and have been improving my metho…
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now