Cisco 6509 - TroubleShooting Ports
Dear Experts,
On our 6509, we have a x6516-GE-TX card with 2 ports that aren't working. I've reset the ports to their default values and matched the config to other 'good' ports. The only thing I've noticed that's different on the 'bad' ports are some 'pkts dropped from output queue' [OQD's] when I do a show int summary. Not sure if this is the problem or not.
The end client for this port is a standard PC with a gigabit nic, nothing special. Works fine if plugged into other switch ports.
Please let me know what other troubleshoot options I could perform.
TIA,
Mike
On our 6509, we have a x6516-GE-TX card with 2 ports that aren't working. I've reset the ports to their default values and matched the config to other 'good' ports. The only thing I've noticed that's different on the 'bad' ports are some 'pkts dropped from output queue' [OQD's] when I do a show int summary. Not sure if this is the problem or not.
The end client for this port is a standard PC with a gigabit nic, nothing special. Works fine if plugged into other switch ports.
Please let me know what other troubleshoot options I could perform.
TIA,
Mike
Can you set the port so it is on a VLAN that has DHCP and then connect a known good laptop with a known good cable? Do you get an IP address? Also try the command "ping -T -L 10000 x.x.x.x" where x.x.x.x is the IP of the switch. If the port is good, the pings will get through and the OQD count will not increase. If the ports is bad, the the pings will fail and the OQD, or some other error counter, will increase.
Are you plugging the laptop directly into the port or patching it through a data port on the floor/wall? sometimes it could be a bad termination on the floor/wall itself.
Moreover, Queue drops are usually associated to high traffic so you can check the interface utilization as well via the "sh int gi 0/1" command. May not be very likely tough :(
Did you clear counters on this interface and then try with a different cat6 cable? How about switchport speed/duplex settings, what is it currently set to?
Basically a layer 1 or 2 problem so have to look around this area ....
Moreover, Queue drops are usually associated to high traffic so you can check the interface utilization as well via the "sh int gi 0/1" command. May not be very likely tough :(
Did you clear counters on this interface and then try with a different cat6 cable? How about switchport speed/duplex settings, what is it currently set to?
Basically a layer 1 or 2 problem so have to look around this area ....
From Cisco Website:
Output Queue Drops
Output drops are caused by a congested interface. For example, the traffic rate on the outgoing interface cannot accept all packets that should be sent out. The ultimate solution to resolve the problem is to increase the line speed. However, there are ways to prevent, decrease, or control output drops when you do not want to increase the line speed. You can prevent output drops only if output drops are a consequence of short bursts of data. If output drops are caused by a constant high-rate flow, you cannot prevent the drops. However, you can control them.
When packets are processed, they are sent to the output queue of the outgoing interface. Issue the show interfaces exec command to view the size of the queue, the current number of packets in the queue, and the number of drops. Based on the type of interface and the type of queueing configured, the number of output queue drops is not explicitly shown, because the output drops counter summarizes the output drops separately at the processing level and at the interrupt level:
router#show interfaces serial 0/0
...
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
...
router#show interfaces serial 0/0
...
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
...
However, it takes longer to process a packet than to sent the packet from the output queue to the wire. Therefore, it is highly unlikely that output queue drops (drops at processing level) can occur without drops at interrupt level. Output queue drops occur only if the interface is already congested at interrupt level, so that packets cannot be pulled out of the output queue before the queue becomes full. Therefore, output drops at processing level (output queue drops) and output drops at interrupt level always occur together, and there is practically no need to distinguish between these two counters.
Note: However, there is one exception. If the output queue is constantly full and if no packets are sent out of the interface at all, you must check for a hardware failure on the interface.
Output Queue Drops
Output drops are caused by a congested interface. For example, the traffic rate on the outgoing interface cannot accept all packets that should be sent out. The ultimate solution to resolve the problem is to increase the line speed. However, there are ways to prevent, decrease, or control output drops when you do not want to increase the line speed. You can prevent output drops only if output drops are a consequence of short bursts of data. If output drops are caused by a constant high-rate flow, you cannot prevent the drops. However, you can control them.
When packets are processed, they are sent to the output queue of the outgoing interface. Issue the show interfaces exec command to view the size of the queue, the current number of packets in the queue, and the number of drops. Based on the type of interface and the type of queueing configured, the number of output queue drops is not explicitly shown, because the output drops counter summarizes the output drops separately at the processing level and at the interrupt level:
router#show interfaces serial 0/0
...
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
...
router#show interfaces serial 0/0
...
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
...
However, it takes longer to process a packet than to sent the packet from the output queue to the wire. Therefore, it is highly unlikely that output queue drops (drops at processing level) can occur without drops at interrupt level. Output queue drops occur only if the interface is already congested at interrupt level, so that packets cannot be pulled out of the output queue before the queue becomes full. Therefore, output drops at processing level (output queue drops) and output drops at interrupt level always occur together, and there is practically no need to distinguish between these two counters.
Note: However, there is one exception. If the output queue is constantly full and if no packets are sent out of the interface at all, you must check for a hardware failure on the interface.
ASKER
Hi Guys,
@eeRoot
Interface is on VLAN with DHCP, using known good notebook and known good cable. Notebook configured with static IP. OQD counter increases as I peform the ping, ping result from the notebook is a 'request timed out'.
@AbuAnaza
Interface currently set to:
Duplex: Full
Speed: 1000
Vlan: Vlan1 [this has dhcp access], should get a valid IP, but doesn't.
Status: Connected
Starting to think the ports are fried :-(
Let me know if there's anything else to check.
Again, thanks for the help.
- Mike
@eeRoot
Interface is on VLAN with DHCP, using known good notebook and known good cable. Notebook configured with static IP. OQD counter increases as I peform the ping, ping result from the notebook is a 'request timed out'.
@AbuAnaza
Laptop connected directly to switchport.
Using known good patch cable
I've tired changing the switchport speed between auto speed/duplex and forcing to 1000mbps & full duplex, no difference.
I've cleared the counters, connected using another 'good' patch cable and the OQD counter continues to increase.
Interface currently set to:
Duplex: Full
Speed: 1000
Vlan: Vlan1 [this has dhcp access], should get a valid IP, but doesn't.
Status: Connected
Starting to think the ports are fried :-(
Let me know if there's anything else to check.
Again, thanks for the help.
- Mike
Can you at least get an IP address via DHCP or ping anything? Does the laptop's MAC address show up in the switch's MAC address table? If not, then it does sound like you have a bad port or two.
ENCL,
If you are not getting an ip address then it means you have a layer 1 or layer 2 problem. What is the status of the port to which the laptop is connected i.e. output of "sh int gi x/x" command? Is the port UP/UP or does it say "not connected"? Did you try a shut/no shut on this port?
eeRoot mentioned a good point - you should check if the MAC of this laptop is showing up in the arp table of your switch. If not, this could very well be a hardware problem.
If you are not getting an ip address then it means you have a layer 1 or layer 2 problem. What is the status of the port to which the laptop is connected i.e. output of "sh int gi x/x" command? Is the port UP/UP or does it say "not connected"? Did you try a shut/no shut on this port?
eeRoot mentioned a good point - you should check if the MAC of this laptop is showing up in the arp table of your switch. If not, this could very well be a hardware problem.
If you are not getting an ip address then it means you have a layer 1 or layer 2 problemerr ... the above is only applicable to this partilcular scenario and its not a rule of thumb ... I feel better now :)
ASKER
Hi Guys,
I cannot get an IP via DHCP, plus even using a static IP doesn't work.
Yes, the notebook's MAC address does show up in the switch's mac address table.
show int gi7/6, the port is up / up. I've tried the shut / no shut a few times.
If the switch is seeing the MAC address of the client, could this still be a config issue on the interface or a layer 1 / 2 issue?
Attached is the output of some of the show commands.
BTW, int gi7/5 works, this has been my baseline. If I plug the notebook into this port all is good.
Thanks again.
- Mike 6509-Port-Troubleshooting.txt
I cannot get an IP via DHCP, plus even using a static IP doesn't work.
Yes, the notebook's MAC address does show up in the switch's mac address table.
show int gi7/6, the port is up / up. I've tried the shut / no shut a few times.
If the switch is seeing the MAC address of the client, could this still be a config issue on the interface or a layer 1 / 2 issue?
Attached is the output of some of the show commands.
BTW, int gi7/5 works, this has been my baseline. If I plug the notebook into this port all is good.
Thanks again.
- Mike 6509-Port-Troubleshooting.txt
it sounds like the port is bad. Receiving, but not transmitting.
output for:
sh run int gi 7/5
sh run int gi 7/6
Secondly, I think it is important at this point to open up your favourite packet capturing software in order to find out what is actually going on; that is if you are curious to find out. Else, if you have a SMARTNET with Cisco then you can open up a TAC and let them sort if out for you.
Commands to setup monitor:
monitor session 10 source interface Gi7/6
monitor session 10 destination interface Gi7/5 (port on which you will receive capture)
This should give you an idea of what is going on exactly.
Oh, and it would be ideal if you can clear the counters of this interface so you get an idea of the IN/OUT activities
clear counters gi 7/6
keep us posted ...
sh run int gi 7/5
sh run int gi 7/6
Secondly, I think it is important at this point to open up your favourite packet capturing software in order to find out what is actually going on; that is if you are curious to find out. Else, if you have a SMARTNET with Cisco then you can open up a TAC and let them sort if out for you.
Commands to setup monitor:
monitor session 10 source interface Gi7/6
monitor session 10 destination interface Gi7/5 (port on which you will receive capture)
This should give you an idea of what is going on exactly.
Oh, and it would be ideal if you can clear the counters of this interface so you get an idea of the IN/OUT activities
clear counters gi 7/6
keep us posted ...
ASKER
Hi Guys,
Here's the output for the sh run int gi 7/5-6
6509#sho run int gi 7/5
Building configuration...
Current configuration : 66 bytes
!
interface GigabitEthernet7/5
description OPEN
switchport
end
6509#sho run int gi 7/6
Building configuration...
Current configuration : 80 bytes
!
interface GigabitEthernet7/6
description OPEN - ### BAD ###
switchport
end
@AbuAnaza
I setup the monitor session as you suggested and capture the traffic with WireShark. But I'm not trainied in network analysis to know what to look for here. If you're willing to take a look at the traffic outsite of this forum please let me know. I would love to know if there's anything of interest in the capture. But if you don't have the time I fully understand, as this is a little above and beyond typical tech forum help.
BTW, we don't have SmartNet on this switch, just to expensive :-(
Regards,
Mike
Mike
Here's the output for the sh run int gi 7/5-6
6509#sho run int gi 7/5
Building configuration...
Current configuration : 66 bytes
!
interface GigabitEthernet7/5
description OPEN
switchport
end
6509#sho run int gi 7/6
Building configuration...
Current configuration : 80 bytes
!
interface GigabitEthernet7/6
description OPEN - ### BAD ###
switchport
end
@AbuAnaza
I setup the monitor session as you suggested and capture the traffic with WireShark. But I'm not trainied in network analysis to know what to look for here. If you're willing to take a look at the traffic outsite of this forum please let me know. I would love to know if there's anything of interest in the capture. But if you don't have the time I fully understand, as this is a little above and beyond typical tech forum help.
BTW, we don't have SmartNet on this switch, just to expensive :-(
Regards,
Mike
Mike
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi AbuAnza,
Port was on default vlan 1, we have one other vlan used as our guest network.
I followed your steps, attached are the results of the capture.
IP 192.168.254.3 was connected to gi 7/6
IP 192.168.254.4 was connected to gi 7/5
The ping to the test vlan [192.168.254.5] failed from the ntoebook on gi 7/6.
Regards,
Mike
9-15-11.txt
Port was on default vlan 1, we have one other vlan used as our guest network.
I followed your steps, attached are the results of the capture.
IP 192.168.254.3 was connected to gi 7/6
IP 192.168.254.4 was connected to gi 7/5
The ping to the test vlan [192.168.254.5] failed from the ntoebook on gi 7/6.
Regards,
Mike
9-15-11.txt
ASKER
bad ports