Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1583
  • Last Modified:

Cisco 6509 - TroubleShooting Ports

Dear Experts,

On our 6509, we have a x6516-GE-TX card with 2 ports that aren't working. I've reset the ports to their default values and matched the config to other 'good' ports. The only thing I've noticed that's different on the 'bad' ports are some 'pkts dropped from output queue' [OQD's] when I do a show int summary. Not sure if this is the problem or not.

The end client for this port is a standard PC with a gigabit nic, nothing special. Works fine if plugged into other switch ports.

Please let me know what other troubleshoot options I could perform.

TIA,

Mike

0
ENCL
Asked:
ENCL
  • 6
  • 5
  • 3
1 Solution
 
eeRootCommented:
Can you set the port so it is on a VLAN that has DHCP and then connect a known good laptop with a known good cable?  Do you get an IP address?  Also try the command "ping -T -L 10000 x.x.x.x" where x.x.x.x is the IP of the switch.  If the port is good, the pings will get through and the OQD count will not increase.  If the ports is bad, the the pings will fail and the OQD, or some other error counter, will increase.
0
 
AbuAnazaCommented:
Are you plugging the laptop directly into the port or patching it through a data port on the floor/wall? sometimes it could be a bad termination on the floor/wall itself.

Moreover, Queue drops are usually associated to high traffic so you can check the interface utilization as well via the "sh int gi 0/1" command. May not be very likely tough :(

Did you clear counters on this interface and then try with a different cat6 cable? How about switchport speed/duplex settings, what is it currently set to?

Basically a layer 1 or 2 problem so have to look around this area ....
0
 
AbuAnazaCommented:
From Cisco Website:

Output Queue Drops
Output drops are caused by a congested interface. For example, the traffic rate on the outgoing interface cannot accept all packets that should be sent out. The ultimate solution to resolve the problem is to increase the line speed. However, there are ways to prevent, decrease, or control output drops when you do not want to increase the line speed. You can prevent output drops only if output drops are a consequence of short bursts of data. If output drops are caused by a constant high-rate flow, you cannot prevent the drops. However, you can control them.

When packets are processed, they are sent to the output queue of the outgoing interface. Issue the show interfaces exec command to view the size of the queue, the current number of packets in the queue, and the number of drops. Based on the type of interface and the type of queueing configured, the number of output queue drops is not explicitly shown, because the output drops counter summarizes the output drops separately at the processing level and at the interrupt level:

router#show interfaces serial 0/0
  ...
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)  
  ...
router#show interfaces serial 0/0
...
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
...
However, it takes longer to process a packet than to sent the packet from the output queue to the wire. Therefore, it is highly unlikely that output queue drops (drops at processing level) can occur without drops at interrupt level. Output queue drops occur only if the interface is already congested at interrupt level, so that packets cannot be pulled out of the output queue before the queue becomes full. Therefore, output drops at processing level (output queue drops) and output drops at interrupt level always occur together, and there is practically no need to distinguish between these two counters.

Note:  However, there is one exception. If the output queue is constantly full and if no packets are sent out of the interface at all, you must check for a hardware failure on the interface.
0
Threat Trends for MSPs to Watch

See the findings.
Despite its humble beginnings, phishing has come a long way since those first crudely constructed emails. Today, phishing sites can appear and disappear in the length of a coffee break, and it takes more than a little know-how to keep your clients secure.

 
ENCLAuthor Commented:
Hi Guys,

@eeRoot

Interface is on VLAN with DHCP, using known good notebook and known good cable. Notebook configured with static IP. OQD counter increases as I peform the ping, ping result from the notebook is a 'request timed out'.

@AbuAnaza

Laptop connected directly to switchport.
Using known good patch cable
I've tired changing the switchport speed between auto speed/duplex and forcing to 1000mbps & full duplex, no difference.
I've cleared the counters, connected using another 'good' patch cable and the OQD counter continues to increase.

Interface currently set to:

Duplex: Full
Speed: 1000
Vlan: Vlan1 [this has dhcp access], should get a valid IP, but doesn't.
Status: Connected

Starting to think the ports are fried :-(

Let me know if there's anything else to check.

Again, thanks for the help.

- Mike
0
 
eeRootCommented:
Can you at least get an IP address via DHCP or ping anything?  Does the laptop's MAC address show up in the switch's MAC address table?  If not, then it does sound like you have a bad port or two.
0
 
AbuAnazaCommented:
ENCL,
If you are not getting an ip address then it means you have a layer 1 or layer 2 problem. What is the status of the port to which the laptop is connected i.e. output of "sh int gi x/x" command? Is the port UP/UP or does it say "not connected"? Did you try a shut/no shut on this port?

eeRoot mentioned a good point - you should check if the MAC of this laptop is showing up in the arp table of your switch. If not, this could very well be a hardware problem.

0
 
AbuAnazaCommented:
If you are not getting an ip address then it means you have a layer 1 or layer 2 problem
err ... the above is only applicable to this partilcular scenario and its not a rule of thumb ... I feel better now :)
0
 
ENCLAuthor Commented:
Hi Guys,

I cannot get an IP via DHCP, plus even using a static IP doesn't work.
Yes, the notebook's MAC address does show up in the switch's mac address table.

show int gi7/6, the port is up / up. I've tried the shut / no shut a few times.

If the switch is seeing the MAC address of the client, could this still be a config issue on the interface or a layer 1 / 2 issue?

Attached is the output of some of the show commands.

BTW, int gi7/5 works, this has been my baseline. If I plug the notebook into this port all is good.

Thanks again.

- Mike 6509-Port-Troubleshooting.txt
0
 
eeRootCommented:
it sounds like the port is bad.  Receiving, but not transmitting.
0
 
AbuAnazaCommented:
output for:

sh run int gi 7/5
sh run int gi 7/6

Secondly, I think it is important at this point to open up your favourite packet capturing software in order to find out what is actually going on; that is if you are curious to find out. Else, if you have a SMARTNET with Cisco then you can open up a TAC and let them sort if out for you.

Commands to setup monitor:
monitor session 10 source interface Gi7/6
monitor session 10 destination interface Gi7/5 (port on which you will receive capture)

This should give you an idea of what is going on exactly.

Oh, and it would be ideal if you can clear the counters of this interface so you get an idea of the IN/OUT activities

clear counters gi 7/6

keep us posted ...
0
 
ENCLAuthor Commented:
Hi Guys,

Here's the output for the sh run int gi 7/5-6

6509#sho run int gi 7/5
Building configuration...

Current configuration : 66 bytes
!
interface GigabitEthernet7/5
 description OPEN
 switchport
end

6509#sho run int gi 7/6
Building configuration...

Current configuration : 80 bytes
!
interface GigabitEthernet7/6
 description OPEN - ### BAD ###
 switchport
end

@AbuAnaza

I setup the monitor session as you suggested and capture the traffic with WireShark. But I'm not trainied in network analysis to know what to look for here. If you're willing to take a look at the traffic outsite of this forum please let me know. I would love to know if there's anything of interest in the capture. But if you don't have the time I fully understand, as this is a little above and beyond typical tech forum help.

BTW, we don't have SmartNet on this switch, just to expensive :-(

Regards,

Mike

Mike
0
 
AbuAnazaCommented:
It seems this port is on the default vlan1, Do you have just 1 vlan on the switch?

Can you assign it to another vlan and see if you get the same result? We can create a test vlan and then see if we get a different response:


config t

vlan 87
 name test
 state active
 exit

int vlan 87
 ip address 192.168.254.5 255.255.255.0
 no shutdown
 exit

int gi 7/6
 switchport mode access
 switchport access vlan 87
 spanning-tree portfast
 exit

Then you have to assign a statis ip to a laptop (192.168.254.3/24) and see if you are able to ping the vlan interface that we created

ping 192.168.254.5

Once you have setup the monitor command on the switch and you connect the test laptop with the above config, you can start the packet capture. You can upload it here and we'll have a look at it, no issues

0
 
ENCLAuthor Commented:
Hi AbuAnza,

Port was on default vlan 1, we have one other vlan used as our guest network.

I followed your steps, attached are the results of the capture.

IP 192.168.254.3 was connected to gi 7/6
IP 192.168.254.4 was connected to gi 7/5

The ping to the test vlan [192.168.254.5] failed from the ntoebook on gi 7/6.

Regards,

Mike

9-15-11.txt
0
 
ENCLAuthor Commented:
bad ports
0

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 6
  • 5
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now