Some packets are drop when ping to hosts in a VLAN

The easy place to start is to double check your speed and duplex modes on each end. Overall make sure you dont have an issue with encapsulation, duplex. or port / int speed negotiation.

We have already checked. They are all access port, Gigabit interface, Full Duplex, 1000Mbps, auto-negotiated

are you using QoS? The main cause for drops typically comes from congestion? What does the utilization look like on the switch?

Thank you for your response. We're on local network. No QoS is applied. The switch utilization is about 10% or less. We have 3 VLANs on this switch but the problem only happens at 1 VLAN.

I know you said this was a 4000 series switch. There is a known issue, depending on SW version. See the notes from the link below. Depending on your version, an update may be needed...


http://www.cisco.com/en/US/ts/fn/100/fn14464.html

Thank you. We're using WS-C4506 which is not listed on the article you sent. The IOS version is "Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASE-M), Version 12.2(37)SG, RELEASE SOFTWARE (fc1)". Please suggest more work around.

I am typically not one to post links as opposed to more direct suggestions, however, in this case the potential number of reasons you are experiencing the congestion could be far too many for me to list. Might I suggest you take a look at the following article. I have used it in the past to help focus my efforts and ID a similar issue...

http://www.cisco.com/en/US/tech/tk389/tk689/technologies_tech_note09186a00801f9eb3.shtml

cheers!

Hi,

Can you post all of the output from the following command:

show int gi2/2

also what is g2/2 connected to?

Gi2/2 connect to IBM server running Redhat 5.0 OS
This is the output of #sh int gi2/2

  Hardware is Gigabit Ethernet Port, address is 0014.a846.4ab4 (bia 0014.a846.4ab4)
  Description: rvc-isms
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 16/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
  input flow-control is on, output flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 17:25:46
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 185817
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 169000 bits/sec, 78 packets/sec
  5 minute output rate 64130000 bits/sec, 6356 packets/sec
     2465682 packets input, 381257469 bytes, 0 no buffer
     Received 3193 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     28632249 packets output, 26685936142 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

Is this the only server which is experiencing problems?

do one thing, get the console connection to the switches, and check if the switches shows you VLAN MISSMATCH error msg, since i had d same problem , and finally found out that some cables from one vlan was connected to another vlan, and in console it was showing vlan mismatch in fastehternet 0/x , then disable or disconnect those mismatch cables , and check back,

cheers,
~RJ

Hi rochey2009,

That switch is hosting 3 VLANs. Only one VLAN get such problem. All servers connected to this particular VLAN experience this problem.

Hi Bawer,

Thank you very much! We'll try then feedback if any.

Regards,

Are all servers showing the output drops on their switch interfaces?

@Bawer: We used console but there is no error message, maybe this is not our case.

@rochey2009: Only servers in VLAN4 have output drops.

Have you tried using wireshark to see where the traffic is coming from?

We tried using ntop to monitor the switch traffic for a week, there's no strange traffic. The highest network load is 300Mbps. We're using Supervisor Engine II+ which has switching capacity about 32Gbps. The switch utilization is always less than 10%.

is it possible to change vlan 4 , only for testing purpose u may have downtime of mins, create another vlan say 20 , and put the server and a host in to that vlan , and check?

Thank you. We changed 1 port in VLAN 4 to other VLAN for 5 minutes. We did not see output drop counter increase. Then we added that port back to VLAN4, the output drop counter is increased significantly. Thanks for your suggestion.

Do you have any multicast or broadcast traffic?

NTOP reported this:

Total Received (ntop)      3,947,577,282
Total Packets Processed      3,947,577,280
Unicast      99.9%      3,944,595,314
Broadcast      0.1%      2,772,108
Multicast      0.0%      209,858

So i don't think we have too much broadcast or multicast.

Thanks for following up our problem.

Try and get a wireshark trace and post some of it here.

I'm don't know much about wireshark, please tell me your intention clearer. Thank you!

Download wireshark from http://www.wireshark.org and install it onto a PC.

Connect the PC to the switch.

Do the following on the switch

mon sess 1 source interface gi2/2 both
mon sess 1 dest interface <interface where wireshark is connected>

In wireshark select capture interfaces and select the interface which is connected to the switch.

Have a look at the output drops from the switch console and when you see it incrementing leave it go for a few minutes to make sure you capture the problem. Stop the capture.

Have a look in the trace for any traffic hitting the server which isn't destined for the server e.g. unknown unicast traffic will have a destination address which doesn't belong to the server. Also look for multicast or broadcast traffic.

When you've finished do

no mon sess 1

on the switch.

1) i m sure there is some loops in VLAN4, have u used CNA if not , following is the link , if u have cisco ID or u can sign up and download the software from cisco, :
http://www.cisco.com/en/US/products/ps5931/index.html
install it , and give ur IP range , that will automatically search for ur switches , its the best tool for monitoring loops or any other faulty in switches, that will show u if there r any loops or errors in ur switches ,

2) if it isnt difficult for u, change all vlan 4 in to any other vlan , say vlan 10,
this may also solve the issue,

Hi rochey2009,

We set up wireshark and found there're lot of broadcast traffic because of Windows NLB unicast mode. We will try to fix this first to see whether the situation is improved. Thanks for your suggestion.

Hi Bawer,

We also set up Cisco CNA but there's no loop found. Thanks for your consideration.

We will update if any.

Hi Everyone,

I'm glad to inform you that our problem is resolved. The cause of problem is Windows NLB cluster is set up in unicast mode. We changed it to multicast with IGMP support then we did not see any output drops any more.
Reference: http://www.cisco.com/en/US/tech/tk870/tk877/tk880/technologies_tech_note09186a008011b481.shtml
Thank you so much for your guideline, comment, suggestion.

Regards,
Chuong Tran

I've requested that this question be closed as follows:

Accepted answer: 0 points for rvc-it's comment http:/Q_27483542.html#37323739

for the following reason:

This help us to fix the problem.

We've made some helpful suggestions pointing to the cause of the problem but no points were awarded. My comment 37287082 suggested broadcasts or multicasts as part of the problem and my comment 37297841 showed how to find the broadcasts or multicasts.

Sorry, i do not know how to awarded point to your comment. I'm new user.

no problem. Thanks for the points.