Solved

Massive in/out errors on 7500 series

Posted on 2004-08-13
16
616 Views
Last Modified: 2012-05-05
Hello,

  I have a 7500 series router with a ds3 card and the fastethernet, the ds3 is unchannelized t3 of bandwidth and has no IP address so only the ethernet has the IP.  I am using Solarwinds to monitor the snmp on both the 7500 and the switch it is connected to.  Here is the problem I experience choppy internet on that provider when connecting through it.  I assumed it was the provider but the snmp polls show me massive IO errors on the interface FastEthnet and not the Serial Connection (DS3), when looking at the port on the switch side it show little to no errors (about 50K on the 7500, and about 10 on the switch side).  So I am back to square one kinda.  THe only thing I can see is that almost ALL of the errors are transmit and none are recieve.  Any thoughts to further debug this?  I have already done the obvious like switching cables and ethernet ports, but the issue still persists....is there a debugging method on the switch or 7500 to see more detailed info about the failures?

Thank you.
0
Comment
Question by:jason987
16 Comments
 
LVL 2

Expert Comment

by:Fire_fly321
ID: 11793215
there is a way on all cisco devices to get further information.

Set the loging leval to debug. To change the level when using a snmp trap (sounds like you are)

config t
  logging trap debug
end


This will send all messages possible to your logging server. (be careful, it will fill up quickly)

If not using a snmp trap then just send the log to the internal buffer

config t
  logging buffered (give max size for buffer)
  logging buffered debug
end

then to look at the log issue this command

sho logging


From your discription, is sounds like you might be having a clocking problem.  I would check with your service provider to see who should supply the clocking.  (ie: set to line, internal ...)

0
 
LVL 79

Expert Comment

by:lrmoore
ID: 11793223
Do you have CEF enabled? dCEF?

Here's another troubleshooting guide:
http://www.cisco.com/univercd/cc/td/doc/cisintwk/itg_v1/tr1904.htm#xtocid6
0
 
LVL 5

Author Comment

by:jason987
ID: 11793304
Here is the two interfaces configs,

FireFly: I am getting all the logging I need I was looking more for a command in debug that can show me more information on why transmits are failing

lrmoore:
See below no cef unless it is in global as a default, I also noticed the line proto on the fastethernet is flopping up/down occasionaly but I think this may be because of the ds3 connect since it is unnumbered in the serial to fa3/0/0? Am I way off here?  On the switch side I don't see the interface going up/down is why I am thinking about the ds3.


!
interface FastEthernet3/0/0
 ip address ____________ 255.255.255.0
 no ip route-cache distributed
 full-duplex
!
interface Serial3/1/0
 bandwidth 12632
 no ip address
 encapsulation frame-relay IETF
 no ip route-cache distributed
 keepalive 8
 scramble
 framing c-bit
 cablelength 50
 dsu bandwidth 12632
 frame-relay lmi-type ansi
!
interface Serial3/1/0.500 point-to-point
 description wcomw0l03954:to UUNET:T3
 bandwidth 12632
 ip unnumbered FastEthernet3/0/0
 no ip redirects
 no ip proxy-arp
 frame-relay interface-dlci 500 IETF  
!
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 11793499
If you look at the interface with Show int serial 3/1/0, what do you see as the MTU?
I would suggest setting the MTU to 1500 to match the Ethernet side. It may default to 4450 or something..
0
 
LVL 5

Author Comment

by:jason987
ID: 11793545
Hmm that was off, so when the serial gets an xunit of 4470 it will chop it and cause issues or is this just a 'should be this way' thing?

Any thoughts on something to run to make sure the serial and eth are playing nicely?


ardware is cyBus PODS3 Serial
  MTU 4470 bytes, BW 12632 Kbit, DLY 200 usec,
     reliability 255/255, txload 3/255, rxload 3/255
0
 
LVL 5

Author Comment

by:jason987
ID: 11793559
on more thought, if I change the MTU on the serial couldn't that cause a mismatch upstream?
0
 
LVL 5

Author Comment

by:jason987
ID: 11793570
arg....to add on above, ...or is that unit only aligned with the ethernet and the upstream is based on the ds3 protocol?
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 11793650
Potentially, yes, you could get a mismatch. Most carriers will tell you that it is not a problem using the large MTU on the serial interface should not cause any issues on high-speed frame-relay.
Why not try putting a real ip address on the serial interface instead of using IP unnumbered? MCI will set you up if you insist on a /30 address fro your serial interface. I NEVER use ipunnumbered on a public interface.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 5

Author Comment

by:jason987
ID: 11793753
So would any of those cause Xmit only errors and the interface 3/0/0 to flop line proto updown, which I see as the main issue now, especiall since it does look to die and come back but on the catalyst side I don't see that interface line proto flop...very confusing
0
 
LVL 5

Author Comment

by:jason987
ID: 11793804
Here is the output of the interfaces, shows after a short uptime but you'll see the failures already starting, would the ethernet show errors that are cause by the ds3 side or only on the ethernet side?


Serial3/1/0 is up, line protocol is up
  Hardware is cyBus PODS3 Serial
  MTU 4470 bytes, BW 12632 Kbit, DLY 200 usec,
     reliability 255/255, txload 3/255, rxload 3/255
  Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
  Keepalive set (8 sec)
  LMI enq sent  63, LMI stat recvd 63, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 0  LMI type is ANSI Annex D  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/256, broadcasts sent/dropped 11/0, interface broadcasts 0
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:08:33
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 0/75, 0 drops
  5 minute input rate 198000 bits/sec, 281 packets/sec
  5 minute output rate 160000 bits/sec, 245 packets/sec
     143579 packets input, 12653685 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     117156 packets output, 9792498 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     1 carrier transitions
     LC=up  CA=down  TM=down LB=down TA=down LA=down


FastEthernet3/0/0 is up, line protocol is up
  Hardware is cyBus FastEthernet Interface, address is
  Internet address is
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 0/75, 0 drops
  5 minute input rate 198000 bits/sec, 240 packets/sec
  5 minute output rate 188000 bits/sec, 274 packets/sec
     110159 packets input, 11803942 bytes, 0 no buffer
     Received 3309 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     132800 packets output, 11201244 bytes, 0 underruns
     4 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     5 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
0
 
LVL 79

Accepted Solution

by:
lrmoore earned 250 total points
ID: 11793987
Agree, very confusing. 5 lost carrier's already. Sounds like you have potential
1) IOS bug
2) Ethernet interface failing
Hope you have SmartNet maintenance!
Any system error messages in your log?
How about on your Catalyst? Which Catalyst do you have?
Is the interface hard set to 100/full, or autonegotiate? Since you have the duplex set to full on the router interface, you MUST also set the switchport to 100/full.

0
 
LVL 5

Author Comment

by:jason987
ID: 11795411
1.  Changed IOS's as a first option a week ago.

2.  I am not in big favor of this because the other side of the connection sees nothing wrong.  Also switch switches and still saw nothign wrong.

Got smart net but I usually get better ideas here! ;)

No errors except the ethernet flopping.

have 3 catalysts tried it on all of them same result.

I always match speeds don't use anything nto hard set to 100/full

I guess I will try to swap chasis/ethernet cars see if that changes behaviour.

0
 
LVL 79

Expert Comment

by:lrmoore
ID: 11796048
> always match speeds don't use anything nto hard set to 100/full
Does this mean that your switchport is set to auto?

Since you have your router interface hard set to full-duplex, you need to also set the switchport to full duplex, or else set the router interface to auto too. You cannot have one end set to auto and one end set to full-duplex, or you will have some of these unexplainable issues. The port can be flapping because it can't negotiate the duplex..
Both hard-set or both auto, no in-between.
0
 
LVL 5

Author Comment

by:jason987
ID: 11796156
That is what I meant by "always match speeds don't use anything nto hard set to 100/full "  I never use auto or anyother combination.

At this point I wish it was mismatched because I would know the logical next step ;)
0
 
LVL 4

Expert Comment

by:celsmk
ID: 11798224
Looked at your interface statistics and seems to be pretty normal. The few fast-ethernet lost carrier errors could be attributed to cable swaps you've done.
Can not understand why you are detecting massive fast-ethernet IO errors with so few error counting shown in "show interface".

Noticed also that you are using Frame Relay on your serial interface and that leads me to blame congestion in your provider frame-relay network.
Do you see non-zero FECN and BECN counters on your 7500? If so, you may be trying to be oversubscribing contracted CIR and have your data being dropped anywhere in frame-relay network and suffer from choppy performance.
0
 
LVL 5

Author Comment

by:jason987
ID: 11806534
Thanks for the help, ends up I think the 7500 is dying somehow I have noticed after restarts the problem resolves itself then abruptly reappears.
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

PRTG Network Monitor lets you monitor your bandwidth usage, so you know who is using up your bandwidth, and what they're using it for.
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now