Solved

Frame-Relay circuit has loss and pinging local interface has lost packets

Posted on 2010-08-17
12
969 Views
Last Modified: 2013-12-12
When I ping local interface ip address xxx.xx.1.1 the ping appears to cross the frame relay network to neighbor xxx.xx.1.2 and then come back instead of replying directly from Hssi1/0.1.

We are also experiencing packet loss across the frame-relay network but I am curious if the ping and traceroute should respond directly from the Hssi1/0.1 or is it normal for frame-relay to send the ping across the cloud and then back?

If this is normal then only the frame-relay circuit has a problem but if not then maybe we have two problems.

I also see the ping going across the frame-relay first when I trace xxx.xx.1.2 on the remote router. (xxx.xx.1.1 responds first and then xxx.xx.1.2 responds)

We have packet loss in both directions across the frame-relay cloud.

I was very surprised to see packet loss pinging a local interface ip address!

172.25.1.1#sh ip int br | ex unassi
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         11.192.1.1      YES NVRAM  up                    up
GigabitEthernet0/1         xxx.xx.192.1    YES NVRAM  up                    up
Hssi1/0.1                  xxx.xx.1.1      YES NVRAM  up                    up
Hssi1/0.2                  xxx.xx.1.5      YES NVRAM  up                    up
Hssi1/0.3                  xxx.xx.1.9      YES NVRAM  up                    up

xxx.xx.1.1#traceroute xxx.xx.1.1

Type escape sequence to abort.
Tracing the route to xxx.xx.1.1

  1 xxx.xx.1.2 8 msec 0 msec 8 msec
  2 xxx.xx.1.1 12 msec 16 msec *



xxx.xx.1.1#ping
Protocol [ip]:
Target IP address: xxx.xx.1.1
Repeat count [5]: 200
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to xxx.xx.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!.!..!!!!!!!!!!!!!!!
!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!..!!!!!!.!!!.!!.!!!...!!.!!
Success rate is 91 percent (183/200), round-trip min/avg/max = 4/9/28 ms

xxx.xx.1.1#ping
Protocol [ip]:
Target IP address: xxx.xx.1.2
Repeat count [5]: 200
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to xxx.xx.1.2, timeout is 2 seconds:
!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!....!!!!
!!!!!!!!!!!.!!.!!!!!.!!..!!!!!!!!!!!!!!!..!!!!!.!!!!!!!!!!!!!!!!!!!!!!
!!!!!!.!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!
Success rate is 90 percent (181/200), round-trip min/avg/max = 1/6/20 ms





xxx.xx.1.1# sh ip ospf neigh

Neighbor ID     Pri   State           Dead Time   Address         Interface
xxx.xx.192.2      1   FULL/DR         00:00:30    xxx.xx.192.2    GigabitEthernet0/1
xxx.xx.208.1      0   FULL/  -        00:00:36    xxx.xx.1.10     Hssi1/0.3
xxx.xx.128.1      0   FULL/  -        00:00:34    xxx.xx.1.6      Hssi1/0.2
xxx.xx.136.1      0   FULL/  -        00:00:32    xxx.xx.1.2      Hssi1/0.1



xxx.xx.1.1#sh run | beg interface Hssi1/0.1
interface Hssi1/0.1 point-to-point
 ip address xxx.xx.1.1 255.255.255.252
 ip ospf cost 200
 frame-relay interface-dlci 19
!
interface Hssi1/0.2 point-to-point
 ip address xxx.xx.1.5 255.255.255.252
 ip ospf cost 200
 frame-relay interface-dlci 20
!
interface Hssi1/0.3 point-to-point
 ip address xxx.xx.1.9 255.255.255.252
 ip ospf cost 200
 frame-relay interface-dlci 21
!
router ospf 777
 log-adjacency-changes
 passive-interface GigabitEthernet0/0
 network 11.192.0.0 0.0.255.255 area 500
 network xxx.xx.1.0 0.0.0.3 area 500
 network xxx.xx.1.4 0.0.0.3 area 500
 network xxx.xx.1.8 0.0.0.3 area 500
 network xxx.xx.2.0 0.0.0.255 area 500
 network xxx.xx.192.0 0.0.0.255 area 500

xxx.xx.1.1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     xxx.xx.0.0/16 is variably subnetted, 8 subnets, 2 masks
O       xxx.xx.128.0/24 [110/201] via xxx.xx.1.6, 00:59:35, Hssi1/0.2
O       xxx.xx.136.0/24 [110/201] via xxx.xx.1.2, 00:59:35, Hssi1/0.1
C       xxx.xx.192.0/24 is directly connected, GigabitEthernet0/1
O       xxx.xx.1.16/30 [110/400] via xxx.xx.1.10, 00:59:35, Hssi1/0.3
                       [110/400] via 172.25.1.2, 00:59:35, Hssi1/0.1
C       xxx.xx.1.4/30 is directly connected, Hssi1/0.2
C       xxx.xx.1.0/30 is directly connected, Hssi1/0.1
O       xxx.xx1.12/30 [110/400] via xxx.xx.1.6, 00:59:35, Hssi1/0.2
                       [110/400] via xxx.xx.1.2, 00:59:35, Hssi1/0.1
C       xxx.xx.1.8/30 is directly connected, Hssi1/0.3
     11.0.0.0/16 is subnetted, 4 subnets
O       11.136.0.0 [110/201] via xxx.xx.1.2, 00:59:35, Hssi1/0.1
O       11.128.0.0 [110/201] via xxx.xx.1.6, 00:59:35, Hssi1/0.2
C       11.192.0.0 is directly connected, GigabitEthernet0/0
O       11.208.0.0 [110/201] via xxx.xx.1.10, 00:59:35, Hssi1/0.3

xxx.xx.1.1#sh frame-relay map
Hssi1/0.3 (up): point-to-point dlci, dlci 21(0x15,0x450), broadcast
          status defined, active
Hssi1/0.1 (up): point-to-point dlci, dlci 19(0x13,0x430), broadcast
          status defined, active
Hssi1/0.2 (up): point-to-point dlci, dlci 20(0x14,0x440), broadcast
          status defined, active


xxx.xx.1.1#sh frame-relay lmi

LMI Statistics for interface Hssi1/0 (Frame Relay DTE) LMI TYPE = CISCO
  Invalid Unnumbered info 0             Invalid Prot Disc 0
  Invalid dummy Call Ref 0              Invalid Msg Type 0
  Invalid Status Message 0              Invalid Lock Shift 0
  Invalid Information ID 0              Invalid Report IE Len 0
  Invalid Report Request 0              Invalid Keep IE Len 0
  Num Status Enq. Sent 2772742          Num Status msgs Rcvd 2772742
  Num Update Status Rcvd 0              Num Status Timeouts 0
  Last Full Status Req 00:00:47         Last Full Status Rcvd 00:00:47




xxx.xx.1.1#sh frame-relay pvc

PVC Statistics for interface Hssi1/0 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          3            0            0            0
  Switched       0            0            0            0
  Unused         0            0            0            0

DLCI = 19, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Hssi1/0.1

  input pkts 137250662     output pkts 104118459    in bytes 916364642
  out bytes 2513817940     dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 667140      in BECN pkts 1304        out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3350691   out bcast bytes 395248923
  5 minute input rate 2977000 bits/sec, 295 packets/sec
  5 minute output rate 136000 bits/sec, 199 packets/sec
  pvc create time 45w5d, last time pvc status changed 45w5d

DLCI = 20, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Hssi1/0.2

  input pkts 132646878     output pkts 179892573    in bytes 1458554381
  out bytes 2621571857     dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 47598       in BECN pkts 16448       out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3366973   out bcast bytes 396883737
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 45w5d, last time pvc status changed 23w2d

DLCI = 21, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Hssi1/0.3

  input pkts 125832671     output pkts 81082180     in bytes 1556550192
  out bytes 88417152       dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 608933      in BECN pkts 3101        out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3375354   out bcast bytes 398059991
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 45w5d, last time pvc status changed 45w5d
0
Comment
Question by:Dragon0x40
  • 6
  • 3
  • 2
  • +1
12 Comments
 
LVL 4

Accepted Solution

by:
cdowdy earned 167 total points
ID: 33463672
This is normal on serial interfaces. In the same way, when you run a remote loopback test on a serial interface and ping your local interface, the packet exits the interface and runs through the loop and back testing the portion looped.
0
 

Author Comment

by:Dragon0x40
ID: 33464284
thanks cdowdy,

I had not seen this behavior before but I guess it makes sense. Sort of like if an Ethernet interface does not have link then it will not respond to ping or show up in the routing table. I guess only a loopback interface will respond to ping regardless of  what the physical state or condition of the links are.

I don't see any dropped packets or interface errors on these routers. Would the switches in the frame relay cloud show the dropped packets and interface errors?

0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33464557
That much packet loss will have a major impact on TCP; I would contact you carrier have them pull PMs (Performance Monitors) on your circuit and then at some point during a maintenance window have them run intrusive testing. Of source you are going to want to rule out your hardware as well (Use spare equipment and replace the HSSI cards [There is not much equipment testing you can do with those old beasts]) and possibly routers.

Billy
0
 
LVL 4

Expert Comment

by:cdowdy
ID: 33465460
I agree with the above. I think you'll find that your provider will pretty easily uncover this much packet loss and see it on their equipment. Personally, I would not look at my equipment as being the cause until they have done their due diligence in testing the circuit. I worked for a company that had literally thousands of frame-relay circuits nationwide, and I can count on one hand how may bad wics we actually had in 7 years..
0
 

Author Comment

by:Dragon0x40
ID: 33470924
thanks rfc1180,

I will have the frame relay circuit checked out.
0
 
LVL 3

Assisted Solution

by:DeltaR7
DeltaR7 earned 167 total points
ID: 33472961
also a good way to check if your circuit on your end has errors is to check the physical interface for errors:
 sh int Hssi1/0
5 minute output rate 910000 bits/sec, 91 packets/sec
     168342587 packets input, 614927758 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     3114 input errors, 3114 CRC, 462 frame, 67 overrun, 0 ignored, 789 abort  ---- for example
     234912161 packets output, 3339762432 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets  ---
     0 output buffer failures, 0 output buffers swapped out
     3 carrier transitions  ----
make sure to clear the counters first
Check for CRC, frame, input errors, interface resets and carrier transitions.
you may want to check the other end of the PVC as well
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:Dragon0x40
ID: 33481510
thanks DeltaR7,

I will check the Hssi interfaces for errors.
0
 

Author Comment

by:Dragon0x40
ID: 33481649
I don't see any errors or dropped packets on the Hssi interface.


#sh int hssi1/0
Hssi1/0 is up, line protocol is up
  Hardware is M1T-HSSI-B
  MTU 4470 bytes, BW 45045 Kbit, DLY 200 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, crc 16, loopback not set
  Keepalive set (10 sec)
  Restart-Delay is 0 secs
  LMI enq sent  2789432, LMI stat recvd 2789432, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  Broadcast queue 0/256, broadcasts sent/dropped 10154170/5, interface broadcasts 8760713
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 46w0d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 9774
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     420029571 packets input, 806256608 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
              0 parity
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 1 abort
     382604792 packets output, 2385058501 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions     LC=down  CA=up  TM=down LB=down TA=up LA=down

#sh int hssi1/0.1
Hssi1/0.1 is up, line protocol is up
  Hardware is M1T-HSSI-B
  Internet address is xxx.xx.1.1/30
  MTU 4470 bytes, BW 45045 Kbit, DLY 200 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY
  Last clearing of "show interface" counters never
0
 
LVL 24

Assisted Solution

by:rfc1180
rfc1180 earned 166 total points
ID: 33481990
hssi looks beautiful! Again, I would have the carrier check the issue (Issue could be related to their network); Also note, that depending how busy the network is, ICMP is low priority, so pinging the router could cause the issues that you are seeing. A router that has to process ICMP packets will have to process the packet in CPU (Software). What is the end to end connectivity like (Host to Host)?

Billy
0
 

Author Closing Comment

by:Dragon0x40
ID: 33590786
This frame relay circuit is a backup to 10gb link using mpls. We reconfigured the mpls so that the traffic is going across the 10gb link. I think maybe some of the traffic was going across the frame and some was trying unsuccessfully to go across the mpls but it is fixed now.
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33590845
>think maybe some of the traffic was going across the frame and some was trying unsuccessfully to go across the mpls but it is fixed now.

Maybe, but you were pinging the remote end IP: xxx.xx.1.2
which is directly connected.	

xxx.xx.1.0/30 is directly connected, Hssi1/0.1

Any ping traffic for xxx.xx.1.2 would have traverse over Hssi1/0.1 and no other interface based on the show ip route that you provided, so if it is fixed, then I am sure there was an issue withing the carrier network that got fixed.

Billy
0
 

Author Comment

by:Dragon0x40
ID: 33591656
That is true. The frame-relay does look good.

Maybe we had two problems but it is working now!

#ping
Protocol [ip]:
Target IP address: xxx.xx.1.2
Repeat count [5]: 200
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to xxx.xx.1.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

#sh ip route xxx.xx.1.2
Routing entry for xxx.xx.1.0/30
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via Hssi1/0.1
      Route metric is 0, traffic share count is 1
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

I have seen some questions on problems with SSH/telnet access to Cisco routers that may occur despite the fact that from a PC connected to your LAN, Internet connectivity is in place and users can access Internet sites without any issues.  There are…
Creating an OSPF network that automatically (dynamically) reroutes network traffic over other connections to prevent network downtime.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now