Link to home
Start Free TrialLog in
Avatar of Dragon0x40
Dragon0x40

asked on

Frame-Relay circuit has loss and pinging local interface has lost packets

When I ping local interface ip address xxx.xx.1.1 the ping appears to cross the frame relay network to neighbor xxx.xx.1.2 and then come back instead of replying directly from Hssi1/0.1.

We are also experiencing packet loss across the frame-relay network but I am curious if the ping and traceroute should respond directly from the Hssi1/0.1 or is it normal for frame-relay to send the ping across the cloud and then back?

If this is normal then only the frame-relay circuit has a problem but if not then maybe we have two problems.

I also see the ping going across the frame-relay first when I trace xxx.xx.1.2 on the remote router. (xxx.xx.1.1 responds first and then xxx.xx.1.2 responds)

We have packet loss in both directions across the frame-relay cloud.

I was very surprised to see packet loss pinging a local interface ip address!

172.25.1.1#sh ip int br | ex unassi
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         11.192.1.1      YES NVRAM  up                    up
GigabitEthernet0/1         xxx.xx.192.1    YES NVRAM  up                    up
Hssi1/0.1                  xxx.xx.1.1      YES NVRAM  up                    up
Hssi1/0.2                  xxx.xx.1.5      YES NVRAM  up                    up
Hssi1/0.3                  xxx.xx.1.9      YES NVRAM  up                    up

xxx.xx.1.1#traceroute xxx.xx.1.1

Type escape sequence to abort.
Tracing the route to xxx.xx.1.1

  1 xxx.xx.1.2 8 msec 0 msec 8 msec
  2 xxx.xx.1.1 12 msec 16 msec *



xxx.xx.1.1#ping
Protocol [ip]:
Target IP address: xxx.xx.1.1
Repeat count [5]: 200
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to xxx.xx.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!.!..!!!!!!!!!!!!!!!
!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!..!!!!!!.!!!.!!.!!!...!!.!!
Success rate is 91 percent (183/200), round-trip min/avg/max = 4/9/28 ms

xxx.xx.1.1#ping
Protocol [ip]:
Target IP address: xxx.xx.1.2
Repeat count [5]: 200
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to xxx.xx.1.2, timeout is 2 seconds:
!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!....!!!!
!!!!!!!!!!!.!!.!!!!!.!!..!!!!!!!!!!!!!!!..!!!!!.!!!!!!!!!!!!!!!!!!!!!!
!!!!!!.!!!!!!!!!!!!!!!.!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!
Success rate is 90 percent (181/200), round-trip min/avg/max = 1/6/20 ms





xxx.xx.1.1# sh ip ospf neigh

Neighbor ID     Pri   State           Dead Time   Address         Interface
xxx.xx.192.2      1   FULL/DR         00:00:30    xxx.xx.192.2    GigabitEthernet0/1
xxx.xx.208.1      0   FULL/  -        00:00:36    xxx.xx.1.10     Hssi1/0.3
xxx.xx.128.1      0   FULL/  -        00:00:34    xxx.xx.1.6      Hssi1/0.2
xxx.xx.136.1      0   FULL/  -        00:00:32    xxx.xx.1.2      Hssi1/0.1



xxx.xx.1.1#sh run | beg interface Hssi1/0.1
interface Hssi1/0.1 point-to-point
 ip address xxx.xx.1.1 255.255.255.252
 ip ospf cost 200
 frame-relay interface-dlci 19
!
interface Hssi1/0.2 point-to-point
 ip address xxx.xx.1.5 255.255.255.252
 ip ospf cost 200
 frame-relay interface-dlci 20
!
interface Hssi1/0.3 point-to-point
 ip address xxx.xx.1.9 255.255.255.252
 ip ospf cost 200
 frame-relay interface-dlci 21
!
router ospf 777
 log-adjacency-changes
 passive-interface GigabitEthernet0/0
 network 11.192.0.0 0.0.255.255 area 500
 network xxx.xx.1.0 0.0.0.3 area 500
 network xxx.xx.1.4 0.0.0.3 area 500
 network xxx.xx.1.8 0.0.0.3 area 500
 network xxx.xx.2.0 0.0.0.255 area 500
 network xxx.xx.192.0 0.0.0.255 area 500

xxx.xx.1.1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     xxx.xx.0.0/16 is variably subnetted, 8 subnets, 2 masks
O       xxx.xx.128.0/24 [110/201] via xxx.xx.1.6, 00:59:35, Hssi1/0.2
O       xxx.xx.136.0/24 [110/201] via xxx.xx.1.2, 00:59:35, Hssi1/0.1
C       xxx.xx.192.0/24 is directly connected, GigabitEthernet0/1
O       xxx.xx.1.16/30 [110/400] via xxx.xx.1.10, 00:59:35, Hssi1/0.3
                       [110/400] via 172.25.1.2, 00:59:35, Hssi1/0.1
C       xxx.xx.1.4/30 is directly connected, Hssi1/0.2
C       xxx.xx.1.0/30 is directly connected, Hssi1/0.1
O       xxx.xx1.12/30 [110/400] via xxx.xx.1.6, 00:59:35, Hssi1/0.2
                       [110/400] via xxx.xx.1.2, 00:59:35, Hssi1/0.1
C       xxx.xx.1.8/30 is directly connected, Hssi1/0.3
     11.0.0.0/16 is subnetted, 4 subnets
O       11.136.0.0 [110/201] via xxx.xx.1.2, 00:59:35, Hssi1/0.1
O       11.128.0.0 [110/201] via xxx.xx.1.6, 00:59:35, Hssi1/0.2
C       11.192.0.0 is directly connected, GigabitEthernet0/0
O       11.208.0.0 [110/201] via xxx.xx.1.10, 00:59:35, Hssi1/0.3

xxx.xx.1.1#sh frame-relay map
Hssi1/0.3 (up): point-to-point dlci, dlci 21(0x15,0x450), broadcast
          status defined, active
Hssi1/0.1 (up): point-to-point dlci, dlci 19(0x13,0x430), broadcast
          status defined, active
Hssi1/0.2 (up): point-to-point dlci, dlci 20(0x14,0x440), broadcast
          status defined, active


xxx.xx.1.1#sh frame-relay lmi

LMI Statistics for interface Hssi1/0 (Frame Relay DTE) LMI TYPE = CISCO
  Invalid Unnumbered info 0             Invalid Prot Disc 0
  Invalid dummy Call Ref 0              Invalid Msg Type 0
  Invalid Status Message 0              Invalid Lock Shift 0
  Invalid Information ID 0              Invalid Report IE Len 0
  Invalid Report Request 0              Invalid Keep IE Len 0
  Num Status Enq. Sent 2772742          Num Status msgs Rcvd 2772742
  Num Update Status Rcvd 0              Num Status Timeouts 0
  Last Full Status Req 00:00:47         Last Full Status Rcvd 00:00:47




xxx.xx.1.1#sh frame-relay pvc

PVC Statistics for interface Hssi1/0 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          3            0            0            0
  Switched       0            0            0            0
  Unused         0            0            0            0

DLCI = 19, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Hssi1/0.1

  input pkts 137250662     output pkts 104118459    in bytes 916364642
  out bytes 2513817940     dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 667140      in BECN pkts 1304        out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3350691   out bcast bytes 395248923
  5 minute input rate 2977000 bits/sec, 295 packets/sec
  5 minute output rate 136000 bits/sec, 199 packets/sec
  pvc create time 45w5d, last time pvc status changed 45w5d

DLCI = 20, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Hssi1/0.2

  input pkts 132646878     output pkts 179892573    in bytes 1458554381
  out bytes 2621571857     dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 47598       in BECN pkts 16448       out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3366973   out bcast bytes 396883737
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 45w5d, last time pvc status changed 23w2d

DLCI = 21, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Hssi1/0.3

  input pkts 125832671     output pkts 81082180     in bytes 1556550192
  out bytes 88417152       dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 608933      in BECN pkts 3101        out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 3375354   out bcast bytes 398059991
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 45w5d, last time pvc status changed 45w5d
ASKER CERTIFIED SOLUTION
Avatar of cdowdy
cdowdy

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Dragon0x40
Dragon0x40

ASKER

thanks cdowdy,

I had not seen this behavior before but I guess it makes sense. Sort of like if an Ethernet interface does not have link then it will not respond to ping or show up in the routing table. I guess only a loopback interface will respond to ping regardless of  what the physical state or condition of the links are.

I don't see any dropped packets or interface errors on these routers. Would the switches in the frame relay cloud show the dropped packets and interface errors?

That much packet loss will have a major impact on TCP; I would contact you carrier have them pull PMs (Performance Monitors) on your circuit and then at some point during a maintenance window have them run intrusive testing. Of source you are going to want to rule out your hardware as well (Use spare equipment and replace the HSSI cards [There is not much equipment testing you can do with those old beasts]) and possibly routers.

Billy
I agree with the above. I think you'll find that your provider will pretty easily uncover this much packet loss and see it on their equipment. Personally, I would not look at my equipment as being the cause until they have done their due diligence in testing the circuit. I worked for a company that had literally thousands of frame-relay circuits nationwide, and I can count on one hand how may bad wics we actually had in 7 years..
thanks rfc1180,

I will have the frame relay circuit checked out.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
thanks DeltaR7,

I will check the Hssi interfaces for errors.
I don't see any errors or dropped packets on the Hssi interface.


#sh int hssi1/0
Hssi1/0 is up, line protocol is up
  Hardware is M1T-HSSI-B
  MTU 4470 bytes, BW 45045 Kbit, DLY 200 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, crc 16, loopback not set
  Keepalive set (10 sec)
  Restart-Delay is 0 secs
  LMI enq sent  2789432, LMI stat recvd 2789432, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  Broadcast queue 0/256, broadcasts sent/dropped 10154170/5, interface broadcasts 8760713
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 46w0d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 9774
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     420029571 packets input, 806256608 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
              0 parity
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 1 abort
     382604792 packets output, 2385058501 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions     LC=down  CA=up  TM=down LB=down TA=up LA=down

#sh int hssi1/0.1
Hssi1/0.1 is up, line protocol is up
  Hardware is M1T-HSSI-B
  Internet address is xxx.xx.1.1/30
  MTU 4470 bytes, BW 45045 Kbit, DLY 200 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY
  Last clearing of "show interface" counters never
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
This frame relay circuit is a backup to 10gb link using mpls. We reconfigured the mpls so that the traffic is going across the 10gb link. I think maybe some of the traffic was going across the frame and some was trying unsuccessfully to go across the mpls but it is fixed now.
>think maybe some of the traffic was going across the frame and some was trying unsuccessfully to go across the mpls but it is fixed now.

Maybe, but you were pinging the remote end IP: xxx.xx.1.2
which is directly connected.	

xxx.xx.1.0/30 is directly connected, Hssi1/0.1

Any ping traffic for xxx.xx.1.2 would have traverse over Hssi1/0.1 and no other interface based on the show ip route that you provided, so if it is fixed, then I am sure there was an issue withing the carrier network that got fixed.

Billy
That is true. The frame-relay does look good.

Maybe we had two problems but it is working now!

#ping
Protocol [ip]:
Target IP address: xxx.xx.1.2
Repeat count [5]: 200
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to xxx.xx.1.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

#sh ip route xxx.xx.1.2
Routing entry for xxx.xx.1.0/30
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via Hssi1/0.1
      Route metric is 0, traffic share count is 1