Solved

Router interface response time isssue

Posted on 2011-03-25
5
1,559 Views
Last Modified: 2012-08-13
I  have been moving from an older Cisco 4500 series switched/routed  network to Core Brocade MLX16 switched/routed network. The project is ¾ complete. I am seeing some weird ICMP responses coming from any of the Cisco VLAN interfaces (not yet migrated) or the Brocade VE’s. As we are have phased the rollover by VLAN’s and their contents. I have set the RSTP priorities to be balanced on the two new Brocade cores. Whichever switch owns the root, also owns the VRRP-E master role. The Cisco switches are setup similar, using HSRP.
If I ping any (brocade) VE or (Cisco) VLAN interfaces, or their VRRP-E/HSRP interface, I am seeing response like the ones below. If I ping any other devices, on the same VLAN or across the routers on other VLAN’s, be it,  Servers, workstations etc, we are seeing no variations. All within 1 MS.

Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW.

It concerns me. Does anyone have an idea where to start?

Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=14ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time=18ms TTL=64
Reply from ??.??.60.1: bytes=32 time=4ms TTL=64
Reply from ??.??.60.1: bytes=32 time=16ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64


0
Comment
Question by:Warhawk72
  • 2
  • 2
5 Comments
 
LVL 4

Expert Comment

by:cdowdy
ID: 35220294
Are your switches utilizing any control plane policing? CoPP can and does cause exactly this behavior on my Nexus core when pinging directly to a destination IP on the core switch itself, but not through it as the pings destined to the switch could be DoS attack and as such, the CoPP configuration rate limits the replies.
0
 
LVL 4

Accepted Solution

by:
cdowdy earned 250 total points
ID: 35220298
In fact, send them faster and see if it begins to drop them entirely..
0
 
LVL 24

Assisted Solution

by:rfc1180
rfc1180 earned 250 total points
ID: 35223163
>Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW

Could be several reasons, one being that with many of the montioring software (SNMP) poll roughly every 5 minutes; this will not catch any microburst traffic. Additionally, with any microburst traffic, you will have to consider queues and buffers and you need to ensure that no packets are being affected due to queing delay or even other delays. Additionally, you have to consider that ICMP packets on most platforms are NOT high priority and will be queued until other traffic such as TCP are forwarded/switched first. What is the CPU like on each of the routers, you also have to indentify which router(s) is causing the delay; remember that ICMP is handled by the router CPU and not by any other router/switching ASICs. This can cause the issues that you are seeing. I would review all SVIs, VEs, Physical interfaces, memory and CPU on all the devices end to end. Finally, you can use tools such as tcpping or tcptraceroute to utilize TCP SYN packets to gather latency information as depending on the platform, TCP traffic will be handled by the data plane and not the control plane.

Billy
0
 

Author Closing Comment

by:Warhawk72
ID: 35335605
thank you for your recommendations.
0
 

Author Comment

by:Warhawk72
ID: 35335659
It turns out that the Processor answers ICMP requests for all VE's and VRRP-E interfaces. Where as icmp traversing to another VLAN is switched/routed via the hardware. So depending on what the processor is doing at any one moment you can see random spikes in ICMP response time.

This is the response I received from Brocade:

1.      Weird ICMP ping time to VE and VRRP interfaces. Intermittently, we see the ping response to the VE and VRRP interfaces on the MLX devices go from <1ms to about 9-10ms. But we do not see any latency while pinging actual devices connected on the same VLAN.

"This is not weird but is normal. While pinging the VE and VRRP interface, we are going through the CPU. Anytime there is higher traffic or any other process that is using CPU resources, we will see some latency in the pings. Whereas, the same issue is not seen on other devices connected as we are going through hardware and are not using CPU resources."[
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Let’s list some of the technologies that enable smooth teleworking. 
Most of the applications these days are on Cloud. Cloud is ubiquitous with many service providers in the market. Since it has many benefits such as cost reduction, software updates, remote access, disaster recovery and much more.
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

775 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question