?
Solved

Router interface response time isssue

Posted on 2011-03-25
5
Medium Priority
?
1,635 Views
Last Modified: 2012-08-13
I  have been moving from an older Cisco 4500 series switched/routed  network to Core Brocade MLX16 switched/routed network. The project is ¾ complete. I am seeing some weird ICMP responses coming from any of the Cisco VLAN interfaces (not yet migrated) or the Brocade VE’s. As we are have phased the rollover by VLAN’s and their contents. I have set the RSTP priorities to be balanced on the two new Brocade cores. Whichever switch owns the root, also owns the VRRP-E master role. The Cisco switches are setup similar, using HSRP.
If I ping any (brocade) VE or (Cisco) VLAN interfaces, or their VRRP-E/HSRP interface, I am seeing response like the ones below. If I ping any other devices, on the same VLAN or across the routers on other VLAN’s, be it,  Servers, workstations etc, we are seeing no variations. All within 1 MS.

Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW.

It concerns me. Does anyone have an idea where to start?

Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=14ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time=18ms TTL=64
Reply from ??.??.60.1: bytes=32 time=4ms TTL=64
Reply from ??.??.60.1: bytes=32 time=16ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64


0
Comment
Question by:Warhawk72
  • 2
  • 2
5 Comments
 
LVL 4

Expert Comment

by:cdowdy
ID: 35220294
Are your switches utilizing any control plane policing? CoPP can and does cause exactly this behavior on my Nexus core when pinging directly to a destination IP on the core switch itself, but not through it as the pings destined to the switch could be DoS attack and as such, the CoPP configuration rate limits the replies.
0
 
LVL 4

Accepted Solution

by:
cdowdy earned 750 total points
ID: 35220298
In fact, send them faster and see if it begins to drop them entirely..
0
 
LVL 24

Assisted Solution

by:rfc1180
rfc1180 earned 750 total points
ID: 35223163
>Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW

Could be several reasons, one being that with many of the montioring software (SNMP) poll roughly every 5 minutes; this will not catch any microburst traffic. Additionally, with any microburst traffic, you will have to consider queues and buffers and you need to ensure that no packets are being affected due to queing delay or even other delays. Additionally, you have to consider that ICMP packets on most platforms are NOT high priority and will be queued until other traffic such as TCP are forwarded/switched first. What is the CPU like on each of the routers, you also have to indentify which router(s) is causing the delay; remember that ICMP is handled by the router CPU and not by any other router/switching ASICs. This can cause the issues that you are seeing. I would review all SVIs, VEs, Physical interfaces, memory and CPU on all the devices end to end. Finally, you can use tools such as tcpping or tcptraceroute to utilize TCP SYN packets to gather latency information as depending on the platform, TCP traffic will be handled by the data plane and not the control plane.

Billy
0
 

Author Closing Comment

by:Warhawk72
ID: 35335605
thank you for your recommendations.
0
 

Author Comment

by:Warhawk72
ID: 35335659
It turns out that the Processor answers ICMP requests for all VE's and VRRP-E interfaces. Where as icmp traversing to another VLAN is switched/routed via the hardware. So depending on what the processor is doing at any one moment you can see random spikes in ICMP response time.

This is the response I received from Brocade:

1.      Weird ICMP ping time to VE and VRRP interfaces. Intermittently, we see the ping response to the VE and VRRP interfaces on the MLX devices go from <1ms to about 9-10ms. But we do not see any latency while pinging actual devices connected on the same VLAN.

"This is not weird but is normal. While pinging the VE and VRRP interface, we are going through the CPU. Anytime there is higher traffic or any other process that is using CPU resources, we will see some latency in the pings. Whereas, the same issue is not seen on other devices connected as we are going through hardware and are not using CPU resources."[
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Make the most of your online learning experience.
LinkedIn blogging is great for networking, building up an audience, and expanding your influence as well. However, if you want to achieve these results, you need to work really hard to make your post worth liking and sharing. Here are 4 tips that ca…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
Michael from AdRem Software outlines event notifications and Automatic Corrective Actions in network monitoring. Automatic Corrective Actions are scripts, which can automatically run upon discovery of a certain undesirable condition in your network.…

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question