Solved

Router interface response time isssue

Posted on 2011-03-25
5
1,551 Views
Last Modified: 2012-08-13
I  have been moving from an older Cisco 4500 series switched/routed  network to Core Brocade MLX16 switched/routed network. The project is ¾ complete. I am seeing some weird ICMP responses coming from any of the Cisco VLAN interfaces (not yet migrated) or the Brocade VE’s. As we are have phased the rollover by VLAN’s and their contents. I have set the RSTP priorities to be balanced on the two new Brocade cores. Whichever switch owns the root, also owns the VRRP-E master role. The Cisco switches are setup similar, using HSRP.
If I ping any (brocade) VE or (Cisco) VLAN interfaces, or their VRRP-E/HSRP interface, I am seeing response like the ones below. If I ping any other devices, on the same VLAN or across the routers on other VLAN’s, be it,  Servers, workstations etc, we are seeing no variations. All within 1 MS.

Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW.

It concerns me. Does anyone have an idea where to start?

Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=14ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time=18ms TTL=64
Reply from ??.??.60.1: bytes=32 time=4ms TTL=64
Reply from ??.??.60.1: bytes=32 time=16ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64


0
Comment
Question by:Warhawk72
  • 2
  • 2
5 Comments
 
LVL 4

Expert Comment

by:cdowdy
ID: 35220294
Are your switches utilizing any control plane policing? CoPP can and does cause exactly this behavior on my Nexus core when pinging directly to a destination IP on the core switch itself, but not through it as the pings destined to the switch could be DoS attack and as such, the CoPP configuration rate limits the replies.
0
 
LVL 4

Accepted Solution

by:
cdowdy earned 250 total points
ID: 35220298
In fact, send them faster and see if it begins to drop them entirely..
0
 
LVL 24

Assisted Solution

by:rfc1180
rfc1180 earned 250 total points
ID: 35223163
>Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW

Could be several reasons, one being that with many of the montioring software (SNMP) poll roughly every 5 minutes; this will not catch any microburst traffic. Additionally, with any microburst traffic, you will have to consider queues and buffers and you need to ensure that no packets are being affected due to queing delay or even other delays. Additionally, you have to consider that ICMP packets on most platforms are NOT high priority and will be queued until other traffic such as TCP are forwarded/switched first. What is the CPU like on each of the routers, you also have to indentify which router(s) is causing the delay; remember that ICMP is handled by the router CPU and not by any other router/switching ASICs. This can cause the issues that you are seeing. I would review all SVIs, VEs, Physical interfaces, memory and CPU on all the devices end to end. Finally, you can use tools such as tcpping or tcptraceroute to utilize TCP SYN packets to gather latency information as depending on the platform, TCP traffic will be handled by the data plane and not the control plane.

Billy
0
 

Author Closing Comment

by:Warhawk72
ID: 35335605
thank you for your recommendations.
0
 

Author Comment

by:Warhawk72
ID: 35335659
It turns out that the Processor answers ICMP requests for all VE's and VRRP-E interfaces. Where as icmp traversing to another VLAN is switched/routed via the hardware. So depending on what the processor is doing at any one moment you can see random spikes in ICMP response time.

This is the response I received from Brocade:

1.      Weird ICMP ping time to VE and VRRP interfaces. Intermittently, we see the ping response to the VE and VRRP interfaces on the MLX devices go from <1ms to about 9-10ms. But we do not see any latency while pinging actual devices connected on the same VLAN.

"This is not weird but is normal. While pinging the VE and VRRP interface, we are going through the CPU. Anytime there is higher traffic or any other process that is using CPU resources, we will see some latency in the pings. Whereas, the same issue is not seen on other devices connected as we are going through hardware and are not using CPU resources."[
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Join & Write a Comment

I eventually solved a perplexing problem setting up telnet for a new switch.  I installed a new Cisco WS-03560X-24P switch connected to an existing Cisco 4506 running a WS-X4013-10GE Sup II-Plus. After configuring vlans and trunking,  I could no…
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now