?
Solved

Router interface response time isssue

Posted on 2011-03-25
5
Medium Priority
?
1,598 Views
Last Modified: 2012-08-13
I  have been moving from an older Cisco 4500 series switched/routed  network to Core Brocade MLX16 switched/routed network. The project is ¾ complete. I am seeing some weird ICMP responses coming from any of the Cisco VLAN interfaces (not yet migrated) or the Brocade VE’s. As we are have phased the rollover by VLAN’s and their contents. I have set the RSTP priorities to be balanced on the two new Brocade cores. Whichever switch owns the root, also owns the VRRP-E master role. The Cisco switches are setup similar, using HSRP.
If I ping any (brocade) VE or (Cisco) VLAN interfaces, or their VRRP-E/HSRP interface, I am seeing response like the ones below. If I ping any other devices, on the same VLAN or across the routers on other VLAN’s, be it,  Servers, workstations etc, we are seeing no variations. All within 1 MS.

Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW.

It concerns me. Does anyone have an idea where to start?

Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=14ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=19ms TTL=64
Reply from ??.??.60.1: bytes=32 time=6ms TTL=64
Reply from ??.??.60.1: bytes=32 time=18ms TTL=64
Reply from ??.??.60.1: bytes=32 time=4ms TTL=64
Reply from ??.??.60.1: bytes=32 time=16ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64
Reply from ??.??.60.1: bytes=32 time<1ms TTL=64


0
Comment
Question by:Warhawk72
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 4

Expert Comment

by:cdowdy
ID: 35220294
Are your switches utilizing any control plane policing? CoPP can and does cause exactly this behavior on my Nexus core when pinging directly to a destination IP on the core switch itself, but not through it as the pings destined to the switch could be DoS attack and as such, the CoPP configuration rate limits the replies.
0
 
LVL 4

Accepted Solution

by:
cdowdy earned 750 total points
ID: 35220298
In fact, send them faster and see if it begins to drop them entirely..
0
 
LVL 24

Assisted Solution

by:rfc1180
rfc1180 earned 750 total points
ID: 35223163
>Not sure why the Router interfaces are responding like that. We are seeing no high utilization via Orion, or SFLOW

Could be several reasons, one being that with many of the montioring software (SNMP) poll roughly every 5 minutes; this will not catch any microburst traffic. Additionally, with any microburst traffic, you will have to consider queues and buffers and you need to ensure that no packets are being affected due to queing delay or even other delays. Additionally, you have to consider that ICMP packets on most platforms are NOT high priority and will be queued until other traffic such as TCP are forwarded/switched first. What is the CPU like on each of the routers, you also have to indentify which router(s) is causing the delay; remember that ICMP is handled by the router CPU and not by any other router/switching ASICs. This can cause the issues that you are seeing. I would review all SVIs, VEs, Physical interfaces, memory and CPU on all the devices end to end. Finally, you can use tools such as tcpping or tcptraceroute to utilize TCP SYN packets to gather latency information as depending on the platform, TCP traffic will be handled by the data plane and not the control plane.

Billy
0
 

Author Closing Comment

by:Warhawk72
ID: 35335605
thank you for your recommendations.
0
 

Author Comment

by:Warhawk72
ID: 35335659
It turns out that the Processor answers ICMP requests for all VE's and VRRP-E interfaces. Where as icmp traversing to another VLAN is switched/routed via the hardware. So depending on what the processor is doing at any one moment you can see random spikes in ICMP response time.

This is the response I received from Brocade:

1.      Weird ICMP ping time to VE and VRRP interfaces. Intermittently, we see the ping response to the VE and VRRP interfaces on the MLX devices go from <1ms to about 9-10ms. But we do not see any latency while pinging actual devices connected on the same VLAN.

"This is not weird but is normal. While pinging the VE and VRRP interface, we are going through the CPU. Anytime there is higher traffic or any other process that is using CPU resources, we will see some latency in the pings. Whereas, the same issue is not seen on other devices connected as we are going through hardware and are not using CPU resources."[
0

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
This article explains the fundamentals of industrial networking which ultimately is the backbone network which is providing communications for process devices like robots and other not so interesting stuff.
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…
Suggested Courses

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question