tmatty102
asked on
LAN connection with high delay on one hop
Hi,
I was getting alert on ping fail from monitoring system on and off for several hours, after several hours, everything goes back to normal.
I looked into the problem and found that there was unusual high delay (between 35 to 60ms) on one hop (Cisco switch to a Cisco router), both ports are set with 1GB bandwidth. Never seen this problem before. No configuration was changed during time.
I have checked the input and output error on the ports, there's no error. Traffic was normal during the time. the ports were not flapping up and down.
What would be some possible causes of this issue?
Please share your knowledge all.
Thank you so much!
I was getting alert on ping fail from monitoring system on and off for several hours, after several hours, everything goes back to normal.
I looked into the problem and found that there was unusual high delay (between 35 to 60ms) on one hop (Cisco switch to a Cisco router), both ports are set with 1GB bandwidth. Never seen this problem before. No configuration was changed during time.
I have checked the input and output error on the ports, there's no error. Traffic was normal during the time. the ports were not flapping up and down.
What would be some possible causes of this issue?
Please share your knowledge all.
Thank you so much!
ASKER
Hi MO,
Thanks for replying.
the topology is access switch to core switch to router and out to the MPLS Cloud. very straight forward.
Spanning tree is enabled to avoid routing loops. the port that connect the switch with the router is configured as layer3 port
Thanks
Thanks for replying.
the topology is access switch to core switch to router and out to the MPLS Cloud. very straight forward.
Spanning tree is enabled to avoid routing loops. the port that connect the switch with the router is configured as layer3 port
Thanks
You're routing between core switch and router? Is that where the latency is occurring? Between core switch and router? Not between access and core switches.
MO
MO
ASKER
yes, correct.
Is there a particular reason you would be routing from your core switch to the local MPLS router? Do you have multiple different networks behind the core switch?
MO
MO
ASKER
No particular reason, the company has this setup when I join the company. Yes, I have multiple different networks behind the core switch.
Have you contacted your MPLS provider to see if they can review interface statistics on there end? Perhaps they were showing errors on the interface that connects to your core switch.
MO
MO
ASKER
yes, They have verified that there's no problem on their end.
I have verified that the problem is between the router ( I managed the router not the MPLS provider) and the core switch. This is verified by using the monitoring system that we have; by comparing the RTT report from remote sites to the router and local probe to the router, the local probe to the core switch and the local probe to the router. The latency is confirmed to be between the core switch and the router.
I have checked the input and output error on the ports, there's no error. Traffic was normal during the time. the ports were not flapping up and down.
I have verified that the problem is between the router ( I managed the router not the MPLS provider) and the core switch. This is verified by using the monitoring system that we have; by comparing the RTT report from remote sites to the router and local probe to the router, the local probe to the core switch and the local probe to the router. The latency is confirmed to be between the core switch and the router.
I have checked the input and output error on the ports, there's no error. Traffic was normal during the time. the ports were not flapping up and down.
Any chance the monitoring tools themselves are reporting inaccurately? Is there any indication of a problem other than what the monitoring tools are reporting?
Does the latency occur regularly? If so, can you run some basic ping tests between the router and core switch to validate the results of the monitoring tools? Maybe setup ping plotter on a host sitting behind the core or access switch to monitor the LAN side IP of the router (e.g. the interface that connects to the core switch) and let it running in eval mode so you can compare results from it to your monitoring tools.
MO
Does the latency occur regularly? If so, can you run some basic ping tests between the router and core switch to validate the results of the monitoring tools? Maybe setup ping plotter on a host sitting behind the core or access switch to monitor the LAN side IP of the router (e.g. the interface that connects to the core switch) and let it running in eval mode so you can compare results from it to your monitoring tools.
MO
ASKER
this issue never happen before. We have an old and a new monitoring system, they both are reporting the same delay. This is the vrry first time it happens. It doesn't happen regularly.
That might be a good idea to set up a ping plotter. However, this issue seems to have resolved itself at this time. I couldn't find any clues after I checked the logs, port errors (there is any but there was no errors) as I mentioned.
Thanks for the help so far.
Please let me know if you can think of anything. I'm placing a call with Cisco to check on this as well in case I have missed anything.
That might be a good idea to set up a ping plotter. However, this issue seems to have resolved itself at this time. I couldn't find any clues after I checked the logs, port errors (there is any but there was no errors) as I mentioned.
Thanks for the help so far.
Please let me know if you can think of anything. I'm placing a call with Cisco to check on this as well in case I have missed anything.
Wouldn't hurt to get those guys on the line for a quick look over. They might coincide your explanation of the issue with a bug in the particular IOS you're running. It happens.
Sorry I couldn't get you any closer to a resolution on this one.
Good luck.
MO
Sorry I couldn't get you any closer to a resolution on this one.
Good luck.
MO
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
The problem resolved by itself over time
MO