causes for high ping times?

I have been logging some pings to random workstations on my network, and have noticed intermittent spikes in ping times. roughly 5% of the time i am getting ping times between 2 and 6 ms. 1% of the time I am getting ping times between 20 and 40ms.

What could be some of the causes for this? How best to troubleshoot?

4 Dell r905 Servers with 10Gb nics in each
2 Dell PowerConnect 6248's with backplane card and dual 10Gb ports.

Bandwidth utilization on the LAN and the servers even during peak times is very low.

comments and suggestions are appreciated. thanks.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

We may need more clues to get a solution. But here's a few ideas.
 I've seen this behavior when an IP pool wasn't quite big enough for the number of devices on the class C subnet, when there weren't quite enough licenses to legitimately cover everyone including Administrator on the windows Server, when a rogue network card had intermittment difficulty, or a bad port and in another incident due to poor wiring (the lady rolled her chair over the network data cable every so often that came from the Cable company).  I've also seen internal issues within a server where the dual ethernet card needed it's bios flashed because of collisions/contention within the backplane or card itself (can I assume you have plenty of diskspace, RAM and paging file is bigger than needed in Adv System Properties?).  Also, sounds like something running out of memory on a router or VLAN misconfiguration.   Perhaps it could be a misconfiguration of the domain or OUs that occasionally gets accessed or a laptop that occasionally connects to the network.  There are a lot of possibilities, maybe one of these will lead you to the solution.
A lot of things can cause longer ping times but high on the list is that the PC or server itself (not the network) is busy. And remember you didn't buy that server to answer pings all day but to move files etc so ping is not a high priority task.

In the networking gear every port is moving packets as fast as they can but if a ping happens to be received while real data is also hitting it could be behind other stuff in buffers and queues.

I don't think to difference you are seeing or the number of pings affected is an issue. If it is happening for long enough periods to capture it on a sniffer you may be able to see what data it is intermixed with to get an idea if there is a bursty application in the mix.
Best way to deal with those kind of problem- put your hardware on SNMP monitoring- 
Or use Kperf soft to test performance quickly point-to-point.
Overall, from what you stated- it's seems to be normal.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Well, you left out some important details, probably. Sure, 10Gb NICs on your servers and a switch  with  10Gb ports for your servers is a recipe for very good network throughput.

Assuming the cabling is correct and all ports are clean (no errors as logged by the switch). However, you say you are pinging these servers from random workstations.

The network performance of the workstations is just as important as the network performance of the servers, when it comes to ping RTT from workstation to servers.

If there are multiple switches in your environment, you may want to examine utilization of every trunk link,  quality of those links, and selection of good hardware switches  (if you have some other switches plugged into your 10Gb switch).

Keeping in mind an issue with any other switch, any other inter-switch link, or any other workstation  or host plugged in the switch can potentially have an eventual effect on your entire network's latency.

What kind of connectivity do the workstations have to the servers?
Are they also 10Gb?  

How are you performing the ping logging?
Are you excluding the results of the first couple pings from a workstation?

A ping to be sent before ARP cache on the workstation has been populated with IP of the server, is expected to show higher latency.

There are also some considerations here such as cabling to the workstation,
if 802.3x Ethernet flow control (PAUSE) frames is in use, a node somewhere on the network with a lower speed connection may create an impact for the same broadcast domain.

Ping latency is not strictly about bandwidth used,  packet rate matters too.
If there is ever a bridging loop on the LAN,  latency could rise, due to packet rate, even with very little computer activity at all.

CPU usage on both workstation and server, and also  the accuracy of system timers  also effect the accuracy of the ping timings.

For example,  ping timings to/from Virtual Machines are notoriously inaccurate.
But many workstations and servers also have system timers that may be inaccurate,  particularly if local clock is undisciplined  (eg non-NTP running server).

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Networking Protocols

From novice to tech pro — start learning today.