Link to home
Start Free TrialLog in
Avatar of rmills8387
rmills8387

asked on

Choppy audio on VOIP system

We started to get choppy audio when we exceed 15 or 18 users. This just started last week and before we had no problem adding up to 30 users. Nothing new was added to our network and all bandwidth test seem to be fine. We did replace our 24 port gigabit switch thinking it was the problem. We are using inContact’s SaaS-based call center software to make our calls which is routed through an edgewater proxy router that they provided. I will note that when we are getting choppy calls using inContact any calls through skype or any other voip system works fine.  They say that the edgewater is working properly and they are not having any problems on their end. Our ISP came out and tested our 50mg fiber circuit and said there were no problems. Below are ping and tracert test to incontacts server that seems to show a problem:

Pinging inlogin.incontact.com [207.166.94.54] with 32 bytes of data:

Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=405ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=484ms TTL=54
Reply from 207.166.94.54: bytes=32 time=404ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=459ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=428ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=138ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=35ms TTL=54
Reply from 207.166.94.54: bytes=32 time=295ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=268ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=61ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=205ms TTL=54
Reply from 207.166.94.54: bytes=32 time=70ms TTL=54
Reply from 207.166.94.54: bytes=32 time=67ms TTL=54
Reply from 207.166.94.54: bytes=32 time=62ms TTL=54
Reply from 207.166.94.54: bytes=32 time=55ms TTL=54
Reply from 207.166.94.54: bytes=32 time=147ms TTL=54
Reply from 207.166.94.54: bytes=32 time=42ms TTL=54
Reply from 207.166.94.54: bytes=32 time=36ms TTL=54
Reply from 207.166.94.54: bytes=32 time=33ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=124ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=348ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=33ms TTL=54
Reply from 207.166.94.54: bytes=32 time=262ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=33ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=250ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54
Reply from 207.166.94.54: bytes=32 time=33ms TTL=54
Reply from 207.166.94.54: bytes=32 time=32ms TTL=54

Ping statistics for 207.166.94.54:
    Packets: Sent = 66, Received = 66, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 32ms, Maximum = 484ms, Average = 91ms


C:\Documents and Settings\Administrator.OFFICE>tracert inlogin.incontact.com

Tracing route to inlogin.incontact.com [207.166.94.54]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  97.67.176.81
  2     1 ms    <1 ms    <1 ms  10.44.9.249
  3     5 ms     4 ms     4 ms  97.67.236.34
  4    11 ms    11 ms    11 ms  te10-0-0d0.cir1.atlanta6-ga.us.xo.net [216.156.108.25]
  5    15 ms    11 ms    11 ms  vb2000d1.rar3.atlanta-ga.us.xo.net [207.88.13.154]
  6    35 ms    35 ms    35 ms  te-3-0-0.rar3.dallas-tx.us.xo.net [207.88.12.2]
  7    26 ms    26 ms    26 ms  ae0d0.mcr1.dallas-tx.us.xo.net [216.156.0.82]
  8    26 ms    26 ms    26 ms  p0-0.chr1.dallas-tx.us.xo.net [207.88.82.10]
  9    27 ms    26 ms    26 ms  ip65-46-140-138.z140-46-65.customer.algx.net [65.46.140.138]
 10    32 ms    32 ms    32 ms  207.166.94.54

Trace complete.

Below is a ping test to another voip server and seems fine:

C:\Users\Administrator>ping montreal.voip.ms -t
Pinging montreal.voip.ms [67.205.74.164] with 32 bytes of data:
Reply from 67.205.74.164: bytes=32 time=61ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=42ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=42ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=42ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=42ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=43ms TTL=53
Reply from 67.205.74.164: bytes=32 time=42ms TTL=53
Reply from 67.205.74.164: bytes=32 time=42ms TTL=53
Ping statistics for 67.205.74.164:
    Packets: Sent = 24, Received = 24, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 42ms, Maximum = 61ms, Average = 43ms

My question is who and where is the problem cause everybody says it's not them. Is there anything else I can test or do to help identify the problem.

Thanks
Avatar of Frosty555
Frosty555
Flag of Canada image

These sort of intermittent latency issues are extremely difficult to debug.

You need a better idea of exactly what is happening on your network in order to isolate the problem to:

1) Your internet service provider
2) Your voip service provider
3) You simply exceeding the bandwidth capabilities of your internet, OR exceeding the capabilities of some network device, switch etc.
4) Hardware failure or issue

The ping tests you have done don't provide a complete picture - they don't show jitter or packet loss, only a crude representation of latency.

I recently setup a "Smokeping" server of my own, which is a package that runs on Ubuntu. It allows me to constantly ping and monitor multiple different targets (both on my local network and out on the internet), and it makes really good graphics of the latency, jitter etc. with all of the sites being pinged lined up and time synchronized so that I can see which ones go out, and when.

If you're comfortable using linux, and you can quickly set up a simple ubuntu LAMP server with smokeping installed you could do the same thing as me and have it monitor ping times to:
      -your router
      -your edgewater proxy server
      -incontact.com's servers
      -voip.ms' servers
      -google

And basically everything else you can think of. Over 24 hours it will give you a good picture of what's going on in your network.

Instructions here: http://www.howtoforge.com/monitoring-network-latency-with-smokeping-ubuntu-9.04

If you're not comfortable with using Ubuntu.... dslreports.com has a hosted "smokeping" service that you can use to ping your own WAN IP address - this will at least give you some indication if your internet is solid or not:

http://www.dslreports.com/smokeping

That's as far as my diagnosis abilities really go... I'd be interested to see what other experts say about troubleshooting this kind of problem, though, because I have issues like this myself on my own VoIP system and I wouldn't mind some more advice.
Hi,
Frosty described already that it is very difficult to pinpoint such issues.
My idea would be to have long time traces on chokepoints eg. connection to the network provider. Setup a mirror port on the switch (i hope that it has such a feature) and make a longterm observation. Write down the time when you observe network problems and let your users do the same.
For long term observation wireshark is not that good. Use something like cascade pilot to do that or  the tool voipmonitor:
http://www.voipmonitor.org/
http://www.riverbed.com/de/products/cascade/cascade_pilot.php

Has your router the capability to show the used bandwidth ?
Setup a logserver or SNMP-collector to get those statistic data.
And use a  program that monitors the connection to your provider:
http://www.myconnectionpc.com/

If you want to test the voip connectivity between the servers (hopefully some Unix system
check out this nice tool, its called "mausezahn":
http://www.perihel.at/sec/mz/mzguide.html
check out chapter 6.6. basically its a traffic generator which shows you the jitter between two communication points. very helpful.

If you want to know what is going on on desktop PCs take wireshark and check factors like jitter and latency or other unusal traffic( packet loss, retransmits Dup acks, delays in HTTP requests or database requests (mysql))

Furthermore what VOip phones are you using ??
We are using snom and i let those devices PUBLISH a RTCP-XR report each time a call is terminated. As a collector i use a simple ngrep which looks for PUBLISH on a designated port. It looks like this:
2012/02/09 18:08:27.342660 10.0.23.97:3072 -> 10.0.23.87:41414
  PUBLISH 10.0.23.87:41414 SIP/2.0..Via: SIP/2.0/UDP 10.0.23.97:3072;branch=z
  9hG4bK-6x7tiyz1v6jx;rport..From: <sip:+4991255600122@happy.hippo.de>;tag=6di6x2ocvr..To: <10.0.23.87:41414>..Call-ID: 197f263c9d03-fs
  23q5asrpht..CSeq: 3 PUBLISH..Max-Forwards: 70..Contact: <sip:+4991255600122
  @10.0.23.97:3072;line=uka034dn>;reg-id=1..User-Agent: snom870/8.4.31..Event
  : vq-rtcpxr..Accept: application/sdp, message/sipfrag..Content-Type: applic
  ation/vq-rtcpxr..Content-Length: 973....VQSessionReport..LocalMetrics:..Tim
  estamps:START=2012-02-09T17:07:20Z STOP=2012-02-09T17:08:26Z..SessionDesc:P
  T=8 PD=G.711A PPS=50 SSUP=off..CallID:d47e263c88e3-j386r5e63djj..x-UserAgen
  t:snom870/8.4.31..FromID:<sip:+4991255600122@happy.hippo.de>..ToID:<sip:091244600177@happy.hippo.de;user=phone>..Loc
  alAddr:IP={x-snom-adr} PORT=56650 SSRC=0x4257AE60..RemoteAddr:IP=x.x.x.x PORT=34742 SSRC=0x7274E230..DialogID:d47e263c88e3-j386r5e63djj;to-tag=
  h7g4Esbg_gyg1isch-4ztn;from-tag=6di6x2ocvr..x-SIPmetrics:SVA=RG SRD=63573 S
  FC=0..x-SIPterm:SDC=OK SDR=OR..JitterBuffer:JBA=0 JBR=0 JBN=0 JBM=0 JBX=655
  35..PacketLoss:NLR=0.0 JDR=0.0..BurstGapLoss:BLD=0.0 BD=0 GLD=0.0 GD=920 GM
  IN=16..Delay:RTD=0 ESD=0 IAJ=0..QualityEst:MOSLQ=4.1 MOSCQ=3.9..RemoteMetri
  cs:..JitterBuffer:JBA=0 JBR=0 JBN=0 JBM=0 JBX=65535..PacketLoss:NLR=0.0 JDR
  =0.0..BurstGapLoss:BLD=0.0 BD=0 GLD=0.0 GD=939 GMIN=16..Delay:RTD=0 ESD=0 I
  AJ=0..QualityEst:MOSLQ=4.1 MOSCQ=3.9..


As you can see i get all interesting QoS data for each and every call. Even a MOS Score:
MOSCQ=3.9.
If you have such a feature on a phone it is very useful, because you can perhaps see a pattern in the network problems !

As you seem to use Windows that tool might be of interest too:
http://www.pingplotter.com/pro/
ASKER CERTIFIED SOLUTION
Avatar of rmills8387
rmills8387

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of rmills8387
rmills8387

ASKER

VOIP provider finally changed their poxey server (Edgemark) and our problems went away.