Poor One Way Audio Quality Via SIP Trunk

Dear Experts Exchange,

QUESTION SUMMARY: My office is having one-way audio quality issues to multiple remote parties over both incoming and outgoing calls.  We in the office always hear the remote parties clearly.  We are not experiencing dropped calls.  Troubleshooting seems to point to our Internet connection via CenturyLink DSL.  I am looking for feedback on the latest MTR reports from our IP PBX (I suggest the Excel file for easiest viewing) and whether you think they still indicate an issue, and if not what do you think is causing our problem?  Suggestions for further troubleshooting steps are also greatly appreciated.



I am in the 5th day of our business phone system essentially being down.  We are experiencing variable yet usually unintelligible one-way audio quality issues with the remote party (i.e. not us in the office) on both inbound and outbound calls.  We can always hear the remote party perfectly (call quality inbound to our office is very good).  All outbound calls connect fine and all inbound calls to the best of our knowledge connect fine as well.  We have not experienced any dropped calls - just very poor one-way audio with the remote party not being able to understand our garbled/dropped speech.  Call quality is excellent extension to extension and extension to voice mail (i.e. all within our LAN).  This phone system in its current configuration has been very stable for years (with the exception of one HDD failure in the server - was RAID 1 so not a real issue - and a serious hack a couple years ago).

I have called Fonality, whose product trixbox Pro we use as our local hosted IP PBX and also through whom we have a support contract, and they say everything looks good on our server and that it must be a carrier issue (e.g. our virtual SIP trunk provider 8x8).  Fonality's tech support has been helpful and knowledgeable in the past.  I have called 8x8 and they said they were not having any service issues.  They placed a call directly from their data center to our IP PBX, essentially testing solely the network connection between our IP PBX and 8x8, and the call quality was similarly poor.  This eliminates the likelihood that it is an upstream carrier issue.  8x8 Tier 2 support was very nice and suggested I run MTR's to their east and west coast data centers after power cycling all my equipment (which I had already done multiple times).

The MTR's (Matt's traceroute, AKA My traceroute) appeared to show significant packet loss and latency both across CenturyLink's network and Level3.  I called CenturyLink and got the run around, but eventually got to a Tier 4 Supervisor who has been helping me.  He said yesterday that their network ops center had done some reconfiguration and to test the WAN connection again.  I have attached to this question the most recent MTR reports in both text format and MS Excel format (much easier to read) from the trixbox Pro server, which is running a slimmed down/custom version of CentOS 4.4 (Final).  These MTR's were run via CLI/Bash on the IP PBX server, whose NIC is set to a static IP, connected via a short patch cord to a newly replaced (yesterday) CenturyLink ZyXEL PK5001Z DSL modem/wireless router in full bridge mode (wireless is of course disabled).  I normally have a ZyXEL ZyWALL 2 Pro firewall in between the modem and server, set in transparent/bridge mode, serving only as a simple firewall, but removed it for the purpose of all MTR tests, both initial and after the changes CenturyLinks ops center supposedly made.  There is no router, no switch, or other intervening device between the IP PBX server and the DSL modem.  Our DSL line is dedicated to our phone system.

The MTR reports do show significant improvement in packet loss and latency since CenturyLink's ops center supposedly made their changes, but our one way audio issue to remote parties is still about as bad as it was before.  For comparison, I have attached MTR reports to the same hosts as I tested from the IP PBX server, but this time using WinMTR on my laptop connected to our office LAN/switch/SonicWALL via Comcast Business cable.  They actually look a lot worse than the current CenturyLink reports.  Any thoughts on why this might be?

My question has two parts: what are your thoughts on the latest MTR reports attached hereto, and also what do you think could be causing our one-way audio quality issues to remote parties?

Thank you very much for any insights or thoughts, and I do apologize for the long explanation/question.

Charles LamAsked:
Who is Participating?
PhonebuffConnect With a Mentor Commented:
One other idea --  You should read this thread, I have used John's Redneck QOS before on ADSL links with good luck.  It's important to note the QOS does not go beyond your router / firewall unless you use someone like CBeyond or One Ring where you are on net until they gateway the IP stream to the Internet after the phone switch.


Couple of other things, if you are happy with 8X8 maybe just move them with a good firewall / router (pfSense is my favorite) to the Comcast link.   You cold also check your CODEC, maybe going to one like GSM would use less bandwith and give you better call quality.

It really sounds like this might be a bandwith / link quality issue if it's on ADSL and the inbound leg of the call is good, but the outbound leg is getting complaints.


First off, yes high packet loss in you UDP audio connection is probably the cause of you voice issues.  Next, MTR really needs to run about 15 minutes or so for the data to be meaningfull, but you definitely have issues in these routes.  

Not sure i understand the topology, you said you were talking to Centurylink, but you have Comcast traces.  Be sure your router is sending packets outbound on the correct / ISP.  

Have you tried another provider IPComms or Vitality in case the issue is 8X8 ?
nociSoftware EngineerCommented:
To what extent can your router support Qos and have VOIP delivered BEFORE any other traffic?

DSL mostly is rather asymetric, so while you have good quality (down speed) the upspeed is limited.  and if you go over the limit some packet will get lost.
Also Routers normally prefer TCP traffic above UDP casing a webaccess/file transfer to intrude on the available BW on the uplink.
Protect Your Employees from Wi-Fi Threats

As Wi-Fi growth and popularity continues to climb, not everyone understands the risks that come with connecting to public Wi-Fi or even offering Wi-Fi to employees, visitors and guests. Download the resource kit to make sure your safe wherever business takes you!

Charles LamAuthor Commented:
Dear Phonebuff and noci,

Thank you so very much for your attention/comments/questions to my issue.  I will address the issues you raised below in order of their appearance:


Thank you - I am very inexperienced with MTR - I will run 15 min traces.  As for topology, I apologize if I was confusing in my original question.  We have both Comcast cable Internet and CenturyLink DSL here at the office.  I have two (2) distinct physical networks - one for the phone system via CenturyLink (exclusively) and one for the office in general (LAN, server, WiFi).  My phone system topology is very simple - my trixbox Pro IP PBX has a patch cable going from it to a ZyXEL ZyWALL 2 Pro hardware firewall in bridge mode, then I have a patch cable going from the ZyWALL to the CenturyLink DSL modem (which I replaced Wednesday with a brand new one) in bridge mode as well.  Nothing else is connected to the DSL modem or the ZyWALL.  I then have a patch cord going from ETH1 to an HP POE switch, and individual Ethernet cords from there directly to all the phones in the office.

I included the MTR's labeled "Comcast" only as a point of comparison to the CenturyLink from my office.  Curiously enough, they look even worse than CenturyLink.

I will try the two other virtual SIP trunk providers you mention if I cannot get this resolved another way today.  8x8 claims they have no issues.  Calls always connect, it is just the one-way audio issue.


As noted above in my response to Phonebuff, there is no active router between my trixbox Pro IP PBX and the CenturyLink DSL modem.  There is the ZyZEL ZyWALL 2 Pro hardware firewall/router, but it is in bridge mode and only acts as a transparent firewall, limiting connections/ports/protocols to only those that are absolutely needed by the trixbox Pro IP PBX.  I have had this firewall in place for several years without issue and have not made any firmware or configuration changes to it.  All the MTR reports attached to my question were taken without the firewall installed, with a direct connection between my IP PBX and the CenturyLink modem in bridge mode.  The only traffic that goes through my CenturyLink DSL connection is traffic to and from my PBX.  This is mostly SIP traffic, with a little VPN action between the PBX and the web admin console, our "HUD" software interfaces on our PC's, and VPN traffic to/from Fonality when I call them for support.


I hope my clarifications above help - again, I REALLY appreciate any thoughts you have on further troubleshooting.  Phonebuff - I will run longer MTR's today, but I take it from your comment that the MTR's I have attached to my question pose a problem?


    Run I tun an MTR to my provider over a 15 minutes period I look for < 5% error anything higher tends to result in quality issues.  

    I am using Comcast Business here --

   An I watching a MTR from my office to yahoo.com, 21 hops and except for one host that is ignoring the ICMP request I have 2% loss on one host an 0% on all others with 300 packets sent..

    Going to 8X8.com I am 19 hops away and I am seeing approximately the same thing with 19 hops.

You might try this -- http://www.8x8.com/Resources/Tools/VoIPTest.aspx

I would also consider putting a wireshark on your network, let it collect data for a while and then look at the traffic source & end IPs to ensure you are carrying what you think you are.
Charles LamAuthor Commented:
OK, thank you Phonebuff for the benchmarks!  I will try to wireshark my traffic on the DSL line and see what's up...  good suggestion, I will let you know what I find out

Again, thank you.  I will award points appropriately as soon as I get the phones working again.

Blue Street TechLast KnightCommented:
Hi Charles Lam,

Have you looked into changing the MTU so that its most optimal, here's a quick guide: http://www.experts-exchange.com/A_12615-Unstable-Slow-Performing-Networks-or-VPNs-just-go-grocery-shopping.html

Let me know how it goes!
Charles LamAuthor Commented:
Thank you all so much for your comments!  I will be implementing your suggestions/additional diagnostics today and will get back.

nociSoftware EngineerCommented:
pchar .... another too to get an estimate of available bandwidth & loss.
it uses various packet sizes and it will estimate available bandwidth's beyond the first hop.
(it does take a while if the target is too far away (network hop count).

Charles LamAuthor Commented:
Just a quick update...  This issue has proved more intractable than I could have ever thought.  A CenturyLink Level 2 line guy was out yesterday and troubleshooted and cleaned the line (from the DSLAM or main distribution frame ?) to the box on the exterior of our office where the home run to our DSL modem connects.  No improvement in MTR.  He then had the brilliant idea of hard resetting the new DSL modem, thus taking it out of bridge mode and then we assigned our static to it.  I changed the config file for the WAN NIC in our IP PBX CentoOS to DHCP, and VOILA - ZERO packet loss to multiple hosts across the Internet, dramatically lower latency (~ 45 ms) and minimal jitter.  We then rebridged the DSL modem, set both and my laptop's NIC's to the static, and still got zero packet loss, though the latency and jitter was up a bit.  Finally, copied over original conf file back to the CentOS box for Eth0 (static) and re-ran MTR for a while - this time, I'm showing low percentage packet loss (~3%) and some latency and jitter (though not terrible) to host google.com (ran the MTR for about 15 min as suggested by Phonebuff)).

These results don't make any sense to me - why would applying the static at the modem/router be significantly better than applying the static to the Eth0 interface of the CentOS 4.4 (Final) IP PBX server?  I have not technical knowledge to explain this.  Any ideas?

I am in early this morning to replace the NIC in my IP PBX server with what I could pick up locally yesterday - a TRENDnet 10/100 Mbps PCI adapter.  I will order a better Intel NIC later today online, in the hopes it will be better (I hear Intel NIC's are the best).

Again, any thoughts are appreciated.  I had a third-party IT guy come in yesterday morning too, but he didn't come up with any ideas other than that our upload speed was too slow and that the resulting congestion was causing the packet loss and latency.  I explained to him that we had had the same DSL speeds for years and they have worked fine.

Thank you so much again for all your guys' patience while I continue to troubleshoot this problem.  As always, any explanation for what I described above, further insight or comments, or troubleshooting tips are always appreciated.

nociConnect With a Mentor Software EngineerCommented:
It may be possible that the software in the modem doesn't handle Bridge mode too well.
Most DSL modems are made to be used in a routing setup.  It can be using more CPU power in that mode due to inefficiencies in software or just run into bugs en recover from them at the cost of packet loss. There is too much that can interfere and not enough info to go on.

I ran a DSL setup like that for long times. I did use a setup up on line limiting the uplink speed to 10% below the maximum, just to prevent queueing in the modem device.
And it needed an extra route setup to make it work so that might have added a bit more overhead then a strait pass through. OTOH it didn;t need to filter packets.

The Settings i used were done using a script  called the wondershaper for Linux Routers
Original source: http://lartc.org/wondershaper/
Others do exist to accomodate for modern developments.  (Ubuntu forums).
The HTB method is the one with least overhead in passing on the packets and allows for (short) bursts of high volume traffic.
It favours short packets (ack, telnet) over large ones(ftp) and ones needing speed (interactive/time critical) over long haul (ftp) and leave some to the middle (http).

This does require you to have a single exit point before the modem...

ie.     FirewallRouter with Wondershaper --> DSLmodem --> DSL --> DSLAM...
And to connect other eq. from within the internal network.

If you cannot do that than you need to have the wondershaper limit bandwidth on all equipment to about 10% below the alloted maximum for that device where the sum of all  maxima is below the DSL speed.

The phenomenon is also known as Buffer Bloat. https://en.wikipedia.org/wiki/Bufferbloat

    It's been a few months since you last updated the thread.  Was wondering the status of your issues is .  

Charles LamAuthor Commented:
Dear Phonebuff & noci,

My sincere apologies for abandoning this thread.  Despite your very helpful suggestions, resources, and insight, the problem devolved (hardware started having issues, finding replacement parts, etc.).  I never resolved the issue or gained any further insight into the issue due to the hardware issues.  I ended up swallowing my pride and contacting a local reputable commercial telephone solutions company/Toshiba re-seller and we installed a PRI and Toshiba IPEdge and all new phones (as Toshiba didn't support, or didn't support very well, our Polycom IP 650's).

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.