Solved

Poor One Way Audio Quality Via SIP Trunk

Posted on 2014-02-06
13
294 Views
Last Modified: 2015-06-17
Dear Experts Exchange,

QUESTION SUMMARY: My office is having one-way audio quality issues to multiple remote parties over both incoming and outgoing calls.  We in the office always hear the remote parties clearly.  We are not experiencing dropped calls.  Troubleshooting seems to point to our Internet connection via CenturyLink DSL.  I am looking for feedback on the latest MTR reports from our IP PBX (I suggest the Excel file for easiest viewing) and whether you think they still indicate an issue, and if not what do you think is causing our problem?  Suggestions for further troubleshooting steps are also greatly appreciated.

---

LONG EXPLANATION

I am in the 5th day of our business phone system essentially being down.  We are experiencing variable yet usually unintelligible one-way audio quality issues with the remote party (i.e. not us in the office) on both inbound and outbound calls.  We can always hear the remote party perfectly (call quality inbound to our office is very good).  All outbound calls connect fine and all inbound calls to the best of our knowledge connect fine as well.  We have not experienced any dropped calls - just very poor one-way audio with the remote party not being able to understand our garbled/dropped speech.  Call quality is excellent extension to extension and extension to voice mail (i.e. all within our LAN).  This phone system in its current configuration has been very stable for years (with the exception of one HDD failure in the server - was RAID 1 so not a real issue - and a serious hack a couple years ago).

I have called Fonality, whose product trixbox Pro we use as our local hosted IP PBX and also through whom we have a support contract, and they say everything looks good on our server and that it must be a carrier issue (e.g. our virtual SIP trunk provider 8x8).  Fonality's tech support has been helpful and knowledgeable in the past.  I have called 8x8 and they said they were not having any service issues.  They placed a call directly from their data center to our IP PBX, essentially testing solely the network connection between our IP PBX and 8x8, and the call quality was similarly poor.  This eliminates the likelihood that it is an upstream carrier issue.  8x8 Tier 2 support was very nice and suggested I run MTR's to their east and west coast data centers after power cycling all my equipment (which I had already done multiple times).

The MTR's (Matt's traceroute, AKA My traceroute) appeared to show significant packet loss and latency both across CenturyLink's network and Level3.  I called CenturyLink and got the run around, but eventually got to a Tier 4 Supervisor who has been helping me.  He said yesterday that their network ops center had done some reconfiguration and to test the WAN connection again.  I have attached to this question the most recent MTR reports in both text format and MS Excel format (much easier to read) from the trixbox Pro server, which is running a slimmed down/custom version of CentOS 4.4 (Final).  These MTR's were run via CLI/Bash on the IP PBX server, whose NIC is set to a static IP, connected via a short patch cord to a newly replaced (yesterday) CenturyLink ZyXEL PK5001Z DSL modem/wireless router in full bridge mode (wireless is of course disabled).  I normally have a ZyXEL ZyWALL 2 Pro firewall in between the modem and server, set in transparent/bridge mode, serving only as a simple firewall, but removed it for the purpose of all MTR tests, both initial and after the changes CenturyLinks ops center supposedly made.  There is no router, no switch, or other intervening device between the IP PBX server and the DSL modem.  Our DSL line is dedicated to our phone system.

The MTR reports do show significant improvement in packet loss and latency since CenturyLink's ops center supposedly made their changes, but our one way audio issue to remote parties is still about as bad as it was before.  For comparison, I have attached MTR reports to the same hosts as I tested from the IP PBX server, but this time using WinMTR on my laptop connected to our office LAN/switch/SonicWALL via Comcast Business cable.  They actually look a lot worse than the current CenturyLink reports.  Any thoughts on why this might be?

My question has two parts: what are your thoughts on the latest MTR reports attached hereto, and also what do you think could be causing our one-way audio quality issues to remote parties?

Thank you very much for any insights or thoughts, and I do apologize for the long explanation/question.

Gratefully,
Charles
MTR-Reports-from-trixbox-Pro-Ser.xlsx
google.com-mtr-Comcast.txt
208.67.222.222-mtr-Comcast.txt
208.67.220.220-mtr-Comcast.txt
205.171.203.226-mtr-Comcast.txt
205.171.2.226-mtr-Comcast.txt
192.84.16.23-mtr-Comcast.txt
8.28.0.59-mtr-Comcast.txt
8.28.0.59-mtr-raw.txt
192.84.16.23-mtr-raw.txt
205.171.2.226-mtr-raw.txt
205.171.203.226-mtr-raw.txt
208.67.220.220-mtr-raw.txt
208.67.222.222-mtr-raw.txt
google.com-mtr.txt
google.com-mtr-raw.txt
0
Comment
Question by:Charles Lam
  • 5
  • 4
  • 3
  • +1
13 Comments
 
LVL 15

Expert Comment

by:Phonebuff
ID: 39841800
First off, yes high packet loss in you UDP audio connection is probably the cause of you voice issues.  Next, MTR really needs to run about 15 minutes or so for the data to be meaningfull, but you definitely have issues in these routes.  

Not sure i understand the topology, you said you were talking to Centurylink, but you have Comcast traces.  Be sure your router is sending packets outbound on the correct / ISP.  

Have you tried another provider IPComms or Vitality in case the issue is 8X8 ?
0
 
LVL 39

Expert Comment

by:noci
ID: 39842230
To what extent can your router support Qos and have VOIP delivered BEFORE any other traffic?

DSL mostly is rather asymetric, so while you have good quality (down speed) the upspeed is limited.  and if you go over the limit some packet will get lost.
Also Routers normally prefer TCP traffic above UDP casing a webaccess/file transfer to intrude on the available BW on the uplink.
0
 

Author Comment

by:Charles Lam
ID: 39842488
Dear Phonebuff and noci,

Thank you so very much for your attention/comments/questions to my issue.  I will address the issues you raised below in order of their appearance:

Phonebuff

Thank you - I am very inexperienced with MTR - I will run 15 min traces.  As for topology, I apologize if I was confusing in my original question.  We have both Comcast cable Internet and CenturyLink DSL here at the office.  I have two (2) distinct physical networks - one for the phone system via CenturyLink (exclusively) and one for the office in general (LAN, server, WiFi).  My phone system topology is very simple - my trixbox Pro IP PBX has a patch cable going from it to a ZyXEL ZyWALL 2 Pro hardware firewall in bridge mode, then I have a patch cable going from the ZyWALL to the CenturyLink DSL modem (which I replaced Wednesday with a brand new one) in bridge mode as well.  Nothing else is connected to the DSL modem or the ZyWALL.  I then have a patch cord going from ETH1 to an HP POE switch, and individual Ethernet cords from there directly to all the phones in the office.

I included the MTR's labeled "Comcast" only as a point of comparison to the CenturyLink from my office.  Curiously enough, they look even worse than CenturyLink.

I will try the two other virtual SIP trunk providers you mention if I cannot get this resolved another way today.  8x8 claims they have no issues.  Calls always connect, it is just the one-way audio issue.

noci

As noted above in my response to Phonebuff, there is no active router between my trixbox Pro IP PBX and the CenturyLink DSL modem.  There is the ZyZEL ZyWALL 2 Pro hardware firewall/router, but it is in bridge mode and only acts as a transparent firewall, limiting connections/ports/protocols to only those that are absolutely needed by the trixbox Pro IP PBX.  I have had this firewall in place for several years without issue and have not made any firmware or configuration changes to it.  All the MTR reports attached to my question were taken without the firewall installed, with a direct connection between my IP PBX and the CenturyLink modem in bridge mode.  The only traffic that goes through my CenturyLink DSL connection is traffic to and from my PBX.  This is mostly SIP traffic, with a little VPN action between the PBX and the web admin console, our "HUD" software interfaces on our PC's, and VPN traffic to/from Fonality when I call them for support.

All

I hope my clarifications above help - again, I REALLY appreciate any thoughts you have on further troubleshooting.  Phonebuff - I will run longer MTR's today, but I take it from your comment that the MTR's I have attached to my question pose a problem?

THANK YOU!
Charles
0
 
LVL 15

Expert Comment

by:Phonebuff
ID: 39842558
Charles,

    Run I tun an MTR to my provider over a 15 minutes period I look for < 5% error anything higher tends to result in quality issues.  

    I am using Comcast Business here --

   An I watching a MTR from my office to yahoo.com, 21 hops and except for one host that is ignoring the ICMP request I have 2% loss on one host an 0% on all others with 300 packets sent..

    Going to 8X8.com I am 19 hops away and I am seeing approximately the same thing with 19 hops.

You might try this -- http://www.8x8.com/Resources/Tools/VoIPTest.aspx

I would also consider putting a wireshark on your network, let it collect data for a while and then look at the traffic source & end IPs to ensure you are carrying what you think you are.
0
 

Author Comment

by:Charles Lam
ID: 39843173
OK, thank you Phonebuff for the benchmarks!  I will try to wireshark my traffic on the DSL line and see what's up...  good suggestion, I will let you know what I find out

Again, thank you.  I will award points appropriately as soon as I get the phones working again.

Sincerely,
Charles
0
 
LVL 15

Accepted Solution

by:
Phonebuff earned 250 total points
ID: 39843239
One other idea --  You should read this thread, I have used John's Redneck QOS before on ADSL links with good luck.  It's important to note the QOS does not go beyond your router / firewall unless you use someone like CBeyond or One Ring where you are on net until they gateway the IP stream to the Internet after the phone switch.

http://pbxinaflash.com/community/index.php?threads/value-of-inbound-qos.9208/

Couple of other things, if you are happy with 8X8 maybe just move them with a good firewall / router (pfSense is my favorite) to the Comcast link.   You cold also check your CODEC, maybe going to one like GSM would use less bandwith and give you better call quality.

It really sounds like this might be a bandwith / link quality issue if it's on ADSL and the inbound leg of the call is good, but the outbound leg is getting complaints.

======================

http://www.bandcalc.com/
0
Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 24

Expert Comment

by:diverseit
ID: 39846991
Hi Charles Lam,

Have you looked into changing the MTU so that its most optimal, here's a quick guide: http://www.experts-exchange.com/A_12615-Unstable-Slow-Performing-Networks-or-VPNs-just-go-grocery-shopping.html

Let me know how it goes!
0
 

Author Comment

by:Charles Lam
ID: 39847386
Thank you all so much for your comments!  I will be implementing your suggestions/additional diagnostics today and will get back.

Sincerely,
Charles
0
 
LVL 39

Expert Comment

by:noci
ID: 39848263
pchar .... another too to get an estimate of available bandwidth & loss.
it uses various packet sizes and it will estimate available bandwidth's beyond the first hop.
(it does take a while if the target is too far away (network hop count).

 http://www.kitchenlab.org/www/bmah/Software/pchar/
0
 

Author Comment

by:Charles Lam
ID: 39852957
Just a quick update...  This issue has proved more intractable than I could have ever thought.  A CenturyLink Level 2 line guy was out yesterday and troubleshooted and cleaned the line (from the DSLAM or main distribution frame ?) to the box on the exterior of our office where the home run to our DSL modem connects.  No improvement in MTR.  He then had the brilliant idea of hard resetting the new DSL modem, thus taking it out of bridge mode and then we assigned our static to it.  I changed the config file for the WAN NIC in our IP PBX CentoOS to DHCP, and VOILA - ZERO packet loss to multiple hosts across the Internet, dramatically lower latency (~ 45 ms) and minimal jitter.  We then rebridged the DSL modem, set both and my laptop's NIC's to the static, and still got zero packet loss, though the latency and jitter was up a bit.  Finally, copied over original conf file back to the CentOS box for Eth0 (static) and re-ran MTR for a while - this time, I'm showing low percentage packet loss (~3%) and some latency and jitter (though not terrible) to host google.com (ran the MTR for about 15 min as suggested by Phonebuff)).

These results don't make any sense to me - why would applying the static at the modem/router be significantly better than applying the static to the Eth0 interface of the CentOS 4.4 (Final) IP PBX server?  I have not technical knowledge to explain this.  Any ideas?

I am in early this morning to replace the NIC in my IP PBX server with what I could pick up locally yesterday - a TRENDnet 10/100 Mbps PCI adapter.  I will order a better Intel NIC later today online, in the hopes it will be better (I hear Intel NIC's are the best).

Again, any thoughts are appreciated.  I had a third-party IT guy come in yesterday morning too, but he didn't come up with any ideas other than that our upload speed was too slow and that the resulting congestion was causing the packet loss and latency.  I explained to him that we had had the same DSL speeds for years and they have worked fine.

Thank you so much again for all your guys' patience while I continue to troubleshoot this problem.  As always, any explanation for what I described above, further insight or comments, or troubleshooting tips are always appreciated.

Respectfully,
Charles
0
 
LVL 39

Assisted Solution

by:noci
noci earned 250 total points
ID: 39853091
It may be possible that the software in the modem doesn't handle Bridge mode too well.
Most DSL modems are made to be used in a routing setup.  It can be using more CPU power in that mode due to inefficiencies in software or just run into bugs en recover from them at the cost of packet loss. There is too much that can interfere and not enough info to go on.

I ran a DSL setup like that for long times. I did use a setup up on line limiting the uplink speed to 10% below the maximum, just to prevent queueing in the modem device.
And it needed an extra route setup to make it work so that might have added a bit more overhead then a strait pass through. OTOH it didn;t need to filter packets.

The Settings i used were done using a script  called the wondershaper for Linux Routers
Original source: http://lartc.org/wondershaper/
Others do exist to accomodate for modern developments.  (Ubuntu forums).
The HTB method is the one with least overhead in passing on the packets and allows for (short) bursts of high volume traffic.
It favours short packets (ack, telnet) over large ones(ftp) and ones needing speed (interactive/time critical) over long haul (ftp) and leave some to the middle (http).

This does require you to have a single exit point before the modem...

ie.     FirewallRouter with Wondershaper --> DSLmodem --> DSL --> DSLAM...
And to connect other eq. from within the internal network.

If you cannot do that than you need to have the wondershaper limit bandwidth on all equipment to about 10% below the alloted maximum for that device where the sum of all  maxima is below the DSL speed.

The phenomenon is also known as Buffer Bloat. https://en.wikipedia.org/wiki/Bufferbloat
0
 
LVL 15

Expert Comment

by:Phonebuff
ID: 40079498
Charles,

    It's been a few months since you last updated the thread.  Was wondering the status of your issues is .  

  ===
0
 

Author Closing Comment

by:Charles Lam
ID: 40835063
Dear Phonebuff & noci,

My sincere apologies for abandoning this thread.  Despite your very helpful suggestions, resources, and insight, the problem devolved (hardware started having issues, finding replacement parts, etc.).  I never resolved the issue or gained any further insight into the issue due to the hardware issues.  I ended up swallowing my pride and contacting a local reputable commercial telephone solutions company/Toshiba re-seller and we installed a PRI and Toshiba IPEdge and all new phones (as Toshiba didn't support, or didn't support very well, our Polycom IP 650's).

Respectfully,
Charles
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

As dyndns has reduced the capabilities of the free service, I looked around for other free providers of Dynamic DNS service. After testing several I decided to move my DNS hosting to Hurricane Electric as then domains that require dynamic hostnam…
Implementing Avaya's One-X portal is pretty painless, until you want to deploy this to the Android and iPhone clients when these clients are outside of your network. The clients will also work within your local network. Here is our experience and so…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now