Poor One Way Audio Quality Via SIP Trunk
Dear Experts Exchange,
QUESTION SUMMARY: My office is having one-way audio quality issues to multiple remote parties over both incoming and outgoing calls. We in the office always hear the remote parties clearly. We are not experiencing dropped calls. Troubleshooting seems to point to our Internet connection via CenturyLink DSL. I am looking for feedback on the latest MTR reports from our IP PBX (I suggest the Excel file for easiest viewing) and whether you think they still indicate an issue, and if not what do you think is causing our problem? Suggestions for further troubleshooting steps are also greatly appreciated.
---
LONG EXPLANATION
I am in the 5th day of our business phone system essentially being down. We are experiencing variable yet usually unintelligible one-way audio quality issues with the remote party (i.e. not us in the office) on both inbound and outbound calls. We can always hear the remote party perfectly (call quality inbound to our office is very good). All outbound calls connect fine and all inbound calls to the best of our knowledge connect fine as well. We have not experienced any dropped calls - just very poor one-way audio with the remote party not being able to understand our garbled/dropped speech. Call quality is excellent extension to extension and extension to voice mail (i.e. all within our LAN). This phone system in its current configuration has been very stable for years (with the exception of one HDD failure in the server - was RAID 1 so not a real issue - and a serious hack a couple years ago).
I have called Fonality, whose product trixbox Pro we use as our local hosted IP PBX and also through whom we have a support contract, and they say everything looks good on our server and that it must be a carrier issue (e.g. our virtual SIP trunk provider 8x8). Fonality's tech support has been helpful and knowledgeable in the past. I have called 8x8 and they said they were not having any service issues. They placed a call directly from their data center to our IP PBX, essentially testing solely the network connection between our IP PBX and 8x8, and the call quality was similarly poor. This eliminates the likelihood that it is an upstream carrier issue. 8x8 Tier 2 support was very nice and suggested I run MTR's to their east and west coast data centers after power cycling all my equipment (which I had already done multiple times).
The MTR's (Matt's traceroute, AKA My traceroute) appeared to show significant packet loss and latency both across CenturyLink's network and Level3. I called CenturyLink and got the run around, but eventually got to a Tier 4 Supervisor who has been helping me. He said yesterday that their network ops center had done some reconfiguration and to test the WAN connection again. I have attached to this question the most recent MTR reports in both text format and MS Excel format (much easier to read) from the trixbox Pro server, which is running a slimmed down/custom version of CentOS 4.4 (Final). These MTR's were run via CLI/Bash on the IP PBX server, whose NIC is set to a static IP, connected via a short patch cord to a newly replaced (yesterday) CenturyLink ZyXEL PK5001Z DSL modem/wireless router in full bridge mode (wireless is of course disabled). I normally have a ZyXEL ZyWALL 2 Pro firewall in between the modem and server, set in transparent/bridge mode, serving only as a simple firewall, but removed it for the purpose of all MTR tests, both initial and after the changes CenturyLinks ops center supposedly made. There is no router, no switch, or other intervening device between the IP PBX server and the DSL modem. Our DSL line is dedicated to our phone system.
The MTR reports do show significant improvement in packet loss and latency since CenturyLink's ops center supposedly made their changes, but our one way audio issue to remote parties is still about as bad as it was before. For comparison, I have attached MTR reports to the same hosts as I tested from the IP PBX server, but this time using WinMTR on my laptop connected to our office LAN/switch/SonicWALL via Comcast Business cable. They actually look a lot worse than the current CenturyLink reports. Any thoughts on why this might be?
My question has two parts: what are your thoughts on the latest MTR reports attached hereto, and also what do you think could be causing our one-way audio quality issues to remote parties?
Thank you very much for any insights or thoughts, and I do apologize for the long explanation/question.
Gratefully,
Charles
MTR-Reports-from-trixbox-Pro-Ser.xlsx
google.com-mtr-Comcast.txt
208.67.222.222-mtr-Comcast.txt
208.67.220.220-mtr-Comcast.txt
205.171.203.226-mtr-Comcast.txt
205.171.2.226-mtr-Comcast.txt
192.84.16.23-mtr-Comcast.txt
8.28.0.59-mtr-Comcast.txt
8.28.0.59-mtr-raw.txt
192.84.16.23-mtr-raw.txt
205.171.2.226-mtr-raw.txt
205.171.203.226-mtr-raw.txt
208.67.220.220-mtr-raw.txt
208.67.222.222-mtr-raw.txt
google.com-mtr.txt
google.com-mtr-raw.txt
QUESTION SUMMARY: My office is having one-way audio quality issues to multiple remote parties over both incoming and outgoing calls. We in the office always hear the remote parties clearly. We are not experiencing dropped calls. Troubleshooting seems to point to our Internet connection via CenturyLink DSL. I am looking for feedback on the latest MTR reports from our IP PBX (I suggest the Excel file for easiest viewing) and whether you think they still indicate an issue, and if not what do you think is causing our problem? Suggestions for further troubleshooting steps are also greatly appreciated.
---
LONG EXPLANATION
I am in the 5th day of our business phone system essentially being down. We are experiencing variable yet usually unintelligible one-way audio quality issues with the remote party (i.e. not us in the office) on both inbound and outbound calls. We can always hear the remote party perfectly (call quality inbound to our office is very good). All outbound calls connect fine and all inbound calls to the best of our knowledge connect fine as well. We have not experienced any dropped calls - just very poor one-way audio with the remote party not being able to understand our garbled/dropped speech. Call quality is excellent extension to extension and extension to voice mail (i.e. all within our LAN). This phone system in its current configuration has been very stable for years (with the exception of one HDD failure in the server - was RAID 1 so not a real issue - and a serious hack a couple years ago).
I have called Fonality, whose product trixbox Pro we use as our local hosted IP PBX and also through whom we have a support contract, and they say everything looks good on our server and that it must be a carrier issue (e.g. our virtual SIP trunk provider 8x8). Fonality's tech support has been helpful and knowledgeable in the past. I have called 8x8 and they said they were not having any service issues. They placed a call directly from their data center to our IP PBX, essentially testing solely the network connection between our IP PBX and 8x8, and the call quality was similarly poor. This eliminates the likelihood that it is an upstream carrier issue. 8x8 Tier 2 support was very nice and suggested I run MTR's to their east and west coast data centers after power cycling all my equipment (which I had already done multiple times).
The MTR's (Matt's traceroute, AKA My traceroute) appeared to show significant packet loss and latency both across CenturyLink's network and Level3. I called CenturyLink and got the run around, but eventually got to a Tier 4 Supervisor who has been helping me. He said yesterday that their network ops center had done some reconfiguration and to test the WAN connection again. I have attached to this question the most recent MTR reports in both text format and MS Excel format (much easier to read) from the trixbox Pro server, which is running a slimmed down/custom version of CentOS 4.4 (Final). These MTR's were run via CLI/Bash on the IP PBX server, whose NIC is set to a static IP, connected via a short patch cord to a newly replaced (yesterday) CenturyLink ZyXEL PK5001Z DSL modem/wireless router in full bridge mode (wireless is of course disabled). I normally have a ZyXEL ZyWALL 2 Pro firewall in between the modem and server, set in transparent/bridge mode, serving only as a simple firewall, but removed it for the purpose of all MTR tests, both initial and after the changes CenturyLinks ops center supposedly made. There is no router, no switch, or other intervening device between the IP PBX server and the DSL modem. Our DSL line is dedicated to our phone system.
The MTR reports do show significant improvement in packet loss and latency since CenturyLink's ops center supposedly made their changes, but our one way audio issue to remote parties is still about as bad as it was before. For comparison, I have attached MTR reports to the same hosts as I tested from the IP PBX server, but this time using WinMTR on my laptop connected to our office LAN/switch/SonicWALL via Comcast Business cable. They actually look a lot worse than the current CenturyLink reports. Any thoughts on why this might be?
My question has two parts: what are your thoughts on the latest MTR reports attached hereto, and also what do you think could be causing our one-way audio quality issues to remote parties?
Thank you very much for any insights or thoughts, and I do apologize for the long explanation/question.
Gratefully,
Charles
MTR-Reports-from-trixbox-Pro-Ser.xlsx
google.com-mtr-Comcast.txt
208.67.222.222-mtr-Comcast.txt
208.67.220.220-mtr-Comcast.txt
205.171.203.226-mtr-Comcast.txt
205.171.2.226-mtr-Comcast.txt
192.84.16.23-mtr-Comcast.txt
8.28.0.59-mtr-Comcast.txt
8.28.0.59-mtr-raw.txt
192.84.16.23-mtr-raw.txt
205.171.2.226-mtr-raw.txt
205.171.203.226-mtr-raw.txt
208.67.220.220-mtr-raw.txt
208.67.222.222-mtr-raw.txt
google.com-mtr.txt
google.com-mtr-raw.txt
To what extent can your router support Qos and have VOIP delivered BEFORE any other traffic?
DSL mostly is rather asymetric, so while you have good quality (down speed) the upspeed is limited. and if you go over the limit some packet will get lost.
Also Routers normally prefer TCP traffic above UDP casing a webaccess/file transfer to intrude on the available BW on the uplink.
DSL mostly is rather asymetric, so while you have good quality (down speed) the upspeed is limited. and if you go over the limit some packet will get lost.
Also Routers normally prefer TCP traffic above UDP casing a webaccess/file transfer to intrude on the available BW on the uplink.
ASKER
Dear Phonebuff and noci,
Thank you so very much for your attention/comments/questio
Phonebuff
Thank you - I am very inexperienced with MTR - I will run 15 min traces. As for topology, I apologize if I was confusing in my original question. We have both Comcast cable Internet and CenturyLink DSL here at the office. I have two (2) distinct physical networks - one for the phone system via CenturyLink (exclusively) and one for the office in general (LAN, server, WiFi). My phone system topology is very simple - my trixbox Pro IP PBX has a patch cable going from it to a ZyXEL ZyWALL 2 Pro hardware firewall in bridge mode, then I have a patch cable going from the ZyWALL to the CenturyLink DSL modem (which I replaced Wednesday with a brand new one) in bridge mode as well. Nothing else is connected to the DSL modem or the ZyWALL. I then have a patch cord going from ETH1 to an HP POE switch, and individual Ethernet cords from there directly to all the phones in the office.
I included the MTR's labeled "Comcast" only as a point of comparison to the CenturyLink from my office. Curiously enough, they look even worse than CenturyLink.
I will try the two other virtual SIP trunk providers you mention if I cannot get this resolved another way today. 8x8 claims they have no issues. Calls always connect, it is just the one-way audio issue.
noci
As noted above in my response to Phonebuff, there is no active router between my trixbox Pro IP PBX and the CenturyLink DSL modem. There is the ZyZEL ZyWALL 2 Pro hardware firewall/router, but it is in bridge mode and only acts as a transparent firewall, limiting connections/ports/protocol
All
I hope my clarifications above help - again, I REALLY appreciate any thoughts you have on further troubleshooting. Phonebuff - I will run longer MTR's today, but I take it from your comment that the MTR's I have attached to my question pose a problem?
THANK YOU!
Charles
Thank you so very much for your attention/comments/questio
Phonebuff
Thank you - I am very inexperienced with MTR - I will run 15 min traces. As for topology, I apologize if I was confusing in my original question. We have both Comcast cable Internet and CenturyLink DSL here at the office. I have two (2) distinct physical networks - one for the phone system via CenturyLink (exclusively) and one for the office in general (LAN, server, WiFi). My phone system topology is very simple - my trixbox Pro IP PBX has a patch cable going from it to a ZyXEL ZyWALL 2 Pro hardware firewall in bridge mode, then I have a patch cable going from the ZyWALL to the CenturyLink DSL modem (which I replaced Wednesday with a brand new one) in bridge mode as well. Nothing else is connected to the DSL modem or the ZyWALL. I then have a patch cord going from ETH1 to an HP POE switch, and individual Ethernet cords from there directly to all the phones in the office.
I included the MTR's labeled "Comcast" only as a point of comparison to the CenturyLink from my office. Curiously enough, they look even worse than CenturyLink.
I will try the two other virtual SIP trunk providers you mention if I cannot get this resolved another way today. 8x8 claims they have no issues. Calls always connect, it is just the one-way audio issue.
noci
As noted above in my response to Phonebuff, there is no active router between my trixbox Pro IP PBX and the CenturyLink DSL modem. There is the ZyZEL ZyWALL 2 Pro hardware firewall/router, but it is in bridge mode and only acts as a transparent firewall, limiting connections/ports/protocol
All
I hope my clarifications above help - again, I REALLY appreciate any thoughts you have on further troubleshooting. Phonebuff - I will run longer MTR's today, but I take it from your comment that the MTR's I have attached to my question pose a problem?
THANK YOU!
Charles
Charles,
Run I tun an MTR to my provider over a 15 minutes period I look for < 5% error anything higher tends to result in quality issues.
I am using Comcast Business here --
An I watching a MTR from my office to yahoo.com, 21 hops and except for one host that is ignoring the ICMP request I have 2% loss on one host an 0% on all others with 300 packets sent..
Going to 8X8.com I am 19 hops away and I am seeing approximately the same thing with 19 hops.
You might try this -- http://www.8x8.com/Resources/Tools/VoIPTest.aspx
I would also consider putting a wireshark on your network, let it collect data for a while and then look at the traffic source & end IPs to ensure you are carrying what you think you are.
Run I tun an MTR to my provider over a 15 minutes period I look for < 5% error anything higher tends to result in quality issues.
I am using Comcast Business here --
An I watching a MTR from my office to yahoo.com, 21 hops and except for one host that is ignoring the ICMP request I have 2% loss on one host an 0% on all others with 300 packets sent..
Going to 8X8.com I am 19 hops away and I am seeing approximately the same thing with 19 hops.
You might try this -- http://www.8x8.com/Resources/Tools/VoIPTest.aspx
I would also consider putting a wireshark on your network, let it collect data for a while and then look at the traffic source & end IPs to ensure you are carrying what you think you are.
ASKER
OK, thank you Phonebuff for the benchmarks! I will try to wireshark my traffic on the DSL line and see what's up... good suggestion, I will let you know what I find out
Again, thank you. I will award points appropriately as soon as I get the phones working again.
Sincerely,
Charles
Again, thank you. I will award points appropriately as soon as I get the phones working again.
Sincerely,
Charles
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Hi Charles Lam,
Have you looked into changing the MTU so that its most optimal, here's a quick guide: https://www.experts-exchange.com/A_12615-Unstable-Slow-Performing-Networks-or-VPNs-just-go-grocery-shopping.html
Let me know how it goes!
Have you looked into changing the MTU so that its most optimal, here's a quick guide: https://www.experts-exchange.com/A_12615-Unstable-Slow-Performing-Networks-or-VPNs-just-go-grocery-shopping.html
Let me know how it goes!
ASKER
Thank you all so much for your comments! I will be implementing your suggestions/additional diagnostics today and will get back.
Sincerely,
Charles
Sincerely,
Charles
pchar .... another too to get an estimate of available bandwidth & loss.
it uses various packet sizes and it will estimate available bandwidth's beyond the first hop.
(it does take a while if the target is too far away (network hop count).
http://www.kitchenlab.org/www/bmah/Software/pchar/
it uses various packet sizes and it will estimate available bandwidth's beyond the first hop.
(it does take a while if the target is too far away (network hop count).
http://www.kitchenlab.org/www/bmah/Software/pchar/
ASKER
Just a quick update... This issue has proved more intractable than I could have ever thought. A CenturyLink Level 2 line guy was out yesterday and troubleshooted and cleaned the line (from the DSLAM or main distribution frame ?) to the box on the exterior of our office where the home run to our DSL modem connects. No improvement in MTR. He then had the brilliant idea of hard resetting the new DSL modem, thus taking it out of bridge mode and then we assigned our static to it. I changed the config file for the WAN NIC in our IP PBX CentoOS to DHCP, and VOILA - ZERO packet loss to multiple hosts across the Internet, dramatically lower latency (~ 45 ms) and minimal jitter. We then rebridged the DSL modem, set both and my laptop's NIC's to the static, and still got zero packet loss, though the latency and jitter was up a bit. Finally, copied over original conf file back to the CentOS box for Eth0 (static) and re-ran MTR for a while - this time, I'm showing low percentage packet loss (~3%) and some latency and jitter (though not terrible) to host google.com (ran the MTR for about 15 min as suggested by Phonebuff)).
These results don't make any sense to me - why would applying the static at the modem/router be significantly better than applying the static to the Eth0 interface of the CentOS 4.4 (Final) IP PBX server? I have not technical knowledge to explain this. Any ideas?
I am in early this morning to replace the NIC in my IP PBX server with what I could pick up locally yesterday - a TRENDnet 10/100 Mbps PCI adapter. I will order a better Intel NIC later today online, in the hopes it will be better (I hear Intel NIC's are the best).
Again, any thoughts are appreciated. I had a third-party IT guy come in yesterday morning too, but he didn't come up with any ideas other than that our upload speed was too slow and that the resulting congestion was causing the packet loss and latency. I explained to him that we had had the same DSL speeds for years and they have worked fine.
Thank you so much again for all your guys' patience while I continue to troubleshoot this problem. As always, any explanation for what I described above, further insight or comments, or troubleshooting tips are always appreciated.
Respectfully,
Charles
These results don't make any sense to me - why would applying the static at the modem/router be significantly better than applying the static to the Eth0 interface of the CentOS 4.4 (Final) IP PBX server? I have not technical knowledge to explain this. Any ideas?
I am in early this morning to replace the NIC in my IP PBX server with what I could pick up locally yesterday - a TRENDnet 10/100 Mbps PCI adapter. I will order a better Intel NIC later today online, in the hopes it will be better (I hear Intel NIC's are the best).
Again, any thoughts are appreciated. I had a third-party IT guy come in yesterday morning too, but he didn't come up with any ideas other than that our upload speed was too slow and that the resulting congestion was causing the packet loss and latency. I explained to him that we had had the same DSL speeds for years and they have worked fine.
Thank you so much again for all your guys' patience while I continue to troubleshoot this problem. As always, any explanation for what I described above, further insight or comments, or troubleshooting tips are always appreciated.
Respectfully,
Charles
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Charles,
It's been a few months since you last updated the thread. Was wondering the status of your issues is .
===
It's been a few months since you last updated the thread. Was wondering the status of your issues is .
===
ASKER
Dear Phonebuff & noci,
My sincere apologies for abandoning this thread. Despite your very helpful suggestions, resources, and insight, the problem devolved (hardware started having issues, finding replacement parts, etc.). I never resolved the issue or gained any further insight into the issue due to the hardware issues. I ended up swallowing my pride and contacting a local reputable commercial telephone solutions company/Toshiba re-seller and we installed a PRI and Toshiba IPEdge and all new phones (as Toshiba didn't support, or didn't support very well, our Polycom IP 650's).
Respectfully,
Charles
My sincere apologies for abandoning this thread. Despite your very helpful suggestions, resources, and insight, the problem devolved (hardware started having issues, finding replacement parts, etc.). I never resolved the issue or gained any further insight into the issue due to the hardware issues. I ended up swallowing my pride and contacting a local reputable commercial telephone solutions company/Toshiba re-seller and we installed a PRI and Toshiba IPEdge and all new phones (as Toshiba didn't support, or didn't support very well, our Polycom IP 650's).
Respectfully,
Charles
Not sure i understand the topology, you said you were talking to Centurylink, but you have Comcast traces. Be sure your router is sending packets outbound on the correct / ISP.
Have you tried another provider IPComms or Vitality in case the issue is 8X8 ?