Asterisk one way audio on random calls

I have a new FreePBX/Asterisk install.  We have approx 35 Snom phones and are using a Audio Codes T1 Gateway.  What happens is 4-5 times a week we will have a call come in where the outside person can hear us but we cannot hear them it appears to be dead air to us, so we hang up and the outside person will just call back in and get through...
We cannot seem to nail down a reason this is happening.

The only seen one thing i wasnt sure what it was and it happened around the same time.
SIP/29041-000021d4 connected line has changed. Saving it until answer for SIP/AudioCodes-000021d1

I have attached the log file of when the issue happened.  The log is from asterisk and from the audio codes gateway.
Incoming call was from 612-916-7443 and the phone that answered the call was 2904 that got the dead air.  Call was around 11:08.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

José MéndezCommented:
You are providing very good information mate, but unfortunately the logs you have won help us much.

This is what I see:

Jan 06 11:09:07 (   lgr_psbrdex)(5378955   )  pstn recv <-- INCOMING_CALL (src:6129167443 dst:2413 redirect1: redirect2:) Trunk:0 Conn:254 BChannel:22 OffhookInd:0

Jan 06 11:09:12 SIP/2.0 200 OK  Via: SIP/2.0/UDP;branch=z9hG4bKac2058463786;received=  From: "MINNESOTA CALL " <sip:6129167443@>;tag=1c2058454348  To: <sip:2413@;user=phone>;tag=as28c8e737  Call-ID: 2058453356282200019614@  CSeq: 1 INVITE  Server: FPBX-2.9.0(  Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY, INFO, PUBLISH  Supported: replaces, timer  Contact: <sip:2413@>  Content-Type: application/sdp  Content-Length: 237    v=0  o=root 2144179463 2144179463 IN IP4  s=Asterisk PBX  c=IN IP4  t=0 0  m=audio 13950 RTP/AVP 0 96  a=rtpmap:0 PCMU/8000  a=rtpmap:96 telephone-event/8000  a=fmtp:96 0-16  a=ptime:20  a=sendrecv  

Jan 06 11:09:30 BYE sip:6129167443@ SIP/2.0  Via: SIP/2.0/UDP;branch=z9hG4bK35c6e5a5  Max-Forwards: 70  From: <sip:2413@;user=phone>;tag=as28c8e737  To: "MINNESOTA CALL " <sip:6129167443@>;tag=1c2058454348  Call-ID: 2058453356282200019614@  CSeq: 102 BYE  User-Agent: FPBX-2.9.0(  X-Asterisk-HangupCause: Normal Clearing  X-Asterisk-HangupCauseCode: 16  Content-Length: 0

Thats all I can relate to the call you described at 11:08 or around.

After that there is just one more call that lasted 1:22 minutes. The funny thing is, that I don't see a complete set of SIP signaling between your Asterisk and your gateway.

We weill need 2 things here: a good SIP debug and a packet capture. Do you know if you PBX is configured to re-invite the phone and get out of the way in terms of RTP, or will the phone and gateway send their audio to Asterisk?
tsukrawAuthor Commented:
I actually do have the wireshark capture from between the phone and the network that goes along with the log files i posted earlier.  I will upload it.  EE wouldnt let me upload a pcap so i changed it to txt just be sure to change it back to pcap to be able to see it all.
José MéndezCommented:
Nice but dun forget the IP addresses involved, and please clarify the answering extension number
Put Your Flow Data to Work

SolarWinds® Flow Tool Bundle combines three easy-to-download, easy-to-use flow analysis tools that can help you quickly distribute, test, and configure your flow traffic.

tsukrawAuthor Commented:
The answering extension is 2904 i had that already listed and the call was from 612-916-7443.
Phone IP:
Asterisk IP:
AudioCodes Gateway IP:
José MéndezCommented:
Yes I saw you listed 2904, but the captures I read pointed to 2413 instead
tsukrawAuthor Commented:
2413 is the ring group.  2904 is the extension inside the ring group that actually answered the phone.
José MéndezCommented:
Man you one weird issue going on.

The call we are after starts at packet 29085

29085      11:07:53.773691      SIP/SDP      Request: INVITE sip:29041@;line=tv8hu305, with session description

From: "FF:MINNESOTA CALL" <sip:6129167443@>;tag=as33bd0a93

Once Wendy answers, I can see a continuous stream of RTP packets towards Asterisk, but not the other way around, until a few seconds later. Here are the 2 audio files I could decode. Note how the sound from the caller is low volume but it is there, as if the call was being monitored. Open them with media player  from-caller.png from-wendy.png
tsukrawAuthor Commented:
So where do you think the issue is possibly at? This is a completely new install and from day one we have had this issue.  Do you know audio Codes units very much?  I could upload the config file from that if that helps at all
José MéndezCommented:
No I don't, I know Asterisk, VOIP protocols and specialize in SIP. if you can, a sniffer capture from Asterisk itself during this bad call will be helpful to narrow down the problem to Asterisk or the gateway.

This is the syntax:

tcpdump -s 0 -w /tmp/onewayaudio.cap

Once we determine if the RTP packets don't arrive to Asterisk at all, then we can focus on the gateway. It may be a DSP issue or a code bug.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
tsukrawAuthor Commented:
Alright i will see what i can do....
Question: does that command make like a never ending cap file?
since the issue only happens a 4-5 times a week it might have to run for a while before the issue happens?

or is there a way to set a size limit say of like 200mb and have it just overwrite its self when it hits that limit that way i dont have like a GBs in size file that i have to try to pick through.
José MéndezCommented:

tcpdump -s 0 -w /path/to/file.cap -C 50 -W 10

where the value after -C representes the maximum allowed size in megabytes for each capture, and the one after W the max amount of files

Be sure to get enough files or to get a report on the problem quickly enough.
tsukrawAuthor Commented:
Sounds good.  I will run the command tomorrow.  Do i need to leave putty open or anything or once the command i running it will stay running until i stop it?
José MéndezCommented:
hum,, no, if you close the putty session the command dies with it I think you may have to prefix a nohup to the command and suffix &

nohup tcpdump -s 0 -w /path/to/file.cap -C 50 -W 10 &

but be careful I haven fully tested that and it may become hard to kill afterward. I would rather leave it on during working hours
tsukrawAuthor Commented:
I am able to leave the putty session open so i will go that route.
The capture is running.  Do you want me to just upload the chunk that contains the data when the issue happened?
José MéndezCommented:
Exactly, because the capture will show me the sip signaling and RTP situation both.

You may want to narrow down the information captured like this:
tcpdump -s 0 -w /path/to/file.cap -C 50 -W 10 ip host <ip address of the mediacodes gw>

That will be easier.

How many SIP devices you have? How many concurrent calls either internal or external (including voicemail checks) are there? And also what hardware platform supports them?
tsukrawAuthor Commented:
Ok i am now running....
tcpdump -s 0 -w /tmp/logs/audioissue.cap -C 50 -W 10 ip host

We have around 40 SNOM 370 phones and a couple PA units. There is almost never more then 5 active external calls and 5 active internal calls.  (total of 10 active calls at once).

Asterisk is running on our VMware server.
José MéndezCommented:
Interesting, the virtualization layer adds one extra failure point, although unlikely. The system should run fine the capture with that load then.
tsukrawAuthor Commented:
Yea we are running multiple systems off VMware and honestly have never had any issues.  I have read stuff online strongly recommending against it but we have never seen any issues.
tsukrawAuthor Commented:
Well since the loggin has been running we have not had the issue.  Go figure...I will post once it happens though it might be a day or to still..
Salah Eddine ELMRABETTechnical Lead Manager (Owner)Commented:
Hi tsukraw,

I the call between IP and PSTN or IP to IP, maybe some where codecs are misconfigured!!

Witch codecs did you configure on the Phone, GW, PBX...

tsukrawAuthor Commented:
was able to determine a firmware update was needed on the gateway from the capture ran on the interface.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Voice Over IP

From novice to tech pro — start learning today.