issues with PBX

We have a internet and phone company hosting our PBX for our IP Phones. The way it currently works is i configure the phone with specific ip including tftp ip that reaches out to the phone company for config and calls are routed through them. For some reason, users can dial out but noone can dial in to reach them. Now, i recently upgraded the router, I don't see how this could cause the issue because I have at least 30 routers in the flee that I upgraded that don't have this issue. The configs are identical and nothing is missing. This is a 2811 router, and 7940 ip phone. All equipment is identical across the board. We're using SIP protocal.

I have reached out to ISP and they said make sure the following ports are open on my ASA but the traffic isn't going through the ASA, I dont have asa at that site. these are the ports they wanted me to open on the ASA just so you can review.

    Ports 5060 & 5061 UDP are used in both directions.
    Ports 69 UDP, 53 UDP, 123 UDP and 9001 UDP are used for outbound requests.
    Ports 10,000 - 20,000 UDP are used for inbound and outbound real-time voice traffic.

I have "no ip nat service sip udp port 5060" which should be on the router, if I delete this entry, in PBX site, I see the extensions unprovisioned. When I re-add the nat entry, It comes back up so this obviosly need to be here.

There is no VOIP configs at all on the router. This is the only router that's not working out of all 30 something. Called ISP phone company, they said everything looks normal. When I dial the ext. I get "The person at ext. is unavailable" FYI, DND is off, I have hard reset the phone to factory already.  Any help? Ideas?
Shark AttackNetwork adminAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

José MéndezCommented:
Please first clarify:
Your phone company hosts your PBX means, that the call processing software is in their control?
Do you know the make/model? Is it a Cisco CallManager system?
Those 30 gateways are not VOIP capable rigth? They just act as routers? Are we talking about 1 gateway per site?
Shark AttackNetwork adminAuthor Commented:
They're hosting the PBX but I have access to the call processing software part.  The gateways are not VoIP devices, just act as routers, correct. 1 gateway per site, I'm not sure which system their using. Their site is I don't think it's a cisco call manager, it doesn't look like it, I'm sure it's their own software they're using.
José MéndezCommented:
Thanks for the clarification. In such case, I don't see how you can correct this issue. You said yourself

and calls are routed through them.

That means that outbound calls (working properly) source from users in one of those 30 sites and are directed to the call processing software to a destination that actually connects to the public telephone network. You have to find out what device is connecting to the public network. I can imagine that the call processing software redirects to an IP/hostname that the IP Phone resolves and contacts on the Internet through the local router, such as  a SIP Provider, and then the SIP provider connects with another IP Voice provider or directly with a Central Office.

Why is this important? Because if we determine the outbound route, we probably determine the inbound route as well.

One way to do this is by collecting a packet capture from the ip phone as indicated here:

If you could send a screen shot of the call processing software you have access to, then maybe I can help a bit further.
Put Your Flow Data to Work

SolarWinds® Flow Tool Bundle combines three easy-to-download, easy-to-use flow analysis tools that can help you quickly distribute, test, and configure your flow traffic.

Shark AttackNetwork adminAuthor Commented:
It's their router connecting to private network

Does each location have their own interface?
What about port 5061 and the 10,000-20,000?
Those seem to be the likely culprits,

Do you have a VPN of sorts to the provider?
José MéndezCommented:
please collect a packet capture from a failed call as shown in the link so we can analyze the situation. Weĺl be able to pin point the characteristics of the audio, which is the component  affected here.
Willy, not sure one can capture info on the outgoing side while determining why an incoming call us not being received.
This is a hosted PBX, unless provides a way to capture data on their side.

Or a way to test from their interface I.e. Mimicking/originating a call ......

Are there multiple public IPs on this 2811?
Shark AttackNetwork adminAuthor Commented:
it was the upgrade that caused this issue, I downgraded the router back to 12.4 from 15.x and it works. I am not sure why its not working on this router while it works on all the other.  The configs are identical and obviously i used the same ios version.
Were all routers upgraded from the same version?

It might be that the up convert of the config for the new version did not go as needed ......
José MéndezCommented:
@arnold: the idea of collecting a phone side packet capture is to clarify if the phone is the source of the disconnection. If the new IOS is sending something that the ip phone does not like, we would be able to detect it in the capture. If the SIP traffic never arrives to the ip phone, then we can start looking up the ladder until we find where the SIP packets are.

@Zack: lets try with the phone capture. Since the gateways are not voice enabled, then we should be looking at routing only. Packets coming in, packets going out to the phone. Captures will bring in a lot of light in my opinion.
Shark AttackNetwork adminAuthor Commented:
i did the captures, i did not see anything there other then ESP packets. I did the captures on site where inbound and outbound calls work fine there was also nothing there but ESP packets.
Check the current config on the working site with the newer iOS version. Check what licenses and how.

If you have a tftp server setup, tftp the working config from that system.
Do those that work only have 5060 or they have both 5060 and 5061 in the no nat rule?

The TFTP copy can be modified with the IP adjustments before loading on the router after update to see if it was an issue with config changes during update.  I.e. The version of iOS on this version is such that you have to go through two update/upgrade cycles.
José MéndezCommented:
what do you mean by ESP packets?
José MéndezCommented:
You should have at least captured packets for the good call.
Shark AttackNetwork adminAuthor Commented:
i have no clue, I have been on a call with cisco and they found nothing, here is their response:

++ Can't enable any logs as there is no VOICE running on this box.
++ Took packet captures on the LAN and WAN interface.
++ Did not get anything except for ESP packets. ESP packets are vpn packets for our tunnel.

when i looked at the configs, they were identical after the upgrade. version and licensing the same as well. It is what it is, they're back in business after the downgrade. I will be looking at a different IOS version down the road.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
José MéndezCommented:
It would have been extremely helpful if you had answered arnold's question about VPNs involved in this scenario.

You can pretty much blame this problem to the VPN in my opinion.
Shark AttackNetwork adminAuthor Commented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
IP Telephony

From novice to tech pro — start learning today.