We're currently experiencing a strange issue at one of our sites (site 2 in this example) that consists of only 2 employees. Here is a brief summary of our environment.
Main site - hosts a single CUCM server and unity server. We have two additional sites that use the VoIP system with cisco routers tied to POTS lines as a fallback service in case they drop connectivity.
Site 1 (large site): is connected via a dedicated circuit. T1 - no issues.
Site 2 (small site): is connected via a VPN tunnel over the cloud (I know, I know).
As of two weeks ago, Site 2 began having issues with intermittent dropped calls. Several times a day, an employee will pick up the phone (to answer a call) and the hears a slight click and notices the call was lost. His phone (7940) then displays "cm fallback service operating" or something similar for about 30 seconds and comes back online. Each phone at this site does this.
Troubleshooting done thus far:
1) Monitored bandwidth to ensure that was not a problem. They aren't even close to maxing their bandwidth. I noticed 300kbps up spikes but this isn't anywhere near the 3mbps up max we purchased.
2) Our SNMP software and the ISP show connectivity has been very stable between sites. I've also run a continuous ping for a couple days and only a few packets have dropped.
3) Our Cisco engineer (outsource hire) doesn't see anything strange on the router/ASA.
4) Power cycled the modem, router, ASA (firewall), and switch.
5) Before I was informed both employees were having issues, I tried swapping phones, new cables, etc.
No changes have been made except the POTS provider did have an outage in the area around the time these problems began. At the time, 5 of our 6 pots lines were out of service. They are telling us the lines are functioning properly at this point. Outgoing calls never seem to drop, only incoming thus far. We've ensured all lines are working by simply calling each line directly.
Any ideas on what we should try next?