Unidirectionnal voice between IP phones and a BCM 400

I am currently experiencing unidirectional voice traffic between our main site and remote locations.  At the main site we have a Nortel BCM400 install and at each remote site has approximately 5-12 Nortel IP-2002 & 2004 phones install.  Each remote site has a 2.4Ghz wireless link back to our main tower site and from there a 20mbs link back to the main office. Each of the sites has a Cisco 2600 router install.  At the remote sites they have a Cisco POE switch and at the main site it has a catalyst switch.   All the routers and switches have been setup with VLANs and QOS.  
The problem I am facing is that two of the sites are experiencing unidirectional voice traffic.  The users can not hear the caller, but the caller can hear them.  The configurations for all the sites are the same and the router is handing out DHCP addresses to the phones.  I have checked the phones and they are getting the correct IP address for the voice VLAN.  This problem doesnt happen all the time, it comes and goes and usually clears itself up.  If it doesnt the user just reboots the phone and it is good to go.  Any help in this matter would be appreciated.