Sporadic audio on VOIP over VPN

I am trying to let our employees use our phone software at home.  Normally we do not do this, so I equipped them with a VPN login and the phone sofware and thought it would work.  They are able to make and receive calls, but not all calls have audio.  That is the strange part, it works some of the time, so I assume that my firewall/ vpn config is okay.  I have had more success with users on Verizon internet than on Comcast.

Is there a reason that RTP packets would flow in each direction for some calls but not others?  It seems like a routing or QoS problem.  It's very strange.  I can make 5 good calls and then 3 will fail.  I'm using an IPSEC vpn client and a softphone application.  It's seems like the SIP side (call setup) works just fine, but audio (RTP) comes and goes.  I tried lowering the firewall on my Comcast connection, but that didn't seem to help.

Thanks!