SBS 2003 Remote Web Workplace Latency

We have a SBS 2003 network with 1 server and about 10 workstations.  Two of our employees connect to 2 of the workstations using Remote Web Workplace.  Frequently, the remote users experience several seconds of latency during the connection where the screen hangs and doesn't refresh itself.  After a few seconds it will correct itself, but it makes working remotely very difficult.  I've run Internet connection tests on both ends and everything appears to be fine.  There is low jitter, zero packet loss, high QOS and consistent, high download/upload speeds.  We've restarted the SBS server, the workstations and the networking equipment several times with no effect.  This problem began about 2-3 months ago for no apparent reason and has progressively gotten worse.  No software, with the exception of Windows security updates, has been installed on either the server or the workstations since this issue began.  Also, we've started to have the remote users connecting using the "Modem (28.8 Kbps)" RWW connection setting thinking this would compress the connection better.  It appeared to work better for about a day, but then the latency returned.