AutoDesk's Revit Slows down in the afternoon

We're having performance issues running Revit that seem to vary based on the time of day.  Users seem to have trouble drawing ducts.   Normally their computers are adequate speed wise but in the afternoon things grind down to the point that it's difficult to do any drawing at all.   We typically run into slowness around 3 PM  with it extending into the 6 to 7 PM time.   Mornings are typically not a problem.

We're running SBS 2003 on our server.   Desktop users are running Dell Precision 490s w Windows XP Pro x64 with 12 GB of RAM and reasonable video cards.   Our network runs through a GB switch.  We have about 10 users on our system at peak times.  Our Network Antivirus is Trend Micro Worry Free Business Advanced 6.0 SP2.   I've killed all of the Trend Micro services on the server and unloaded the workstation client thinking that could be the issue but it doesn't seem to make much difference.  

I've used Task Manager to look at the running processes (one user has 41 the other 49).   I don't see anything that is close to consuming the computers resources.   Typically we're using about 3.5 GB of RAM according to Task Manager.

Any thoughts as to what my phantom is would be appreciated.