VMware VDI Latency

We have a server farm hosting our virtual desktop infrastructure. We have 3 Dell PowerEdge R900, 16 CPUs x2.393 GHz, Intel Xeon processors with 4 nics and 128GB memory each. We have approximately 110 virtual desktops running XP. Each VM is configured with 1 vCPU and 1GB memory and 1 network connection. We are running vSphere 4.0 update 1.  Each physical host is utilizing about 9GHZ processor and 24G of memory. Sporadically, the virtual desktops will slow to almost a crawl. There is not rhyme or reason to which ones, as it has happened to the majority of them, however, we do see it slightly more frequently on those that utilize AS400 sessions frequently. The perfomance of the physical hosts appears to be fine, but the virtual desktops are not. I do not know what other settings to modify to increase performance.