Short version: From one remote site, Citrix connections to our office are slow; everything else works as it should, including Citrix connections to other sites. From everywhere else I have available to test, Citrix connections into our office work perfectly.
I have a single PS4 server in my office that functions for remote access to apps and data. It has worked nearly flawlessly since I put it in until two weeks ago (Friday, 9/1/6 to be exact), but since that time three users in a remote office (A company we just bought but don't have room for until we move, next month.) are seeing unbearably slow performance. Prior to 9/1 they had acceptable but less than stellar performance, fraction of a second but noticeable lag when clicking, etc. Since 9/1 they're seeing waits of over a minute for responses to clicks or keystrokes, with sessions occasionally hanging completely, disconnecting and reconnecting. Interestingly, sessions get slower the longer users stay on. When I shadow remote users from inside the main office, I see things moving faster than they do, the server itself does not appear to be slow, just the remote display of what's happening.
If I have the users connect via RDP, performance is as expected for RDP. Not as fast as with Citrix but quick. It also has all the issues of RDP, the biggest one being that I can't get their network copier printer to work.
From that remote site I am able to connect to another CItrix server (Access Essentials, at one of our client companies) to test, performance is fabulous. From other remote locations I am able to connect to my Citrix server, performance is fabulous. All but one of our employees use the same ISP and hardware from their home as is at the problem site, and everything works perfectly for all of us except when at this one site. All three of the PCs at the problem site are portables, and all work fine from other locations.
I, Of course, was off 8/31 and 9/1 to take a longer long weekend for Labor Day. Everyone swears that they did nothing TO the server on 8/31 or 9/1, and the logs support this.
All remote users are using Outlook 2003 with the RPC over HTTPS feature to get to the Exchange server on our network, this works as expected everywhere, including the problem site. Access to our pubic website, which is inside our network, works fine from everywhere, as does the Citrix web interface, which is on the metaframe server.
Citrix server is an HP Proliant ML150, 4G ram, Xeon 3GHz, 72Gb HDD, Windows 2003 SP1.
Main office internet connection is a T-1, two firewalls/DMZ. Outer firewall is a Cisco Pix, inner is a Sonicwall. I am assured that nothing has been touched on either in months, but I don't speak Cisco and try not to touch SonicWalls if I can help it, so I am relying on third party information.
Remote office Internet is Cable (Charter) 3Mb Down / 256Kb Up. Ambit modem, Linksys router, wired network only. Removal of the router and attaching a PC directly to the cable modem produces identical results.
I'm planning to replace our firewalls with a single one to eliminate the multiple firewalls / multiple NAT as a point-of-failure, but that seems unlikely since it worked for several months and stopped suddenly. I am also going to replace the modem with the one form my house, since the ISP won't (if web browsing works, the service works for the class of service that is subscribed to at this office), but likewise that seems low likelihood of success.