Web Interface/CSG stops functioning with Presentation Server 4.0

Recently this has been a bit of a problem. The web interface will work fine for a while and then suddenly it will stop, giving a "Credentials could not be validated" error when authorized users attempt to log in. My Citrix farm consists of three servers: Citrix1, Citrix2, and Citrix3. We are running Presentation Server 4.0 Enterprise on these servers, and are using Citrix Secure Gateway 3.0 in our DMZ.

The XML service is running on all three servers on port 80. I have verified this port setting through the management console. On the Secure Gateway end of things, I get some interesting behavior though, and I've seen this for a while. I just haven't been able to resolve it. When I run the Secure Gateway Diagnostics, everything passes except for Authority Servers. Under this section, it only shows Citrix2 as passing, but still users can't log in. So my next step was to verify that the XML service is in fact running on these servers -- it is. All three servers have these services running:

Citrix SMA Service
Citrix WMI Service
Citrix XML Service
Citrix XTE Server

Furthermore I've verified that each of these servers still has the correct ALTADDR set.

At the very least, Citrix2 should work. It has in the past, although at times it's stopped working. Usually this was because the XML service stopped for some reason. It's definitely running now. The problem with Citrix1 and Citrix3 not configuring properly in the CSG I suspect may be a firewall issue, even though the consultant who set up our new firewall states this isn't the case. I'm not really familiar with the new SonicWALL firewall that we have (and I don't particularly like it), but there appears to be a rule that allows any traffic from the DMZ to hit my Citrix servers' public addresses on ports 80 and 443 (and there is also a rule that opens port 1494 to WAN traffic). What's interesting though is that if I try to telnet into port 80 from the CSG server, the connection is refused for Citrix1 and Citrix3:

C:\> telnet citrix1.mydomain.com 80
Connecting To citrix1.mydomain.com... Could not open connection to the host, on port 80: Connect failed

C:\> telnet citrix2.mydomain.com 80
(command prompt box goes blank, so it appears to connect OK)


However, from inside my LAN if I try to telnet to these same servers it works fine.

C:\> telnet citrix1 80
(command prompt box goes blank)

C:\> telnet citrix2 80
(same as above, command prompt box goes blank)


My priority issue is getting to the bottom of Citrix2 not working. On Citrix2's event logs, the only errors I see are printer creation errors. I tried restarting the XML service but the login problem persists. The CSG still says Citrix2 tests OK. In the Secure Gateway event logs, there are a bunch of warnings. There are "CORE" category events which indicate that "connection was broken by either client or server," and then there are "SCHANNEL" category events which indicate "SSL handshake from client failed." There aren't many errors. The last time an error shows up here was on 12/3/08: "Connection Timed out." The last time I know a user was able to log in was this past Thursday, and nothing on the network has changed since then.

The full error message that the users get when they try to log in is, "The supplied credentials could not be validated. Either they are incorrect or there is a problem with the authentication system." Any idea what's going on here? I've included the CSG report (with private information changed).

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:

Version = 3.0.1
 
Computer NetBIOS Name: CTXWEB
Configuration captured on: 12/15/2008 10:33:59 AM
-------------------------------------------------
 
Secure Gateway Global Settings
------------------------------
  Version = 3.0.1
  Product secured = MetaFrame Presentation Server only
  Logging level = 2 (Warning, errors, and fatal errors)
  Maximum concurrent connections = 250
  Certificate FQDN = ctxweb.mydomain.com
 
Interfaces
----------
  172.16.1.3 : 443
  ----------------
    Protocol = SSL, TLS
    Cipher suites = ALL
    Secured = YES
    HTTP = No
    ICA = Yes
    SOCKS = Yes
    Gateway Client = No
    LoadBalancerIPs = None defined
 
Web Interface
-------------
  FQDN = localhost
  Port = 80
  Secured = No
  Protocol = SSL, TLS
  Cipher suites = ALL
  Access mode = Indirect
  Tested OK
 
Authority Servers
-----------------
 
  ID = STA.........
  -----------------
    FQDN = citrix3.mydomain.com
    Port = 80
    Path = /Scripts/CtxSTA.dll
    Type = STA
    Secured = No
    Protocol = SSL, TLS
    Cipher Suites = ALL
 
  ID = STA.........
  -----------------
    FQDN = citrix2.mydomain.com
    Port = 80
    Path = /Scripts/CtxSTA.dll
    Type = STA
    Secured = No
    Protocol = SSL, TLS
    Cipher Suites = ALL
    Tested OK
 
  ID = STA.........
  -----------------
    FQDN = citrix1.mydomain.com
    Port = 80
    Path = /Scripts/CtxSTA.dll
    Type = STA
    Secured = No
    Protocol = SSL, TLS
    Cipher Suites = ALL
 
Certificate Check
-----------------
  FQDN = ctxweb.mydomain.com
  This certificate is currently valid.
  
EOF