jnordeng
asked on
Unable to launch applications when routed through Netscaler, but works via StoreFront directly
We have an existing environment, Netscaler MPX9700, v11.1 63.15.nc, Storefront v 3.15.0.18019, XenApp 6.5 Farm using port 8080.
We are setting up a new backend XenApp 7.15 farm. The structure will be the same existing Netscaler MPX9700 and Storefront 3.15 System and then point to the XenApp 7.15 farm.
I've setup a second store on StoreFront for the XenApp 7.15 farm so as to not interrupt production XenApp 6.5. On the Netscaler I've setup a secondary Netscaler Gateway VIP and set the Profile to point to the Secondary Store on StoreFront.
What we are seeing is when we login to https://cwig.domain.com/Citrix/CitrixStore_cwisfWeb/ and authenticate, we then switch the URL to the second store, https://cwig.domain.com/Citrix/CitrixStore_cwisf7Web/. We see the applications populate from StoreFront, but when we launch an application, prompted to Open Citrix Receiver launch, but then nothing happens, no errors, don't see anything in the Client's Connection Center. Doing a packet capture from both the firewall and performing an Nstrace on the Netscaler, we are not seeing traffic pass through to the backend XenApp servers or traffic in general when launching the app when routing through the Netscaler.
When I go directly to the StoreFront URL, https://cwisf.domain.com/Citrix/CitrixStore_cwisf7Web, I am able to authenticate, see the applications and launch the applications.
So our issue seems to be related to the communicate from Netscaler receiving the traffic from the XenApp systems.
Since this Netscaler/Storefront is already in use in production and working with our XenApp 6.5 farm we are trying to understand what would be preventing the traffic from flowing to the new XenApp 7.15 backend servers? We do have this working in our non-production environment which is why w e do not understand why we are unable to have this working in production.
Thanks in advance for your help.
We are setting up a new backend XenApp 7.15 farm. The structure will be the same existing Netscaler MPX9700 and Storefront 3.15 System and then point to the XenApp 7.15 farm.
I've setup a second store on StoreFront for the XenApp 7.15 farm so as to not interrupt production XenApp 6.5. On the Netscaler I've setup a secondary Netscaler Gateway VIP and set the Profile to point to the Secondary Store on StoreFront.
What we are seeing is when we login to https://cwig.domain.com/Citrix/CitrixStore_cwisfWeb/ and authenticate, we then switch the URL to the second store, https://cwig.domain.com/Citrix/CitrixStore_cwisf7Web/. We see the applications populate from StoreFront, but when we launch an application, prompted to Open Citrix Receiver launch, but then nothing happens, no errors, don't see anything in the Client's Connection Center. Doing a packet capture from both the firewall and performing an Nstrace on the Netscaler, we are not seeing traffic pass through to the backend XenApp servers or traffic in general when launching the app when routing through the Netscaler.
When I go directly to the StoreFront URL, https://cwisf.domain.com/Citrix/CitrixStore_cwisf7Web, I am able to authenticate, see the applications and launch the applications.
So our issue seems to be related to the communicate from Netscaler receiving the traffic from the XenApp systems.
Since this Netscaler/Storefront is already in use in production and working with our XenApp 6.5 farm we are trying to understand what would be preventing the traffic from flowing to the new XenApp 7.15 backend servers? We do have this working in our non-production environment which is why w e do not understand why we are unable to have this working in production.
Thanks in advance for your help.
ASKER
Hello. I posted this while waiting for Citrix Support. Working with them yesterday, still not resolved but confirmed that we are seeing the .ica file get generated when switching the extension to .txt. So the support tech thinks the issue is in the SSL Handshake between the Netscaler and Delivery Controller. So that's where we're at. Thought we had everything from non-prod and prod configured the same, but this isn't working, so something is slightly different.
That's good.. can you post a sanitized version of the ICA file?
But - what version is your Netscaler? It could be a TLS support issue where your DDC wants to run TLS 1.2 and your Netscaler doesn't support it. (That support started with 10.5, and I forget the build number.. but I'd know it if I saw it again). Also, do you have disabled ciphers on either your DDC or your Netscaler? If they can't agree on a cipher, that could definitely cause it.
Coralon
But - what version is your Netscaler? It could be a TLS support issue where your DDC wants to run TLS 1.2 and your Netscaler doesn't support it. (That support started with 10.5, and I forget the build number.. but I'd know it if I saw it again). Also, do you have disabled ciphers on either your DDC or your Netscaler? If they can't agree on a cipher, that could definitely cause it.
Coralon
ASKER
Thanks for the follow up, Support is still analyzing the situation.
I looked at the TLS settings for both the non-prod (working) and prod (Not Working) DDC's and their Cipher lists. They are set the same from my eye point of view. The Netscaler Gateway's are using the same Cipher Group including the same setting in non-prod and prod.
Netscaler is running NS11.1 63.15.nc.
Sure - I was able to get output from the .ica file yesterday and can from the other sites, but for some reason is failing today. If I can get that generated, I'll post here.
Thanks in advance.
I looked at the TLS settings for both the non-prod (working) and prod (Not Working) DDC's and their Cipher lists. They are set the same from my eye point of view. The Netscaler Gateway's are using the same Cipher Group including the same setting in non-prod and prod.
Netscaler is running NS11.1 63.15.nc.
Sure - I was able to get output from the .ica file yesterday and can from the other sites, but for some reason is failing today. If I can get that generated, I'll post here.
Thanks in advance.
Ok.. 11.1 is definitely new enough for TLS 1.2.
I think it's almost going to have to be a beaconing problem. The ICA file should confirm that.
Coralon
I think it's almost going to have to be a beaconing problem. The ICA file should confirm that.
Coralon
ASKER
Any idea what I would look for in the .ica file if I can get one again?
Thanks
Thanks
ASKER
Since I've been troubleshooting have determined the main issue i believe is the Delivery Controller machines. I have another set of Netscaler's and StoreFront that point to these same DC's and they are experiencing the same issue. I've been looking around for logs and in the Event Viewer but not finding much. Is there another tool or location where more details would reside?
Thanks
Thanks
Ok.. 11.1 is definitely new enough for TLS 1.2.
It could be a beaconing problem. The ICA file would confirm that.
In the ICA file, look at the Address line.. if it starts out with something like Address=10;STAxxxxx then the beacon side is working well. If it has an IP address, then it's definitely *not* working.
But the information for the other DC's etc.is definitely helpful.
I'd suggest running a network capture on a DDC and the Netscaler at the same time, and see if you get the TLS session information.
Coralon
It could be a beaconing problem. The ICA file would confirm that.
In the ICA file, look at the Address line.. if it starts out with something like Address=10;STAxxxxx then the beacon side is working well. If it has an IP address, then it's definitely *not* working.
But the information for the other DC's etc.is definitely helpful.
I'd suggest running a network capture on a DDC and the Netscaler at the same time, and see if you get the TLS session information.
Coralon
ASKER
Support wasn't too helpful. I had shared this with them:
CitrixSupport.png
I'm still looking through my packet captures, but need to figure this out. So hoping today is a fresh perspective and it stands out. :)
CitrixSupport.png
I'm still looking through my packet captures, but need to figure this out. So hoping today is a fresh perspective and it stands out. :)
ASKER
I did notice something, I changed the .ica file extension to open notepad. I am able to do this for the working environments, I did notice that for the new production environments will prompt me to 'open Citrix Receiver Launcher' and then the program doesn't launch. Instead now is giving "Unable to connect to the server. Contact your system administrator with the following error: (no error text available)". The working environment simply opens the notepad. So, trying to understand what is initiating the 'prompt'. Maybe that will clue me into what needs to be fixed here.
ASKER
I removed the additional DC's and ran another packet capture, was able to get the text pad to generate and here are the contents.
[Encoding]
InputEncoding=UTF8
[WFClient]
ProxyFavorIEConnectionSetting=Yes
ProxyTimeout=30000
ProxyType=Auto
ProxyUseFQDN=Off
RemoveICAFile=yes
TransparentKeyPassthrough=Local
TransportReconnectEnabled=Off
Version=2
VirtualCOMPortEmulation=On
[ApplicationServers]
TextPad_CTXPRD2X_NR=
[TextPad_CTXPRD2X_NR]
Address=;40;STA397466046;DE198F4C297A8B1C3490728E5C1BAF
AutologonAllowed=ON
BrowserProtocol=HTTPonTCP
CGPSecurityTicket=On
ClearPassword=12003B5955538D
ClientAudio=On
DesiredColor=8
DesiredHRES=0
DesiredVRES=0
Domain=\C12DFC8BEDD92EE3
DoNotUseDefaultCSL=On
FontSmoothingType=0
HDXoverUDP=Off
HTTPBrowserAddress=!
InitialProgram=#TextPad_CTXPRD2X_NR
Launcher=WI
LaunchReference=4A25D728E96F64F4226217C0E5985B
LocHttpBrowserAddress=!
LogonTicket=12003B5955538DC12DFC8BEDD92EE3
LogonTicketType=CTXS1
LongCommandLine=
LPWD=471
NRWD=173
ProxyTimeout=30000
ProxyType=Auto
SecureChannelProtocol=Detect
SessionsharingKey=w+O5ITQ82JZ4n3rXrbvZeMckoz1nIxWY
SFRAllowed=Off
SSLCiphers=all
SSLEnable=On
SSLProxyHost=cwig7.domain.com:443
startSCD=1594231252394
Title=TextPad_NRZ_CTXPRD2X
TransportDriver=TCP/IP
TRWD=0
TWIMode=On
WinStationDriver=ICA 3.0
[Compress]
DriverNameWin16=pdcompw.dll
DriverNameWin32=pdcompn.dll
[EncRC5-0]
DriverNameWin16=pdc0w.dll
DriverNameWin32=pdc0n.dll
[EncRC5-128]
DriverNameWin16=pdc128w.dll
DriverNameWin32=pdc128n.dll
[EncRC5-40]
DriverNameWin16=pdc40w.dll
DriverNameWin32=pdc40n.dll
[EncRC5-56]
DriverNameWin16=pdc56w.dll
DriverNameWin32=pdc56n.dll
[Encoding]
InputEncoding=UTF8
[WFClient]
ProxyFavorIEConnectionSetting=Yes
ProxyTimeout=30000
ProxyType=Auto
ProxyUseFQDN=Off
RemoveICAFile=yes
TransparentKeyPassthrough=Local
TransportReconnectEnabled=Off
Version=2
VirtualCOMPortEmulation=On
[ApplicationServers]
TextPad_CTXPRD2X_NR=
[TextPad_CTXPRD2X_NR]
Address=;40;STA397466046;DE198F4C297A8B1C3490728E5C1BAF
AutologonAllowed=ON
BrowserProtocol=HTTPonTCP
CGPSecurityTicket=On
ClearPassword=12003B5955538D
ClientAudio=On
DesiredColor=8
DesiredHRES=0
DesiredVRES=0
Domain=\C12DFC8BEDD92EE3
DoNotUseDefaultCSL=On
FontSmoothingType=0
HDXoverUDP=Off
HTTPBrowserAddress=!
InitialProgram=#TextPad_CTXPRD2X_NR
Launcher=WI
LaunchReference=4A25D728E96F64F4226217C0E5985B
LocHttpBrowserAddress=!
LogonTicket=12003B5955538DC12DFC8BEDD92EE3
LogonTicketType=CTXS1
LongCommandLine=
LPWD=471
NRWD=173
ProxyTimeout=30000
ProxyType=Auto
SecureChannelProtocol=Detect
SessionsharingKey=w+O5ITQ82JZ4n3rXrbvZeMckoz1nIxWY
SFRAllowed=Off
SSLCiphers=all
SSLEnable=On
SSLProxyHost=cwig7.domain.com:443
startSCD=1594231252394
Title=TextPad_NRZ_CTXPRD2X
TransportDriver=TCP/IP
TRWD=0
TWIMode=On
WinStationDriver=ICA 3.0
[Compress]
DriverNameWin16=pdcompw.dll
DriverNameWin32=pdcompn.dll
[EncRC5-0]
DriverNameWin16=pdc0w.dll
DriverNameWin32=pdc0n.dll
[EncRC5-128]
DriverNameWin16=pdc128w.dll
DriverNameWin32=pdc128n.dll
[EncRC5-40]
DriverNameWin16=pdc40w.dll
DriverNameWin32=pdc40n.dll
[EncRC5-56]
DriverNameWin16=pdc56w.dll
DriverNameWin32=pdc56n.dll
ASKER
This seems intermittent, I went through one by one to all DC's and was able to generate the text file. I have then thrown it back to the primary DC and gotten the original error message again and it doesn't show the text file. So something is causing this to be intermittent.... Are there better troubleshooting tools? I'm going to analyze my latest packet capture and see if that helps... but that was taken when the .ica file was generated.
The prompt is the browser figuring out what to do with the file. It's a "feature" of the newer clients. But it doesn't really change anything. The address line is probably correct.
But that does make me think of one of thing..
Your netscaler gateway must contain the correct STA addresses. If there is a mismatch in the STA, then when the gateway goes to configure the connection, it will try to verify the wrong STA service.
And of course, your CVAD farm must have the Netscaler added properly.
David F.
But that does make me think of one of thing..
Your netscaler gateway must contain the correct STA addresses. If there is a mismatch in the STA, then when the gateway goes to configure the connection, it will try to verify the wrong STA service.
And of course, your CVAD farm must have the Netscaler added properly.
David F.
ASKER
Thanks, is there a way to verify what STA ID is assigned on the Delivery Controller versus what registers in the Netscaler? I narrowed it down to run just one STA in the Netscaler Gateway as well as the Gateway in StoreFront.
Thanks
Thanks
Yes. It's in the registry. From this article(https://support.citrix.com/article/CTX231451) it's under HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer, and the value is XmlStaIdentity.
Coralon
Coralon
ASKER
Nice, thanks for that, that can come in handy. Checked, this matches what is registering on the Netscaler.
Looking through your ICA file there.. I don't see anything referencing the user ID.. I can see the domain, but it starts with a '\', but it doesn't have anything for the userid, encrypted or otherwise.
How is your authentication set up on the Netscaler? I'm wondering it is not being passed properly?
You'll want to check the AAA log on the Netscaler to verify if that end of it is working properly. The fact that a login ticket is being generated tells me that at least part of the authentication process is working. But normally, you'd see the User ID somewhere in the ICA file, and that is sent to the STA along with the domain, the encrypted password and the ticket.
Coralon
How is your authentication set up on the Netscaler? I'm wondering it is not being passed properly?
You'll want to check the AAA log on the Netscaler to verify if that end of it is working properly. The fact that a login ticket is being generated tells me that at least part of the authentication process is working. But normally, you'd see the User ID somewhere in the ICA file, and that is sent to the STA along with the domain, the encrypted password and the ticket.
Coralon
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Is the beaconing process actually working? I'm guessing probably not.
Do you ever get an ICA file on the outside?
Coralon