We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you a podcast all about Citrix Workspace, moving to the cloud, and analytics & intelligence. Episode 2 coming soon!Listen Now

x

All of a sudden Published Apps won't launch - seems to be related to Netscaler perhaps

Medium Priority
304 Views
Last Modified: 2020-03-16
I have a StoreFront system running version 3.15.  This is sitting on a Windows 2012 server.  There are 2 stores configured.  one store is pointed to our existing XenApp 6.5 environment, the other to our new XenApp 7.15 environment.  

All was going well yesterday and then all of a sudden for every published app regardless of whether or not it is hitting the XenApp 6.5 or XenApp 7.15 backend servers provides the error "Unable to connect to the server.  Contact your system administrator with the following error:  Socket operation on non-socket (Socket Error 10038)" or "The published resource is not available currently.  Contact your system administrator for further assistance."  This is happening with all applications.

I have ruled out the backend XenApp 6.5 servers in testing as I can reach those going through Secure Gateway/Web Interface instead of Netscaler/Storefront.  I can't test XenApp 7.15 the same way.  I am not seeing anything in the Netscaler Logs, however; maybe I'm not looking in the right place.  However; it seems the issue is with  Netscaler.  If I go direct to StoreFront, the apps launch.  

To aid in troubleshooting, I rebooted the Netscaler and StoreFront systems yesterday - no change.

Troubleshooting is more difficult as I am not able to find anything logged in Event viewer on the StoreFront system in the App, System or Citrix Delivery Services logs.

On the XenApp servers, I am not finding anything in the Event logs either.  How in the heck do I troubleshoot this?  Why did this just flake out?  Thankfully only in test right now, but if this happens in production, we are SOL and will be a major issue.

Your assistance is apprecaited.
Comment
Watch Question

Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
Can you download the ICA file and post it here to analyze the content.
(remove the automatic assiociation of .ICA file with Receiver to be able to download it as a file)

I have done a quick search and it's something probably related to the STA configuration, I remember that you changed your Delivery Controller port to 8080 recently?

In StoreFront, if you are routing everything through NetScaler, you could create a "internal test" store (you can create it as hidden so no one can accidentally reach it) and change the routing to direct acess, then you are always able to test a connection without NetScaler.
How to hide a test store: https://docs.citrix.com/en-us/storefront/current-release/configure-manage-stores/hide-stores.html

Author

Commented:
Here is the ica file for one of the apps, example below, I obtained this from going direct to the StoreFront server.

[Encoding]
InputEncoding=UTF8

[WFClient]
ProxyFavorIEConnectionSetting=Yes
ProxyTimeout=30000
ProxyType=Auto
ProxyUseFQDN=Off
RemoveICAFile=yes
TransparentKeyPassthrough=Local
TransportReconnectEnabled=On
Version=2
VirtualCOMPortEmulation=On

[ApplicationServers]
Error Manager v23 - ST1=

[Error Manager v23 - ST1]
Address=10.25.x.x:1494
AutologonAllowed=ON
BrowserProtocol=HTTPonTCP
CGPAddress=*:2598
ClearPassword=ADD2FDCE003387
ClientAudio=Off
DesiredColor=4
DesiredHRES=0
DesiredVRES=0
Domain=\5C191AE823B599E5
DoNotUseDefaultCSL=On
EncryptionLevelSession=EncRC5-128
FontSmoothingType=0
HDXoverUDP=Off
InitialProgram=#Error Manager v23 - ST1
Launcher=WI
LaunchReference=2WK/Fu8u/25seh4hwbpE8DcrFVdamtwzUi355o8jpRI=
LocHttpBrowserAddress=!
LogonTicket=ADD2FDCE0033875C191AE823B599E5
LogonTicketType=CTXS1
LongCommandLine=
LPWD=234
NRWD=33
ProxyTimeout=30000
ProxyType=Auto
SessionsharingKey=-fnDN1qU3ju8QJmAaagiLAt
SFRAllowed=Off
SSLEnable=Off
startSCD=1571845463893
Title=Error Manager v23 - ST1
TransportDriver=TCP/IP
TRWD=7
TWIMode=On
WinStationDriver=ICA 3.0

[Compress]
DriverNameWin16=pdcompw.dll
DriverNameWin32=pdcompn.dll

[EncRC5-0]
DriverNameWin16=pdc0w.dll
DriverNameWin32=pdc0n.dll

[EncRC5-128]
DriverNameWin16=pdc128w.dll
DriverNameWin32=pdc128n.dll

[EncRC5-40]
DriverNameWin16=pdc40w.dll
DriverNameWin32=pdc40n.dll

[EncRC5-56]
DriverNameWin16=pdc56w.dll
DriverNameWin32=pdc56n.dll

Author

Commented:
Yes, I recently changed the port to 8080 and its been working as expected,  I've been able to setup and test many apps in the new farm as well as go back to the original StoreFront store and test the old.  It just stopped.  That's why I'm perplexed.  I haven't changed the Netscaler or StoreFront configuration since I got this working initially.  I've just been adding published apps to the Delivery Controller including trying various attempts to launch a RDP session, again though a published app.

Author

Commented:
Here's one initiated from Netscaler

[Encoding]
InputEncoding=UTF8

[WFClient]
ProxyFavorIEConnectionSetting=Yes
ProxyTimeout=30000
ProxyType=Auto
ProxyUseFQDN=Off
RemoveICAFile=yes
TransparentKeyPassthrough=Local
TransportReconnectEnabled=Off
Version=2
VirtualCOMPortEmulation=On

[ApplicationServers]
Error Manager v23 - ST1=

[Error Manager v23 - ST1]
Address=;40;STA954496338;119A83719730FE8E2838861AFBF30D
AutologonAllowed=ON
BrowserProtocol=HTTPonTCP
CGPSecurityTicket=On
ClearPassword=54EB6EA20DFF67
ClientAudio=Off
DesiredColor=4
DesiredHRES=0
DesiredVRES=0
Domain=\5D9A17025DA47ED0
DoNotUseDefaultCSL=On
EncryptionLevelSession=EncRC5-128
FontSmoothingType=0
HDXoverUDP=Off
HTTPBrowserAddress=!
InitialProgram=#Error Manager v23 - ST1
Launcher=WI
LaunchReference=Q9lHiYmNVTHzKAZkA/j1CYcVXYkt/223/sH7QiFjeqE=
LocHttpBrowserAddress=!
LogonTicket=54EB6EA20DFF675D9A17025DA47ED0
LogonTicketType=CTXS1
LongCommandLine=
LPWD=288
NRWD=36
ProxyTimeout=30000
ProxyType=Auto
SecureChannelProtocol=Detect
SessionsharingKey=-UJHRAcXNvJAbIjxvYZHFvQ
SFRAllowed=Off
SSLCiphers=all
SSLEnable=On
SSLProxyHost=cwgctt.domain.com:443
startSCD=1571846375930
Title=Error Manager v23 - ST1
TransportDriver=TCP/IP
TRWD=7
TWIMode=On
WinStationDriver=ICA 3.0

[Compress]
DriverNameWin16=pdcompw.dll
DriverNameWin32=pdcompn.dll

[EncRC5-0]
DriverNameWin16=pdc0w.dll
DriverNameWin32=pdc0n.dll

[EncRC5-128]
DriverNameWin16=pdc128w.dll
DriverNameWin32=pdc128n.dll

[EncRC5-40]
DriverNameWin16=pdc40w.dll
DriverNameWin32=pdc40n.dll

[EncRC5-56]
DriverNameWin16=pdc56w.dll
DriverNameWin32=pdc56n.dll
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
For the second ICA file, is the following host address valid?

SSLProxyHost=cwgctt.domain.com:443

(this should be your NetScaler address)

Author

Commented:
Awe, I think I just figured out what happened.  So I am setting up 2 stores, ie on StoreFront I have (Attachment StoreFront)

Therefore they both use the same prefix in the Store URL, https://cwsfctt.domain.com/Citrix/<Appropriate Store>

So in the StoreFront configuration, I had under Manage Netscaler Gateway different STA's/Delivery Controller for each 'store' to point to the appropriate backend farm.  So the primary points to XenApp 6.5 the secondary with the 7 in the name points to the XenApp 7.15 farm.  For some reason while testing I was able to get one 'farm' to work and not the other.  Realized I needed to update the STA's/Delivery Controller's per farm.  What I didn't realize is that this is a shared configuration.  It did seem previously this was not the case but I likely was just testing one farm at a time so didn't realize this was being updated.  So though I thought I was editing for one store or the other, it is changing in both.  

So, seems I must create a secondary Item, one per store, in the Manage Netscaler Gateway.  That way they are separate entries for each store but the main URL that users will use uses the same prefix.

The reason it was setup as 2 stores on this system is to make it easier to move users when we transition from XenApp 6.5 to 7.15.  There won't be an additional URL for them, it's simply changing/removing the one store from within StoreFront.

My bigger concern is I have no idea what log would assist with troubleshooting this and where it would live?  I just decided to look here as I know there are 3 places in the StroreFront config where the STA's/Delivery Controller's must be set and use the same ones.

I'll try out my theory and report back.   IN the meantime, is there good documentation around how/where I look at such logs?

Thanks in Advance.

Author

Commented:
That alone doesn't seem to help.  So think the original thought isn't going to work.  So, maybe someone can provide some suggestions on the best way to do this?  Ultimate goal, need a simple way to just make a change to have the users still go to the same URL and simply switch the route/farm on the StoreFront system?
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
Have you tried to add both XenApp 7.15 and XenApp 6.5 delivery controllers to one store?
Or do you want to keep them separate for your migration?
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
In your ICA file, you can check the STA value and make sure that it's the correct server for the correct farm:

Address=;40;STA954496338;119A83719730FE8E2838861AFBF30D

You can see the STA number associated to the DDC server in the STA configuration page for your Citrix Netscaler Gateway.

If you are using 2 stores, you will need 2 distinct NetScaler Gateways.
The STA configuration of the NS Gateway must match the STA configuration of the StoreFront store.

Author

Commented:
The weird thing is I have this working again, so its flaky.  I dropped the STA's in the Store's Netscaler config again and re-added, now it's working with both farm.  I believe before I edited rather than drop/re-add.  So I don't know.  

So I'm back to the troubleshooting/logs.  And why this is flaky?
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
If you have 2 stores you need 2 NetScaler Gateway.

Author

Commented:
Ok I left the second one configured, so now there's two.  So how does it know which one to use?
Storefront_Netscaler.png

Author

Commented:
I mean because the url starts the same?
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
I mean, 2 NetScaler Gateways in NetScaler, not in StoreFront

Author

Commented:
Oh, Yes, I already had that setup
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
Ok. So when you create multiple Citrix Gateways in StoreFront, you can name them differently (the "display name").
Configure each one with the appropriate settings and STA servers.
When it's done, right click on each store and go to "Configure Remote Access Settings", then select the corresponding gateway for that store.
It's how it should work. A good option for testing is to read the STA value in the downloaded ICA file, and go to the NetScaler Gateway STA setting and check if the STA number correspond to the correct STA server.

Author

Commented:
Thanks, good to know.  Do you have any other locations where I could have looked to help troubleshoot this issue?  Logs generated?
Olivier MARCHETTACitrix Support and Infrastructure Engineer
CERTIFIED EXPERT

Commented:
I would check StoreFront first, usually with newer versions you find useful information in the logs (Even Viewer):

Application and Services Logs > Citrix Delivery Services
 or
Windows Logs > Application

It seems that the error happens on the client side, though, you can follow these steps, on your client computer:

To enable logging of the launch.ica file
Save the information in the launch.ica file to the client computer to troubleshoot multiple issues. The launch.ica file is generated by Citrix Web Interface or Citrix StoreFront Servers.

To enable logging of the launch.ica file, complete the following steps:

    Navigate to the following registry key by using the registry editor:

    32-bit Systems: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\Logging

    64-bit Systems: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\Logging

    Set the following two string key values:
        LogFile=”path to the log file”
        LogICAFile=true

    For example:

    copy
    LogFile=C:\ica\ica.log
    LogICAFile=true

Then check the C:\ica\ica.log file.

Just for my information, did you say that the application can launch directly from StoreFront without NetScaler?

Author

Commented:
Yes, the applications can launch from both places now.  When I was troubleshooting, I went straight to StoreFront and was able to launch without going through Netscaler.  However; after determining my issue, not quite sure if I had adjusted for the appropriate STA prior to the test and therefore didn't see the failure.  The only items changed were in the StoreFront layer, in the Netscaler Gateway configuration, so perhaps as from what I understand it is using this information for the callback through Netscaler.  So perhaps bypassing it it doesn't use this?

Author

Commented:
Thanks for the information on logging, seems this is more scattered than the Secure Gateway/Web Interface days when everything was very much in one place and pretty concise.  As I work through problems, trying to figure out how to read logs and where to go for that type of information as I'll have to share this with the rest of my team for troubleshooting tools going forward once we're live.
Citrix Support and Infrastructure Engineer
CERTIFIED EXPERT
Commented:
Yes, Storefront will identify the request coming from the NetScaler address and use the remote access configuration.
If you use StoreFront directly, you bypass the remote access configuration.

Maybe reconfiguring your remote access full chain and STAs did solve the issue.
It's never easy to troubleshoot it when it's broken but usually if you set it right from the start it's stable.

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.