Citrix Errors on Xenapp 6.5 farm Event Id : 30003, 30005, 30401, 31003

We have three farm and all three farms (Staging Xenapp 6.5, PS 4.5 Legacy PROD, PROD Xenapp 6.5) are configured in same web interface, We are getting these error frequently only on PROD Xenapp 6.5 farm other two farms fine. Whenever we receive this errors users are unable to launch applications, To fix this issue I need to reboot the data collector and web interface servers. Need help to fix this issue permanently.

Installed RP02 and SP1, XML port 8080.

Farm Name : CITRIXFARM
DC : DataCollector1 (Most Preferred), Datacollector2 (Preferred), DataCollector3(Default)
WI : WEBINTERFACE1, WEBINTERFACE2 (Both are Load balanced)
S : SQL 2008 R2

Once you log a case with Citrix, Please let me know the ticket details.
Let me know if you need any more details.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Name:      Application
Source:        XenApp Services
Date:          10/5/2013 2:01:08 AM
Event ID:      30005
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WEBINTERFACE.be.soc01.monitoredsecurity.com
Description:
Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

The Citrix servers sent incorrect HTTP syntax. Verify that the current Web Interface version is compatible with the servers being used. This message was reported from the XML Service at address http://DataCollector:8080/scripts/wpnbr.dll [com.citrix.xml.NFuseProtocol.RequestAppData]. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services. [Unique Log ID: 817e8638]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="XenApp Services" />
    <EventID Qualifiers="0">30005</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-05T02:01:08.000000000Z" />
    <EventRecordID>20956</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WEBINTERFACE.be.soc01.monitoredsecurity.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

The Citrix servers sent incorrect HTTP syntax. Verify that the current Web Interface version is compatible with the servers being used. This message was reported from the XML Service at address http://DataCollector:8080/scripts/wpnbr.dll [com.citrix.xml.NFuseProtocol.RequestAppData]. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services. [Unique Log ID: 817e8638]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.</Data>
 </EventData>
</Event>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Name:      Application
Source:        XenApp Services
Date:          10/5/2013 1:38:49 AM
Event ID:      30003
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WEBINTERFACE.be.soc01.monitoredsecurity.com
Description:
Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

An error occurred while attempting to connect to the server DataCollector on port 8080. Verify that the Citrix XML Service is running and is using the correct port. If the XML Service is configured to share ports with Microsoft Internet Information Services (IIS), verify that IIS is running. This message was reported from the XML Service at address . The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services. [Unique Log ID: e51769f9]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="XenApp Services" />
    <EventID Qualifiers="0">30003</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-05T01:38:49.000000000Z" />
    <EventRecordID>20905</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WEBINTERFACE.be.soc01.monitoredsecurity.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

An error occurred while attempting to connect to the server DataCollector on port 8080. Verify that the Citrix XML Service is running and is using the correct port. If the XML Service is configured to share ports with Microsoft Internet Information Services (IIS), verify that IIS is running. This message was reported from the XML Service at address . The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services. [Unique Log ID: e51769f9]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.</Data>
  </EventData>
</Event>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Name:      Application
Source:        XenApp Services
Date:          10/5/2013 1:42:08 AM
Event ID:      30401
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WEBINTERFACE.be.soc01.monitoredsecurity.com
Description:
Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

A socket has been forcibly destroyed by the transaction layer. [Unique Log ID: 3a3736a9]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="XenApp Services" />
    <EventID Qualifiers="0">30401</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-05T01:42:08.000000000Z" />
    <EventRecordID>20906</EventRecordID>
  </System>
  <EventData>
    <Data>Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

A socket has been forcibly destroyed by the transaction layer. [Unique Log ID: 3a3736a9]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.</Data>
  </EventData>
</Event>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Name:      Application
Source:        XenApp Services
Date:          10/5/2013 1:42:08 AM
Event ID:      30005
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WEBINTERFACE.be.soc01.monitoredsecurity.com
Description:
Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

The Citrix servers sent incorrect HTTP syntax. Verify that the current Web Interface version is compatible with the servers being used. This message was reported from the XML Service at address http://DataCollector:8080/scripts/wpnbr.dll [com.citrix.xml.NFuseProtocol.RequestAddress]. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services. [Unique Log ID: bdc66a0e]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="XenApp Services" />
    <EventID Qualifiers="0">30005</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-05T01:42:08.000000000Z" />
    <EventRecordID>20909</EventRecordID>
  <EventData>
    <Data>Site path: C:\inetpub\wwwroot\Citrix\PNAgent.
The Citrix servers sent incorrect HTTP syntax. Verify that the current Web Interface version is compatible with the servers being used. This message was reported from the XML Service at address http://DataCollector:8080/scripts/wpnbr.dll [com.citrix.xml.NFuseProtocol.RequestAddress]. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services. [Unique Log ID: bdc66a0e]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.</Data>
  </EventData>
</Event>

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Name:      Application
Source:        XenApp Services
Date:          10/5/2013 1:42:08 AM
Event ID:      31003
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WEBINTERFACE.be.soc01.monitoredsecurity.com
Description:
Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

All the Citrix XML Services configured for farm CITRIXFARM failed to respond to this XML Service transaction. [Unique Log ID: 3062138d]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="XenApp Services" />
    <EventID Qualifiers="0">31003</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-05T01:42:08.000000000Z" />
    <EventRecordID>20910</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WEBINTERFACE.be.soc01.monitoredsecurity.com</Computer>
    < 
>
  <EventData>
    <Data>Site path: C:\inetpub\wwwroot\Citrix\PNAgent.

All the Citrix XML Services configured for farm CITRIXFARM failed to respond to this XML Service transaction. [Unique Log ID: 3062138d]

For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.</Data>
  </EventData>
</Event>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
LVL 26
Sekar ChinnakannuStaff EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ayman BakrSenior ConsultantCommented:
First thing to check is to ensure that all your XA 6.5 farm servers are at Hotfix RP 02, especially the data collectors.

Is the above OK, but still the issue occurs? Then:
Are you using HTTPS - secure socket transaction? If not then go to the WI web site -> Server Farms -> Advanced button -> uncheck Enable Socket Pooling which is enabled by default.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Sekar ChinnakannuStaff EngineerAuthor Commented:
Yes checked all servers are at RP2 and enable socket pooling is unchecked.
0
Ayman BakrSenior ConsultantCommented:
Are they all at the same FP?

Please check the farm servers (specifically the XML brokers) for any errors - do you find errors related to not allowing users to log on?

Please check CrashOnAuditFail setting under HKLM\System\CurrentControlSet\Lsa. Is it set to 0x2? Then check this article:

http://technet.microsoft.com/en-us/library/cc963220.aspx

If you have enough disk space for audits, then set it to 0
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

Sekar ChinnakannuStaff EngineerAuthor Commented:
Yes all are in same farm and no errors in XML brokers only i can see lot of errors in web interface. Also you want me to add value 0x2? because currently it set to 0 in DC's.
0
Ayman BakrSenior ConsultantCommented:
No do not add a value of 0x2. I meant are they at the same feature pack?
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
Yeah all are in Same FP
0
Ayman BakrSenior ConsultantCommented:
Do you have CSG or Access Gateway configured? Can your issue be similar to this: http://www.experts-exchange.com/Software/System_Utilities/Remote_Access/Citrix/Q_24415807.html   ?

Moreover, you said two WI servers and load balanced. How are they load balanced? Do you get the errors on both the WI servers?
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
Yes on both the web interface, we load balanced using A10 and no CSg or AG in our environment.
0
Ayman BakrSenior ConsultantCommented:
Please try the following, testing after each step for resolution:

1. Change the XML broker name on both the web interface servers from the hostname to FQDN and test.

2. Does telneting ports 1494 and 2598 from the Web Interface servers to the XML brokers work fine during the reported events?

3. Are you able to launch applications through RDP?

4. Try changing the Data Collector role to another server, make it the primary XML broker on the Web Interface servers and observe testing to see whether the events re-occur. If they do not occur, then I believe the initial XML Data Collector has an issue.

5. Restart IMA service and recreate the local host cache on the XML brokers

6. Repair the XenApp 6.5 installation from 'Program and Features' on the affected Data Collectors. This time also ensure that it is at the same FP, RP and hotfixes.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
please find the below update

Step 1 - Tried no luck
Steps 2 - success
Steps 3 - Yes
Step 4 - After changing the DC so far no issues, Checked with initial DC event no erros in system\application, Still trying find the root cause
Step 5 - Tried no luck
Step 6 - will this cause any issue? when we repair
0
Ayman BakrSenior ConsultantCommented:
If you have already changed the DC to another server, then there should be no issues repairing the problematic server/s, provided no users are connected.

But, since the new XML are now fine with no issues reported so far, let's wait until we are satisfied that these new servers will not reproduce the errors - at that moment you will carry out the repair.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
After changing th DC today again we received same error for all the DC's it's frustrating because out of three farms issue happening to only one farm
0
Ayman BakrSenior ConsultantCommented:
The errors clearly indicate communication problem between the web interface and the XML brokers. It could be any of the following:

1. Windows patch on either or both the web interface and XML broker/s that broke the communication

2. A Hotfix that is causing the issue

3. A misconfiguration

4. XA servers, and especially your DCs, are not at the same Hotfix list, rollup packs feature packs or editions.

5. Improper LB on the WI servers

In a previous post you explained that the issue did not occur after changing the DC. Then now you explained that this reoccurred after again changing the DC. What did you do? Did you revert back to the same DC? Have you repaired it prior to reverting back? Can you post the XML configuration, and STA configuration on the WI servers.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
after changing the DC everything is fine for last 4 days, again today same error occur

1. Windows patch on either or both the web interface and XML broker/s that broke the communication - Other two farms are fine, only issue with PROD xenapp 6.5

2. A Hotfix that is causing the issue - after RP2 nothing i installed

3. A misconfiguration - what exactly you mean

4. XA servers, and especially your DCs, are not at the same Hotfix list, rollup packs feature packs or editions. - once again verified all are in same level

5. Improper LB on the WI servers - Checked LB logs no errors
0
Ayman BakrSenior ConsultantCommented:
Actually I was not implying that you need to check the five points. I was only trying to list what might be the possible causes in order to troubleshoot.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
just few mins back same issue and we rebooted both the Dc's and WI to fix the issue.  Also STA is set to direct. I am facing this issue frequently in last 4 weeks prior to that everything is fine. Only change is patch was installed, If the patch is issue then staging is same as prod, Staging working fine with same config and wi
ss.png
0
Ayman BakrSenior ConsultantCommented:
Now you are giving a clue. I should expect the same as you do; logically if the patch was tested on the staging environment and deemed fine for production then it should consequently be fine in production. But, needless to say, however you make your staging environment resemble your production, systems are unique and a difference will always exist. Thus, out of the many successful staging - production deployments, you should always expect, and be prepared, that things might not go as well as you have tested.

Back to the issue. I see from your configuration that you have implemented load balancing for your XML brokers. So even though you have changed your DC, you still have a 1/3 chance that the problematic DC would respond for XML brokerage and thus the issue again. So, in order to pinpoint whether the issue comes from that problematic DC or from all the DCs regardless, then you have two options - either:
1. Remove load balancing in the XML configuration on the WI and put your problematic DC in the third position (this way you implement failover and you ensure that no XML requests go to the problematic DC unless the first two are down)     OR
2. Remove for the moment the problematic DC from the list of XML brokers in the WI

If you can't afford further testing you can have another approach! Remove the patch that was installed and restart the WI and DCs; hoping that we would have removed the potential culprit.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
Its a strange issue :( finally remove problematic DC from XML brokers and recreated LHC in entire,reason to recreate when we face the servers stooped accepting connection and after rebooting DC's no luck so recreated LHC. In last one hour no error in WI.

Should I need to open any other ports from LB to Xenapp servers, WI, Because I think we opened only 8080 & 80 between WI & LB.

Can I try to revert sml port from 8080 to 80? if so any process you prefer
0
Ayman BakrSenior ConsultantCommented:
I don't think you need to open ports other than that of your XML ports of your three farms.

Currently, I would prefer to do one troubleshooting step at a time; especially that I don't believe it has anything to do with the port number. Therefore, now that you have removed the problematic DC from the list, stick to monitoring the environment.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
Also same time i can see the Event 10001: A usable server cannot be found on which to launch the application error in DC's. All the server are set default load evaluators. Verified and all the server only one nic card available in device manger and no worker group issues.

sure will monitor the same last 2 hour no issue and no error. Let you know the update
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
Again today same error
0
Ayman BakrSenior ConsultantCommented:
Strange enough, I just got to learn that a corrupted published application might affect the whole XML service - and it could be well your issue.

There might be a specific application that is causing the whole problem and it did not manifest since yesterday till today, maybe because a user just started that app today. Please check for that corrupted app, delete it from app center, carry out a dscheck on your datastore and then properly create that app.

Please check this thread:
http://forums.citrix.com/thread.jspa?threadID=260496
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
I already ran dscheck /clean and no errors, Any other commands you want me to run?
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
see dscheck no error in attachment
dscheck.JPG
0
Ayman BakrSenior ConsultantCommented:
Good. But, did you check the published applications? Any space characters or icons from executable files?
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
ran dscheck /full apps no errors, also we have around 350+ apps, how can i check all at same. is there any possibility to use a script.
0
Ayman BakrSenior ConsultantCommented:
Hmmm!! Which applications were launched just before the issue occurred today. Maybe, you can start by excluding these and checking the rest.

Or maybe, you can go a little back in memory and see which applications were published just before the issue started happening.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
First step already checked.
Working on second one, will update
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
today when i rebooted data colelctor noticed xml service is missing in preferred data colelctor, immediately started receiving error. To fix the issue registed xml on 8080 port. Not sure why xml service was unregistered and what cause the issue.
0
Ayman BakrSenior ConsultantCommented:
Apart from finding the application causing the issue, you can go about and repair your XML brokers. However, if our thinking that a published application is causing the issue is correct then even if you repair the XML brokers the issue will re-occur.

Are you able to find what application was launched the time you discovered the XML service suddenly was missing?

If you are not able to recall what applications were recently published when you started to get the errors, then maybe during a maintenance window you can start an application one by one and check which one would produce the errors.
0
Sekar ChinnakannuStaff EngineerAuthor Commented:
- repaired all the xml servers
- but still receiving the errors for all dc's
- using DsCheckMaintAssist no single single erros
- let me try to find the application, try to delete the application and recreate the same
0
LewygCommented:
This last post leaves this problem unsolved.  was there an update since 10/19/2013?  I am having a similar issue with all of the same error codes and a mix of Citrix farm versions aggregated.  Only the 6.5 Citrix XML brokers seem to be having an issue.
0
LewygCommented:
we have 9 farms that are all aggregated and are comprised of Citrix versions 4.0, 4.5 and 6.5.   We have 5 PNA servers all behind a Cisco VIP.  we ran into a similar issue to what was described above by secarc4u.  The Citrix 6.5 farms were the only ones impacted.   Only one of the 5 PNA servers was getting blasted with the errors noted above.  So 1 out of 5 login requests would be impacted by their application ICON not getting to their desktop and of those there was a 1/5 th chance they wouldn't get their icons.  We worked with Citrix for a solid week checking everything.  we finally found a workstation (out of 40,000) that was in an endless loop trying to log on to Citrix every 30 seconds or so.  The error returned by XML was "invalid credentials". This desktop was in a "training area" (we are having it shipped to us for analysis).  Once identified we got the device turned off and all errors regarding this issue went away.  I will say that Citrix error messages that appear in the PNA application event logs are far to generic and not very helpful.  I am writing this in hope that it might help someone else out there with the same issue.
0
Ayman BakrSenior ConsultantCommented:
I agree with you; with the last comment of the author, it seems unresolved. However, he had specifically mentioned in the last point that he will try to find the application, try to delete it etc... So after some time he accepted the solution, albeit without putting a further comment - which I have assumed then that he found the culprit application.

As for your last comment, I am glad you were able to solve your issue and would actually thank you for the valuable addition you made to this thread.
0
NewbNukemCommented:
So could these errors be generated just from a user simply inputting the wrong credentials?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Citrix

From novice to tech pro — start learning today.