Microsoft Terminal Server is becoming un-responsive, a timeout (30000 milliseconds) was reached

Hello:

We have A Citrix Farm with 3 Microsoft Server 20018 R2 Terminal Servers (Citrix Servers) everything was working just fine until yesterday (June 15, 2015) around 9:00 am (when our users were starting to logon and use the Citrix servers).  Then all of the sudden some Terminal Server (Citrix Server) users began to report that they could not access their printers and the logon process was hanging up and eventually they could not use Windows explorer.

This problem was specific for users that were on '1' of our '3' Microsoft Terminal Servers(Citrix Servers).  I ended up having the same users logon to the other 2 Citrix servers and those work fine.  At the end of the day I ended up restarting the Citrix1 server and then I can print and access Windows Explorer again.  I ended up trying Citrix1 out all night long as did other users ( 1 other user) and there were no problems in the evening.  Backups worked and everything appears to be normal again at 9:00 am this morning; but, at 10:00 am the same problem came back.

I see in the System event logs that numerous services became un-responsive:  

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the UxSms service.

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the UmRdpService service.

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Netman service.

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the AudioEndpointBuilder service.

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the WPDBusEnum service.


I tried to stop and start the print spooler service; but I was not able to it took too long and it eventually timed out. I plan on having the users log into the other Citrix 2 servers until this is resolved.  Any ideas on how to permanently fix this because I think a restart will only delay the problem from coming back?

Perhaps I should run a virus scan and try a new virus software program?  We use Symantec Anti-Virus on Microsoft Terminal Server that is having problems and we use Forticlient on the server that are not presently having problems.

So on the bad Server (Citrix1) I cannot open 'Devices and Printers' and I cannot use Windows Explorer.  But the other 2 Citrix server are working just fine.  It appears that some services have stopped; but if I check them in the Services module, all of the automatic services' say 'started'.
LVL 1
PkafkasNetwork EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

PkafkasNetwork EngineerAuthor Commented:
I did run a full scan last night with the Symantec Antivirus Software.

Please let me know what you think.  I am open to suggestions.  My plan, as of now, is to:

1.  Restart the Server after hours (because this is also our Citrix Web server) and if anyone is logged on via the Citrix web portal then they will be kicked off.

2.  Remove this server from the 3 available to logon from the Citrix web portal.  
      a.  This way our users will only be able to logon to the 2 good servers, from the portal.

3.  Un-install Symantec anti-virus and install the Forticlient anti-virus.
      a.  Run a full scan on the Citrix1 server.

4.  I can logon and a select few can logon and see what happens to morrow.
       a.  I wonder if a user is triggering the problem; but, if that was the case the same problem would follow them to the Citrix 2 and Citrix 3 Server.

Again my plan if the above does not work, I will try to restore from backups, from last week or 2 weeks ago.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
At first I was going to say its not a virus but check for it anyway, as its a server please try the following steps and specifically the citrix web interface server try this.
1. Reboot the server to safe mode with networking.
2. Run a combo of spybot search and destroy free version is fine, trend micro housecall and mcafee stinger, forget about your AV if you have a bad virus your AV will just be bypassed these tools are specifically virus removal tools.

I think its a bit convenient its the webinterface server having the issue, its been a while since i did citrix but i remember the principles well, is this the citrix licencing server? citrix 1 that is? look into this.

regarding you talking of terminal server which is a windows product and citrix server this is confusing me, are you using terminal server role installed on a citrix server? is this the case on citrix 1, 2 and 3?

check the event logger again, check the citrix events, make sure all relevant citrix updates and sp are installed through the citrix management console.

also citrix web interface last time i had it under my belt which was recently enough is heavily linked into IIS in windows on the server it runs from.

please post more specific enviroment data too, ie.. are your servers VM's or not? are you running TS role on Citrix servers in conjunction? etc etc the more info the better, even to the point of are you using Citrix Cag?
0
PkafkasNetwork EngineerAuthor Commented:
Hello Mark Bill,

Yes this is also the Citrix Licensing server and Web Interface Server.  This server also has applications installed and is used for users to logon and use the applications.

Sorry to confuse you but this is a Windows 2008 R2 Sp1 server.  We use it for Citrix, I just assumed it is a Microsoft Terminal Server.  I am trying to check the server roles; but I cannot because it is so un-responsive.  I do think running hitmanpro.exe on the Citrix1 server is a good idea.  What do you think?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
hm i like combo fix actually forgot about that, this is a server, be very careful running these tools on them. I highly recommend combo fix actually but I do not think a virus is the source of a problem here so start with the three above and go from there. Im actually not going to recommend running hitman pro or combofix on a citrix server of that importance tbh.

go all the other routes first.

Do you have image backups of this box?

I mentioned alot of troubleshooting steps above.
0
PkafkasNetwork EngineerAuthor Commented:
Hello mark Bill:

1.  These Servers are Virtual Machines with VMware 5.5 Update 2.

2.  This is a very old version of XenApp and it ahs not been updated in a very long time.
        a.  These Citrix Servers have the "Remote Desktop Services" role installed on them.

3.  We are actually testing a new Citrix Farm that is working very well.

4.  For now, I have reconfigured the Web portal not to put users on the bad Citrix Server.
        a.  Hence they are putting users on Citrix 2 and Citrix3.

5.  Will reboot the Citrix1 server soon.

6.  Why do you not recommend running hitman pro?  Do you think it will screw something up?
           a.  Do you think that comboxfix might screw something up as well?

Which 3 steps are you referencing to try first?  I am not sure which 3 steps you are referring to.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
1. reboot the server into safe mode with networking and run spybot s&d and stinger or housecall on it.
2. install any relevant citrix updates.
3. check the event logs in windows and the logs in citrix management console too.

start here.

the web interface side of it is working fine, why dont you force the users who dont use it to use a different citrix server and then wait for the upgrade.

what version of windows and sp level are these servers?

i dont recommend it because (1) its a citrix licencing and web interface server (2) its a server (3) i dont know if you have any backups or not (4) these tools are not really designed for servers.
0
PkafkasNetwork EngineerAuthor Commented:
OK,  I would like to provide a status update.

I re-configured the Web portal to direct our users to the other 2 servers and not to the bad Citrix1 users.  That should provide some temporary relief.  As does the fact that we are close to migrating to our new Citrix XenApp Environment.  We are close to finishing our testing phase before we migrate over.  Hence we have 25% of the staff using the new Citrix system.

The Citrix1 Server is Windows 2008 R2 Sp1.  I ran the Hitman Pro software on the Citrix1 server and it did not find anything except for some riskware for a shopping website installed.  I ended up quarantining that shopping installer .exe file.

I did restart the Server again (not in safe mode), after hitmanpro. ran its scan.  I wish I knew your warning before then; but, so far everyone can work just fine in the Citrix1 and all of the old Citrix Farm servers.  Right now I and 2 other users are logged on to Citrix1 server as of 9:14 am.  I was logged on pretty much all night to the Citrix1 server as well (from 2:00 am) and I fell a sleep, No problems thus far.

I will check the event logs through out the day.  If the server is fine with a select number of users.  Perhaps a user's device is triggering the problem.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
good to hear :)
0
PkafkasNetwork EngineerAuthor Commented:
So far, I have been on the "Bad" citrix1 all day through Remote Desktop and I have not had any problems.  Equally important one other user has been on the same Citrix1 Server all day as well.

I recently asked another employee to logon to the "Bad" Citrix1 server at 3:30 pm.  The work day is until 5:00 pm.  it is now 3:50 pm and all is well so far.  I supposed we need to proceed with "Baby Steps" to find out what the issue is.
0
PkafkasNetwork EngineerAuthor Commented:
I will have more users try out the "bad" citrix1 server today and see what happens.  I and 2 other people were on it all day yesterday.  I do not see any error messages in the event viewer.  I used to see things like could not load driver but no errors from yesterday.

The thing to try is to add more users a little at a time and see if and what may trigger this problem.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
If your allowing users to load manufacturer third party drivers it is not uncommon to get these errors in event viewer.
ideally for citrix printing should be using citrix universal print drivers only.

as you are moving enviroments and i dont think this is causing the issue here we can just leave that one for now.  
just an fyi regarding the errors.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
PkafkasNetwork EngineerAuthor Commented:
I should correct myself.  The remote users do not have the privledges to install new drivers; but, I was thinking that perhaps when their machines connect then that triggers some kind of problem.

Today I had a few other users try out the "Bad" citrix1 server and there were no problems.  I will allow the existing users to continue to connect and allow a few more going forward.  A little bit by little bit.
0
PkafkasNetwork EngineerAuthor Commented:
I remember when the server was having problems that the PrintIsolationhost.exe was going berserk on the processes tab.  many different instances and I tried ending the process but that did not help.  Only a restart helped.

http://discussions.citrix.com/topic/310089-printisolationhostexe-and-spoolsvexe-pinning-cpu/

The problem has not come up in a couple of days and if it was a driver problem, then wouldn't it be more consistant.  Considering we use pretty much  the same driver for almost all of our printers and I have not added any drivers to the Citrix server because we just use a separate printer server for printing.  No printer drivers are installed on the Citrix1 server I n a very, very long time.  Before my time, when it was first setup perhaps.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
I think its funny its a printing service going nuts tbh, do you use only citrix universal print drivers or are you using third party print drivers?

Ive had to strip down massive citrix print drivers before from using 3rd party drivers.

How are things going now?
0
PkafkasNetwork EngineerAuthor Commented:
For the last 3 days things have been working just fine.  I have had a select number of power users logon to the "Bad" Citrix1 server and there have been no issues.  We do use a couple of specific drivers; but not a lot and those same drivers are installed on our Printer server not our Citrix servers that have been in production for years.

Everyone else is using the other 2 servers and the other 2 Citrix Servers are also working just fine.  I did run "hitmanpro" pro on the Citrix1 server Tuesday night.  Maybe that helped.  I am considering letting more users logon to the Citrix1 server every day.  I can test it over the weekend as well and watch and see.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
if there being carried onto citrix sessions by users there being used on the citrix sessions unless citrix is set to use universal only drivers which it should be. I think this is the issue, as your moving to a new enviroment shortly i dont think it matters.

I would definetly recomend forcing universal drivers for citrix on this new enviroment you can learn from this one no doubts.

http://discussions.citrix.com/topic/310089-printisolationhostexe-and-spoolsvexe-pinning-cpu/ 

I think this is the issue your having funnily enough its the link you also posted, its also known as a rogue driver and is a frigging nightmare to resolve, i could be mistaken in this line but if a user is bringing a new printer onto the server through a citrix session along with a third party driver this could be the cause.

just force universal print drivers to fix, better still you have a workaround in place just get by on it until you move to the new enviroment and configure universal drivers only on the new citrix enviroment. what versions are you moving too on the new enviroment?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
its clearly not the server by the sounds of it with the power users and your admins working fine, no errors in citrix also points to this as its a windows issue not logged in citrix consoles.

im especially sure this is the issue if the print spooler is failing crashing non responsive, thats a common one for this.
0
PkafkasNetwork EngineerAuthor Commented:
How does one force force universal drivers for citrix on its environment?  I may have to ask the Citrix consultant about that.

So Citrix is supposed to only work with universal print drivers only?  So if you have an HP Laserjet 4000n (for example) one should use a universal print driver with that printer?  What if a user is connecting from a home PC?

It is peculiar that the other 2 Citrix servers are working just fine.
0
PkafkasNetwork EngineerAuthor Commented:
We are dealing with some separate performance issues on the new Test Citrix environment; hence we will try to figure that out before we move everyone over to that new Citrix System.

My plan remains to add more users to the Citrix 1 serer and to see if any specific users are triggering the problems.
0
PkafkasNetwork EngineerAuthor Commented:
This printer isolation problem happened again with a select n umber of users.  Each of these users have logged on to the server in the past during the test week.  I think the problem is with the server and not with the user devices.

Those same users just logged on to the other Citrix Server (that is still in production and everyone is functioning just fine on the other servers.  Hence the same devices are loggining on to teh other Citrix servers (Citrix 2) and no problems.

My plan for now is to:

1.  Wait for everyone to be out of Citrix and then restart the Citrix1 server.
         a.  Then see if the server is responsive again.

2.  If it is responsive again, I plan to document any printers that are not using an HP universal printer driver.
          a.  Then remove those specific printers from printers and devices.

3.  This sounds like a Citrix and printer specific issue.
          a.  Perhaps we have a specialist look into this or have the users use the other 2 Citrix servers.
          b.  Until we move to the new Citrix system (Which 20% of the users are already using).

I just am afraid if this problem follows over to the new system.  That is why I would like to know a fix for this problem.  In the mean time, restarting a server in the middle of the day may not be a good option if it kicks off any other users.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
I think option 2 you just posted will resolve this issue, there is also an overall setting in citrix somewhere.

you basically stop it from allowing people to use these drivers, like an over ride.

I understand what your saying regards to different users, with the service thats crashing i honestly dont see how its not a print driver tbh.
0
PkafkasNetwork EngineerAuthor Commented:
The same print drivers are installed on the other 2 Citrix servers as well.  The same users are logged on to the other 2 Citrix servers as well.  I do not know.   I think it's a printer driver problem but why us it server specific?
0
PkafkasNetwork EngineerAuthor Commented:
I am not familiar enough with Citrix to begin making changes on the Citrix configuration:  http://support.citrix.com/article/CTX128786

http://support.citrix.com/proddocs/topic/xenapp65-admin/ps-console-policies-rules-printer-drivers-v2.html

http://support.citrix.com/proddocs/topic/xenapp65-admin/ps-console-policies-rules-printer-drivers-v2.html

I am not familiar enough with Citrix to fix this printer driver problem.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Universal print driver usage
This setting specifies when to use universal printing. By default, universal printing is used only if the requested driver is unavailable.

Universal printing employs generic printer drivers instead of standard model-specific drivers, potentially simplifying the burden of driver management on host computers. The availability of universal print drivers depends on the capabilities of the user device, host, and print server software. In certain configurations, universal printing might not be available.

When adding this setting to a policy, select an option:
Use only printer model specific drivers specifies that the client printer use only the standard model-specific drivers that are auto-created at logon. If the requested driver is unavailable, the client printer cannot be auto-created.
Use universal printing only specifies that no standard model-specific drivers are used. Only universal print drivers are used to create printers.
Use universal printing only if requested driver is unavailable uses standard model-specific drivers for printer creation if they are available. If the driver is not available on the server, the client printer is created automatically with the appropriate universal driver.
Use printer model specific drivers only if universal printing is unavailable uses the universal printer driver if it is available. If the driver is not available on the server, the client printer is created automatically with the appropriate model-specific printer driver.

-----

you can implement these changes safely, think about it worst case someone cant print. these issues are pretty messy to clear up which is why i suggested to leave it.

ive done this once before, only once, i remember i felt nervous doing it but it isnt a big deal.
If this is a VM just take a full backup first before you do anything, it becomes alot easier then.
use FASTSCP by VEAM out of hours to back it up and make the changes or use your VM backup software if you have some, you could even take a snapshop in vmware or hyperv
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
first thing to do is identify the settings your going to change in the citrix console and ensure that it is currently set incorrectly. then take it from here above post.
0
PkafkasNetwork EngineerAuthor Commented:
Mark Bill:

Can this policy be set for 1 specific VM or is a policy for the entire Citrix Farm?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Im pretty sure its entire farm cant say with certainty as im not using citrix these days, although i have alot in the past obviously, I think the settings in the print admin section of the citrix program files folder.
0
PkafkasNetwork EngineerAuthor Commented:
Here is the deal, we do have some Cannon Copiers that are heavily used and a Dell Printer that is an older Dell that does use a specific model driver.  Those have been in place for years and years before my time with the company.

I will take a look and see what and if any printer policies are already in place for printing.  If I need to reverse the policy or remove it will the user that need to print need to log off and then to log back on? In case I need to back track?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
its a setting that states allow use drivers or not, if you click no it forces the server to use a universal and stops the client trying to bring the driver through onto the citrix server through an ica session basically an rdp session.

if the setting is on it will work just as it does now.

it is a simple on off switch as far as i remember, do not be afraid ;). i used to be scared but i lost that along time ago.
need to break eggs to make an omelette.
0
PkafkasNetwork EngineerAuthor Commented:
Thanks Mark Bill, I will need to review the material and see what the next step should be.  I would like to fix this issue just in case it shows up with the new system.

Like I said I will need to review the material.  Right now I just have a select few users use that server.  Today I had 4 users and there were no problems.  I suspect one use on this use server is triggering the problem via a  specific printer.

On another note a Citrix consultant replied back to me with this:

"Interesting scenario. If this is believed to be relating to the printer subsystem the I would recommend using the Citrix Stress Printers utility and check out each printer driver to confirm there are no issues there. Just follow through this article and it will guide you through the process http://support.citrix.com/article/CTX109374

Also this discussion shows how to troubleshoot printer related issues. I use the recommendation regarding the change to the concurrent add events and number of retries settings from 1 to 5 which will simulate better how concurrent logins to the server are being done in reality.

http://discussions.citrix.com/topic/308407-xenapp-6x-printing-best-practices-recommendations-and-requirements/
"
0
PkafkasNetwork EngineerAuthor Commented:
I think I may know what the problem is.  I remember from the policies that were already created that the Citrix session did not allow for locally installed printers (on the client Device) to be created in the new Citrix session.

This does actually occur on Ctrix1; but, that does not occur on Citrix2, or Citrix3.  I have to look at that policy on that server and see how that is applied.  That could be the problem because someone is logging in and trying to print to a locally default printer and then the problem begins.
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
good stuff
0
PkafkasNetwork EngineerAuthor Commented:
It appears that there is a policy to not allow client printers from being added.  But the locally installed printers are being added when one logs on to Citrix1

Please see the screen shots.

screen0
screen1
Screen shot of Citrix1

screen3
Screen shot of Citrix2 server

screen4
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Hey

Can you see if this article does the job, looks like your on Xen 5.0 or above.

http://support.citrix.com/article/CTX106812

M
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
this one will be useful aswell http://support.citrix.com/article/CTX124817
0
PkafkasNetwork EngineerAuthor Commented:
This question is getting very long and confusing.  Basically:

1.  Could the fact that client printers are being added to this Citrix1 server cause the printer services to hang?

2.  The fact that the other 2 Citrix servers, do not allow the locally installed printers to be setup and they work with the same users and devices logging would indicate the problem is Server Specific.

How can we fix this server to obey the Citrix policies and to not auto-create local printers?

I wonder if no one logs on to this server, as an experiment will the printers and devices option hang up? Will other service hang up as they did before?
0
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
1. Definetly. It is quite clear it is, the services crashing, the related articles we both found. The fact I knew prior to the articles this is a known issue with Citrix.

2. I think this point is backing up point 1 no?

So we have established Universal print drivers are being used on Citrix2 and Citrix3 correct? and not on Citrix1 aka allowing third party drivers come onboard like dell and hp for example.

This clearly backs up the rogue driver theory, as discussed force Citrix1 to stop bringing third party drivers onboard if this is the case.

Thanks for the solution.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.