Terminal Server 2008 Slow

I know it sounds odd, but I've found that Win2008 has performance problems when it's IP address doesn't have a valid PTR record in DNS. For some reason Win2008 queries DNS for its own IP address, and if it doesn't get an answer or gets "not found"  it keeps repeating the query. I first noticed this myself with TS -- the DNS server was getting 100 queries per second with several users logged in, all for the TS servers own IP address! Adding the PTR record in our master DNS (which wasn't running on the problem server) immediately stopped the repetitive queries.

Thanks for the suggestions.
I do have a valid PTR record in DNS.

Like I said I have a single LAN nic for all the VMs.
I will disable the NIC teaming and see if that helps at all. We only have a single switch, I think you need both cables plugged into the same switch for NIC teaming (unlike multipath)
I'm not sure if it's related to the other VMs only because they seem to be functioning fine. The network usage on the VMs (and the physical server they're on) seems fine.

Thanks!

NExt time it happends can you log into the server locally and see if the permormance is degraded?

Plimpias. I logged on locally to the TS today when it was slow, about 1pm. Locally it was just as slow as it is over RDP.

I did notice the CPU is higher than normal. Spoolsv.exe and msiexec.exe being 2 of the highest. I disabled Windows Installer service and the performance was much better. I noticed that in the eventvwr this was there:
"Windows Installer reconfigured the product. Product Name: 64 Bit HP BiDi Channel Components Installer. Product Version: 1.2.0.2. Product Language: 1033. Reconfiguration success or error status: 0."

It happens every few seconds. We have about 15 HP printers....

Ah!  Spoolsv.exe is a huge clue! Try this:

 1. Go to Control Panel > Administrative Tools > Services and stop the Print Spooler.

  2. Turn off any printers attached to the server.

  3. Navigate to C:\WINDOWS\system32\spool\PRINTERS and delete any .SHD or .SPL fils you find there.

  4, Turn the printers back on if necessary

  5,. Restart the Print Spooler service

This may clear up your problem. You're getting corrupt spool files from another user or server on the network, which Spoolsv.exe is famous for choking on.

I see,

instead of using local printers on the TS server, move those printers to another server and use the terminal server as a client. You may have some corrupt drivers. install the latest certified drivers by HP and try not to use any HP tools while doing it. Use on the Basic driver for each printer.

MSIExec is probably an installation of some sort that is running, were u installign something while you were monitoring?

Hey Guys.
Here is what I've done so far.
We still were having this problem; Spoolsv.exe was spiking around 20% very often. This happens mainly when people are logging in, adding the printers. Eventually we called up Microsoft PSS and opened a case. We got a very unknowledgeable guy; I thought PSS was supposed to be high-quality engineers...either way. He couldn't get it. We eventually did a reinstall of the TS and got a refund.

To get around it we started installing all printers via login script on everyone's local machines. Then using TS Easy Print to print. This isn't a good solution because some applications don't support XPS fully and it is very slow to print, large files will eventually time out. We're starting to install all the printers again on our print server and deploy via GPO (Printer Management in Server 2008, similar to 2003 R2).

We are getting the same thing, when people login Spoolsv.exe goes up to 20%, the CPU itself is completely at 100% and everything is slow. Not as bad as last time, we are only testing 1/5th of our users with GPO printers.

We are using only 2 print drivers. HP Universal Print Driver PS for all HP printers. And then the new Xerox Global print driver, it is supposed to work on all Xerox printers and most of any other printer. We've had good luck with it on all our printers. Using GPO it should only deploy those 2 drivers with about 7 printers. A login can sometimes be 10 minutes.

What's annoying is that it isn't just the first login when printers are being installed. We can login and logout and login again and it's the same time to install printers. I thought that after it installed it once it wouldn't have to again.

Another wierd thing is that sometimes printers are there and sometimes they aren't. When they disappear and then the user logs in the default printer switches, which is annoying. GPO seems normal. No errors in the eventvwr. When the printers don't get installed the login is very fast. We logout and relogin and typically the login is slow and printers get installed.

We have a dedicated print server, which is pushing out the printers via GPO. It is local, 1Gbps, to the terminal servers. We have 2 physical terminal servers (round robin load balancing) and both have the same problem, fresh installs. Can anyone give any suggestions or insight?

Everything else is working fine. Just printers are the issue. When Spoolsv is taking up so much CPU then everything is slow, but when we disable the spooler or wait until people are logged in everything is fine. This is a daily thing though.

Thanks!

Another update. I have tried to install the printers via login script, gpo, cse. They all take the same amount of time. It seems like it's 1minute per printer.

I can connect to the print server via UNC and right-click on a printer, hit connect, and it takes over a minute. Even if the driver is installed locally.

I have a bare 2008 server with no roles and I add the printer via UNC and it takes 1 minute and 15 seconds. I delete the printer and add it again (driver is installed) and it takes 45 seconds.

I have a freshly installed 2003 server and add the printer and it takes 25 seconds. I delete and readd and it takes 3 seconds to add.

Both servers have same hardware and are on the LAN with the print server.

Is there any known issues with adding printers in 2008? Our terminal servers are 2008.

Thanks!

I have the same problem and it has something to do with an msi file which is in de dependency list of the printer driver. So for some reason on Windows Server 2008 it keeps calling that MSI file.
This fix is not very nice but effective, basically it just prevent "cioum32.msi" from running on your system. To apply the fix run "cacls %windir%\system32\spool\drivers\w32x86\3\*.msi /E /D USERS SYSTEM" from a command line.

Thanks for the reply. This was an issue, that MSIExec would run dozens of times. I just disabled Windows Installer and it fixed this.

The issue still exists. Adding a printer on Server 2008 is incredibly slow. Everything else is fine. This happens on 2 TS 2008 servers even after complete reinstalls. They are fairly locked down servers with users not having access to much besides their programs.

Plz try it and tell me what happens. i tried disabling installer as well.

Hello.
I'm sorry but this did not work. Adding a printer still takes a long time. We got it to not install the printers during every login, only when a printer is added to the network or when a user needs a new printer. Or when a new user is added. It takes about 1 minute per printer.

BiSHGoD,

Would I be corrrect if I said that your users are likely mapping network printers in their terminal server 2008 sessions, and that these printers are made by Hewlett-Packard?

If so, I have a solution for you.  There is a known issue with the HP BiDi and CIO components included in the discrete and UPD drivers (respectively).  This issue affects both Vista and Server 2008 (including Terminal Services).  You are likely seeing a huge number of MSInstaller events in your application log stating the following:

Product: 32 Bit HP CIO Components Installer -- Configuration completed successfully

and

Windows Installer reconfigured the product. Product Name: 32 Bit HP CIO Components Installer. Product Version: 2.1.4. Product Language: 1033. Reconfiguration success or error status: 0.

If my assumptions about your environment are correct, please go to this thread in the HP Community Forum for a workaround, as well as steps to take to contact HP for updated DLL files for your printer drivers.

http://forums11.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=1259636

This issue will be fixed in the next revision of the Universal Print Driver which HP has said will be released in either December 08/January 09.

Thanks,
Haas Computing Services

Thanks for the info. It is very similar to what I'm having. You are correct on my setup besides I also have multiple copy machines that are installed via GPO using the Xerox Global driver. These printers are also slow but if the server is slow because of the HP printers this could cause the copy machines to be slow loading.

I did notice the MSIExec taking up high CPU. What I did was disable Windows Installer and that fixed the problem. Do you still think that deleting those registry keys on the TS and getting the DLLs from HP will further help the issue?

We've had a different problem with the server freezing up that we thought was unrelated (still might be). We have a MSFT PSS case open and they think that the crashes happen when we have a lot of Inactive TS Ports in the registry. We can get over 1500 of these. MSFT has me delete the following subkeys every day before my daily reboot. It's helped a bit with the freezing, only happens once every 2 weeks or so. Slowness hasn't gotten better:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\

Thanks!

What version of HP drivers are you using?  I haven't had any problems with Xerox printers on 2008, but we mainly use HP drivers in our organization, so I really couldn't tell you.

While you've already disabled the Microsoft Installer service, I do highly recommend contacting HP for the updated DLL files, because the HP driver will still attempt to run MSIExec everytime a user maps a network printer; even though the service is disabled, I think you should stop the problem at the source if possible (instead of using workaround).

Before you stopped the Microsoft Installer service, the HP driver likely created the millions of registry entries on your Terminal Server and other Vista/2008 based printing clients.  These also need to get cleaned out.

Check these registry keys and you will likely see what I'm talking about.  The first key would only be found on your terminal servers--however the second key will be associated with any Vista/2008 machine that has mapped a printer.
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Hewlett-Packard
HKU\ .DEFAULT\Software\Hewlett-Packard

I've configured GPO Clientside Preferences to delete both of the registry keys above..

Note that the DEFAULT profile is mentioned as one of the registry keys containing a lot of garbage.  These locations are where the millions of registry API calls are being made. That could be contributing to your problem.  It inflated the size of our user profiles causing exceeded quotas and increased login times (we use roaming profiles).

Based on what you've said about already disabling the Microsoft Installer Service (I think I missed that point in the earlier posts)--I think there could something else going on.  I would definitely contact HP for the updated DLL's and re-enable Microsoft Installer Service as a first step, then check the registry entries I mentioned above and ensure they are cleaned out.  

FYI: We typically have no more than 20-30 users max on each of our terminal servers, so our org hasn't experienced the load that you are shooting for.  We run 64-bit and 32-bit terminal servers (separate farms for separate purposes).  Something else to consider is that you are running 32-bit app's in a 64-bit OS.  I'm not sure if there are any scaling constraints in the 32-bit emulation?  We typically run only 64-bit native apps on our 64-bit terminal servers (unless absolutely necessary in the case of Word/Excel).

Here are our specs:
32-bit farm, serving ~150 thin clients for general computing use
4 - Dell PowerEdge 1950's
2 - Quad-Core Xeon processors
4GB RAM
Windows Server 2008 Standard (x86)
MSOffice Suite (minus Outlook)
Educational Applications

64-bit farm server farm (for CPU/memory intensive research app's)
2 - Dell PowerEdge 1950's, on average about 5-6 connections/node
2 - Quad-Core Xeon processors
16GB RAM
Windows Server 2008 Enterprise (x64)
MS Word & Excel only
Research App's (SAS, MATLAB, Stata, Mathematica)

Let me know how things go...

We're using HP Universal Printing PS (61.81.41.2). My COIUM.DLL is 1.2.1.51.

I'm talking to HP right now. They're trying to troubleshoot the issue, not giving me the DLLs direct. I've referenced them to your HP case and emailed them your forum post but they're still giving me the run around.

I'll delete those registry files tonight when no one is on. We have about 60-70 users on each server, 32GB ram each.

The first TS registry key has 4 hex subkeys under it.

The HKU/Default key has 2277 records under HP. I'll defintely delete those. Can I just delete the "Hewlett-Packard" key directly and it won't mess with anything? Just want to ensure any settings or printers users currently have don't have to be recreated.

Do you use 2008 TS Session Broker for your farm?

Thanks a lot!

HCSSYS.
I think HP added me to the ticket you originally opened. I referenced it and they just me as a contact even though I said I was totally seperate. They are very confused. I'm awaiting a call back so they can escalate me again to the right people. Does the small team who handles these DLLs have a name that I can tell HP?

Hopefully shortly they'll give me the DLLs.

Thanks!

Has anyone found a fix for this yet?
I have endless events with 32 bit hp cio components installer configuration completed successfully.
I am running a 2003 TS using the HP UPD. My PrintSpooler service is currently taking up 75,64K in memory.

I would recommend getting in touch with HP to get the drivers. It seems that it's a Windows 2008/64bit issue but maybe it's similar. Disabling Windows Installer got rid of those events and made the server much faster (MSI was getting installed everytime someone logged in). Or you could 0 out the MSIs like others have.

I got the 64bit CIO dlls finally and things are much better.In the Feb '09 universal print driver this should be fixed.

Hey BiSHGoD--sorry, been out of the country for most of December and catching up on e-mails.  I can drop you a name of someone who had helped me out, but I probably shouldn't give direct contact information:

This is the person who help me out:
brian dalley
ats \ ipg gcc
hp imaging and printing group

I am cuurently implementing a 22000 user Citrix environment. we using Thin Print RES PowerFuse and a Xerox print solution. I see you ar4e investigating the HP driver?! what about the Xerox UPD. I have been reliably told by RES that the Xerox universal print driver is know to impact the performance on terminal Services and therefore it is not advisable to use. I am not suggesting that this is the sole cause to your problems however it could be a contributing factor. Becuase I am roling out a Xerox print solution with Citrix I will be doing a lot of testing with the Xerox UPD

Microsoft did find a bunch of errors with the Xerox universal print driver that might have been causing our freezes. But the slowness they attributed to the HP driver. I think both need to be tested better at a rollout location before going to production.

I think that link bellow condenses some issues that are related for this problem. Problems that apply to 2003 Server too.
 
This document points out problems related to HP drivers and DLL´s.

Hope this helps.

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA0-8465ENW.pdf

We had this problem, and we fixed it by stopping and disabling the Net Driver HPZ12, and Plm Driver HPZ12 servcie on the client side.  One of these services allows a scan to be done across the network, and one queries an HP printer for ink levels.  Our problem came back recently, and I'm not sure if iwas due to an HP update or a Windows update.

We too had the same issue on a Windows 2003 server with Citrix XenApp. I retrieved an updated hpzbdi.dll (c:\windows\system32\spool\drivers\w32x86\3\) which was only available from HP. The old version which causes problems is 1.1.1.21, the updated version is 1.1.2.51.

More information can be found here, but contact HP and ask for the dll.

http://forums.citrix.com/thread.jspa?threadID=103686&tstart=15