Link to home
Start Free TrialLog in
Avatar of BiSHGoD
BiSHGoD

asked on

Terminal Server 2008 Slow

Hello.
I am having general slowness issues on my Server 2008 terminal server. The issue only happens when over 40-50 users are online, I need 100-150 users on it. This is their only way to work, they use thin clients over RDP.

I have 3 physical servers, all the same general specs. All are new dual xeon quad core HP servers. 2 of the servers have 12GB of RAM and are used for VMWare. They all have mirrored 147GB 15kRPM Enterprise SAS drives but most data is stored on a SANs (Raid 10, SAS drives).

VM1 (Physical. 12GB RAM). Windows Server 2008 VMWare Server 1.06
  DC1      Domain Controller (1.5GB RAM). Windows Server 2008
  PS1      Print Server (2GB RAM). Windows Server 2008
  EX1      Exchange 2007 Server (3.5GB RAM). Windows Server 2008

VM2 (Physical 12GB RAM). Windows Server 2008. VMWare Server 1.06
  DC2      Domain Controller (1.5GB). Windows Server 2008
  FS1      File Server (2GB RAM). Windows Server 2008
  SQL1      SQL Server 2005 (3.5GB RAM). Windows Server 2008


My last physical server is my terminal server, TS1. TS1 is the same as the other 2 servers besides this one has 32GB of RAM. It doesn't run VMWare at all. It has Server 2008 installed with Terminal Services. The only applications installed are Microsoft Office 2007. Outlook 2007 is connected to the Exchange server in it's default way.

No one thing is slow. Everything gets slow when too many people are logged in: Outlook, Word, Excel, Internet Explorer, browsing My Computer. The lag can get really bad. Upwards of 30 seconds waiting for a window to open. Typing lag in Word is bad too.

There is nothing blatently wrong in the performance monitor. The CPU/Disk spikes every once and a while but that can be normal. The only thing that I know is good is the RAM. Page file is never used.

Here is some info:
Roaming profiles are not used.
My Documents are redirected using GPO to a SANs (via FS1).
All data is accessed via mapped drive from FS1. The data on FS1 is on a SANs, using Microsoft ISCSI Initiator.
Desktop is redirected to C:\Users\Universal, a write protected folder. This gives us a universal desktop users can't modify.
Printers are shared from the Print server
Redirected printers are disabled
Disk Defrag is set to nightly
Nightly reboot of TS1
The other servers aren't noticeably slow to RDP into when TS1 is slow.
Vista graphics/user experience is disabled.
The Vmware physical servers have dual nics. 1 is for LAN and 1 is for SANs on different subnets.
The TS1 physical server also has dual nics but it is configured for NIC Teaming via HP's utility. I've done ipef benchmarks on a gigabit switch and I get about 500Mbps up/down (250Mbps without teaming, which is about Windows' max throughput even on gigabit).

Microsoft says hundreds of users can use a single terminal server in their TS white paper. The main thing users use is RAM, which we're fine on.


I can provide any information. Including benchmarks/perf logs done over the day.

Also if someone can include their own setup with how many users their Windows 2008 TS supports(Hopefully using Office 2007 as it uses more resources than 2003).

Thanks!
Avatar of plimpias
plimpias
Flag of United States of America image

A good test would be to see if the same slowness happends when you log in locally, without using RDP. For each VMware server, do you have a dedicated nic?

I would take that teaming on unless you are have two switches and are using redudency, the less you have the better your server.
I know it sounds odd, but I've found that Win2008 has performance problems when it's IP address doesn't have a valid PTR record in DNS. For some reason Win2008 queries DNS for its own IP address, and if it doesn't get an answer or gets "not found"  it keeps repeating the query. I first noticed this myself with TS -- the DNS server was getting 100 queries per second with several users logged in, all for the TS servers own IP address! Adding the PTR record in our master DNS (which wasn't running on the problem server) immediately stopped the repetitive queries.
Avatar of BiSHGoD
BiSHGoD

ASKER

Thanks for the suggestions.
I do have a valid PTR record in DNS.

Like I said I have a single LAN nic for all the VMs.
I will disable the NIC teaming and see if that helps at all. We only have a single switch, I think you need both cables plugged into the same switch for NIC teaming (unlike multipath)
I'm not sure if it's related to the other VMs only because they seem to be functioning fine. The network usage on the VMs (and the physical server they're on) seems fine.

Thanks!
NExt time it happends can you log into the server locally and see if the permormance is degraded?
Avatar of BiSHGoD

ASKER

Plimpias. I logged on locally to the TS today when it was slow, about 1pm. Locally it was just as slow as it is over RDP.

I did notice the CPU is higher than normal. Spoolsv.exe and msiexec.exe being 2 of the highest. I disabled Windows Installer service and the performance was much better. I noticed that in the eventvwr this was there:
"Windows Installer reconfigured the product. Product Name: 64 Bit HP BiDi Channel Components Installer. Product Version: 1.2.0.2. Product Language: 1033. Reconfiguration success or error status: 0."

It happens every few seconds. We have about 15 HP printers....
Ah!  Spoolsv.exe is a huge clue! Try this:

 1. Go to Control Panel > Administrative Tools > Services and stop the Print Spooler.

  2. Turn off any printers attached to the server.

  3. Navigate to C:\WINDOWS\system32\spool\PRINTERS and delete any .SHD or .SPL fils you find there.

  4, Turn the printers back on if necessary

  5,. Restart the Print Spooler service

This may clear up your problem. You're getting corrupt spool files from another user or server on the network, which Spoolsv.exe is famous for choking on.

I see,

instead of using local printers on the TS server, move those printers to another server and use the terminal server as a client. You may have some corrupt drivers. install the latest certified drivers by HP and try not to use any HP tools while doing it. Use on the Basic driver for each printer.

MSIExec is probably an installation of some sort that is running, were u installign something while you were monitoring?
Avatar of BiSHGoD

ASKER

Hey Guys.
Here is what I've done so far.
We still were having this problem; Spoolsv.exe was spiking around 20% very often. This happens mainly when people are logging in, adding the printers. Eventually we called up Microsoft PSS and opened a case. We got a very unknowledgeable guy; I thought PSS was supposed to be high-quality engineers...either way. He couldn't get it. We eventually did a reinstall of the TS and got a refund.

To get around it we started installing all printers via login script on everyone's local machines. Then using TS Easy Print to print. This isn't a good solution because some applications don't support XPS fully and it is very slow to print, large files will eventually time out. We're starting to install all the printers again on our print server and deploy via GPO (Printer Management in Server 2008, similar to 2003 R2).

We are getting the same thing, when people login Spoolsv.exe goes up to 20%, the CPU itself is completely at 100% and everything is slow. Not as bad as last time, we are only testing 1/5th of our users with GPO printers.

We are using only 2 print drivers. HP Universal Print Driver PS for all HP printers. And then the new Xerox Global print driver, it is supposed to work on all Xerox printers and most of any other printer. We've had good luck with it on all our printers. Using GPO it should only deploy those 2 drivers with about 7 printers. A login can sometimes be 10 minutes.

What's annoying is that it isn't just the first login when printers are being installed. We can login and logout and login again and it's the same time to install printers. I thought that after it installed it once it wouldn't have to again.

Another wierd thing is that sometimes printers are there and sometimes they aren't. When they disappear and then the user logs in the default printer switches, which is annoying. GPO seems normal. No errors in the eventvwr. When the printers don't get installed the login is very fast. We logout and relogin and typically the login is slow and printers get installed.

We have a dedicated print server, which is pushing out the printers via GPO. It is local, 1Gbps, to the terminal servers. We have 2 physical terminal servers (round robin load balancing) and both have the same problem, fresh installs. Can anyone give any suggestions or insight?

Everything else is working fine. Just printers are the issue. When Spoolsv is taking up so much CPU then everything is slow, but when we disable the spooler or wait until people are logged in everything is fine. This is a daily thing though.

Thanks!
Avatar of BiSHGoD

ASKER

Another update. I have tried to install the printers via login script, gpo, cse. They all take the same amount of time. It seems like it's 1minute per printer.

I can connect to the print server via UNC and right-click on a printer, hit connect, and it takes over a minute. Even if the driver is installed locally.

I have a bare 2008 server with no roles and I add the printer via UNC and it takes 1 minute and 15 seconds. I delete the printer and add it again (driver is installed) and it takes 45 seconds.

I have a freshly installed 2003 server and add the printer and it takes 25 seconds. I delete and readd and it takes 3 seconds to add.

Both servers have same hardware and are on the LAN with the print server.

Is there any known issues with adding printers in 2008? Our terminal servers are 2008.

Thanks!
I have the same problem and it has something to do with an msi file which is in de dependency list of the printer driver. So for some reason on Windows Server 2008 it keeps calling that MSI file.
This fix is not very nice but effective, basically it just prevent "cioum32.msi" from running on your system. To apply the fix run "cacls %windir%\system32\spool\drivers\w32x86\3\*.msi /E /D USERS SYSTEM" from a command line.
Avatar of BiSHGoD

ASKER

Thanks for the reply. This was an issue, that MSIExec would run dozens of times. I just disabled Windows Installer and it fixed this.

The issue still exists. Adding a printer on Server 2008 is incredibly slow. Everything else is fine. This happens on 2 TS 2008 servers even after complete reinstalls. They are fairly locked down servers with users not having access to much besides their programs.
Plz try it and tell me what happens. i tried disabling installer as well.
Avatar of BiSHGoD

ASKER

Hello.
I'm sorry but this did not work. Adding a printer still takes a long time. We got it to not install the printers during every login, only when a printer is added to the network or when a user needs a new printer. Or when a new user is added. It takes about 1 minute per printer.
BiSHGoD,

Would I be corrrect if I said that your users are likely mapping network printers in their terminal server 2008 sessions, and that these printers are made by Hewlett-Packard?

If so, I have a solution for you.  There is a known issue with the HP BiDi and CIO components included in the discrete and UPD drivers (respectively).  This issue affects both Vista and Server 2008 (including Terminal Services).  You are likely seeing a huge number of MSInstaller events in your application log stating the following:

Product: 32 Bit HP CIO Components Installer -- Configuration completed successfully

and

Windows Installer reconfigured the product. Product Name: 32 Bit HP CIO Components Installer. Product Version: 2.1.4. Product Language: 1033. Reconfiguration success or error status: 0.

If my assumptions about your environment are correct, please go to this thread in the HP Community Forum for a workaround, as well as steps to take to contact HP for updated DLL files for your printer drivers.

http://forums11.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=1259636

This issue will be fixed in the next revision of the Universal Print Driver which HP has said will be released in either December 08/January 09.

Thanks,
Haas Computing Services
Avatar of BiSHGoD

ASKER

Thanks for the info. It is very similar to what I'm having. You are correct on my setup besides I also have multiple copy machines that are installed via GPO using the Xerox Global driver. These printers are also slow but if the server is slow because of the HP printers this could cause the copy machines to be slow loading.

I did notice the MSIExec taking up high CPU. What I did was disable Windows Installer and that fixed the problem. Do you still think that deleting those registry keys on the TS and getting the DLLs from HP will further help the issue?

We've had a different problem with the server freezing up that we thought was unrelated (still might be). We have a MSFT PSS case open and they think that the crashes happen when we have a lot of Inactive TS Ports in the registry. We can get over 1500 of these. MSFT has me delete the following subkeys every day before my daily reboot. It's helped a bit with the freezing, only happens once every 2 weeks or so. Slowness hasn't gotten better:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{28d78fad-5a12-11d1-ae5b-0000f803a8c2}\

Thanks!
What version of HP drivers are you using?  I haven't had any problems with Xerox printers on 2008, but we mainly use HP drivers in our organization, so I really couldn't tell you.

While you've already disabled the Microsoft Installer service, I do highly recommend contacting HP for the updated DLL files, because the HP driver will still attempt to run MSIExec everytime a user maps a network printer; even though the service is disabled, I think you should stop the problem at the source if possible (instead of using workaround).

Before you stopped the Microsoft Installer service, the HP driver likely created the millions of registry entries on your Terminal Server and other Vista/2008 based printing clients.  These also need to get cleaned out.

Check these registry keys and you will likely see what I'm talking about.  The first key would only be found on your terminal servers--however the second key will be associated with any Vista/2008 machine that has mapped a printer.
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Hewlett-Packard
HKU\ .DEFAULT\Software\Hewlett-Packard

I've configured GPO Clientside Preferences to delete both of the registry keys above..

Note that the DEFAULT profile is mentioned as one of the registry keys containing a lot of garbage.  These locations are where the millions of registry API calls are being made. That could be contributing to your problem.  It inflated the size of our user profiles causing exceeded quotas and increased login times (we use roaming profiles).

Based on what you've said about already disabling the Microsoft Installer Service (I think I missed that point in the earlier posts)--I think there could something else going on.  I would definitely contact HP for the updated DLL's and re-enable Microsoft Installer Service as a first step, then check the registry entries I mentioned above and ensure they are cleaned out.  

FYI: We typically have no more than 20-30 users max on each of our terminal servers, so our org hasn't experienced the load that you are shooting for.  We run 64-bit and 32-bit terminal servers (separate farms for separate purposes).  Something else to consider is that you are running 32-bit app's in a 64-bit OS.  I'm not sure if there are any scaling constraints in the 32-bit emulation?  We typically run only 64-bit native apps on our 64-bit terminal servers (unless absolutely necessary in the case of Word/Excel).

Here are our specs:
32-bit farm, serving ~150 thin clients for general computing use
4 - Dell PowerEdge 1950's
2 - Quad-Core Xeon processors
4GB RAM
Windows Server 2008 Standard (x86)
MSOffice Suite (minus Outlook)
Educational Applications

64-bit farm server farm (for CPU/memory intensive research app's)
2 - Dell PowerEdge 1950's, on average about 5-6 connections/node
2 - Quad-Core Xeon processors
16GB RAM
Windows Server 2008 Enterprise (x64)
MS Word & Excel only
Research App's (SAS, MATLAB, Stata, Mathematica)

Let me know how things go...
Avatar of BiSHGoD

ASKER

We're using HP Universal Printing PS (61.81.41.2). My COIUM.DLL is 1.2.1.51.

I'm talking to HP right now. They're trying to troubleshoot the issue, not giving me the DLLs direct. I've referenced them to your HP case and emailed them your forum post but they're still giving me the run around.

I'll delete those registry files tonight when no one is on. We have about 60-70 users on each server, 32GB ram each.

The first TS registry key has 4 hex subkeys under it.

The HKU/Default key has 2277 records under HP. I'll defintely delete those. Can I just delete the "Hewlett-Packard" key directly and it won't mess with anything? Just want to ensure any settings or printers users currently have don't have to be recreated.

Do you use 2008 TS Session Broker for your farm?

Thanks a lot!
ASKER CERTIFIED SOLUTION
Avatar of Robert Lozano
Robert Lozano
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of BiSHGoD

ASKER

HCSSYS.
I think HP added me to the ticket you originally opened. I referenced it and they just me as a contact even though I said I was totally seperate. They are very confused. I'm awaiting a call back so they can escalate me again to the right people. Does the small team who handles these DLLs have a name that I can tell HP?

Hopefully shortly they'll give me the DLLs.

Thanks!
Has anyone found a fix for this yet?
I have endless events with 32 bit hp cio components installer configuration completed successfully.
I am running a 2003 TS using the HP UPD. My PrintSpooler service is currently taking up 75,64K in memory.
Avatar of BiSHGoD

ASKER

I would recommend getting in touch with HP to get the drivers. It seems that it's a Windows 2008/64bit issue but maybe it's similar. Disabling Windows Installer got rid of those events and made the server much faster (MSI was getting installed everytime someone logged in). Or you could 0 out the MSIs like others have.

I got the 64bit CIO dlls finally and things are much better.In the Feb '09 universal print driver this should be fixed.
Hey BiSHGoD--sorry, been out of the country for most of December and catching up on e-mails.  I can drop you a name of someone who had helped me out, but I probably shouldn't give direct contact information:

This is the person who help me out:
brian dalley
ats \ ipg gcc
hp imaging and printing group
I am cuurently implementing a 22000 user Citrix environment. we using Thin Print RES PowerFuse and a Xerox print solution. I see you ar4e investigating the HP driver?! what about the Xerox UPD. I have been reliably told by RES that the Xerox universal print driver is know to impact the performance on terminal Services and therefore it is not advisable to use. I am not suggesting that this is the sole cause to your problems however it could be a contributing factor. Becuase I am roling out a Xerox print solution with Citrix I will be doing a lot of testing with the Xerox UPD
 
Avatar of BiSHGoD

ASKER

Microsoft did find a bunch of errors with the Xerox universal print driver that might have been causing our freezes. But the slowness they attributed to the HP driver. I think both need to be tested better at a rollout location before going to production.
I think that link bellow condenses some issues that are related for this problem. Problems that apply to 2003 Server too.
 
This document points out problems related to HP drivers and DLL´s.

Hope this helps.

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA0-8465ENW.pdf
We had this problem, and we fixed it by stopping and disabling the Net Driver HPZ12, and Plm Driver HPZ12 servcie on the client side.  One of these services allows a scan to be done across the network, and one queries an HP printer for ink levels.  Our problem came back recently, and I'm not sure if iwas due to an HP update or a Windows update.
Avatar of Lee Osborne
We too had the same issue on a Windows 2003 server with Citrix XenApp. I retrieved an updated hpzbdi.dll (c:\windows\system32\spool\drivers\w32x86\3\) which was only available from HP. The old version which causes problems is 1.1.1.21, the updated version is 1.1.2.51.

More information can be found here, but contact HP and ask for the dll.

http://forums.citrix.com/thread.jspa?threadID=103686&tstart=15
I had similar Problems, I had to switch to all Native HP Drivers, and then install all Printers AS TCP/IP Printers on every TS, and Permission them out.  It seemed to solved most of the issues,   but TS2008 no R2   in general seems slower then 2003 TS