?
Solved

VM Terminal Servers displaying periodic overload

Posted on 2011-05-02
13
Medium Priority
?
618 Views
Last Modified: 2012-06-21
Folks,

I am in need of some serious guidance here..

First a bit of background, we have two main applications, one being a financial package and secondly a web based clinical system. Previously they were both served from a single physical server. This server crashed and as part of a recovery plan we built two VM terminal servers to separate out the applications/load and provide a bit of redundancy. Both these VM are accessed via RDP, over our WAN and from external connections. WAN is a microwave linked series of VPNs

We are now frequently finding that over the course of the day, but particularly between 2:30 & 3:30pm one or both of these VMs are being seriously hit in the CPU stakes causing very poor response times at the user interface. (typically 40 users on one box, about a dozen on the other))

From the VM host perspective, looking at the resources for the particular VM - nothing seems to be over the top and each VM is on a separate host. (VM hosts are IBM Blades with SAS external storage, ds3200, Database and web server are also separate blades - but physical)

A collegue of mine is suggesting the network, I'm thinking the VMs themselves as we never had this problem when it was a physical device, we're looking to monitor all network devices, but would love any additional thoughts anyone might have as to where to look for the bottleneck!

thanks in advance for your advices
0
Comment
Question by:wspall
  • 4
  • 4
  • 2
  • +2
13 Comments
 
LVL 16

Expert Comment

by:Danny McDaniel
ID: 35510272
Just to clarify, you built new, clean VM's and installed the apps onto them and not P2V'd the original server and then split out the roles, right?  If you P2V'd, go into add/remove programs and uninstall everything that is not needed, especially any hardware related utilities.

-when the CPU spikes, what process in task manager is using the CPU???

-is either system running off of a snapshot (VMware snapshot, not SAN)

-have you checked the performance charts for disk command latency spikes during the times you see CPU go up?  If there is a storage delay, it will make the CPU artificially spike as processor requests queue up waiting for storage.  Network can do the same, but storage is more likely to be an issue.

-Do you have anything scheduled on these VM's or others on the same host?  anti-virus scans, defrag's or backup's would be likely culprits.

-do you see other VM's on the same hosts experience performance problems at same time?  this would suggest something wrong with the host/storage/network and not the VM's themselves.  check performance charts for the other VM's to see if there's a history of simultaneous CPU/Network/Disk spikes among the VM's.
0
 
LVL 8

Expert Comment

by:PenguinN
ID: 35510296
What micht give you some more insight in your situation is to use procexp inside the vm to see if there are any processes opening a lot of subprocesses.

If you want to check your vm environmet i would suggest to install Veeam monitor server on a seprate box. Just use the eval version. It gives a good insight in what goeing on. Check vm ready time and io latancy.

On the esx itself you can use esxtop to see how your box is behaving in combination with your performance monitor. But veeam wil report any issue's directly in it's management console, and in most cases report the solution to your problem as wel.
0
 

Author Comment

by:wspall
ID: 35510400
Thanks for your quick responses...

VMs were clean builds of server 2003,  nothing scheduled (but will check this again) - A/V has even been removed incase this was causing grief (locked down by GPO anyway). sanpshots taken, but should be running VM (will check this too)

looked to install Hyper9 to view VM environment, but as this is a VM itself so might try the Veeam option (currently use their backup/copy tool)

On one of the VMs, main procceses tend to be iexplorer (ie the browser based app), have seen a few processes pulling 100Mb+

Currently offsite (but had the ubiquitous phone call) will look further into your thoughts on the morrow, VMs have now returned to normality  - ave 4% CPU, few spikes 2.5Gb of memory....

thanks again
0
Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

 
LVL 8

Expert Comment

by:PenguinN
ID: 35510545
If you have a full licenced suite of veeam maybe you also have a monitoring license.
0
 
LVL 124
ID: 35510815
Install and use Veeam Monitor and Reporting (free download), to check the performance of the servers.

Did you say you had 40 terminal server users per VM?
0
 
LVL 21

Expert Comment

by:mastoo
ID: 35513515
How many logical processors did the physical have compared to the virtual?  For instance, you might have had a dual quad core physical (8 logical processors) and possibly you only have a couple of virtual processors allocated?  Those would quickly show up as highly utilized.
0
 

Author Comment

by:wspall
ID: 35710363
Folks thanks for your continued input, have installed prtg monitor a it happens to examine, network,VMs,SQL DB.....etc

First port of call looks like a difference between original physical and running virtual. will ramp this up over the week end, but didn't want to impact the other vms on the respective hosts too much - lookslike this may now be necessary!

How much faith do we put into the warning that changing the number of cpus after the os build can make the vm unstable?

Also found a slight bottleneck in one of the connecting switch interfacesis only running at 100mbps, looking to change to gigabit....
0
 
LVL 124
ID: 35710374
Well the slow switch could impact performance, if you were Network Bound, check Network Utilization history.

As for "changing the number of cpus" after the build was complete, we do it all the time, if warrantied, and the application supports vSMP. But don't get too carried away because you can, adding processors, sometimes adding additional processors can slow the VM down, due to scheduling.
0
 
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 999 total points
ID: 35710381
oh, and as for physical to virtual terminal server users, doesn't surprise me, I've seen physical servers cope comfortably at 75-85 concurrent users, and once virtualised, could only handle 20-25!
0
 
LVL 16

Assisted Solution

by:Danny McDaniel
Danny McDaniel earned 501 total points
ID: 35710421
the old rule of thumb with Citrix and TS servers was 2 vcpu and 4GB of RAM for each server and scale out as many as needed.  plan on 15-25 users per server.

going from single vcpu to smp can cause issues with some rare thing... I think it was .net trash collection, but otherwise adding vcpu's usually doesn't make an impact.
0
 

Accepted Solution

by:
wspall earned 0 total points
ID: 35813117
Folks,

sorry for the absence, by way of an update, over the past week we have poured over monitoring (PRTG) of servers (both physical & virtual), CPU & RAM & disk, network etc etc. Following many more web searches and subtle changes to VM settings thereby, the general concensus seems to be that whilst terminal servers can be virtualised there are some limitations (much like danm66 mentioned) or certainly some gotchas for the unwarey.

It would appear in our case that disk connectivity between the VM hosts and stored VMs may have been our bottleneck (VMs stored on SAS storage array attached to blade chassis) - mind you our used base has also doubled in the past 6 months.

We have rolled out a new physical TS over the weekend and are watching for our usual window of anxiety this afternoon....
0
 
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 999 total points
ID: 35813899
Personally, I would never deploy a farm of virtual terminal servers, the costs don't stack up, with the cost of hypervisor and terminal servcies, and few users per virtual server. But if your policy is to have only virtual then that's a different matter. You'll just end up with double the number of virtual terminal servers compared to physical to manage.
0
 

Author Closing Comment

by:wspall
ID: 37648577
All answers accepted have pointed us in necessary directions to clean up possible flaws in our environment. WHilst the actual problem still is not resolved completely, hopefully these answers will benefit others who find themselves in similar situations !!
0

Featured Post

Free recovery tool for Microsoft Active Directory

Veeam Explorer for Microsoft Active Directory provides fast and reliable object-level recovery for Active Directory from a single-pass, agentless backup or storage snapshot — without the need to restore an entire virtual machine or use third-party tools.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
In this article, I will show you HOW TO: Install VMware Tools for Windows on a VMware Windows virtual machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, using the VMware Host Client. The virtual machine has Windows Server 2016 instal…
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
Suggested Courses

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question