[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 370
  • Last Modified:

How to perform a proper sizing for multiple Terminal servers?

s,

We have a customer with 6 terminal VM servers on a single ESXi 5.5 host. The server conf is: Dell R520, 2x E5-2620 (12 cores in total), 192GB RAM, 3x local SSD.
During the busy hours of the day all 6 VM’s performs poorly, and when digging into it I’ve found the below (`amazing`) findings:

1) Each one of the 6x VM's was allocated with 2x vCPU's and 6 cores (!) – 12 cores in total, which IMHO means that each TS VM thinks that he's eligible to take ownership of the entire physical CPU's resources, right?
2) The ESXi host shows more than 100% CPU usage (see attached screenshot - R520 6xTS, fully loaded CPU's.jpg)
3) Running ESXTOP, I've witnessed even a greater pleasure. Hint: watch the %RDY + %CSTP per VM (see attached screenshot - R520 6xTS - ESXTOP.jpg). Amazing ah?

The customer told me that when he tried to allocate less vCPU’s the each VM was stuck on 100% CPU usage, so lowering the amount of vCPU’s per VM doesn’t seems like an option.
BTW, the host is currently a standalone host with no VC which manage it. There’re additional 3x ESXi hosts which are ~50% loaded each – in the organization which are being managed by a VC with an Essentials plus license (which is limited to 3x hosts).

In the bottom line, I need to do sizing for the customer for 170 terminal users and allow him to scale while he grow in the future. Currently there are 70 users and the system is dying.
What's the best way to size a TS session; Should I understand which applications the user is running? I guess there's a smarter way right, especially that the system is already configured, up and running?

The issue is urgent, and your prompt reply will be more than appreciated!

Thanks in advance
R520-6xTS--fully-loaded-CPU-s.jpg
R520-6xTS---ESXTOP.jpg
0
IT_Group1
Asked:
IT_Group1
  • 5
  • 4
  • 3
  • +1
3 Solutions
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would start with 2 vCPUs per VM, and scale out the TS farm.

However virtual servers with TS can perform really poorly compared to physical servers.

How many concurrent users, and what applications are they running?
0
 
IT_Group1Author Commented:
Andrew, thanks,
2 vCPUs per VM has been tried, 100% flat in the OS.
I'm not sure yet which application the users are using, it's part of the sizing we're about to do.

What else can we do to get a proper sizing?

Thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Proper sizing is performed by Testing and Proof of Concept, we have never managed to get more than 25 user per 2 vCPU Server!

It is difficult for anyone to tell you how many concurrent users, you will get per virtual server.

Compared to 55 concurrent users per physical server of the same specification used by the hypervisor, with less memory.
0
2017 Webroot Threat Report

MSPs: Get the facts you need to protect your clients.
The 2017 Webroot Threat Report provides a uniquely insightful global view into the analysis and discoveries made by the Webroot® Threat Intelligence Platform to provide insights on key trends and risks as seen by our users.

 
skullnobrainsCommented:
the sizing depends greatly on what apps the users will run

note that in such a case, running a single TS on the physical host would perform much better. if the guests provide the same functionality, there is little to no reason not to do so.

also note that it is possible that a handful of users eat up all the resources. most likely applying per user cpu limitations would be a great help, and you might not need to use different hardware for that number of users if you do so.
0
 
IT_Group1Author Commented:
Guys, thanks alot.
I've got a couple of questions:

1) Why virtual TS servers will perform poorly compared to physical servers?
2) Given that today 70 users on 6x TS servers are killing the server - but users are ABLE to work, how much improvement do you think we'll benefit from a single TS (Physical OR virtual)? How many concurrent users?

Thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
1. Because the hypervisor takes away too much performance, and TS servers are generally maxed out at 90%-100% which makes them unsuitable candidates for viurtualisation.

2. You would probably be able to reduce the number of servers by half, e.g. 2 Servers for that load, but a third for resilience, should a server fail. 70 concurrent users on a server, I've not seen for a long time, we used to be able to get 77 concurrent users on a DL360G5, Windows 2003, Citrix MetaFRame 1.8, 4GB, Dual Processor, Quad Core.

3. Or, just scale out the number of TS, you require virtually. I would think another 6.
0
 
skullnobrainsCommented:
i'm not too sure that there is an easy answer for 2).

like mentioned above, it is quite likely that a few users are using 100% of their available CPU. if 5 of your users are in this case, you can use 15 servers and end up in the same situation because 3 or 4 of your servers will still host at least one such user so there will still be a significant number of users to complain

i don't think it is reasonable to try to guess about sizing before looking at which user and/or app eats up the resources. also alternate solutions might emerge if you detail what are your needs. for example it may be worth a shot to virtualize a single app rather than a full desktop if that's all that is needed, or use restrained profiles that are not cluttered with various useless bloatware the users tend to install on their desktops.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You may also want to consider using tools...and these are the tools we use for large Citrix/TS farms we build, e.g. 1,500 - 26,000 concurrent users.

Login VSI
http://www.loginvsi.com/product-overview

CitraTest
http://www.tevron.com/citratest.aspx
0
 
asavenerCommented:
2 vCPUs per VM has been tried, 100% flat in the OS.
How are you measuring the CPU usage?  Are you looking at the stats on the host or the stats within the VM?

Using performance metrics for CPU from within the guest is not reliable.  The hypervisor will only offer the number of CPU cycles requested, so often the CPU looks maxed out on the guest while the host still has resources in reserve.


To determine whether you're running into resource contention, look at the CPU_Ready performance counter on the host.  This will tell you how long the guest has to wait after requesting CPU.


Another place to look for a performance bottleneck is disk.  You typically need 150 IOPS per user in order to guarantee performance.  Average IOPS will be lower, but you need to guarantee performance even during peak times.
0
 
IT_Group1Author Commented:
Thank you all.
asavener - The IOPS are SSD based, so I think it's not the issue.
In the bottom line - what would YOU do with the current situation (physical/virtual/more VM's/extra server etc..)

Thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Either test your current servers, to give you an actual figure using software available, or pilot and work out a figure.

I remember in the early 90s, that implementations of terminal services, implementors, would just state, oh 25 concurrent users per processor, organisations purchased quad processor server boxes, hoping to get 100 concurrent users per server, and, we all know now, that they did not scale linearly.

and dual processor servers were a better purchase.

Either scale out, with more virtual servers, or switch to physical servers.

I would do physical servers.
0
 
asavenerCommented:
OK, in this situation I would advise that they've reached (and perhaps exceeded) capacity on the ESXi box that hosts the terminal servers, and they need to purchase additional capacity.  Personally, I think the benefits of virtualization outweigh the (very) slight performance hit, so I would keep the RDP servers virtualized.

Everyone focuses on server consolidation when talking about virtualization, but there are many additional benefits:  hardware independence, non-intrusive hardware maintenance, non-intrusive hardware upgrades, simplified backups, ability to scale out, ability to snapshot before risky operations, etc.

The optimum solution, IMO, would be to upgrade the vCenter license and get at least one more server to host the RDP servers.
0
 
skullnobrainsCommented:
The IOPS are SSD based, so I think it's not the issue.
even SSDs can end up saturated, and given the crazy quantity of RAM your server has, you probably would hardly make the difference w/o SSDs since the hypervisor probably ignores the flush commands issued by the guests. but then my hunch would be the problem lies elsewhere

what would YOU do with the current situation (physical/virtual/more VM's/extra server etc..)
my 2 cents is the following piece of advise : do not try to think up a remedy before you get a proper diagnosis... which you do not have. you may end up lucky (working setup), mildly lucky (working setup but get the same problem in a fortnight), or just end up buying extra hardware and licenses to no avail.
0
 
IT_Group1Author Commented:
Thanks for the effort guys!
0

Featured Post

Fill in the form and get your FREE NFR key NOW!

Veeam is happy to provide a FREE NFR server license to certified engineers, trainers, and bloggers.  It allows for the non‑production use of Veeam Agent for Microsoft Windows. This license is valid for five workstations and two servers.

  • 5
  • 4
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now