Solved

Initial Configurations for Virtual Machines when considering NUMA Nodes

Posted on 2014-10-27
5
218 Views
Last Modified: 2014-11-16
I am brand new to virtualization so this will be a bit of a long winded question.  I've been doing a lot of reading of previous questions here on EE and I am still a bit confused regarding the best configuration to use for my VMs when considering NUMA Nodes.  
I am using ESXi 5.5 and have not yet purchased any other VMware products so I'm just using the vSphere Client to configure the VMs.  My host has 2-10 core processors with HT, so the vSphere Client is showing 40 logical processors.  I also have 256 GB of RAM.  It appears that I have two NUMA nodes, each with 20 logical processors and 128 GB of RAM.  From everything I've been able to understand, I should configure my VMs as a variant of the 20 logical processors per NUMA node.  So I can have 1, 2, 4, 5, 10, or 20 virtual processors.  I am limited to 8 vCPU per VM due to using the free version of ESXi 5.5 and don't have a need to have more than 8 vCPU anyway.  I've also read that I should just plan for one or two vCPU and then monitor performance and only add another vCPU if the performance requires it.

Question #1:  Is my understanding of how the processors and memory are associated with the NUMA nodes correct?
Question #2:  If a VM's performance does require another vCPU, is my thinking correct that I should only use 2 vCPU or 4 vCPU and not 3 vCPU due to degraded performance?
Question #3:  I haven't read anything that outlines how you should allocate memory to your VM when considering NUMA nodes.  Is there something that outlines the best approach when it comes to RAM?
Question #4:  I'm planning on creating six VMs initially.  Does the total number of vCPUs and memory, when added together, matter with regard to NUMA nodes, or does ESXi take care of balancing the VMs with respect to those nodes?  I'm aware that you can direct what node the VM should access but I'm thinking you leave it as "No Affinity" and let ESXi do the work.
Thank you for any assistance you can provide.
0
Comment
Question by:sber1988
  • 3
  • 2
5 Comments
 
LVL 118
ID: 40407214
I would not get too bogged down with NUMA....assign Sockets and the Hypervisor will do all the work for you.....(it's get a little more complicated if you assign Cores, which you would only do for licensing reasons!)

see here

https://blogs.vmware.com/vsphere/2013/10/does-corespersocket-affect-performance.html

vSMP (virtual SMP) can affect virtual machine performance, when adding too many vCPUs to virtual machines that cannot use the vCPUs effectly, e.g. Servers than can use vSMP correctly :- SQL Server, Exchange Server.

This is true, many VMware Administrators, think adding lots of processors, will increase performance - wrong! (and because they can, they just go silly!). Sometimes there is confusion between cores and processors. But what we are adding is additional processors in the virtual machine.

So 4 vCPU, to the VM is a 4 Way SMP (Quad Processor Server), if you have Enterprise Plus license you can add 8, (and only if you have the correct OS License will the OS recognise them all).

If applications, can take advantage e.g. Exchange, SQL, adding additional processors, can/may increase performance.

So usual rule of thumb is try 1 vCPU, then try 2 vCPU, knock back to 1 vCPU if performance is affected. and only use vSMP if the VM can take advantage.

Example, VM with 4 vCPUs allocated!

My simple laymans explaination of the "scheduler!"

As you have assigned 4 vCPUs, to this VM, the VMware scheulder, has to wait until 4 cores are free and available, to do this, it has to pause the first cores, until the 4th is available, during this timeframe, the paused cores are not available for processes, this is my simplistic view, but bottom line is adding more vCPUs to a VM, may not give you the performance benefits you think, unless the VM, it's applications are optimised for additional vCPUs.

See here
http://www.vmware.com/resources/techresources/10131

see here
http://www.gabesvirtualworld.com/how-too-many-vcpus-can-negatively-affect-your-performance/

http://www.zdnet.com/virtual-cpus-the-overprovisioning-penalty-of-vcpu-to-pcpu-ratios-4010025185/

also there is a document here about the CPU scheduler

www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf

https://blogs.vmware.com/vsphere/2013/10/does-corespersocket-affect-performance.html
0
 

Author Comment

by:sber1988
ID: 40416224
Thank you for your response.

I've read the information you provided and I had read most of it before as I researched my questions and I believe I have a sufficient understanding of the pitfalls of overprovisioning a VM.

You're advising me to not to worry about NUMA, but the information I've read seems to state that if you create your VMs without considering NUMA it will affect performance to some degree.  So is your statement reflective of the performance impact being negligible or is it based upon the resources of this particular host, or should I never worry about NUMA?  I mean no disrespect as you are clearly superior in your knowledge of virtualization, but I have three other ESXi hosts that were installed by a consultant that I no longer employ and I'm trying to gain a deeper understanding so I can manage them effectively as well add new VMs to them as necessary.

Regarding your statement of assigning Sockets in lieu of Cores.  Can you clarify the difference because what I've read and understand (maybe incorrectly) is the VM will see them the same whether it is 1 socket with 2 cores or 2 sockets with 1 core.  I'm running Windows Server Standard, and it is my understanding of the licensing that Windows is limited to two physical processors on the host, so am I incorrect to think that I need to limit the VM to no more than 2 sockets per Windows license?
0
 
LVL 118
ID: 40416241
Sockets and Cores can be regards as the same, in terms of performance (Assigning Cores does affect the NUMA balance!).

(VM sees the same! yes, but...hypervisor does not!)

If you have a license constraint based on cores, of you must ONLY run the VM, on 2 physical cores of the host CPU, assign cores to be within your license.

otherwise continue to assign sockets....assigning sockets and the hypervisor will balance NUMA requirement....

you can assign 64 sockets to Windows Standard.
0
 

Author Comment

by:sber1988
ID: 40438996
Thanks for your feedback, but could you review the initial questions I wrote and offer your specific thoughts on those?

I've continued to read and I do understand that your saying don't get bogged down worrying about NUMA, but is there an instance when we should give it consideration?  

It sure seems there is a lot of information out there stating that you should take NUMA into account when planning your VMs and maybe with the VMs I would deploy I would never get to the point where it is necessary.
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 40439014
I've continued to read and I do understand that your saying don't get bogged down worrying about NUMA, but is there an instance when we should give it consideration?  

If you are constrained by license restrictions, where you are licensed by a product on CORES! and you start to allocate and deploy VMs, by specifying Cores, and not Sockets!

If you deploy VMs as recommended, based on Socket provision, the Hypervisor, balances the selection of physical cores in a NUMA configuration.

Question #1:  Is my understanding of how the processors and memory are associated with the NUMA nodes correct?

Correct.

Question #2:  If a VM's performance does require another vCPU, is my thinking correct that I should only use 2 vCPU or 4 vCPU and not 3 vCPU due to degraded performance?

Correct, biggest mistake admins make is oversubscribbing a VM, by adding to many vCPUs. Very few VMs require more than 2 vCPUs. Check performance, and increase slowly, if after adding no more improvement, reduce. Adding processors is not linear, so adding 2 does not double the performance, adding 4 does not quadruple!

Question #3:  I haven't read anything that outlines how you should allocate memory to your VM when considering NUMA nodes.  Is there something that outlines the best approach when it comes to RAM?


there is not much to say...

ESXi uses a sophisticated NUMA scheduler, which load balances, vCPU and memory requirements, and ensures thay are used from the same NUMA home group.

Question #4:  I'm planning on creating six VMs initially.  Does the total number of vCPUs and memory, when added together, matter with regard to NUMA nodes, or does ESXi take care of balancing the VMs with respect to those nodes?  I'm aware that you can direct what node the VM should access but I'm thinking you leave it as "No Affinity" and let ESXi do the work.
Thank you for any assistance you can provide.

Also correct, if you statically allocate all your cores, you got to have good documents and management, knowing where all the VMs are using which cores, why do this, when the hypervisor does a good job of this already! So no messing with affinity values!

You will run out of memory before you run out of CPU, and processing power, Memory is always these bottleneck!
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

It Is not possible to enable LLDP in vSwitch(at least is not supported by VMware), so in this article we will enable this, and also go trough how to enabled CDP and how to get this information in vSwitches and also in vDS.
David Varnum recently wrote up his impressions of PRTG, based on a presentation by my colleague Christian at Tech Field Day at VMworld in Barcelona. Thanks David, for your detailed and honest evaluation!
Teach the user how to use vSphere Update Manager to update the VMware Tools and virtual machine hardware version Open vSphere Client: Review manual processes for updating VMware Tools and virtual hardware versions: Create a new baseline group in vSp…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now