ESX 4.0 Best practices

We are running ESXi 4.0 in our environment with Dell blade hardware most servers have 8GB ram and 3.0 xeon processors. We currently have 2 ESXi servers and are adding 2 more.  Most VM's are just for backup of physical servers.  To start off we did just like many people and just installed it and started using it without doing a ton of research.  Now we are looking to use it more as a true production platform and want to make sure we are using ESX to its full potential.  I have several general questions for people using ESX in production environments.  I know some of these are vague but I'm really just looking to see how other people approach using VM"s.  I've done some research and read a lot of information but wanted to actually talk to some people and see how they have done it.  

1. What is the best way to setup authentication?  We currently use LDAP for our other boxes.
2. What are the recommended network setups?  We have recreated all of our vlans on the ESX server and trunk them to two cisco switches for redundancy.  Works fine but is this the best way?
 3. How do you manage moving servers around without using the full blown virtual center or vphere I think it's called now?
4. What is your ram recommendation for around 5 - 12 servers.  None of them get hammered
5. How do you recommend configuring resource allocation?  We don't really do it at all now.
6. How to permanently fix time drifting problems?
7. how to verify that your server is running at peak performance or close to it?
8. What design problems to look out for?
9. What would you do differently?
10. What is the best software to manage VM's outside of vmware, free prefereable but not mandatory

Thanks for any information anyone could provide.
bevegeAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

coolsport00Commented:
1. Depends...authentication to what?...using the vSphere Client to log into the host or to log into the VMs?
2. Sounds good here.
3. Need more info on what you mean by 'moving servers around'? Do you mean moving *VMs* around between ESXi hosts? There are a couple ways to achieve this -> vSphere Converter Standalone (http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vcenter_converter_standalone/4_0) or Veeam FastSCP (http://www.veeam.com/vmware-esxi-fastscp.html).
4. Do you mean 5-12 VMs? Is this per host or total? This depends on what those VM servers are doing...email? IIS/Web? DB? I recommend a minimum of 12GB of RAM, but suggest getting 32GB as a) RAM is cheap and it's good to have more than you *think* you may need for growth/expansion and to cover whatever extra resources current VMs may require.
5. There is no need to do resource pool allocation. ESX/i by default does a pretty good job of allocating resources on its own.
6. Need more info...what do you mean "drifting problems"?
7. There is a performance tab when you log into your ESXi hosts you can monitor.
8. Design problems?...what are you looking for specifically here? Your setup is pretty basic, so I guess the only recommendation I have as far as implementation is to spread your VMs across hosts, have shared storage (SAN) if possible, and have a backup solution.
9. Since you're running ESXi, you're limited in feature/functionality (also since you don't use vCenter), so there isn't much that can be done differently.
10. I'm not aware of any other mgmt tool for VMware's infrastructure besides vCenter. I believe Microsoft's SCVMM is able to manage VMware hosts/VMs, but I haven't used it. I was testing with it a year and a half ago, but full functionality wasn't yet implemented within SCVMM to do any in depth testing.

Regards,
~coolsport00
0
vmwarun - ArunCommented:
1. What is the best way to setup authentication?  We currently use LDAP for our other boxes.
Using vCenter or VirtualCenter is the best option to setup when you are managing multiple ESX/ESXi Hosts

2. What are the recommended network setups?  We have recreated all of our vlans on the ESX server and trunk them to two cisco switches for redundancy.  Works fine but is this the best way?
VMware Best practice recommends a total of 6 NICs (2 for Service Console, 2 for VMKernel and 2 for Virtual Machine Traffic), if you are usign VLANs then a single vSwitch with 6 NICs teamed would be the best option. Configure Route based on IP Hash on the vSwitch and configure Etherchannel at the physical switch level.

 3. How do you manage moving servers around without using the full blown virtual center or vphere I think it's called now?
You can use VMware vConverter 4.0 or Veeam FastSCP to move VMs from one ESX/ESXi Host to another.

4. What is your ram recommendation for around 5 - 12 servers.  None of them get hammered
Its based on flexibility. Normally I would recommend 16GB or 24GB RAM per host.

5. How do you recommend configuring resource allocation?  We don't really do it at all now.
Resource Allocation is taken care by ESX by default. However you can start tuning them once you have a good resource monitoring tool in place.

6. How to permanently fix time drifting problems?
Do not sync your Virtual Machines time with the ESX Host.
In an Active Directory environment, the PDC Emulator acts as a NTP Server for all Workstations and Servers within the Domain and PDC Emulator fetches its time from an internet server.

7. how to verify that your server is running at peak performance or close to it?
This is pretty difficult with standalone ESX since Resource Logs are purged once in every 24 hours. Using vCenter is recommended.

8. What design problems to look out for?
Redundancy is the name of the game. Analyze your existing environment and then take a detailed inventory of the same. This will be a good starting point to look for designing problems.

9. What would you do differently?
Start migrating to SAN if using DAS or NAS at present.

10. What is the best software to manage VM's outside of vmware, free prefereable but not mandatory

Easy Management always comes at a cost. VM Explorer can be used to manage Virtual Machines outside vCenter.
0
IanThCommented:
as far as memory goes make sure the esxi host has enough ram. The vmware blurb says it can do memory paging but its disk intensive and if the paging is on the same datastore as the vm's it dish thrashing time so what I do is make sure no host is memory maxxed in the vm's on that host
0
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

VMwareGuyCommented:
1. What is the best way to setup authentication?  We currently use LDAP for our other boxes.

You want to use MS AD?  follow this:
 - From the Service Console, run the following command:
-  esxcfg-auth --enabelad --addomain=FQDN of domain --addc=fqdn of domain controller name
- Run the service mgmt-vmware restart command.

2. What are the recommended network setups?  We have recreated all of our vlans on the ESX server and trunk them to two cisco switches for redundancy.  Works fine but is this the best way?

Create a fault tolerant team for your service console and vmotion network, set vmnnic0 as primary for the service console port group and vmnic2 for the vmotion network, notice how i selected vmnic2 instead of 1, that is because it is better to create a team with separate physical nic adaptors as opposed to creating it with a single dual port nic - got it?
 
 3. How do you manage moving servers around without using the full blown virtual center or vphere I think it's called now?

if you have shared storage - You power down the VM, right click and select remove from inventory, log into your other ESX host, browse the datastore the .vmx file resides on, right click the .vmx file and select register

if you don't have shared storage - download free copy of fastSCP by VEEAM, its on their web site, it allows you to connect to ESX, it can automatically allow you to switch to root, then select the folder that contains all of the VMs files, and copy it to your new ESX storage, then browse the destination datastore and register the .vmx file.  There isn't a SCP utility out there that works faster

4. What is your ram recommendation for around 5 - 12 servers.  None of them get hammered

although you can over commit up to twice the amount, I wouldn't, and keep in mind you need to be able to handle the workload of other VMs from another host should it fail.. if you think you have enough for the VMs running, and then there is an outage forcing you to bring up the other VMs on this same host server, good luck.. use your head, do your math, this isn't surgery.  Look at your perf counters and examine the peaks and averages and this will give you an idea of what your consumption is and how much RAM you will need to handle a failover event

5. How do you recommend configuring resource allocation?  We don't really do it at all now.

It works best the way VMware desgined it, keep it default, unless you have a department that you are using charge back and you don't want them to have more than a certain amount of CPU or memory, the you can set the limits, don't use shares unless there is contention because it won't matter if there isn't, and you can also create clusters with vCenter to limit resource usage by VMs that are members of the cluster.  

Also, try to only allocate 1 vCPU per VM if possible unless the apps running are multithreaded, than go with 2.  

6. How to permanently fix time drifting problems?

Keep host CPU utilization under 60%, configure VMs to time sync with ESX service console, it syncs every minute this way, make sure ESX has a valid time provider in NTP settings

7. how to verify that your server is running at peak performance or close to it?

Get the performance white paper and do some reading for once, if you did any at all you wouldn't be posting this question, it is what separates good engineers from the hacks:
http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf
 
8. What design problems to look out for?  

Shared storage - make sure you have redundant paths to storage, avoid link aggregation \ ether channel if using iSCSI for storage, make sure you measure IOPs if you know how and calculate your storage needs, best to have as many disks as possible in your RAID 10 or RAID 5..

9. What would you do differently?

Differently that what?  VMware has testing their product thoroughly and it runs best in its default mode, I have fine tuned hosts to no end and there was never a noticeable difference in performance - except when I had to set CPU affinity for some VMs in a particular cluster that was over worked and was experiencing time drift

10. What is the best software to manage VM's outside of vmware, free prefereable but not mandatory

Sorry, but vCenter is the way to go.  otherwise, if you want to try out some free utilities, there are plenty out there to test and try out, but they typically have a specific purpose, not over all management, that is what vCenter is for and there isn't any product that will allow you to get the most out of VMware than vCenter.  

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
namoomCommented:
I disagree with this:

VMware Best practice recommends a total of 6 NICs (2 for Service Console, 2 for VMKernel and 2 for Virtual Machine Traffic), if you are usign VLANs then a single vSwitch with 6 NICs teamed would be the best option. Configure Route based on IP Hash on the vSwitch and configure Etherchannel at the physical switch level.

I would like to see documentation for this recommendation as 2 service consoles can do a great deal more harm then good on esxi (not sure about esx)
0
VMwareGuyCommented:
namoom, show me one doc where this is stated - this is only true if you don't have VLAN trunking on your network, with VLAN trunking you don't need this, all you really need is 4, 2 for service console and VMkernel, they share these adaptors, and the other 2 for production VMs, however, if you are using software  iSCSI then you will need 2 more for storage connection.  
0
bevegeAuthor Commented:
Thank you all so much for the comments.  I just got really slammed at work the last two days.  I'll post replies to everything later tonight.  I have a few other questions but this feedback is awesome!
0
bevegeAuthor Commented:
Ok, finally a chance to respond.

First off a few things I left out.  We have no shared storage. Any low budget suggestions are welcomed.  I know cheap is not necessarily the best way to go but we all know how that goes.  We are using ESXi 4 so no vmotion

Question 1 - Authentication
Looks like AD is the way to go.  If anyone has done LDAP let me know

Question 2 - Neworking
Using dell blade chassis with 2 blade switches.  Right now each blade switch has two port-channels trunked to 2 separate  cisco switches.  This gives each blade slot 2 ethernet connections with built in redundancy.  

In ESX which is on say blade 1 there are 2 ethernet connections listed (vmnic0 and vmnic1).   Sounds like the way to go is use vmnic0 for ONLY they service console and use vmnic1 for VM network.   Especially if we want to eventually use Vmotion.  BTW, our blade network looks just like the one here http://support.dell.com/support/edocs/network/LAG1855/LAGConsiderationv0.5.pdf

I assume that this means no Nic teaming inside of ESXi because you need more than 1 nic to team them?  Not sure how to add 2 or more nics to a blade chassis but I haven't researched it at all.  Any other input welcome.  I've also printed out the esxi networking pdf for some late night reading.

3. Moving VM's
As I thought, Virtual Center is really the only way to go. Which makes sense.  I tried Veem and also the VMware converter methods.  Veem is very slow and the Vm converter kept giving me all kinds of problems.  Gloing to try converter again when I get some time.

4. Ram Recommendations
Excellent information thank you all.

5. Resource Allocation
 Excellent information thank you all.

6.  Time syncronization
Excellent information thank you all.

7. Performance
Excellent information thank you all.  Already had that document but have not read the entire thing yet.  Just asking for expert advice.

8. What design problems to look out for?  
Excellent information thank you all.   Will probably have more questions if I can talk someone into getting shared storage.

9. What would you do differently?
What I meant by this was.  Did anyone design their systems, set it all up and then say man I wish I would have ........  Did anyone start with a certain storage scenario only to have to upgrade to something else due to performance or whatever?  That type of thing.  Basically things I can use as an argument to do this the right way and not the cheapest way.

10. What is the best software to manage VM's outside of vmware, free  prefereable but not mandatory
Enough said. Just need to figure out a good way to convince the higher ups on why this will save a lot of time and money.
   
A big thank you to everyone.  I'll hand out some points after this round of questions.
0
vmwarun - ArunCommented:
Keeping in mind the current and future Storage requirements, I would suggest Dell PowerVault MD3000i SAN Array.
Fore more info, check the URL
http://www1.ap.dell.com/au/en/enterprise/storage/pvaul_md3000i/pd.aspx?refid=pvaul_md3000i&s=lca&cs=aulca1 
0
coolsport00Commented:
For question 2, you really don't have to separate your traffic to different NICs, especially since you don't use/have VMotion capability. If anything, I would 'team' them for failover capability, but what you're suggesting is certainly fine.

#3 Yes, you do need vCenter to be able to 'migrate' VMs (not necessarily VMotion..i.e. live migration, but to even cold migrate). I'm curious as to what problems you had with vCenter Converter Standalone (may need another EE post on that) :)

#8/9 The only thing I did when implementing my infrastructure...or didn't do, but knew it...was not having NIC failover. I do now, but I just wanted to get my virtual infrastructure up and going and just implemented the basics initially. Really, reading through the vSphere Guides (ESX Config, Res Mgmt, Basic Admin, etc.) is really what helped me. I really took my time to read through and *understand* them.
http://www.vmware.com/support/pubs/vs_pages/vsp_pubs_esx40_vc40.html

Hope this hels.

Regards,
~coolsport00
0
VMwareGuyCommented:
Question 1 - Authentication
Looks like AD is the way to go.  If anyone has done LDAP let me know

Did you even read my earlier post? The command parameters are there for you.

Question 2 - Neworking
Using dell blade chassis with 2 blade switches.  Right now each blade switch has two port-channels trunked to 2 separate  cisco switches.  This gives each blade slot 2 ethernet connections with built in redundancy.  

In ESX which is on say blade 1 there are 2 ethernet connections listed (vmnic0 and vmnic1).   Sounds like the way to go is use vmnic0 for ONLY they service console and use vmnic1 for VM network.   Especially if we want to eventually use Vmotion.  BTW, our blade network looks just like the one here http://support.dell.com/support/edocs/network/LAG1855/LAGConsiderationv0.5.pdf

if you have only 2 nic connections you should combine them into a team within you v-switch and then use VLAN trunking with your port groups, this will give you the networking security and flexiblility you need.  Vmotion, if ever used in the future, doesn't need to be on separate physical, it simply needs logical network security and isolation.  Why would you even think about only using a single NIC and leave yourself a single point of failure?  

3. Moving VM's
As I thought, Virtual Center is really the only way to go. Which makes sense.  I tried Veem and also the VMware converter methods.  Veem is very slow and the Vm converter kept giving me all kinds of problems.  Gloing to try converter again when I get some time.

I have been using fastscp for many years, there isn't anything faster to my knowledge, and vmnware converter is super slow. I don't think you did adaquate testing with it.  vCenter will allow you to migrate files around, I haven't used storage vmotion in a standalone environment that didn't include shared storage, but I don't see why it wouldn't work if your vCenter licensing can handle it.
0
bevegeAuthor Commented:
Thanks everyone.

Vmwareguy:

Authentication
I did read your post.  We do not use AD in our network at all.  In fact we do not have a single windows server in our entire network.  I should have said openldap instead of just stating LDAP. I know how to make Centos work with openldap but not sure how ESX would work.  Just looking for resources and pointers until I have a few hours to research the subject myself in more detail.

Networking
You are right I wasn't thinking clearly. Need to stop doing this late at night.  So we already have it setup the best way if you only have 2 nics.

Moving VM's
I'm not bashing Veeam at all but it seemed really slow to me just like when copying over files manually.  I don't have the numbers right in front of me but I remember that it was saying it would take 4 or 5 hrs to copy a 20gb vm.  This on a gig network on the same vlan on the same switch.

What type of performance are you guys getting out of veem for a 20-30 GB vm?  I know there are a million and one factors but I'm just trying to get an idea.  I read many, many posts about people getting slow transfer speeds.

I 100% admit I did not do any thorough testing on either product so it could very well be the way I set it up.  I'm going to try it again when I get more time.

Thanks again for everyone's help.

0
IanThCommented:
the pdf says the blade chassis you are using has advanced network fault tollerance
0
VMwareGuyCommented:
I have seen 20GB files copy over in just minutes.  I was blown away by fastSCP, and they recently made further improvements to it.  The reason why vCenter works so well at it is becuas they aren't copying the white space of a virtual disk, just the data, then the actual vmdk is recompiled.  I haven't confirmed this, but I believe Veeam is leveraging the same native ESX utility under the covers that does this, vmkfstools -i.  you would have to pop them an email to confirm it.  
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server OS

From novice to tech pro — start learning today.