We help IT Professionals succeed at work.

Setup advice for VMWare ESXi 4.1

Matthew_Way
Matthew_Way asked
on
1,896 Views
Last Modified: 2012-05-10
We are about to setup a small VMWare server.

The machine has 4 x 1TB SATA disks running as a RAID 5 Array.

Question is should we boot the VMWare server from the RAID array or internal USB flash drive.

What are the pros and cons of either solution ?
Comment
Watch Question

Commented:
The cons of the internal USB flash drive is that if that drive has a fault or fails, the host goes down. Any running VMs on it that are not in FT mode will also go down (in an HA cluster they can be restarted but will still go down for a moment). Since you mentioned 4x 1TB drives, I'm assuming you are using local storage to FT and HA are not possible. It's not very likely, but using the usb drive is relying on a non-redundant disk to run the OS.

If installing to the local disks, make sure to partition them otherwise you will end up with the OS on partition formatted with a 1MB block size (having a max file size of 256GB). It is generally recommended (when financially possible) to have the OS on a redundant RAID volume, an the data/VMs on another redundant RAID volume. I don't know if this is possible in your situation, but I wanted to share the knowledge.
Danny McDanielClinical Systems Analyst

Commented:
you will also want to keep any RAID volumes to 2TB-512B or smaller.  ESXi will not recognize anything 2TB or larger.

Commented:
Good point. Keep each partition under 2TB. 4.1 is supposed to recognize a single local volume over 2TB (broken into partitions), but there are some reports of issues with this. Keep each volume under 2TB to be safe.

Author

Commented:
Yes it's DAS storage via Adaptec 5405 PCI-E RAID controller.
 
With the partition size over 2TB we have already had that issue with ESXi 4.1

We setup a single 3GB LUN and while VMWare 'sees' the LUN @ 3TB
Once formatted as a datastore it's size becomes 740GB

So yes will partition to smaller than 2TB LUN's on the RAID controller.

Should the VMWare OS be on it's own LUN of say 40GB
Then two LUNs at 1.5TB for datastores ?
Top Expert 2010

Commented:
Yes "matthew_way"...I HIGHLY recommend segregating your hypervisor install from your VM datastore storage. A very common option you can do is install ESXi on a USB stick (see here for instructions: http://kb.vmware.com/kb/1020655), then you can use all your disks for datastore storage.

Hope that helps.

Regards,
~coolsport00

Commented:
Matthew_Way:

Using a memory stick you have the advange of solid state storage (more with reliability than speed), so it's unlinkely to get damaged as a HDD, however there is no way to provide HA on that scenerio in case the stick dies the good part do it's you separate HyperVisor load out of the VM's store.

If it's possible do 2 x RAID 1 rather one big R5, remember all your VM and Hypervisor would use it at once.

Commented:
Yes, 40GB for the OS should be good, allowing 2 x 1.5 TB datastores.

The USB stick isn't a bad option, but the inability to have redundancy is a dealbreaker for many people. Flash devices are quite reliable, but a single bad block could mean all your VMs on the host are unusable for an extended period of time as the host is diagnosed, a replacement drive is installed, and esx is reinstalled and reconfigured. It's unlikely, but possible - it depends on the risk you are willing to take.

For me, giving up 40GB of local storage to have redundancy is totally worth it.
Top Expert 2010

Commented:
The way to mitigate redundancy issue is have a 2nd USB stick available in case of some hypervisor failure. That still leaves some downtime, though. I personally have yet to see a hypervisor install fail though...it's pretty stable. But, you always want to prepare for the worst. Regardless, it is still a viable option.

Regards,
~coolsport00
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Thanks for all the advice.

The system isn't HA as that would require us to move from DAS storage to SAN, plus the licensing etc etc.

Having said that we want to make the system reasonably reliable.
We don't actually have a big requirement for storage beyond backup.
So I'm thinking of setting up the 4x1TB drives as a RAID 6 array.
Giving us a usable capacity of 2TB.

Thinking about it if the RAID setup fails we are out of action anyway.
So booting off USB flash may just be a second failure point.

The 2TB of usable space I will setup three LUNs
LUN 1 - 40GB - VMWare OS
LUN 2 - 1TB - Datastore 1
LUN 3 - 1TB - Datastore 2

Top Expert 2010

Commented:
Unless you need to create some volume of extreme large size "matthew_way", I would further split your 2TB array into at least 2 1-TB arrays, at a minimum. This will give you more flexibility as well as redundancy, and HA. If you have 1 array, you have the potential to have all your VMs be down. With separate arrays, you can split up where your VMs are and be at least 1/2 up in case 1 of your RAID arrays fails. That would be my suggestion...

What you have will certainly work, but having as much of a HA scenario as you can based off your budget is the best way to go. And, segregating things is also general best practice. It's really up to you.

Hope that helps.
Regards,
~coolsport00

Commented:
@coolsport00
I agree with that thought process. However, in terms of actual redundancy, RAID 6 would be preferable since it can handle 2 drive failures. With 2 x RAID-1 arrays, it's possible (50% chance) that 2 drive failures would take out one of the arrays. You will only lose 50% of your data (and possibly the OS since it is be shared with 1 of the arrays), but that's still a huge loss that would require backup restoration procedures and significant downtime anyway. Personally, I would rather provide better protection from the downtime via a RAID-6 array. Plus it can more easily grow.

Just a different approach - both have their merits.
Top Expert 2010

Commented:
True and agree that both have merits & downsides....I still like separate datastores personally :) It's really more up to the posters performance requirements and SLAs I guess. :)

Regards,
~coolsport00

Author

Commented:
Once again thanks for all the advice.

We have probably got a requirement for around 1TB storage total.
Which is why I have decided to use RAID 6 over 4 1TB drives which as pointed out by geowrian will handle two drive failures.
The largest VMDK we require would be around 200GB and if we require more we can use LVM on the guest ( Ubuntu Linux ).

SLA's we don't have but I did look into HA using a mirrored pair of SAN controllers but with that and the VMWare extra licenses it took the project budget from about $10K to $25K

One last parting question what size should I allocate for the VMWare OS datastore ?
Top Expert 2010
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Thanks for the great advice.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.