Setup advice for VMWare ESXi 4.1

The cons of the internal USB flash drive is that if that drive has a fault or fails, the host goes down. Any running VMs on it that are not in FT mode will also go down (in an HA cluster they can be restarted but will still go down for a moment). Since you mentioned 4x 1TB drives, I'm assuming you are using local storage to FT and HA are not possible. It's not very likely, but using the usb drive is relying on a non-redundant disk to run the OS.

If installing to the local disks, make sure to partition them otherwise you will end up with the OS on partition formatted with a 1MB block size (having a max file size of 256GB). It is generally recommended (when financially possible) to have the OS on a redundant RAID volume, an the data/VMs on another redundant RAID volume. I don't know if this is possible in your situation, but I wanted to share the knowledge.

you will also want to keep any RAID volumes to 2TB-512B or smaller.  ESXi will not recognize anything 2TB or larger.

Good point. Keep each partition under 2TB. 4.1 is supposed to recognize a single local volume over 2TB (broken into partitions), but there are some reports of issues with this. Keep each volume under 2TB to be safe.

Yes it's DAS storage via Adaptec 5405 PCI-E RAID controller.
 
With the partition size over 2TB we have already had that issue with ESXi 4.1

We setup a single 3GB LUN and while VMWare 'sees' the LUN @ 3TB
Once formatted as a datastore it's size becomes 740GB

So yes will partition to smaller than 2TB LUN's on the RAID controller.

Should the VMWare OS be on it's own LUN of say 40GB
Then two LUNs at 1.5TB for datastores ?

Yes "matthew_way"...I HIGHLY recommend segregating your hypervisor install from your VM datastore storage. A very common option you can do is install ESXi on a USB stick (see here for instructions: http://kb.vmware.com/kb/1020655), then you can use all your disks for datastore storage.

Hope that helps.

Regards,
~coolsport00

Matthew_Way:

Using a memory stick you have the advange of solid state storage (more with reliability than speed), so it's unlinkely to get damaged as a HDD, however there is no way to provide HA on that scenerio in case the stick dies the good part do it's you separate HyperVisor load out of the VM's store.

If it's possible do 2 x RAID 1 rather one big R5, remember all your VM and Hypervisor would use it at once.

Yes, 40GB for the OS should be good, allowing 2 x 1.5 TB datastores.

The USB stick isn't a bad option, but the inability to have redundancy is a dealbreaker for many people. Flash devices are quite reliable, but a single bad block could mean all your VMs on the host are unusable for an extended period of time as the host is diagnosed, a replacement drive is installed, and esx is reinstalled and reconfigured. It's unlikely, but possible - it depends on the risk you are willing to take.

For me, giving up 40GB of local storage to have redundancy is totally worth it.

The way to mitigate redundancy issue is have a 2nd USB stick available in case of some hypervisor failure. That still leaves some downtime, though. I personally have yet to see a hypervisor install fail though...it's pretty stable. But, you always want to prepare for the worst. Regardless, it is still a viable option.

Regards,
~coolsport00

Thanks for all the advice.

The system isn't HA as that would require us to move from DAS storage to SAN, plus the licensing etc etc.

Having said that we want to make the system reasonably reliable.
We don't actually have a big requirement for storage beyond backup.
So I'm thinking of setting up the 4x1TB drives as a RAID 6 array.
Giving us a usable capacity of 2TB.

Thinking about it if the RAID setup fails we are out of action anyway.
So booting off USB flash may just be a second failure point.

The 2TB of usable space I will setup three LUNs
LUN 1 - 40GB - VMWare OS
LUN 2 - 1TB - Datastore 1
LUN 3 - 1TB - Datastore 2

Unless you need to create some volume of extreme large size "matthew_way", I would further split your 2TB array into at least 2 1-TB arrays, at a minimum. This will give you more flexibility as well as redundancy, and HA. If you have 1 array, you have the potential to have all your VMs be down. With separate arrays, you can split up where your VMs are and be at least 1/2 up in case 1 of your RAID arrays fails. That would be my suggestion...

What you have will certainly work, but having as much of a HA scenario as you can based off your budget is the best way to go. And, segregating things is also general best practice. It's really up to you.

Hope that helps.
Regards,
~coolsport00

@coolsport00
I agree with that thought process. However, in terms of actual redundancy, RAID 6 would be preferable since it can handle 2 drive failures. With 2 x RAID-1 arrays, it's possible (50% chance) that 2 drive failures would take out one of the arrays. You will only lose 50% of your data (and possibly the OS since it is be shared with 1 of the arrays), but that's still a huge loss that would require backup restoration procedures and significant downtime anyway. Personally, I would rather provide better protection from the downtime via a RAID-6 array. Plus it can more easily grow.

Just a different approach - both have their merits.

True and agree that both have merits & downsides....I still like separate datastores personally :) It's really more up to the posters performance requirements and SLAs I guess. :)

Regards,
~coolsport00

Once again thanks for all the advice.

We have probably got a requirement for around 1TB storage total.
Which is why I have decided to use RAID 6 over 4 1TB drives which as pointed out by geowrian will handle two drive failures.
The largest VMDK we require would be around 200GB and if we require more we can use LVM on the guest ( Ubuntu Linux ).

SLA's we don't have but I did look into HA using a mirrored pair of SAN controllers but with that and the VMWare extra licenses it took the project budget from about $10K to $25K

One last parting question what size should I allocate for the VMWare OS datastore ?

Thanks for the great advice.