Hot Plug Enclosure/Setup for ESXi box
Hi folks...need some assistance with a number of questions, hope someone can help...
I've built a new server that is running ESXi 5.1 with 2 VM's - Nexenta(for ZFS) and Windows 7
This is the hardware I've used to get this working
Motherboard - SuperMicro X8ST3
Hard drives - 16GB USB(for ESXi), 250GB Sata(for datastore), 2 x 3TB Seagate Constellation E3's - Model ST33000650SS
How I've set this up is I have setup 'Passthrough mode' on the onboard LSI1068E controller to pass to the Nexenta VM and access the 2 3TB drives for ZFS...
This motherboard has 8 SAS ports and 6 SATA ports - the SAS are passing to the Nexenta VM so all I'm left with for passthrough to the Windows 7 VM are the SATA ports...
Ok here's the issue
I want to pass drives through to the Windows 7 VM - these drives I want to be 'hotplug' capable...here's why
I am using this box for both storage and diagnostics
The Nexenta setup with ZFS running as storage(already shared out)
Now I want to be able to use Windows 7 to run various hard drive diags applications and things like Runtime's Raid Reconstructor for example...so with the 6 onboard SATA ports this should work I assume...but I then realised what I'd really like is a hotpluggable setup - where I've the controller passing direct to the VM but able to handle hotplug drives...
This is where I am lost through and need assistance...not sure about hotplug and what is needed to make this work? Do I need an external enclosure that connects to a sata port?
I assume the onboard sata ports will not work with hotplug drives? Or can anybody give me details on what makes it 'hotplugable'?
Anybody with any experience with something like this? Or is this even feasible?
Any other info you need just ask...
All ideas welcome...thanks
Shane
I've built a new server that is running ESXi 5.1 with 2 VM's - Nexenta(for ZFS) and Windows 7
This is the hardware I've used to get this working
Motherboard - SuperMicro X8ST3
Hard drives - 16GB USB(for ESXi), 250GB Sata(for datastore), 2 x 3TB Seagate Constellation E3's - Model ST33000650SS
How I've set this up is I have setup 'Passthrough mode' on the onboard LSI1068E controller to pass to the Nexenta VM and access the 2 3TB drives for ZFS...
This motherboard has 8 SAS ports and 6 SATA ports - the SAS are passing to the Nexenta VM so all I'm left with for passthrough to the Windows 7 VM are the SATA ports...
Ok here's the issue
I want to pass drives through to the Windows 7 VM - these drives I want to be 'hotplug' capable...here's why
I am using this box for both storage and diagnostics
The Nexenta setup with ZFS running as storage(already shared out)
Now I want to be able to use Windows 7 to run various hard drive diags applications and things like Runtime's Raid Reconstructor for example...so with the 6 onboard SATA ports this should work I assume...but I then realised what I'd really like is a hotpluggable setup - where I've the controller passing direct to the VM but able to handle hotplug drives...
This is where I am lost through and need assistance...not sure about hotplug and what is needed to make this work? Do I need an external enclosure that connects to a sata port?
I assume the onboard sata ports will not work with hotplug drives? Or can anybody give me details on what makes it 'hotplugable'?
Anybody with any experience with something like this? Or is this even feasible?
Any other info you need just ask...
All ideas welcome...thanks
Shane
For your Nexenta NAS with the 2x SATA disks... you probably want this to be a physical piece of hardware, not virtual. You can run this even on commodity hardware.
The NAS provides the storage to your ESXi host. The ESXi host uses that storage to store the data for it's VMs.
While it is technically possible to make a VM and install Nexenta on it, and then use THAT for storage of your VMs, it's a bit of a chicken-and-egg problem and not really ideal. Nexenta works best when it connects talks to the disks, uninhibited by virtualization, RAID configuration etc.
If you just want to play with ZFS a bit and get a feel for it, sure, make a VM and play with it, but in that case just provision half a dozen small virtual disks as VMDK files and use those as your disks in Nexenta to get a feel for what you can do with it, creating, extending, and destroying various zpools and playing with it.
But when you're ready to use it for real, you want it to be installed physically on separate hardware. Besides, Nexenta is going to gobble up at least 4GB out of your 16GB available on your host. That's memory that could have been better used to power your VMs.
The NAS provides the storage to your ESXi host. The ESXi host uses that storage to store the data for it's VMs.
While it is technically possible to make a VM and install Nexenta on it, and then use THAT for storage of your VMs, it's a bit of a chicken-and-egg problem and not really ideal. Nexenta works best when it connects talks to the disks, uninhibited by virtualization, RAID configuration etc.
If you just want to play with ZFS a bit and get a feel for it, sure, make a VM and play with it, but in that case just provision half a dozen small virtual disks as VMDK files and use those as your disks in Nexenta to get a feel for what you can do with it, creating, extending, and destroying various zpools and playing with it.
But when you're ready to use it for real, you want it to be installed physically on separate hardware. Besides, Nexenta is going to gobble up at least 4GB out of your 16GB available on your host. That's memory that could have been better used to power your VMs.
ASKER
Frosty555...
The reason I am doing this like I've built it is to take advantage of this hardware to do 2 things - storage and diags...
The Nexenta is just creating a simple share(and running on ZFS of course) to allow me to store recovery data/misc data - not for storing VM's or anything like that...so just simple cifs share
As for the Win7 VM I figured since passthrough means passing the hardware layer to the VM - is this not correct?
If this isn't going to work then cool...this is an experiment so I wanted to get some thoughts/feedback on the setup
As for your enclosures - so from what I understand to get this working all I need is a motherboard that is AHCI compatible and nothing else? If so great...just want to check
I don't really need it to be internal...can be an external if that makes life easier...
The reason I am doing this like I've built it is to take advantage of this hardware to do 2 things - storage and diags...
The Nexenta is just creating a simple share(and running on ZFS of course) to allow me to store recovery data/misc data - not for storing VM's or anything like that...so just simple cifs share
As for the Win7 VM I figured since passthrough means passing the hardware layer to the VM - is this not correct?
If this isn't going to work then cool...this is an experiment so I wanted to get some thoughts/feedback on the setup
As for your enclosures - so from what I understand to get this working all I need is a motherboard that is AHCI compatible and nothing else? If so great...just want to check
I don't really need it to be internal...can be an external if that makes life easier...
If Nexenta is being used simply for it's fileserver capabilities, I'd suggest you set up RAID in hardware, and provide a single VMDK to your Nexenta VM, which will be seen as a single disk. If you're virtualization your fileserver, redundancy should be provided by your hardware, rather than the VM. You can use ZFS but it won't provide any additional benefits over a regular, less memory hungry filesystem.
Any enclosure will do, internal or external. The enclosure in general is just a piece of plastic that positions the connectors and cradles the disk. The only requirement, as you said, is a motherboard which supports AHCI.
This is the same reason why connecting your disk directly to the IDE header on the motherboard is preferable to connecting it to an IDE->USB adapter / enclosure.
Any enclosure will do, internal or external. The enclosure in general is just a piece of plastic that positions the connectors and cradles the disk. The only requirement, as you said, is a motherboard which supports AHCI.
As for the Win7 VM I figured since passthrough means passing the hardware layer to the VM - is this not correct?You are correct, but this works reliably only if your hardware is also working reliably, and makes assumptions about what kinds of responses it will be getting from the hardware. If you're working with potentially-defective disks which you are doing data recovery on, every layer you put inbetween your recovery application and the hardware is a potential place where problems could occur.
This is the same reason why connecting your disk directly to the IDE header on the motherboard is preferable to connecting it to an IDE->USB adapter / enclosure.
ASKER
Thanks Frosty555...
Reason I setup Nexenta like I did is to allow ZFS to be the raid(since ZFS doesn't play with underlying raid controllers very well) so that's already taken care of
Main question was about hotplug - wasn't sure what all was needed to make that part work
I've now passed the onboard SATA ports using 'Passthrough' and its working like I expected...all I need now is the enclosure to make plugging in the drives easier...something with 4 drive bays should do the trick - any recommendations on what I could use(if you've worked with enclosures before?)
As for the setup...this is not in production...this is a simple experiment to see if things work...I'm in learning mode and want to see what is possible
I fully understand that in 'data recovery' mode that connecting directly to the ports is recommended...no arguements there...
Thanks for the assistance.
Reason I setup Nexenta like I did is to allow ZFS to be the raid(since ZFS doesn't play with underlying raid controllers very well) so that's already taken care of
Main question was about hotplug - wasn't sure what all was needed to make that part work
I've now passed the onboard SATA ports using 'Passthrough' and its working like I expected...all I need now is the enclosure to make plugging in the drives easier...something with 4 drive bays should do the trick - any recommendations on what I could use(if you've worked with enclosures before?)
As for the setup...this is not in production...this is a simple experiment to see if things work...I'm in learning mode and want to see what is possible
I fully understand that in 'data recovery' mode that connecting directly to the ports is recommended...no arguements there...
Thanks for the assistance.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Least I was looking at the right product line then! I was thinking of getting this one...
http://www.icydock.com/goods.php?id=48
Good stuff, I'll see where I can source one and go from there...thanks for the input, will close this one out for now
http://www.icydock.com/goods.php?id=48
Good stuff, I'll see where I can source one and go from there...thanks for the input, will close this one out for now
If your primary goal is to be able to do data recovery - raid reconstruction, etc. or arbitrary disks that a client gives you, you really want to be plugging those disks directly into the SATA ports on the machine you're working on - a direct hardware connection gives the best chance of success especially when your disks are in questionable condition. Putting too many layers of abstraction - virtualization, passthrough etc. - is going to cause issues when you're dealing with disks that are in questionable condition.
There's a bazillion hot-swap bays on Newegg. They all pretty much work the same way. They're just a physical tray / carriage which lets you slot the disk in and out easily. Nothing special to them, really. If your motherboard's sata ports supports hot-swap (if you're AHCI, it probably does), then you can use a hot-swap bay:
http://www.newegg.ca/Product/Product.aspx?Item=N82E16817994027
But... you want to install this in a physical desktop workstation.