Seeking storage solution advice for VMWare ESXi rollout.

The time has come to replace our servers and I am looking to take advantage of virtualization. I am having a hard time deciding whether or not I want to go with servers that have plenty of (yet finite) internal storage or go with small (1U) servers and a SAN. I've been toying with VMWare ESXi and so far, I love it. I noticed that it will allow me to add networked storage as datastores to store virtual machines.

Since I am looking at about 3 TB of total storage I was thinking that a SAN and many smaller disk may be the way to go for performance. However, I can get 3 TB in a single 2U server with SFF disk. Below is a list of all the servers I need to virtualize...

4 - Novell Netware 6.5 (to be upgraded to Suse Linux ES 10 SP2 with OES)
6 - Windows Server 2003 (already running under MS Virt Server 2005)
1 - Suse Linux ES 10 SP2

Combined current storage usage is about 1.5 TB so I want to double that across the board to 3 TB total. In order to plan for a spike in data storage I am leaning towards providing about 4TB. I serve about 100 users with mainly File/Print/E-Mail. We do have a few specialized applications that require the Windows boxes.

Does anyone have advice on whether to go with a couple large form factor machines with internal storage or a SAN? If you like SANs do you recommend iSCSI or Fibre?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

For SAN the backup is easier to manage especially if you decide to use VCB with your favourite backup software
Another advantage of have SAN as storage is you can share your storage among multiple hosts and use the esx advance feature, this is where the fun begins!
Both iscsi and fibre is good technology, everyone thinks that fibre is really fast(well it is)
iscsi can be fast too, depending on your design and budget, if you can allocate a pure dedicated IP network for your iscsi storage, it will be fast and cheap as well
and you are right, smaller disks for your VM are good for performance, in several enterprise customers i supported before, they normally present the LUN as required/needed, not really an estimation, so if you think you want 300GB now for 3 VMS
you can add a new 500GB datastore later for another few VMs
i always like to work with storage engineer, we may think that we know a lot about storage but when we deal with a dedicated storage guy, we'll learn a lot more :)


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
I would go for ESX servers with shared storage (ryder0707 says, iscsi or FC does not matter, it depends on your needs and budget). The advantage with a san is that you can expand. You can now buy what you need and next year, if you need more, expand it.
If you need more processing power, add an ESX server to it. If more storage is needed, add disks or an expansion bay. Offcourse this will cost you money at first, but gives you a lot of flexibility later on. Advantages of more ESX servers with a san is that you can use vmotion. When the server needs service, you can vmotion your vm's off to an other esx and bring the esx down or do what ever you like.
11 virtuals normally can run on one hardware box (depending on your load offcourse), but then it's normally full and you do not have room for expansion. Begin with two ESX servers and a shared storage, expand later. If you need to watch your money, you can go for a single esx with local storage, but a few advantages are gone. When you have a hardware failure, all virtuals are down. If you want to expand and need a second ESX box, and you want to vmotion, you still need shared storage and rebuild your environment.
Watch performance though. IF you run 11 virtuals on the same disks (local storage), it needs IO. It could be that your local disks aren't that fast enough to provide you with enough io to give you the performance you need (disk bottleneck).
za_mkhIT ManagerCommented:
In addition to the above, be aware that the max size for a LUN or datastore that ESX/i supports is 2TB. You can use Extents to increase this to a total of 64TB but think about that too when you do this.
We currently have a few Novell Servers running on ESX and they are great. At the moment, they the IPP/Zenworks servers and soon our Fileservers too.
Check your IO Load for Storage - that will dictate how you will carve up your SAN / underlying disks (ie do you use RAID1, RAID5 or RAID10).
In my experience for VM Load testing, the amount of load I generated one VM (with IOMeter) connected to our FC SAN, was so high, that comparing this to the IO Load generated by our production VM guests put together (that were all running on local storage) didn't come close to matching it! So I knew that our SAN was up to the task. And I am talking about Oracle Servers / SQL / CRM, etc ...  We are site that is double your size.
If you looking at SAN solutions. I highly recommend SanMelody ( It is a great way of using disparate storage to build your SAN. In addition, to Windows, ESX, etc, we also have our Physical Novell Servers connected to our FC SAN via SanMelody, etc ... works great (but took a while to setup)..
Powerful Yet Easy-to-Use Network Monitoring

Identify excessive bandwidth utilization or unexpected application traffic with SolarWinds Bandwidth Analyzer Pack.

nappy_dThere are a 1000 ways to skin the technology cat.Commented:
I have never been a fan of internal storage for any ESX environment except for a test lab.

VMWare recommends using the best practice of n+1 for ESX host planning. This practice means "n" is the number of ESX servers you want to have and then add 1 more for redundancy.

Shared storage is definitely the way to go.

I would point you in the direction of the Dell 2950III servers with dual quad core processors, and 32gb of RAM. Each processor core can support between 4-6 guest OS's.

For storage take a look at the Dell MD3000 DAS units. The MD3000 can support upto 4 servers connected for shared storage.

Along with this config, take a look at getting Virtual Center Foundations with 1year gold support.

For backups, consider adding VCB for accessing and backing up your ESX guests.
Keep in mind the MD3000 has SAS and/or SATA disks, this could be a performance bottleneck. You have also the i version (do not know that one) which can do iscsi. Both are supported bij the vwmare hcl list (
andyalderSaggar maker's bottom knockerCommented:
If you've got internal drive bays and want to use them you can use LeftHand Virtual San Appliance to turn the internal sorage into an iSCSI SAN and use network RAID1 across two VMware servers running VSA to make the SAN redundant.

As far as hardware manufacturers go it doesn't really matter, they all perform similarly because they all use the same CPUs.

SFF disks rock, take a look at the HP MSA2324 in iSCSI, FC or SAS (up to 4 hosts max with SAS unless using blades) if you don't want a software iSCSI target running as a WM.
nappy_dThere are a 1000 ways to skin the technology cat.Commented:
Bleeuem, not sure I would consider 15K SAS drives slow by any stretch of the imagination.

I have in one environment an HP EVA 4400 san with SAS and it runs great.

The MD3000 comes in at about half the cost.
Sure it runs great, but every environment has it own needs. You need to design your environment to your needs, and ofcourse everything has it's price. If you build your san with multiple large 1 TB sas disks you have  less performance then multiple 300 GB sas disks. But for the same amount of storage you need much more disks, and more expansionsbay's (more money). So sometimes faster is better, but if you do not need it, it is a waste of money.
So in this case the MD3000 can be a great solution, but i do not know the requirements in this specific case.
nappy_dThere are a 1000 ways to skin the technology cat.Commented:
Keep in mind, the MD300, Like the HP MSA series or the HP EVA series are expandable.  So with all of these and other solutions, you can add additional trays for storage.  I would suggest you start with one try and fill it with drives.  As the environment grows, you can always add trays and configure additional LUNs as required.
jasonsfa98Author Commented:
It was like Christmas waking up to all this info, thanks guys.

One thing I do not quite understand with VMWare ESXi is it's own storage needs. For instance, do I even need physical disk installed in the server? It seems like it takes up so little space that any drives I get would be wasted. I suppose an SSD or something would be fine? Looks like HP does this.

Also, once I create a VM for say- Netware. Will it hurt performance to have the OS sitting on one datastore and the "data" volumes sitting on another? I ask because I always have the OS installed to a separate set of disk. It seems like there would be a lot of SAN traffic just to operate normally.
with esxi you also can run it on an usb key:
Download ESXi 4.o VMware-VMvisor-Installer-4.0.0-164009.x86_64.iso
Open image.tgz
Open image.tgz\usr\lib\vmware\installer\VMware-VMvisor-big-164009-x86_64.dd.bz2
Extract VMware-VMvisor-big-164009-x86_64.dd to your local hardrive
Attach the USB flash drive and make sure you no longer need the data on it
Use WinImage to transfer VMware-VMvisor-big-164009-x86_64.dd to the USB flash drive
       Disk->Restore Virtual Hard Disk image on physical drive&
       Select the USB flash drive (Warning: If you select the wrong disk you will lose data!)
       Select the image file VMware-VMvisor-big-164009-x86_64.dd
       Confirm the warning message
       Wait for the transfer to complete
Unplug the USB flash drive (Warning: If you forget to unplug the flash drive from the PC you might lose the data on your hard drives the next time you boot!)
Attach the USB flash drive to the machine you want to boot (Warning: If ESX Server 3i recognizes local drives, you might lose the data on it, so make sure you don´t need it anymore or unplug all hard drives!)
Turn the machine on and make sure the USB flash drive is selected as boot device  

nappy_dThere are a 1000 ways to skin the technology cat.Commented:
You can get servers with ESX embedded or you can just simply run it from a 2B USB key.

The best practices is you go iSCSI is to place this traffic on it's on vLAN or separate network switch from the rest of your network.

It will not hurt performance to have your data and OS on separate sets of disks.  The only performance hit to keep an eye on is the IO for read/writes to your shared storage.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.