Setting up New Redundant VMware ESX Server Based Network & Servers - Need Advice!

Hi there,

I am designing up a new server solution and as the technology is fairly cutting edge and the investment is large, I thought I would run it past people to see what they think, and garner any suggestions. Also I am aware that people may have set up systems like this before and have found\created set up guides, lists of gotchas, etc. I am aiming for redundancy across Email, web servers, network shares, and hardware. On top of this I would like the system to be expandable, to be able to put email servers in our international offices, in order to provide greater redundancy would be excellent.  Basically any constructive input would be GREATLY appreciated! This is what I have planned:

Server 1
HP DL380 G5 12GB RAM, 8 x 146GB SAS Disks, 2 x Quad Core Xeon
Array 1  2 x 146GB SAS in Raid 1 Array 2  2 x 146GB in Raid 1 Array 3  4 x 146GB SAS in Raid 5
Server running VMware ESX Server 3.5
VMs running Windows Server2003 Standard x64 & x86
VM1  Exchange Server 2007 Standard (6GB Memory), VM2  Windows 2003 Standard x 64 for file sharing, VM3  Windows 2003 Standard x86 for IIS, VM4  Domain Controller / Active Directory

Server 2
HP DL380 G5, 12GB RAM, 8 x 146GB SAS Disks, 2 x Quad Core Xeon
Server running VMware ESX Server 3.5
VMs running Windows Server2003 Standard x64 & x86
Array 1  2 x 146GB SAS in Raid 1 Array 2  2 x 146GB in Raid 1 Array 3  4 x 146GB SAS in Raid 5
VM1  Exchange Server 2007 Standard (6GB Memory),  VM2  Terminal Server VM3  Backup Server

Is VMware ESX server the best way to go for using virtual machines?
Backup  What should I be running here? If there is a good way to backup VMware then I would love to hear about it, I am thinking of backing everything up to 1TB USB Drives. Is Symantec Backup Exec & System Recovery the way to go here?

Misc Hardware
UPS  2 x 3000VA UPS

Big Questions
Does VMware work well on this hardware? I am assuming that VMware ESX is best in this scenario.
What is the best way to make Exchange 2007 redundant here  would that be clustering?

So that is where I am at now, if you can see any glaring problems or simply have found a way to do things in a way I havent considered please post away!
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Is VMware ESX server the best way to go for using virtual machines?:
Yes, it has the most features. But ofcourse you need to look at your own specific situation. Virtual Server or Virtualization from Microsoft can sometimes be better in a specific situation.

Backup  What should I be running here? You could use any backup solution. Make sure the backup software has also a linux agent to backup the esx server itself. I've used TSM and Commvault, both are suitable for the backup.

Does VMware work well on this hardware?
HP is supported by vmware. Look at for compatibility guides.

Maybe a san is an option in your enviroment so you could use HA of vmware. This way you could use the failover software from vmware. Also you can set-up a cluster enviroment for your exchange server (node 1 on esx1, node2 on esx2). The benefit of a san is that u can vmotion the virtuals to both esx machines. This way the downtime (when you need maintenance one esx server) of virtuals can be less.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Q: What is the best way to make Exchange 2007 redundant here  would that be clustering?
A: Yes, you can create active/passive cluster of virtual machines but you must have a SAN in order to this. In order to implement clustering you need to have shared storage between those virtual nodes. You do need SAN when the cluster is on the same ESX host, only when there are two physical hosts. I have deployed this scenario and it works (both of them). If you need more detailed guidelines feel free to ask.

In my opinion the best high availability is achieved when using a SAN. You will need two FC HBA controllers (per physical host) or you can use dual-port FC HBA controllers (this is for redundancy), two FC Switches and one SAN with two FC Controllers with cache on them. You can rely on the VMware high availability (VMotion as bleeuwen said).

If you need more details please ask,

the-wavesAuthor Commented:
Hi Guys,

Thanks for you suggestions, when you say a SAN, would something like the HP MSA 70 be adequate? Also would I need enterprise editions of Exchange 2007 or would standard do it? I like the look of Commvault, I have never heard of them before now but they look great, would it be possible to use them as the only backup solution for the entire network?
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

The HP MSA 70 is a standard san, i think this will do for your environment. Watch out that this enclosure is not extreem fast (but the question is what do you need for performance?).

Exchange 2007 standard does not support single copy clusters (, for more info about SCC look at

Commvault is an enterprise solution. You work with agents on the servers which you will backup, then the main server (or multiple main servers, you can use mediaagents) will backup the agents. This way you can backup your entire serverfarm very fast. TSM is also an enterprise backup solution which works almost the same way. These kinds of backup solutions can backup your entire environment. Commvault also has a VCB (backup for vmware) implemented, so you can backup full vm's (or parts of it) very fast.
the-wavesAuthor Commented:

I noticed on the MSA 70 page there is no support for VMware ESX- mentioned:

If I didn't go for that do you have any other SAN recommendations for 50 -100 users & 6 VM's?
When you have HP servers, you can use the HP EVA storage (maybe a bit to large for your environment). The EVA Storage is from HP so it will work with this servers. There is a HP selector which gives a indication which device you can use:
typically you have 2 main choices: A SAN, or no SAN.
If you dont want a SAN, you would be limited in cluster functionality regarding VMware ESX and MS products like Exchange. I dont recommend ESX in this setup. VMware server (free) will do just fine on those machines.

If you do want SAN, you can think about a ESX3.5 / VCenter combination (VMware enterprise bundle). I would recommend iSCSI storage in your environment. Have a look at the new EMC AX4 iSCSI storage array. its scalable and can give you enough performance.
iSCSI is cheaper to implement than FC. You Could use your simple 1Gbit NICs in the server to connect to the AX4, instead of using $500+ FC HBAs. However, iSCSI HBAs from Qlogic would fit better than standard NICs from the HA point of view.
With a SAN, you'll have the possibility to use VMware ESX Clustering features. HA and DRS.  I've seen HA clustering from vmware replace MS clustering completely. Much less complex and will do the job just fine in environments like yours.

I have experience with CommVault and i think it's a great product. VMware integration is not that great yet (with ESX), due to scripting procedures. However, Agent-backups are great. Exchange Archiving is really simple too, all from the same gui.
I would recommend a product like Vizioncore's vRanger PRO for simple (image) backups of Virtual Machines to USB disks.

Please check that Exchange 2007 is fully supported on a VMware platform. I'm not sure about that... :)

I am running exchange 2007 active/passive cluster for the last year and I have no issues at all. The cluster has been working without any kind of problems.

When VMware has no storage supported it doesn't mean that it will not work. I will ask you to try and see yourself. I have two "unsupported" storages and they are running really good, no problems at all. We had a employee of VMware and we told him that we are running on unsupported storage and he said, the support for VMware on that storage (technical support) is not present because they didn't try to run VMware on it, that doesn't mean that it will not work!

So I will tell you to test it for a while and see for yourself.

I do not agree completly with Clustor. VMware server (free) has a memory limitation of 3.6 GB. So the guest can not have more than 3.6 GB of memory. Your exchange servers has 6, this mean you have the choice between workstation and ESX. Workstation has disadvantages (you need to be logged on).

ISCSI is a nice feature and is sometimes (what is your bottleneck, disk, cpu, network?) faster than plain old SCSI, but when you use ISCSI make sure you have dedicated hba for ISCSI with an offload engine. If you use standard 1 GB nics the processor of the server will process the storage network traffic. This will give an additional load of 10%-20% on your cpu power. With Offload on your iscs hba, it will only take 3 procent or less (for exact figures you need to google. This figures i had were during hardware workshops to demonstrate iscsi). ISCSI nics with the offload engine are not cheap, just like FC hba's. FC costs are in my opinion the switches which are needed. Advantage off ISCSI is also the distance it can travel.

Clustering (HA) from ESX is not a typical replacement of Microsoft Clustering. If your ESX crashes, it will vmotion (migrate) your running vm's to your other esx machines and will start the virtuals. Starting just means starting, if your windows box stops with a BSOD it is also started. Microsoft Clustering is a different mechanism. Normally when a node crashes, the other node takes the resources and it will work. Offcourse you can use microsoft clustering inside vmware. You can do this with rdm's, more info about mscs and vmware is desribed in
@ predragpetrovic
How would you run exchange clustering without shared storage?

Ok there is a limit on memory for vmware server. Version 2.0, which will come available soon, will fix this and will have a 8 GB RAM limit.
Workstation is just a no-go.

I cant see into the authors budget, but with 50-100 users, i would not go into any FC-like solutions.

10-20% CPU for iSCSI handling? interesting figures i must say! But again, from the cost perspective in this environment, and with dual quadcores, i dont think this is a big issue. You could start with gbit nics and later on switch to iSCSI HBAs.
The real reason i would go for iSCSI HBAs is Availability and failover. Its the only way to get good loadbalancing and failover on IP networks.

Technically, bleeuwen is right. But again, whats really needed here?  VMware HA is much less complex, and CAN be sufficient. I have seen alot of offices use MS clusters that are not maintained properly because sys admins are "afraid"  it will break (lack of knowledge). MS clustering and VMware HA are not on the same level, as bleeuwen pointed out. VMware HA restarts VM's on available hosts in case of ESX cluster node failures, as MS clustering is more fluent and more integrated in the clustered application.

Supported hardware:
There is a reason why compatability matrixes, it's a guarantee in case something does NOT work. Always a fun discussion :) Not a risk worthwhile taking however. VMware is very easy going with this though as stated above.
Vmware server 2 (beta) looks promising. It has a 8 GB limit and you can (if you have a shared storage) build windows clustering on it. The advantage is the price of the software (vmware server is free).

Cisco has a lot of iscsi information, in this article you can see a good comparison of the traffic (different block sizes) with FC, TOE and HBA's.
Remember, if you are considering ISCSI read the vmware white paper (

So like i said earlier, you have to look at your own situation. What do you need (want) for tools for your management, do you have enough skills/personel, do you have the money, what will bring the future (if you go for FC, SCSI, ISCSI, Local (sata) storage make sure it is expandable and you can easily switch to  other techniques.......

the-wavesAuthor Commented:
Hi guys,

That got complex pretty quickly! I have done a bit of research and I have found a couple of good SAN Solutions, first the new HP MSA 2000i

and also the Wasabi iSCSI SAN

For redundancy, I think I will run Exchange 2007 Standard over a number of AD sites, if one goes down the next server on the MX record can receive mail and vice versa, and sync back up when the AD site comes back online.

In regards to Backup it's a jungle out there! I think I will use Visioncore vRanger Pro for snapshots, and maybe Acronis True Image Enterprise Pro Server somewhere in there? For Offsite Backup I am keen on EMC's new Mozy enterprise - maybe that is the way forward for file level backups. I didn't realise this before hand but apparently I will need a separate physical server from the VM servers to run backups also. I would like to award some point for this as you guys have been very generous with your time and expertise, and last contributions?

Cheers again!
the-wavesAuthor Commented:

In the End I went with a Wasabi IP-SAN on an Intel Chassis, it is working well, but at this tage is incredibly loud. I went with an Intel SSR212MC2R Storage Server for Hardware. All of this Intel gear seem sreally loud - I am pretty keen to silence it so I will keep you posted if I have any success. In addition, installing the Dongle and getting wasabi working on this box was a bit fiddly on the Intel side, Wasabi support was great but if you can just buy it preconfigured!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.