Link to home
Start Free TrialLog in
Avatar of sustrans
sustransFlag for United Kingdom of Great Britain and Northern Ireland

asked on

Veeam Backup or Replication causing Exchange 2003 VM to Max CPU

Here is my environment;

Vmware V4.0 essentials bundle installed on 3 hosts with VCenter and Veeam Backup and Replication 6.5 installed as separate VM's within one of the hosts and happily backing up all VM's.

We are in the process of migrating all mailboxes from a physical exchange 2003 sp1 server to a virtual exchange 2003 sp1 server over a period of two months, and all has been going fine however last week after 6 weeks of happily working both backing up and replicating every 3 hours the growing store, our virtual exchange server ground to a halt and stopped responding, the CPU maxed out at 100% as soon as the Veeam replication job started, this also occurred again when the Veeam server tried to backup the server later that day. In each case I terminated the veeam job and within 15mins or so the server returned to normal operation or sometimes required a restart if this did not occur in a timely manner.

If a restart was forced the snapshot Veeam created for copying etc. then needed to be manually re-combined to ensure stability etc..

It looks like store.exe is using high CPU amongst others rising to around 50% during backup or replication along with a few other exchange tasks. Store.exe during normal operation runs at around 2 or 3%

The exchange VM becomes so busy it starts to reject and drop outlook client connections, rejects RDP and becomes pretty much unresponsive forcing a hot reboot at worst

All other VM's being backed up are fine and show no signs of a performance hit whilst being backed up. so I am thinking this is not a host performance issue however the RAID controller Battery has failed but is due to be replaced ASAP, but as the other VM's on the host with the failed battery seem fine I don't think it is related. But I know without battery the RAID controller may have changed write back/write through etc..

It seems something within Veeam is causing MS Exchange to max its CPU out. For reference the virtual exchange server has 4 vCPU and 4GB Ram and plenty of disk space, which is the same spec as the Veeam Server, and Vcenter Servers.

The 3 Virtual Host Servers in use are Dell R710 with dual quad core zeons @ 2.6GHz and 24GB RAM each. Also for storage the servers have internal storage of 6 x 15,000rpm SAS 600GB in a RAID 5 setup.

At present I can only run the backup out of hours to ensure no production problems which before i could do this during the day and replicating has been suspended during production hours to ensure exchange remains up. Replicating worked fine every 3 hours before last week but now I am unable to replicate

Please Help..
Avatar of chubby_informer
chubby_informer
Flag of Trinidad and Tobago image

disbale backup while u are moving across.  

or if you can...shutdown the other virtuals while u migrating
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
do you have 2 vCPU, and 4096 of memory in the Veeam Server?

is this when the snapshot has been created?
Avatar of sustrans

ASKER

The veeam server has 4 vCPU and 4GB Ram running Win Server 2008R2

When the snapshot has been created the exchange server starts to slow down and comes to an almost stop refusing any more connections and dropping exisiting connections.

I ran a test and excluded Veeam by creating a snapshot within VMWARE only and same thing happens, after the snapshot is created store.exe climbs in cpu use and the the server grinds to a halt.

So this seems to be a VMware issue ?? or Exchange Bug ?? not Veeam..
It's caused by the Snapshot, which causes performance issues, on slow datastores.

It's very common, and there have been a few related questions on EE, about this same error.
Would a RAID controller battery failure make the situation worse?

The data-store is on top of a RAID 5 Set on 15k SAS Drives
ASKER CERTIFIED SOLUTION
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I will be replacing the battery today so hopefully this will resolve the issue, many thanks..

I will report back..
Battery replaced and Data store performance back to normal..