• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 706
  • Last Modified:

How do I increase backup rate on Symantec backup exec after virtualizing the Exchange Server 2007?

2 weeks ago, Exchange Server 2007 crashed and we had to create a new server. We decided to Virtualize Exchange this time. However, Backups have not been lasting about 3 hours more and there fore timing out(23 Hrs) before it gets the rest of the servers. The transfer rate has essentially been cut in half going from 1000 MB/Min to 500 MB/Min on the Exchange Server. All appropriate exclusions have been added to the Antivirus scanning. The AV has officially been ruling out as the culprit since everything occurred after virtualizing Exchange. The hardware specs are:

OS: Windows Server 2003 R2
Exchange Server 2007 - 64 Bit Windows Server 2003
Running Vcenter 2.5 on Windows Server 2003

Any suggestions on trouble shooting this? Very much appreciated. Thanks.
0
fstinc
Asked:
fstinc
  • 11
  • 9
  • 3
  • +1
1 Solution
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What VMware product are you using VMware Server?

How is the Exchange Server currently being backed up?
0
 
fstincAuthor Commented:

Running VSphere. Exchange is backed up to Tape. Is used to run complete the backup of Exchange 2007 in less than 1:30. Now its doubled. Is it more likely to be a problem in the hardware or a setting on VMware. I was thinking hardware but I am not sure.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So this is vSphere ESX or ESXi.

If you are performing an Agent based backup of the VM to tape, across the network, I'm afraid this is to be expected, tape based backup of VMs, based on file/folder copy, can be slow.

Performance can be increased,  but this depends on how fast the storage system underneath ESX is, what RAID, disk speed, type are you using?

The bottleneck is likely to be the VM/datastore read, unless you have other evidence (e.g. network utilization).

I know it's not what you want to hear, but I would recommend, investigation other Backup solutions, e.g. Veeam Backup and Replication, Quest vRanger Backup, which will backup the VM, at a virtual machine block level, rather than  file/folder based, this will be much quicker, these backups can then be saved to tape if you wish.
0
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

 
Danny McDanielClinical Systems AnalystCommented:
use the performance charts on the VM within the vsphere client to see what latency goes up to for the datastore/disk during the backups.  also check network utilization and packet drops to verify that you're not getting clogged at that level
0
 
Gerald ConnollyCommented:
What tape drive are you using?  1000MB/min = 16.7MB/sec, could be verging on shoe-shining speed for even a fairly old tape drive.
0
 
fstincAuthor Commented:
The VM is under a Raid 5.  

Disk Write Rate Averages: 390.42 KBps Max: 3879 KBps Min: 33
Disk Read Rate Averages: 144.98 KBps Max 1709 KBps  Min: 1
Disk Usage (Averages): Average: 532.62  Max: 4269 Min: 36

Network Usage (Avg): Max: 843 KBps Min: 8 Avg: 114.96
Network Data Receive Rate: Max: 521 Min: 4 Avg: 41.41
Network Data Transmit Rate: Max: 710 Min: 3 Avg: 70.73

The tapes are definitely not the issue. We have been using the same ones for the 2 years.

I am currently trying to compare the disk reads to other servers to verify the problem . . .
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
VM file/directory read is probably the bottleneck. If it's causing you an issue, it may be worth starting to investigate disk based backups of the VM. (e.g. Veeam Backup etc)
0
 
fstincAuthor Commented:
I wish this was an option for me, but #1 there is no budget. #2 Big guy doesnt take "not possible" for an answer. . . Anything ideas as too increase this speed to a point where the backups can complete under my 23 hr caps?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What is the underlying datastore which the VM resides on?

and if the "big guy" doesn't take "not possible" for an answer, ask him to call us! and we'll will ask him did he complete any proof of concepts, pilot tests before committing to virtualise the server.

"If you have a look through EE, this is a common question, we were backing up a physical to tape, and we virtualised it, and now it takes long why!"

I know it's not want you wanted to hear, but let me know the VMware ESX hosts specifications, datastore, RAID 5 is not enough, what speed our the disks, how many disks, 7,200, 10,000 or 15,000 rpm, SATA or SAS etc What is the RAID controller, do you have a BBWC (battery backup raid controller)

and we will see what we can advise, RAID 5 (fyi), is not the fatest datastore to use, if you require performance, RAID 10 is better. (but that would require, a destroy datastore and re-format) unless you can create a new datastore with new disks.
0
 
Danny McDanielClinical Systems AnalystCommented:
The performance measurement that you want to look at is Datastore latency.  If you click on the performance tab at the host level, then the advanced button, select Datastore in the 'Switch to:" dropdown.

That should show all of the datastores and the last hour of statistics.  Click on 'Chart Options' to change the time period to something that includes the last backup run.
0
 
Gerald ConnollyCommented:
So what tape drive do you have?

If your tape write speed was only just above the minimum write speed for the tape (before descending into shoe-shining), and if your change to using ESX has added a tiny percentage of latency then you might be in the shoe-shining zone, different generations of tape drive have different minimums and in the case of LTO different minimums for diferent manufacturers in the same generation.
0
 
fstincAuthor Commented:
1) Im need to prove to my manager that the disk reads have decreased. Not sure how to get it from the past server since the backup history is no more.

2) Here is more info:

ESXi Servers are running on Dell PowerEdge 1950.
SAS Drives of Hitachi: 73G, SAS, 15 K, HIT VIPER
DataStore: Sata Raid 5 1.36 TB capacity  
Free: 172.72 GB

@connollyg The tape drives that I am using are SONY LTO Ultrium . 400GB/800 GB Compressed

@handcocka. I am still trying to gather all of that information. . . Havent been here long enough to have memorized that info

@danm66: Dont see those options. I have included screen shots.


He wants hard numbers, that I am doing my best to get a hold of. Any suggestions? Very much appreciated.



ScreenShot300.jpg
ScreenShot301.jpg
ScreenShot302.jpg
ScreenShot303.jpg
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Other than moving the Virtual Machine to a new RAID 10 set (and I don't see the performance increasing that much), or Fusion IO-Drive (SSD) - now these are difference you will see a performance increase in using these, we use these for VMware View 4.6 VDI desktops for rapid provisioning, lots and read and writes.

If the history is gone, not too worry, check it from the present now - (future), when backups are running, and checking datastore latency. (read and write!)
0
 
fstincAuthor Commented:
More info:

2 Hosts: ESX(1) and ESX(2)
2 Networks
4 Datastores
36 Virtual Machines
6 NICs/ Host
2 CPU / Host

Looking at these numbers, it might most likely be the NIC's, since I would highly doubt that Exchange has a dedicated NIC. Is there a way to verify?
0
 
fstincAuthor Commented:
What is considered a good disk read/write (KBps) on a physical server as opposed to a virtualized server.

I know that his server is giving me these numbers:
Disk Write Rate Averages: 390.42 KBps Max: 3879 KBps Min: 33
Disk Read Rate Averages: 144.98 KBps Max 1709 KBps  Min: 1
Disk Usage (Averages): Average: 532.62  Max: 4269 Min: 36

But where can I get the numbers on an Physical Exchange Server. I want to let my manager know that, "On physical servers, we were getting disk reads at (#) and now we are getting 29 KBps. For this reason, there is not much we can do to speed up out Exchange running on VM other than changing RAIDs or writting to disks"?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Look at the performance of the NIC, and check the utilization.

also, there is a flaw in the network reporting of vCenter, The vSphere Client can indicate that VM network traffic is causing a 1 GB Ethernet adapter to have a 99% utilization rate. But strangely, it doesn't display which kind of traffic is going across the virtual networks, where it came from or where it's going.

Put this in your VMware Toolbox (for free!)

To learn which traffic is going across a virtual network, there's another free tool for vSphere: Xangati for ESX, a virtual appliance that tracks conversations on the virtual network. It's great for troubleshooting any virtual network issue, analyzing virtual desktop infrastructure and correlating vCenter performance stats with virtual network stats.

http://xangati.com/try-it-free/
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You would have to look at performance benchmarks, using IOMeter in the VM and on a Physical Server.
0
 
Danny McDanielClinical Systems AnalystCommented:
Here's a quick screen grab of a 4.1 performance chart, which should be fairly close to what you see in vCenter 2.5...  click on the Chart Options to pick specific date periods and/or monitors.
disk-latency-snapshot.JPG
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You could also use

http://www.softsea.com/review/DiskTT.html

Disk Throughput Tester, this is a cheap and chearful application, which can generate benchmarks, which you can compare VM versus Physical.

CrystalDiskMark is also a good one, we test SDDs and VMs, FC SAN LUNs with this one!

http://crystalmark.info/software/index-e.html

You can generate nice screen grabs and colour pictures for the "big man"!
0
 
fstincAuthor Commented:
@hanccocka Ok. The full backup will not run until tomorrow, so I should be able to compare physical and VM disk and network rates at that time. The "Big Man" doesnt like screen grabs, he would rather me explain things to him so that later when things go wrong, he can act like we misunderstood each other!

@danm66 Heres a few screen shots of version 2.5. There is no options for latency.
ScreenShot321.jpg
ScreenShot320.jpg
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
run the disktt program and email the big man the figures, then you have got an audit trail to produce later!
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
i think what you need to demonstrate is physical disk is faster than virtual disk in mbps.

iometer or disktt will give you the figures.
0
 
fstincAuthor Commented:
Yeah, so what I did was use Disktt and recorded the read rate on a server, my computer and compared it to the vmware statistics. Here are the results.

Physical Server: 50  MB p/sec
Desktop: 28.5  MB p/sec
VM Exchange: 8 MB p/sec

Hopefully this is enough. Is there really any solution to this problem? Maybe I could reduce the size of the backups so that they backup in the 23 hour window? Really thats they only solution I can think of. . .
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
things you can do, not weve highlighted these.

1. change the vmfs datastore for RAID 10
2. I dont know the model of server or storage array controller, but if you have HP server or controller, a battery backup write cache module, boosts read and write performance, and can be configured as 25/75 percent reaud and write.
3. Finally, the use of SSD solid state drives, we use these for read/write performance solutions, weve been using Fusion IO drives, but we are now testing consumer Intel, Kingston and OCZ offerings in ESX.
4. Backup using a virtual machine backup software, that does block level backup, e.g Veeam Backup replicator, investigate the benefits of vm backup, in favour of vintage tape backup.

Good Luck with the Big Man.......
0
 
fstincAuthor Commented:
Good Stuff
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 11
  • 9
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now