Solved

Exchange 2003 Info Store Backup Slow

Posted on 2007-11-20
12
2,741 Views
Last Modified: 2009-07-21
Was on Exchange 2003SP2 Enterprise on Server 1.  Saw backup speeds of over 2GB/min.  Server needed to be replaced.  

Server 2 enters the scene.  Pretty beefy, this time has SAS drives instead of SCSI.  Go from an IBM x236 to an IBM x3650.  Moved mailboxes through Exchange interface, and everything transitions over fine.  

Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store (not doing brick level).  Seeing speeds of 160MB/min.  On the Information Store.  Both server 1 and 2 were/are connected to the same network segment, same switch as backup server.  All 3 are/were connected via Gigabit ports.  End result, backup window went from approx 4-5 hours to 40-50 hours.  Can't even complete it in a weekend (we're talking over 300GB of email).

ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI (being very generous), but not sure why it's coming up so slowly.  The disk activity lights on Server2 don't seem to indicate they're doing a ton of reading and trying to send it across the network to the backup server.  Perfmon didn't seem to indicate any high number of MAPI sessions open, or high disk queue.  Processor not over utilized, so it doesn't look like it's trying to process anything.

Any ideas?
0
Comment
Question by:dav-i-son
  • 5
  • 3
  • 2
12 Comments
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20323806
Hi,

What was the backup speed of flat files from the other server earlier. Any change in the memory allocation or any noticable changes from server 1 to server 2. Any anti-virus installed on this new server. Normally we did not see that kind of speed (low) for a exchange info store backup. normally if the flat file backups are around 1500 mb then the info store backup would be around 700/900 mb.

Check for the network card drivers on both the servers. upgrade the network card drivers to the latest.
Try to run a ntbackup of exchange info store and see what is the througput you are able to achive.
This article would help up run ntbackup from the backup server.

http://www.msexchange.org/tutorials/Exchange-2003-Backup-Restore-NTBACKUP.html

Stop all arcserve services and enable Removable storage service and also the tape drives and the medium changer. To run the backups to the tape library.

See if you are getting the same throughput.

bhanu
0
 

Author Comment

by:dav-i-son
ID: 20327120
The test backup I ran on the files portion yielded 1398MB/min (from server2 to the backup server, which has the tape drive attached).

Both servers have 4GB of RAM, and both have McAfee Enterprise VirusScan (same version) and GroupShield for Exchange (same version).

I tried running NTBackup on the backup server to try doing Exchange on server2, but it wasn't able to see the Exchange environment.  If I run NTBackup on server2, (if I did the conversion right--100GB--yea I know, completed in 4.25 hours) I get about 392MB/min (triple what ArcServe is giving).  That's also including sending the backup past a firewall to a disk (my Exchange server doesn't have enough room to backup 1 storage group to the same drive that it exists on), and the only thing with enough storage space is behind a firewall.  Looks like it could potentially see 500MB if it were unimpeded (but I'd still like to get back to the original speed server1 gave--near 2GB/min).
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20331799
Hi,

one question did you say information store backup or document level backup.

"Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store "

Are you seeing 160 MB when you use document level backup or database level backup. And what was the speed on the earlier server for flat files. Was it the same 1200/1500 MB. Or was it higher. What are the kind of hard disks used SCSI/SATA any raid implementation.

Because though the the network and the memory are the same these factors do add up overheads. Disable anti-virus during the backup and see what is the throughput you achieve.

Bhanu
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 

Author Comment

by:dav-i-son
ID: 20351676
Bhanu-  

I'm using the Information Store backup.  Wouldn't even try the document-level (we've always just done the IS backups).  The 160MB/min is coming from the ArcServe backup job, through the agent, for information store.  

Speed on server1 doing flat files (for some odd reason) just came up w/ 202MB/min... but it's gotten average 650MB/min in the past.  Server1 has SCSI 15k drives, server2 has SAS15k drives, both in a RAID5 configuration.  

I'll try anti-virus is configured to skip the folders the Exchange stores (ebd/stm files) are in, but I'll try with McAfee completely disabled.  I'll post results tomorrow--it'll throw off the backup schedule if I start it now--critical items won't get backed up if I do Exchange now).
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20355398
Hi,

"ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI "

Is this the speed on the new server for flat files. As we are currently trying to understand what is the issue with server 2 right.

If the flat file backup has slowed down on server 2 then need to check if it is running on sp2 and if that is the case check this MS article and disable tcp chimney

http://support.microsoft.com/kb/912222

bhanu
0
 

Author Comment

by:dav-i-son
ID: 20357411
Bhanu-

Yes, server2 is on 2003 SP2.  Referencing that MS article, I see TCP Chimney is enabled, but not all of the entries they mention in the article appear in my registry:

EnableTCPChimney -   Enabled
EnableRSS              -   Enabled
EnableTCPA            -  Enabled
OffloadExcludeDestinationPorts  -- Does not appear
OffloadIncludeDestinationPorts  --  Does not appear
OffloadExcludeSourcePorts       -- Does not appear
OffloadIncludeSourcePorts        --  Does not appear
MinPacketSizeToDma                 --  Does not appear
DmaSyncCompletionHighThreshold  -- Does not appear

Although it says I can turn chimney off via the netsh commands, I'd rather wait for my next maintenance window, to do this off-line.  Unfortunately, I'm expecting the next window to appear around 12/13.

As a stop-gap for now, I can use NTBackup, and back those BKF files up to tape.

I'll let you know how I make out after disabling the TCP Chimney

Ken
0
 
LVL 23

Accepted Solution

by:
bhanukir7 earned 150 total points
ID: 20362348
Hi Dav,

Disabling tcp chimney would not require a reboot. Though its always good.
You can disable tcp chimney and then try to run a test backup.

A good way of testing would be to

Try to identify around 1 gb of data:

1. Run a windows copy from that exchange server to the backup server.
2. Run a copy job using ARCserve for the same set of data
3. Create a backup job using the client agent and run the backup to FSD
4. Modify the same job and run it to a tape drive.

Notice the amount of time it is taking for this data to be copied, backedup.
This would give a clear idea if the issue is with network or the software.

Run ntbackup on the exchange server See the throughput.
Run NTbackup from the backup server refer to that link i have provided in the earlier post.

This would give you a clear picture of exchange backup when it is local and remote with ntbackup.

You can try this before disabling the tcpchimney and then try the same set of tests after disabling the TCP chimney.

Bhanu
0
 
LVL 2

Expert Comment

by:mathiesen-data
ID: 21415074
I just solved my problems at 2 customers by upgrading firmware on both troublesome servers in the equation.
server1<->server2 slow
server1<->server3 fast
server2<->server3 fast

The I was going nuts, because they were connected to the same switch, so I couldnt figure it out.
0
 
LVL 2

Assisted Solution

by:mathiesen-data
mathiesen-data earned 150 total points
ID: 21415083
Firmware on nic's that is.
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 24906814
from the recommendations made the asker could have resolved the issue. As he wanted to work when he can test them during a scheduled maintenace window.

This should rather be closed as answered and not deleted.
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This process describes the steps required to Import and Export data from and to .pst files using Exchange 2010. We can use these steps to export data from a user to a .pst file, import data back to the same or a different user, or even import data t…
This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…

825 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question