Solved

Exchange 2003 Info Store Backup Slow

Posted on 2007-11-20
12
2,736 Views
Last Modified: 2009-07-21
Was on Exchange 2003SP2 Enterprise on Server 1.  Saw backup speeds of over 2GB/min.  Server needed to be replaced.  

Server 2 enters the scene.  Pretty beefy, this time has SAS drives instead of SCSI.  Go from an IBM x236 to an IBM x3650.  Moved mailboxes through Exchange interface, and everything transitions over fine.  

Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store (not doing brick level).  Seeing speeds of 160MB/min.  On the Information Store.  Both server 1 and 2 were/are connected to the same network segment, same switch as backup server.  All 3 are/were connected via Gigabit ports.  End result, backup window went from approx 4-5 hours to 40-50 hours.  Can't even complete it in a weekend (we're talking over 300GB of email).

ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI (being very generous), but not sure why it's coming up so slowly.  The disk activity lights on Server2 don't seem to indicate they're doing a ton of reading and trying to send it across the network to the backup server.  Perfmon didn't seem to indicate any high number of MAPI sessions open, or high disk queue.  Processor not over utilized, so it doesn't look like it's trying to process anything.

Any ideas?
0
Comment
Question by:dav-i-son
  • 5
  • 3
  • 2
12 Comments
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20323806
Hi,

What was the backup speed of flat files from the other server earlier. Any change in the memory allocation or any noticable changes from server 1 to server 2. Any anti-virus installed on this new server. Normally we did not see that kind of speed (low) for a exchange info store backup. normally if the flat file backups are around 1500 mb then the info store backup would be around 700/900 mb.

Check for the network card drivers on both the servers. upgrade the network card drivers to the latest.
Try to run a ntbackup of exchange info store and see what is the througput you are able to achive.
This article would help up run ntbackup from the backup server.

http://www.msexchange.org/tutorials/Exchange-2003-Backup-Restore-NTBACKUP.html

Stop all arcserve services and enable Removable storage service and also the tape drives and the medium changer. To run the backups to the tape library.

See if you are getting the same throughput.

bhanu
0
 

Author Comment

by:dav-i-son
ID: 20327120
The test backup I ran on the files portion yielded 1398MB/min (from server2 to the backup server, which has the tape drive attached).

Both servers have 4GB of RAM, and both have McAfee Enterprise VirusScan (same version) and GroupShield for Exchange (same version).

I tried running NTBackup on the backup server to try doing Exchange on server2, but it wasn't able to see the Exchange environment.  If I run NTBackup on server2, (if I did the conversion right--100GB--yea I know, completed in 4.25 hours) I get about 392MB/min (triple what ArcServe is giving).  That's also including sending the backup past a firewall to a disk (my Exchange server doesn't have enough room to backup 1 storage group to the same drive that it exists on), and the only thing with enough storage space is behind a firewall.  Looks like it could potentially see 500MB if it were unimpeded (but I'd still like to get back to the original speed server1 gave--near 2GB/min).
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20331799
Hi,

one question did you say information store backup or document level backup.

"Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store "

Are you seeing 160 MB when you use document level backup or database level backup. And what was the speed on the earlier server for flat files. Was it the same 1200/1500 MB. Or was it higher. What are the kind of hard disks used SCSI/SATA any raid implementation.

Because though the the network and the memory are the same these factors do add up overheads. Disable anti-virus during the backup and see what is the throughput you achieve.

Bhanu
0
 

Author Comment

by:dav-i-son
ID: 20351676
Bhanu-  

I'm using the Information Store backup.  Wouldn't even try the document-level (we've always just done the IS backups).  The 160MB/min is coming from the ArcServe backup job, through the agent, for information store.  

Speed on server1 doing flat files (for some odd reason) just came up w/ 202MB/min... but it's gotten average 650MB/min in the past.  Server1 has SCSI 15k drives, server2 has SAS15k drives, both in a RAID5 configuration.  

I'll try anti-virus is configured to skip the folders the Exchange stores (ebd/stm files) are in, but I'll try with McAfee completely disabled.  I'll post results tomorrow--it'll throw off the backup schedule if I start it now--critical items won't get backed up if I do Exchange now).
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20355398
Hi,

"ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI "

Is this the speed on the new server for flat files. As we are currently trying to understand what is the issue with server 2 right.

If the flat file backup has slowed down on server 2 then need to check if it is running on sp2 and if that is the case check this MS article and disable tcp chimney

http://support.microsoft.com/kb/912222

bhanu
0
Shouldn't all users have the same email signature?

You wouldn't let your users design their own business cards, would you? So, why do you let them design their own email signatures? Think of the damage they could be doing to your brand reputation! Choose the easy way to manage set up and add email signatures for all users.

 

Author Comment

by:dav-i-son
ID: 20357411
Bhanu-

Yes, server2 is on 2003 SP2.  Referencing that MS article, I see TCP Chimney is enabled, but not all of the entries they mention in the article appear in my registry:

EnableTCPChimney -   Enabled
EnableRSS              -   Enabled
EnableTCPA            -  Enabled
OffloadExcludeDestinationPorts  -- Does not appear
OffloadIncludeDestinationPorts  --  Does not appear
OffloadExcludeSourcePorts       -- Does not appear
OffloadIncludeSourcePorts        --  Does not appear
MinPacketSizeToDma                 --  Does not appear
DmaSyncCompletionHighThreshold  -- Does not appear

Although it says I can turn chimney off via the netsh commands, I'd rather wait for my next maintenance window, to do this off-line.  Unfortunately, I'm expecting the next window to appear around 12/13.

As a stop-gap for now, I can use NTBackup, and back those BKF files up to tape.

I'll let you know how I make out after disabling the TCP Chimney

Ken
0
 
LVL 23

Accepted Solution

by:
bhanukir7 earned 150 total points
ID: 20362348
Hi Dav,

Disabling tcp chimney would not require a reboot. Though its always good.
You can disable tcp chimney and then try to run a test backup.

A good way of testing would be to

Try to identify around 1 gb of data:

1. Run a windows copy from that exchange server to the backup server.
2. Run a copy job using ARCserve for the same set of data
3. Create a backup job using the client agent and run the backup to FSD
4. Modify the same job and run it to a tape drive.

Notice the amount of time it is taking for this data to be copied, backedup.
This would give a clear idea if the issue is with network or the software.

Run ntbackup on the exchange server See the throughput.
Run NTbackup from the backup server refer to that link i have provided in the earlier post.

This would give you a clear picture of exchange backup when it is local and remote with ntbackup.

You can try this before disabling the tcpchimney and then try the same set of tests after disabling the TCP chimney.

Bhanu
0
 
LVL 2

Expert Comment

by:mathiesen-data
ID: 21415074
I just solved my problems at 2 customers by upgrading firmware on both troublesome servers in the equation.
server1<->server2 slow
server1<->server3 fast
server2<->server3 fast

The I was going nuts, because they were connected to the same switch, so I couldnt figure it out.
0
 
LVL 2

Assisted Solution

by:mathiesen-data
mathiesen-data earned 150 total points
ID: 21415083
Firmware on nic's that is.
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 24906814
from the recommendations made the asker could have resolved the issue. As he wanted to work when he can test them during a scheduled maintenace window.

This should rather be closed as answered and not deleted.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Resolve DNS query failed errors for Exchange
Scam emails are a huge burden for many businesses. Spotting one is not always easy. Follow our tips to identify if an email you receive is a scam.
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now