Solved

Exchange 2003 Info Store Backup Slow

Posted on 2007-11-20
12
2,745 Views
Last Modified: 2009-07-21
Was on Exchange 2003SP2 Enterprise on Server 1.  Saw backup speeds of over 2GB/min.  Server needed to be replaced.  

Server 2 enters the scene.  Pretty beefy, this time has SAS drives instead of SCSI.  Go from an IBM x236 to an IBM x3650.  Moved mailboxes through Exchange interface, and everything transitions over fine.  

Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store (not doing brick level).  Seeing speeds of 160MB/min.  On the Information Store.  Both server 1 and 2 were/are connected to the same network segment, same switch as backup server.  All 3 are/were connected via Gigabit ports.  End result, backup window went from approx 4-5 hours to 40-50 hours.  Can't even complete it in a weekend (we're talking over 300GB of email).

ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI (being very generous), but not sure why it's coming up so slowly.  The disk activity lights on Server2 don't seem to indicate they're doing a ton of reading and trying to send it across the network to the backup server.  Perfmon didn't seem to indicate any high number of MAPI sessions open, or high disk queue.  Processor not over utilized, so it doesn't look like it's trying to process anything.

Any ideas?
0
Comment
Question by:dav-i-son
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 3
  • 2
12 Comments
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20323806
Hi,

What was the backup speed of flat files from the other server earlier. Any change in the memory allocation or any noticable changes from server 1 to server 2. Any anti-virus installed on this new server. Normally we did not see that kind of speed (low) for a exchange info store backup. normally if the flat file backups are around 1500 mb then the info store backup would be around 700/900 mb.

Check for the network card drivers on both the servers. upgrade the network card drivers to the latest.
Try to run a ntbackup of exchange info store and see what is the througput you are able to achive.
This article would help up run ntbackup from the backup server.

http://www.msexchange.org/tutorials/Exchange-2003-Backup-Restore-NTBACKUP.html

Stop all arcserve services and enable Removable storage service and also the tape drives and the medium changer. To run the backups to the tape library.

See if you are getting the same throughput.

bhanu
0
 

Author Comment

by:dav-i-son
ID: 20327120
The test backup I ran on the files portion yielded 1398MB/min (from server2 to the backup server, which has the tape drive attached).

Both servers have 4GB of RAM, and both have McAfee Enterprise VirusScan (same version) and GroupShield for Exchange (same version).

I tried running NTBackup on the backup server to try doing Exchange on server2, but it wasn't able to see the Exchange environment.  If I run NTBackup on server2, (if I did the conversion right--100GB--yea I know, completed in 4.25 hours) I get about 392MB/min (triple what ArcServe is giving).  That's also including sending the backup past a firewall to a disk (my Exchange server doesn't have enough room to backup 1 storage group to the same drive that it exists on), and the only thing with enough storage space is behind a firewall.  Looks like it could potentially see 500MB if it were unimpeded (but I'd still like to get back to the original speed server1 gave--near 2GB/min).
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20331799
Hi,

one question did you say information store backup or document level backup.

"Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store "

Are you seeing 160 MB when you use document level backup or database level backup. And what was the speed on the earlier server for flat files. Was it the same 1200/1500 MB. Or was it higher. What are the kind of hard disks used SCSI/SATA any raid implementation.

Because though the the network and the memory are the same these factors do add up overheads. Disable anti-virus during the backup and see what is the throughput you achieve.

Bhanu
0
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

 

Author Comment

by:dav-i-son
ID: 20351676
Bhanu-  

I'm using the Information Store backup.  Wouldn't even try the document-level (we've always just done the IS backups).  The 160MB/min is coming from the ArcServe backup job, through the agent, for information store.  

Speed on server1 doing flat files (for some odd reason) just came up w/ 202MB/min... but it's gotten average 650MB/min in the past.  Server1 has SCSI 15k drives, server2 has SAS15k drives, both in a RAID5 configuration.  

I'll try anti-virus is configured to skip the folders the Exchange stores (ebd/stm files) are in, but I'll try with McAfee completely disabled.  I'll post results tomorrow--it'll throw off the backup schedule if I start it now--critical items won't get backed up if I do Exchange now).
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 20355398
Hi,

"ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI "

Is this the speed on the new server for flat files. As we are currently trying to understand what is the issue with server 2 right.

If the flat file backup has slowed down on server 2 then need to check if it is running on sp2 and if that is the case check this MS article and disable tcp chimney

http://support.microsoft.com/kb/912222

bhanu
0
 

Author Comment

by:dav-i-son
ID: 20357411
Bhanu-

Yes, server2 is on 2003 SP2.  Referencing that MS article, I see TCP Chimney is enabled, but not all of the entries they mention in the article appear in my registry:

EnableTCPChimney -   Enabled
EnableRSS              -   Enabled
EnableTCPA            -  Enabled
OffloadExcludeDestinationPorts  -- Does not appear
OffloadIncludeDestinationPorts  --  Does not appear
OffloadExcludeSourcePorts       -- Does not appear
OffloadIncludeSourcePorts        --  Does not appear
MinPacketSizeToDma                 --  Does not appear
DmaSyncCompletionHighThreshold  -- Does not appear

Although it says I can turn chimney off via the netsh commands, I'd rather wait for my next maintenance window, to do this off-line.  Unfortunately, I'm expecting the next window to appear around 12/13.

As a stop-gap for now, I can use NTBackup, and back those BKF files up to tape.

I'll let you know how I make out after disabling the TCP Chimney

Ken
0
 
LVL 23

Accepted Solution

by:
bhanukir7 earned 150 total points
ID: 20362348
Hi Dav,

Disabling tcp chimney would not require a reboot. Though its always good.
You can disable tcp chimney and then try to run a test backup.

A good way of testing would be to

Try to identify around 1 gb of data:

1. Run a windows copy from that exchange server to the backup server.
2. Run a copy job using ARCserve for the same set of data
3. Create a backup job using the client agent and run the backup to FSD
4. Modify the same job and run it to a tape drive.

Notice the amount of time it is taking for this data to be copied, backedup.
This would give a clear idea if the issue is with network or the software.

Run ntbackup on the exchange server See the throughput.
Run NTbackup from the backup server refer to that link i have provided in the earlier post.

This would give you a clear picture of exchange backup when it is local and remote with ntbackup.

You can try this before disabling the tcpchimney and then try the same set of tests after disabling the TCP chimney.

Bhanu
0
 
LVL 2

Expert Comment

by:mathiesen-data
ID: 21415074
I just solved my problems at 2 customers by upgrading firmware on both troublesome servers in the equation.
server1<->server2 slow
server1<->server3 fast
server2<->server3 fast

The I was going nuts, because they were connected to the same switch, so I couldnt figure it out.
0
 
LVL 2

Assisted Solution

by:mathiesen-data
mathiesen-data earned 150 total points
ID: 21415083
Firmware on nic's that is.
0
 
LVL 23

Expert Comment

by:bhanukir7
ID: 24906814
from the recommendations made the asker could have resolved the issue. As he wanted to work when he can test them during a scheduled maintenace window.

This should rather be closed as answered and not deleted.
0

Featured Post

Are your AD admin tools letting you down?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

MS Outlook is a world-class email client application that is mainly used for e-communication globally.  In this article, we will discuss the basic idea about MS Outlook, its advanced features, and types of MS Outlook File formats.
Microsoft will be releasing the Windows 10 Creators Update in just a matter of weeks. Are you prepared? Follow these steps to ensure everything goes smoothly and you don't lose valuable data on your PC.
This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…
The basic steps you have just learned will be implemented in this video. The basic steps are shown to configure an Exchange DAG in a live working Exchange Server Environment and manage the same (Exchange Server 2010 Software is used in a Windows Ser…

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question