• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2755
  • Last Modified:

Exchange 2003 Info Store Backup Slow

Was on Exchange 2003SP2 Enterprise on Server 1.  Saw backup speeds of over 2GB/min.  Server needed to be replaced.  

Server 2 enters the scene.  Pretty beefy, this time has SAS drives instead of SCSI.  Go from an IBM x236 to an IBM x3650.  Moved mailboxes through Exchange interface, and everything transitions over fine.  

Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store (not doing brick level).  Seeing speeds of 160MB/min.  On the Information Store.  Both server 1 and 2 were/are connected to the same network segment, same switch as backup server.  All 3 are/were connected via Gigabit ports.  End result, backup window went from approx 4-5 hours to 40-50 hours.  Can't even complete it in a weekend (we're talking over 300GB of email).

ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI (being very generous), but not sure why it's coming up so slowly.  The disk activity lights on Server2 don't seem to indicate they're doing a ton of reading and trying to send it across the network to the backup server.  Perfmon didn't seem to indicate any high number of MAPI sessions open, or high disk queue.  Processor not over utilized, so it doesn't look like it's trying to process anything.

Any ideas?
0
dav-i-son
Asked:
dav-i-son
  • 5
  • 3
  • 2
2 Solutions
 
bhanukir7Commented:
Hi,

What was the backup speed of flat files from the other server earlier. Any change in the memory allocation or any noticable changes from server 1 to server 2. Any anti-virus installed on this new server. Normally we did not see that kind of speed (low) for a exchange info store backup. normally if the flat file backups are around 1500 mb then the info store backup would be around 700/900 mb.

Check for the network card drivers on both the servers. upgrade the network card drivers to the latest.
Try to run a ntbackup of exchange info store and see what is the througput you are able to achive.
This article would help up run ntbackup from the backup server.

http://www.msexchange.org/tutorials/Exchange-2003-Backup-Restore-NTBACKUP.html

Stop all arcserve services and enable Removable storage service and also the tape drives and the medium changer. To run the backups to the tape library.

See if you are getting the same throughput.

bhanu
0
 
dav-i-sonAuthor Commented:
The test backup I ran on the files portion yielded 1398MB/min (from server2 to the backup server, which has the tape drive attached).

Both servers have 4GB of RAM, and both have McAfee Enterprise VirusScan (same version) and GroupShield for Exchange (same version).

I tried running NTBackup on the backup server to try doing Exchange on server2, but it wasn't able to see the Exchange environment.  If I run NTBackup on server2, (if I did the conversion right--100GB--yea I know, completed in 4.25 hours) I get about 392MB/min (triple what ArcServe is giving).  That's also including sending the backup past a firewall to a disk (my Exchange server doesn't have enough room to backup 1 storage group to the same drive that it exists on), and the only thing with enough storage space is behind a firewall.  Looks like it could potentially see 500MB if it were unimpeded (but I'd still like to get back to the original speed server1 gave--near 2GB/min).
0
 
bhanukir7Commented:
Hi,

one question did you say information store backup or document level backup.

"Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on.  Configure backup job, just to backup the information store "

Are you seeing 160 MB when you use document level backup or database level backup. And what was the speed on the earlier server for flat files. Was it the same 1200/1500 MB. Or was it higher. What are the kind of hard disks used SCSI/SATA any raid implementation.

Because though the the network and the memory are the same these factors do add up overheads. Disable anti-virus during the backup and see what is the throughput you achieve.

Bhanu
0
Nothing ever in the clear!

This technical paper will help you implement VMware’s VM encryption as well as implement Veeam encryption which together will achieve the nothing ever in the clear goal. If a bad guy steals VMs, backups or traffic they get nothing.

 
dav-i-sonAuthor Commented:
Bhanu-  

I'm using the Information Store backup.  Wouldn't even try the document-level (we've always just done the IS backups).  The 160MB/min is coming from the ArcServe backup job, through the agent, for information store.  

Speed on server1 doing flat files (for some odd reason) just came up w/ 202MB/min... but it's gotten average 650MB/min in the past.  Server1 has SCSI 15k drives, server2 has SAS15k drives, both in a RAID5 configuration.  

I'll try anti-virus is configured to skip the folders the Exchange stores (ebd/stm files) are in, but I'll try with McAfee completely disabled.  I'll post results tomorrow--it'll throw off the backup schedule if I start it now--critical items won't get backed up if I do Exchange now).
0
 
bhanukir7Commented:
Hi,

"ArcServe will backup straight files from the server at approx 1200-1500MB/min.  I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI "

Is this the speed on the new server for flat files. As we are currently trying to understand what is the issue with server 2 right.

If the flat file backup has slowed down on server 2 then need to check if it is running on sp2 and if that is the case check this MS article and disable tcp chimney

http://support.microsoft.com/kb/912222

bhanu
0
 
dav-i-sonAuthor Commented:
Bhanu-

Yes, server2 is on 2003 SP2.  Referencing that MS article, I see TCP Chimney is enabled, but not all of the entries they mention in the article appear in my registry:

EnableTCPChimney -   Enabled
EnableRSS              -   Enabled
EnableTCPA            -  Enabled
OffloadExcludeDestinationPorts  -- Does not appear
OffloadIncludeDestinationPorts  --  Does not appear
OffloadExcludeSourcePorts       -- Does not appear
OffloadIncludeSourcePorts        --  Does not appear
MinPacketSizeToDma                 --  Does not appear
DmaSyncCompletionHighThreshold  -- Does not appear

Although it says I can turn chimney off via the netsh commands, I'd rather wait for my next maintenance window, to do this off-line.  Unfortunately, I'm expecting the next window to appear around 12/13.

As a stop-gap for now, I can use NTBackup, and back those BKF files up to tape.

I'll let you know how I make out after disabling the TCP Chimney

Ken
0
 
bhanukir7Commented:
Hi Dav,

Disabling tcp chimney would not require a reboot. Though its always good.
You can disable tcp chimney and then try to run a test backup.

A good way of testing would be to

Try to identify around 1 gb of data:

1. Run a windows copy from that exchange server to the backup server.
2. Run a copy job using ARCserve for the same set of data
3. Create a backup job using the client agent and run the backup to FSD
4. Modify the same job and run it to a tape drive.

Notice the amount of time it is taking for this data to be copied, backedup.
This would give a clear idea if the issue is with network or the software.

Run ntbackup on the exchange server See the throughput.
Run NTbackup from the backup server refer to that link i have provided in the earlier post.

This would give you a clear picture of exchange backup when it is local and remote with ntbackup.

You can try this before disabling the tcpchimney and then try the same set of tests after disabling the TCP chimney.

Bhanu
0
 
mathiesen-dataCommented:
I just solved my problems at 2 customers by upgrading firmware on both troublesome servers in the equation.
server1<->server2 slow
server1<->server3 fast
server2<->server3 fast

The I was going nuts, because they were connected to the same switch, so I couldnt figure it out.
0
 
mathiesen-dataCommented:
Firmware on nic's that is.
0
 
bhanukir7Commented:
from the recommendations made the asker could have resolved the issue. As he wanted to work when he can test them during a scheduled maintenace window.

This should rather be closed as answered and not deleted.
0

Featured Post

Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

  • 5
  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now