dav-i-son
asked on
Exchange 2003 Info Store Backup Slow
Was on Exchange 2003SP2 Enterprise on Server 1. Saw backup speeds of over 2GB/min. Server needed to be replaced.
Server 2 enters the scene. Pretty beefy, this time has SAS drives instead of SCSI. Go from an IBM x236 to an IBM x3650. Moved mailboxes through Exchange interface, and everything transitions over fine.
Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on. Configure backup job, just to backup the information store (not doing brick level). Seeing speeds of 160MB/min. On the Information Store. Both server 1 and 2 were/are connected to the same network segment, same switch as backup server. All 3 are/were connected via Gigabit ports. End result, backup window went from approx 4-5 hours to 40-50 hours. Can't even complete it in a weekend (we're talking over 300GB of email).
ArcServe will backup straight files from the server at approx 1200-1500MB/min. I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI (being very generous), but not sure why it's coming up so slowly. The disk activity lights on Server2 don't seem to indicate they're doing a ton of reading and trying to send it across the network to the backup server. Perfmon didn't seem to indicate any high number of MAPI sessions open, or high disk queue. Processor not over utilized, so it doesn't look like it's trying to process anything.
Any ideas?
Server 2 enters the scene. Pretty beefy, this time has SAS drives instead of SCSI. Go from an IBM x236 to an IBM x3650. Moved mailboxes through Exchange interface, and everything transitions over fine.
Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on. Configure backup job, just to backup the information store (not doing brick level). Seeing speeds of 160MB/min. On the Information Store. Both server 1 and 2 were/are connected to the same network segment, same switch as backup server. All 3 are/were connected via Gigabit ports. End result, backup window went from approx 4-5 hours to 40-50 hours. Can't even complete it in a weekend (we're talking over 300GB of email).
ArcServe will backup straight files from the server at approx 1200-1500MB/min. I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI (being very generous), but not sure why it's coming up so slowly. The disk activity lights on Server2 don't seem to indicate they're doing a ton of reading and trying to send it across the network to the backup server. Perfmon didn't seem to indicate any high number of MAPI sessions open, or high disk queue. Processor not over utilized, so it doesn't look like it's trying to process anything.
Any ideas?
ASKER
The test backup I ran on the files portion yielded 1398MB/min (from server2 to the backup server, which has the tape drive attached).
Both servers have 4GB of RAM, and both have McAfee Enterprise VirusScan (same version) and GroupShield for Exchange (same version).
I tried running NTBackup on the backup server to try doing Exchange on server2, but it wasn't able to see the Exchange environment. If I run NTBackup on server2, (if I did the conversion right--100GB--yea I know, completed in 4.25 hours) I get about 392MB/min (triple what ArcServe is giving). That's also including sending the backup past a firewall to a disk (my Exchange server doesn't have enough room to backup 1 storage group to the same drive that it exists on), and the only thing with enough storage space is behind a firewall. Looks like it could potentially see 500MB if it were unimpeded (but I'd still like to get back to the original speed server1 gave--near 2GB/min).
Both servers have 4GB of RAM, and both have McAfee Enterprise VirusScan (same version) and GroupShield for Exchange (same version).
I tried running NTBackup on the backup server to try doing Exchange on server2, but it wasn't able to see the Exchange environment. If I run NTBackup on server2, (if I did the conversion right--100GB--yea I know, completed in 4.25 hours) I get about 392MB/min (triple what ArcServe is giving). That's also including sending the backup past a firewall to a disk (my Exchange server doesn't have enough room to backup 1 storage group to the same drive that it exists on), and the only thing with enough storage space is behind a firewall. Looks like it could potentially see 500MB if it were unimpeded (but I'd still like to get back to the original speed server1 gave--near 2GB/min).
Hi,
one question did you say information store backup or document level backup.
"Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on. Configure backup job, just to backup the information store "
Are you seeing 160 MB when you use document level backup or database level backup. And what was the speed on the earlier server for flat files. Was it the same 1200/1500 MB. Or was it higher. What are the kind of hard disks used SCSI/SATA any raid implementation.
Because though the the network and the memory are the same these factors do add up overheads. Disable anti-virus during the backup and see what is the throughput you achieve.
Bhanu
one question did you say information store backup or document level backup.
"Install Exchange Agent for ArcServe backup, and it installs the Exchange Premium Add-on. Configure backup job, just to backup the information store "
Are you seeing 160 MB when you use document level backup or database level backup. And what was the speed on the earlier server for flat files. Was it the same 1200/1500 MB. Or was it higher. What are the kind of hard disks used SCSI/SATA any raid implementation.
Because though the the network and the memory are the same these factors do add up overheads. Disable anti-virus during the backup and see what is the throughput you achieve.
Bhanu
ASKER
Bhanu-
I'm using the Information Store backup. Wouldn't even try the document-level (we've always just done the IS backups). The 160MB/min is coming from the ArcServe backup job, through the agent, for information store.
Speed on server1 doing flat files (for some odd reason) just came up w/ 202MB/min... but it's gotten average 650MB/min in the past. Server1 has SCSI 15k drives, server2 has SAS15k drives, both in a RAID5 configuration.
I'll try anti-virus is configured to skip the folders the Exchange stores (ebd/stm files) are in, but I'll try with McAfee completely disabled. I'll post results tomorrow--it'll throw off the backup schedule if I start it now--critical items won't get backed up if I do Exchange now).
I'm using the Information Store backup. Wouldn't even try the document-level (we've always just done the IS backups). The 160MB/min is coming from the ArcServe backup job, through the agent, for information store.
Speed on server1 doing flat files (for some odd reason) just came up w/ 202MB/min... but it's gotten average 650MB/min in the past. Server1 has SCSI 15k drives, server2 has SAS15k drives, both in a RAID5 configuration.
I'll try anti-virus is configured to skip the folders the Exchange stores (ebd/stm files) are in, but I'll try with McAfee completely disabled. I'll post results tomorrow--it'll throw off the backup schedule if I start it now--critical items won't get backed up if I do Exchange now).
Hi,
"ArcServe will backup straight files from the server at approx 1200-1500MB/min. I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI "
Is this the speed on the new server for flat files. As we are currently trying to understand what is the issue with server 2 right.
If the flat file backup has slowed down on server 2 then need to check if it is running on sp2 and if that is the case check this MS article and disable tcp chimney
http://support.microsoft.com/kb/912222
bhanu
"ArcServe will backup straight files from the server at approx 1200-1500MB/min. I could reasonably understand 1000-1300MB/min due to the overhead and accessing MAPI "
Is this the speed on the new server for flat files. As we are currently trying to understand what is the issue with server 2 right.
If the flat file backup has slowed down on server 2 then need to check if it is running on sp2 and if that is the case check this MS article and disable tcp chimney
http://support.microsoft.com/kb/912222
bhanu
ASKER
Bhanu-
Yes, server2 is on 2003 SP2. Referencing that MS article, I see TCP Chimney is enabled, but not all of the entries they mention in the article appear in my registry:
EnableTCPChimney - Enabled
EnableRSS - Enabled
EnableTCPA - Enabled
OffloadExcludeDestinationP
OffloadIncludeDestinationP
OffloadExcludeSourcePorts -- Does not appear
OffloadIncludeSourcePorts -- Does not appear
MinPacketSizeToDma -- Does not appear
DmaSyncCompletionHighThres
Although it says I can turn chimney off via the netsh commands, I'd rather wait for my next maintenance window, to do this off-line. Unfortunately, I'm expecting the next window to appear around 12/13.
As a stop-gap for now, I can use NTBackup, and back those BKF files up to tape.
I'll let you know how I make out after disabling the TCP Chimney
Ken
Yes, server2 is on 2003 SP2. Referencing that MS article, I see TCP Chimney is enabled, but not all of the entries they mention in the article appear in my registry:
EnableTCPChimney - Enabled
EnableRSS - Enabled
EnableTCPA - Enabled
OffloadExcludeDestinationP
OffloadIncludeDestinationP
OffloadExcludeSourcePorts -- Does not appear
OffloadIncludeSourcePorts -- Does not appear
MinPacketSizeToDma -- Does not appear
DmaSyncCompletionHighThres
Although it says I can turn chimney off via the netsh commands, I'd rather wait for my next maintenance window, to do this off-line. Unfortunately, I'm expecting the next window to appear around 12/13.
As a stop-gap for now, I can use NTBackup, and back those BKF files up to tape.
I'll let you know how I make out after disabling the TCP Chimney
Ken
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I just solved my problems at 2 customers by upgrading firmware on both troublesome servers in the equation.
server1<->server2 slow
server1<->server3 fast
server2<->server3 fast
The I was going nuts, because they were connected to the same switch, so I couldnt figure it out.
server1<->server2 slow
server1<->server3 fast
server2<->server3 fast
The I was going nuts, because they were connected to the same switch, so I couldnt figure it out.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
from the recommendations made the asker could have resolved the issue. As he wanted to work when he can test them during a scheduled maintenace window.
This should rather be closed as answered and not deleted.
This should rather be closed as answered and not deleted.
What was the backup speed of flat files from the other server earlier. Any change in the memory allocation or any noticable changes from server 1 to server 2. Any anti-virus installed on this new server. Normally we did not see that kind of speed (low) for a exchange info store backup. normally if the flat file backups are around 1500 mb then the info store backup would be around 700/900 mb.
Check for the network card drivers on both the servers. upgrade the network card drivers to the latest.
Try to run a ntbackup of exchange info store and see what is the througput you are able to achive.
This article would help up run ntbackup from the backup server.
http://www.msexchange.org/tutorials/Exchange-2003-Backup-Restore-NTBACKUP.html
Stop all arcserve services and enable Removable storage service and also the tape drives and the medium changer. To run the backups to the tape library.
See if you are getting the same throughput.
bhanu