Link to home
Start Free TrialLog in
Avatar of Jono Martin
Jono MartinFlag for United States of America

asked on

I need to rebuild the Exchange database to get clear out corruption - Last Resort Action

Time to rebuild the mail database.  I'm hoping to do it this Saturday morning.

Background details are here: https://www.experts-exchange.com/questions/24159611/Repartition-the-Domain-Controller-Running-Exchange.html

Tigermatt mentioned in the other question...

"How many users are running on this server? It's a drastic move, but you might want to consider using Exmerge to export mailboxes to PST files, then delete your mail database and recreate it before re-importing the mailboxes.

"That would give you a clean mail store which will not have any corruption whatsoever.

"Alternatively, if you are running Exchange 2003 Enterprise Edition, you could create a second mailbox store and move mailboxes over, one by one."

I am running Exchange Server 2003 with 116 mailboxes.  I have a couple of questions on the above comment.  If I create a 2nd mailbox store and move mailboxes over one by one, will that remove corruption?  That option sounds safer as I won't need to delete anything until it's all over.  Can you give me some more information on what's involved in that?  Since the backups aren't working completely, I can take the stores offline and back up the .ebd files manually, by copying them to another location.

Thanks for your help!

ASKER CERTIFIED SOLUTION
Avatar of Justin Owens
Justin Owens
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Avatar of tigermatt
tigermatt
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Jono Martin

ASKER

Ok.  Sounds simple enough.  I just created the new store.  I am keeping the default public store since there's no corruption there.

Questions:
1. Will the old and new mailbox stores need to be dismounted before I can move the mailboxes?
2. Is it as simple as choosing to move the mailbox stores in the Exchange Tasks and that's it, or is there something else I need to be concerned about afterward?
3. Due to the partition reconfiguration in the other ticket, I now have enough room to do this, but room may become a little bit short (though not exhausted).  At what point should I feel comfortable deleting the corrupted mail store?  Once this is done (this Saturday), should I keep the old one around for a while as part of a backout plan?
4. Will Outlook on the client computers automatically pull up the mailboxes in the new store, or will each client need to be reconfigured?  The mail server should tell the client where to get the mailboxes, right?
5. Are there any concerns regarding remote users using...
     a. OWA
     b. Outlook on a VPN connection
6. Any other items that a nervous man like myself needs to know?

Thank you both for your assistance!

Jono

>>Will the old and new mailbox stores need to be dismounted before I can move the mailboxes?

Not at all. In fact, it's quite the opposite; the stores need to be mounted for the operation to be successful.

>> Is it as simple as choosing to move the mailbox stores in the Exchange Tasks and that's it, or is there something else I need to be concerned about afterward?

It's that simple. Simply choose to Move Mailbox and specify the new database as the destination. Ideally, do it at night or over a weekend or something, so traffic on the server is at its lowest.

>> Due to the partition reconfiguration in the other ticket, I now have enough room to do this, but room may become a little bit short (though not exhausted).  At what point should I feel comfortable deleting the corrupted mail store?  Once this is done (this Saturday), should I keep the old one around for a while as part of a backout plan?

You could do if you wished. After the move operation completes, I'd dismount the old store. If you work for a week or more with the store dismounted (if it's dismounted, it won't be accessible) it would be safe to go ahead and delete it. Of course, it would be a good idea to have a backup (try taking an Exchange-aware one with NTBackup). But even if it's just of the raw EDB/STM files, it won't be ideal, but it would be sufficient.

>> Will Outlook on the client computers automatically pull up the mailboxes in the new store, or will each client need to be reconfigured?  The mail server should tell the client where to get the mailboxes, right?

Outlook just looks at the mailbox server. It's then the server's job to match user to their mailbox, and locate the mail store in which it's located. Therefore, no client-side changes are necessary.

>> Are there any concerns regarding remote users using...

When each user's individual mailbox is moved, they may be temporarily kicked off and may have to initiate their connection again. This will only happen when their particular mailbox is moved, so disruption to each user should last no longer than 5 - 10 minutes for an average sized mailbox.

>> Any other items that a nervous man like myself needs to know?

There will be a lot of transaction logs generated as a result of data being removed from one store and added to the other. If it's going to be an issue on disk space, I'd enable circular logging before hand, to keep the transaction logs down to 20MB for the duration of the move.

-Matt
Just remember to turn circular logging off once you have cleared up the space issue.
Ok.  It took quite a while, but I have finally finished this.  There were a couple of errors, but they are resolved now.

One thing that I didn't expect is that the new database is larger than the old one by about 8GB.  Why would that be?  I was expecting it to be a little bit smaller or, at the most, the same size.

I will be running a backup job on it tonight (using Backup Exec) to see if the backup succeeds.  It constantly failed on the old one.  I'll post the results here.

Jono

Depending on how you migrated the mailboxes to the new database, you may have lost Single Instance Storage in some or all cases. That would be a plausible explanation for the database size growing.

-Matt
Well, the backup using Veritas Backup Exec failed (timed out) after 22 hours--the limit I set for the backup job--and only backed up about 3½GB of the 35GB database.  I am backing it up using NT backup now, going to a network drive--I have the space on another server.  I will then use Veritas to backup that NT backup file.  I have a weekly job in Veritas to do a full backup, not including the mailboxes; it always succeeds.

If that works without issue, I'll set the schedules and have it done that way from now on.

Matt - you mentioned Single Instance Storage.  Is that for attachments?  How do I check that?

The big test will be tomorrow when people are accessing their mail.  I'd like to keep this case open for a couple of days.  I'll post all results here.

Thanks!
Jono
The NT Backup job backed up the entire database in just under 2 hours.  I'll have NT Backup backup the Exchange store and have Veritas backup everything else.  Anything I should be aware of when using NT Backup?

Veritas used to give me a report showing corrupted items in the mailboxes.  Does NT Backup have any such feature?  I didn't see anything besides full reporting that details every item that's backed up, successful or not.

Also, I was told that with NT Backup, there's no way to restore single items at a time.  Do you know if this is true?  If it is, do you know of a utility that will do this for me from an NT Backup file?

So far, there have been no reports from users about mail issues.

Thanks again guys!
Jono

Provided you are backing up the Information Store with NTBackup, there is virtually no difference between that and Backup Exec.

Backup Exec has a feature known as Granular Restore Technology (GRT) which allows you to restore individual items. NTBackup does not have this technology. You could perform a brick-level backup, but these are old, slow and unreliable. Instead, you'd be looking to do a full Information Store backup, then use a Recovery Storage Group to recover items if necessary.

Single Instance Storage is essentially a technology in the Exchange database whereby an email message sent to multiple users is stored only once, and a reference to that message simply added to each recipient's mailbox. The move operation can break Single Instance Storage, which may mean you now have duplicates of larger messages in the databases.

Exchange 2003 and lower has SIS for ALL messages, so a large email can be subject to SIS; Exchange 2007 recognises that most larger emails will have some form of attachment, so will store multiple copies of an email message and only perform SIS on any attachments.

-Matt
The procedure was successful!  Thanks to both of you for your assistance.  The only thing outstanding is that I'm getting an error in Task Scheduler (exit code: 1f) when trying to run NTBackup on the Info Store.  I can run the job manually without issue, but the scheduler fails.  I'll search EE for an answer to that and post another question if necessary.

Thanks again!
Jono
Quick question for anyone still monitoring this...

I tried to delete the old store in Exchange Manager and I get this error message:
"One or more users currently use this mailbox store.  These users must be moved to a different mailbox store or be mail disabled before deleting this store."

I mounted the store and then checked in the Mailboxes folder under the store and it's empty, as is the Logons folder.

Any idea how else I can check for mailboxes in this store?  Or how I can delete it to reclaim the space?

Thanks,
Jono
Correction:

I ran a refresh on the mailboxes and logons folders and now it's showing things.

In Logons I have:
3 SMTP accounts
2 System Mailbox accounts
1 System Attendant account

In Mailboxes I have:
10 mailboxes with red "x"s over them.
1 SMTP mailbox
1 System Attendant mailbox
1 SystemMailbox mailbox

I tried to purge one of the user mailboxes and it said that it cannot be purged because it's been reconnected to another user.
 
Any help?  I'd like to delete this store to reclaim the disk space.

Thanks,
Jono
In fairness to the experts here, I have started a new (related) question for this.
https://www.experts-exchange.com/questions/24359103/How-do-I-delete-a-corrupted-Mailbox-Store.html