Solved

I need to rebuild the Exchange database to get clear out corruption - Last Resort Action

Posted on 2009-04-13
14
507 Views
Last Modified: 2012-05-06
Time to rebuild the mail database.  I'm hoping to do it this Saturday morning.

Background details are here: http://www.experts-exchange.com/Storage/Misc/Q_24159611.html

Tigermatt mentioned in the other question...

"How many users are running on this server? It's a drastic move, but you might want to consider using Exmerge to export mailboxes to PST files, then delete your mail database and recreate it before re-importing the mailboxes.

"That would give you a clean mail store which will not have any corruption whatsoever.

"Alternatively, if you are running Exchange 2003 Enterprise Edition, you could create a second mailbox store and move mailboxes over, one by one."

I am running Exchange Server 2003 with 116 mailboxes.  I have a couple of questions on the above comment.  If I create a 2nd mailbox store and move mailboxes over one by one, will that remove corruption?  That option sounds safer as I won't need to delete anything until it's all over.  Can you give me some more information on what's involved in that?  Since the backups aren't working completely, I can take the stores offline and back up the .ebd files manually, by copying them to another location.

Thanks for your help!

0
Comment
Question by:Jono Martin
  • 8
  • 4
  • 2
14 Comments
 
LVL 31

Accepted Solution

by:
DrUltima earned 250 total points
ID: 24132158
Since space seems to be an issue for you, I will let you know that your original EDB file will NOT shrink as users are moved out of it, but the new one WILL grow as you move users into it.  The only way to reduce the size of an EDB is an offline backup, which was already discussed in your previous issue.

Moving the users is done from either the Exchange Management Console or the Active Directory Users and Computer (from a machine that has the Exchange Management features installed).  

1) Right click on the user from ADUaC or mailbox on EMC.  
2) Select Exchange Tasks.  
3) Select Move Mailbox.
4) Select your desired Server/Message Store.

Make SURE your target drive (where the new EDB resides) is:

1) (Ideally) In the permanent new location (easier recovery if log files have to be replayed).
2) Has enough space to accommodate the EDB and whatever growth it will have.
0
 
LVL 58

Assisted Solution

by:tigermatt
tigermatt earned 250 total points
ID: 24132451

Whether or not you can follow the approach of creating a new store and migrating across depends on whether you are running Exchange Server 2003 Enterprise Edition. Your question doesn't specifically say so, but remember that Exchange 2003 Enterprise is not the same as Windows Server 2003 Enterprise Edition. They are two different platforms and two different terms, so don't confuse them!

Assuming you are running Enterprise Edition, the process of moving mailboxes to a brand-new mailbox store will eliminate any corruption in the correct store. The move procedure has the option to 'Skip' over any items which are corrupted; in other words, leaving them behind and logging this fact for you to follow-up on later.

If you only run Exchange Standard Edition, you have two options. Either the approach I described previously where you export the mailboxes, delete the store then re-create and re-import. Alternatively, you would need to use the appropriate eseutil tools to repair the store. This is a difficult and potentially dangerous procedure which I'd only attempt as a last resort.

The previous poster is correct in stating that store size does NOT reduce when items are permanently deleted from the store. Exchange store maintenance simply marks those deleted database pages as "white space"; the disk space remains allocated to the Exchange store and is used for new items entering the store.

-Matt

-Matt
0
 

Author Comment

by:Jono Martin
ID: 24140859
Ok.  Sounds simple enough.  I just created the new store.  I am keeping the default public store since there's no corruption there.

Questions:
1. Will the old and new mailbox stores need to be dismounted before I can move the mailboxes?
2. Is it as simple as choosing to move the mailbox stores in the Exchange Tasks and that's it, or is there something else I need to be concerned about afterward?
3. Due to the partition reconfiguration in the other ticket, I now have enough room to do this, but room may become a little bit short (though not exhausted).  At what point should I feel comfortable deleting the corrupted mail store?  Once this is done (this Saturday), should I keep the old one around for a while as part of a backout plan?
4. Will Outlook on the client computers automatically pull up the mailboxes in the new store, or will each client need to be reconfigured?  The mail server should tell the client where to get the mailboxes, right?
5. Are there any concerns regarding remote users using...
     a. OWA
     b. Outlook on a VPN connection
6. Any other items that a nervous man like myself needs to know?

Thank you both for your assistance!

Jono
0
 
LVL 58

Expert Comment

by:tigermatt
ID: 24140926

>>Will the old and new mailbox stores need to be dismounted before I can move the mailboxes?

Not at all. In fact, it's quite the opposite; the stores need to be mounted for the operation to be successful.

>> Is it as simple as choosing to move the mailbox stores in the Exchange Tasks and that's it, or is there something else I need to be concerned about afterward?

It's that simple. Simply choose to Move Mailbox and specify the new database as the destination. Ideally, do it at night or over a weekend or something, so traffic on the server is at its lowest.

>> Due to the partition reconfiguration in the other ticket, I now have enough room to do this, but room may become a little bit short (though not exhausted).  At what point should I feel comfortable deleting the corrupted mail store?  Once this is done (this Saturday), should I keep the old one around for a while as part of a backout plan?

You could do if you wished. After the move operation completes, I'd dismount the old store. If you work for a week or more with the store dismounted (if it's dismounted, it won't be accessible) it would be safe to go ahead and delete it. Of course, it would be a good idea to have a backup (try taking an Exchange-aware one with NTBackup). But even if it's just of the raw EDB/STM files, it won't be ideal, but it would be sufficient.

>> Will Outlook on the client computers automatically pull up the mailboxes in the new store, or will each client need to be reconfigured?  The mail server should tell the client where to get the mailboxes, right?

Outlook just looks at the mailbox server. It's then the server's job to match user to their mailbox, and locate the mail store in which it's located. Therefore, no client-side changes are necessary.

>> Are there any concerns regarding remote users using...

When each user's individual mailbox is moved, they may be temporarily kicked off and may have to initiate their connection again. This will only happen when their particular mailbox is moved, so disruption to each user should last no longer than 5 - 10 minutes for an average sized mailbox.

>> Any other items that a nervous man like myself needs to know?

There will be a lot of transaction logs generated as a result of data being removed from one store and added to the other. If it's going to be an issue on disk space, I'd enable circular logging before hand, to keep the transaction logs down to 20MB for the duration of the move.

-Matt
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 24143781
Just remember to turn circular logging off once you have cleared up the space issue.
0
 

Author Comment

by:Jono Martin
ID: 24176938
Ok.  It took quite a while, but I have finally finished this.  There were a couple of errors, but they are resolved now.

One thing that I didn't expect is that the new database is larger than the old one by about 8GB.  Why would that be?  I was expecting it to be a little bit smaller or, at the most, the same size.

I will be running a backup job on it tonight (using Backup Exec) to see if the backup succeeds.  It constantly failed on the old one.  I'll post the results here.

Jono
0
 
LVL 58

Expert Comment

by:tigermatt
ID: 24178347

Depending on how you migrated the mailboxes to the new database, you may have lost Single Instance Storage in some or all cases. That would be a plausible explanation for the database size growing.

-Matt
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 

Author Comment

by:Jono Martin
ID: 24181005
Well, the backup using Veritas Backup Exec failed (timed out) after 22 hours--the limit I set for the backup job--and only backed up about 3½GB of the 35GB database.  I am backing it up using NT backup now, going to a network drive--I have the space on another server.  I will then use Veritas to backup that NT backup file.  I have a weekly job in Veritas to do a full backup, not including the mailboxes; it always succeeds.

If that works without issue, I'll set the schedules and have it done that way from now on.

Matt - you mentioned Single Instance Storage.  Is that for attachments?  How do I check that?

The big test will be tomorrow when people are accessing their mail.  I'd like to keep this case open for a couple of days.  I'll post all results here.

Thanks!
Jono
0
 

Author Comment

by:Jono Martin
ID: 24186882
The NT Backup job backed up the entire database in just under 2 hours.  I'll have NT Backup backup the Exchange store and have Veritas backup everything else.  Anything I should be aware of when using NT Backup?

Veritas used to give me a report showing corrupted items in the mailboxes.  Does NT Backup have any such feature?  I didn't see anything besides full reporting that details every item that's backed up, successful or not.

Also, I was told that with NT Backup, there's no way to restore single items at a time.  Do you know if this is true?  If it is, do you know of a utility that will do this for me from an NT Backup file?

So far, there have been no reports from users about mail issues.

Thanks again guys!
Jono
0
 
LVL 58

Expert Comment

by:tigermatt
ID: 24187217

Provided you are backing up the Information Store with NTBackup, there is virtually no difference between that and Backup Exec.

Backup Exec has a feature known as Granular Restore Technology (GRT) which allows you to restore individual items. NTBackup does not have this technology. You could perform a brick-level backup, but these are old, slow and unreliable. Instead, you'd be looking to do a full Information Store backup, then use a Recovery Storage Group to recover items if necessary.

Single Instance Storage is essentially a technology in the Exchange database whereby an email message sent to multiple users is stored only once, and a reference to that message simply added to each recipient's mailbox. The move operation can break Single Instance Storage, which may mean you now have duplicates of larger messages in the databases.

Exchange 2003 and lower has SIS for ALL messages, so a large email can be subject to SIS; Exchange 2007 recognises that most larger emails will have some form of attachment, so will store multiple copies of an email message and only perform SIS on any attachments.

-Matt
0
 

Author Closing Comment

by:Jono Martin
ID: 31569630
The procedure was successful!  Thanks to both of you for your assistance.  The only thing outstanding is that I'm getting an error in Task Scheduler (exit code: 1f) when trying to run NTBackup on the Info Store.  I can run the job manually without issue, but the scheduler fails.  I'll search EE for an answer to that and post another question if necessary.

Thanks again!
Jono
0
 

Author Comment

by:Jono Martin
ID: 24243891
Quick question for anyone still monitoring this...

I tried to delete the old store in Exchange Manager and I get this error message:
"One or more users currently use this mailbox store.  These users must be moved to a different mailbox store or be mail disabled before deleting this store."

I mounted the store and then checked in the Mailboxes folder under the store and it's empty, as is the Logons folder.

Any idea how else I can check for mailboxes in this store?  Or how I can delete it to reclaim the space?

Thanks,
Jono
0
 

Author Comment

by:Jono Martin
ID: 24243980
Correction:

I ran a refresh on the mailboxes and logons folders and now it's showing things.

In Logons I have:
3 SMTP accounts
2 System Mailbox accounts
1 System Attendant account

In Mailboxes I have:
10 mailboxes with red "x"s over them.
1 SMTP mailbox
1 System Attendant mailbox
1 SystemMailbox mailbox

I tried to purge one of the user mailboxes and it said that it cannot be purged because it's been reconnected to another user.
 
Any help?  I'd like to delete this store to reclaim the disk space.

Thanks,
Jono
0
 

Author Comment

by:Jono Martin
ID: 24244032
In fairness to the experts here, I have started a new (related) question for this.
http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/Q_24359103.html
0

Featured Post

Why do Marketing keep bothering you?

Is your marketing department constantly asking for new email signature updates? Are they requesting a different design for every department? Do they need yet another banner added? Don’t let it get you down! There is an easy way to manage all of these requests...

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
In this video we show how to create a Contact in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Contact ta…
In this video we show how to create a Resource Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: Navigate to the Recipients >> Resources tab.: "Recipients" is our default selection …

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now