Exchange server 2003 Information Store / Mailbox Store is dismounted almost every night.

Hi guys,

  Let me start by saying that I'm kind of a newcomer to servers and Exchange. The IT manager at the company I work at resigned so it has left me (The helpdesk guy) as the network admin as well. Ok here's my problem. Almost every night our Exchange server email system "goes down" as in noone can get mail and outlook says the connection has been lost. When I invetigated this further I found that in Exchange system manager the mailbox store is red and appears to be dismounted....if you right click it and select to mount it it mounts fine and the email system comes back up. Ok so my question is why would this be happening and how can I prevent it. This is happening 3-4 days a week and is causing me to rush in at early hours. This machine runs Exchange server 2003 and Veritas backup exec. Our backups run over night and I was starting to think that may have been the cause until I looked in event viewer and say the error that I am going to include in a screen shot. It appears that the mailbox store database may be too big? Can anyone please offer some tips to an overworked newcomer on how te resolve this? Thanks!
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

I'm guessing you currently use Exchange Server 2003 Standard.  That allows one storage group of 75GB (with SP2 and the Registry key set).
Do you use mailbox size limits?
JReacherAuthor Commented:
I thought I also might mention that I just looked up the database and it doesn't appear to be over 75gb as the error implied...I have included a screen shot.
JReacherAuthor Commented:
Yes I think she did incorporate mailbox size limits because I have seen the email go to some people that they are over their size limits. I'm not sure what that limit would be or how to change it though.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

You need to add together the priv1.edb & priv1.stm to get the total database size.  From a quick look (as well as I can see in the screenshot) 85GB does seem possible.
Exactly. Nightly, Exchange will run an online degrag. If you have exceeded your limit, the store will dismount. Here is the KB. 
Check your Mailbox Store Mailboxex in ESM and sort by mailbox size. You will see how large some accounts are. How many employees attach to this exchange server? Do some quick math and you may realize you need to move up to Enterprise edition.
I'd guess that unfortunatley the max mailbox size x total users is greater than 75GB.
As a quick "fix" I'd get the users to purge as many old emails as they can and reduce the deleted item retention period to say zero.  To do this Open Exchange System Manager, Expand until you can see the mailbox store.  Right Click and select properties.  On the Limits tab amend the "Keep deleted items for (days).
After 24 hours have elapsed you can then defrgament the exchange DB using ESEUtil as shown on

Hope that helps, please post back if you need any clarification.
You need to free up space in your store.  

I would recommend finding the largest users mailboxes and either having them SHIFT-Delete the items from Outlook, or forcably remove them yourself using EXMERGE.  You can find the largest offenders in Exchange System Manager.

Attachments are the biggest culprints. You can always remove the attachment and leave the body of the email.
You need to get the users to delete some of the e-mail's and then do an offline deframentation of the Exchange databases as specified in the article:

This is the only way to shrink the database and stop it dismounting.

The offline defragmentation can take a while to run and can be as slow as 4GB per hour, you will therefore need to run this out of hours over a weekend.

You need to do this as soon as possible as the database will get to the stage where it will not let you mount it until you have recovered some space. This could mean your system being out of action for a number of days during normal week working hours which would not be very good
JReacherAuthor Commented:
Thanks for all the advice, I made a how-to for archiving and cleaning up your email folders and sent it out to all employees. We will see how well they actually respond ;). I am going to try the offline defrag this weekend. Has anyone seen or heard of any potential dangers from that? I'm just kind of new to exchange and it would be a nightmare to mess up anyone's email. Thanks for all the tips!
Make sure you have a good backup prior.
The offline defrag is pretty standard and if you follow the instructions you should be ok. It creates another smaller version of the database before overwriting it.

As bcrosby007 says - Do a full backup of Exchange first just in case something does go wrong.
JReacherAuthor Commented:
What method of backup do you suggest? And should I just backup the 2 Exchange databse files that were listed in the screenshot I provided earlier?
You should use a program that works for open files, like Backup Exec with an exchange agent.
JReacherAuthor Commented:
We have that, But which files specifically do I need to backup to cover myself? I think our nightly backup gets peoples mailboxes but I'm not sure about the database files.
JReacherAuthor Commented:
It looks like she also has a windows backup set up to backup exchange log files? I remember her mentioning that those log files would help restore the email server if anything went wrong?
You should do a backup with an Exchange Aware backup program.  I don't recommend using one that backs up open files.  Before you run ESEUTIL, dismount the database.  Then do a file backup of the .EDB and .STM files, along with your transaction logs.

FYI, I don't believe you need to run the ESEUTIL offline defrag to fix your problem.  I believe Exchange only counts the storage space used within the database to determine the 75/85Gig limits, as noted in the Event Log entry, "(the logical size equals the physical size of the .edb file and the .stm file minus the logical free space in each)".
JReacherAuthor Commented:
Rob, So what do you suggest to fix it? Please try to put it as simple as possible for me, I can follow directions well but I really have never worked with exchange and am kind of sweating that I will mess up their email server. Thanks!!
I would say for now, your issue is that the store is dismounting due to the size.  Address that issue alone - for now.

Do so by having your users clean up their mailboxes.  Delete items they do not need.  Have them empty their Deleted Items folders in Outlook.  If the users will not remove the data, you can do it for them, depending on the policies in place.  You can open their mailbox with Outlook and manually delete data.  You can also open their mailbox and move data to PST files.  Another option is to use the EXMERGE utility from the mail server.  The EXMERGE utility will parse any to all of the mailboxes and either delete or archive data.

Once you get your storage problem under control, you should run the offline defrag.  As the mailbox items are deleted from exchange, the sizes of your .EDB and .STM files will not shrink.   There will be white space (empty space) within those files that Exchange isn't using.  To recover that space, you need to run the offline defrag.  It's not the safest utility so you should only run it if you need to, and make sure you have a solid backup before doing so.  There are also size requirements to run it.  When it comes to that stage you should look into it further.

But for now, get users to delete emails they don't need (and empty deleted items folder).

Oh.  You can also create a Mailbox Manager policy that will manually deleted items from peoples folders, such as the deleted items folder.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JReacherAuthor Commented:
Thank you very much for your advice. I am going to give everything u suggested a try. Im still having some trouble getting users to comply with cleaning their email boxes out. I am going to try an offline defrag this weekend so cross your fingers for me and hope it goes well. Thanks again for all your help! I'll keep you posted on how it goes.
Here's a link to another EE article that explains how to use the Mailbox Manager to empty everyone's deleted items folder:

JReacherAuthor Commented:
Ok sorry I didnt get back to you sooner, due to fathers day and long backup times this weekend I wasn't able to try the defrag yet. I am however going to do it saturday. I have a question first. On this server we have a C: which is only 12gb and almost full. We also have a D,E, and F drive. The database files are being stored in F:\Exchsrvr\Mdbdata and we have plenty of free space on that drive (400gb). My question is if I run the utility from the command line while in that directory will it create the temporary database on the F:\ in that directory or will it try to put it on the C:\ unless I point it somewhere. Sorry I'm sure this is a silly question but I just dont want to leave anything to chance. Also what would be the command format to point the temporary database to a specific location? Below is the command format I was going to run this weekend so please anyone tell me if I'm not correct on this or any suggestions on how to run this offline defrag safer / better.

F:\Exchsrvr\Mdbdata> Eseutil /d "f:\exchsrvr\mdbdata\priv1.edb"

The Tempdfrg.edb file is created on the logical drive from which the eseutil /d command is run unless you use the /t switch. For example, to create a Tempdfrg.edb on the root of drive D, run the following command:
D:\>eseutil /d /ispriv;en-us;328804

So if you have sufficient space on that drive it will be ok
JReacherAuthor Commented:
Ok thanks. and in your opinion how safe is this to try? Recently I have been getting this error in Veritas Backup Exec 10:

0xe000032c - The consistency check of the snapshot for the Microsoft Exchange database has failed. You should check for possible data corruption in the Exchange Server that is online.
Final error category: Resource Errors

It worries me that the database might be corrupt and have issues with the defrag, have u heard of anyone having trouble with this?
There is always a slight possibility that something can go wrong so it is always wise to run a backup prior to running. The process is usually very reliable as it defrags the database in to a new database (thus the requirement for haveing 110% free space) once it has created the new database it then becomes live, if it cannot make it live the old one is still there. It has never gone wrong for me but I do know of instances when it has.
JReacherAuthor Commented:
So when it's done and it goes live, should I backup and then delete the old one or it will get rid of it for me if it goes live?
It will create the temporary one, defragment the old one in to it, remove the original and replace it with the new one, if there are no errors and the defrag completes you don't have to do anything
Prior to running the defrag, make sure you have a flat file backup of the edb, stm, and log files.  Having the flat file saved me once.  HIGHLY recommended.

JReacherAuthor Commented:
Ok sorry....can you explain what a flat file backup is? I know where all these files are located but im not sure what you mean by that. Thanks.
Flat file is a regular file backup when exchange isn't accessing them (services stopped).  
JReacherAuthor Commented:
Hey guys, here's the update.

So I found out that my Mail store database was corrupt. I couldn't get a backup of he database because i would always fail either through NT backup or Backup Exec. I did a ESEUtil /g to check integrity of the databse and it of course came back that it was corrupted and also informed me that the databse had not been backed up since 3/10/2008. So at the advice of you guys and one of my friends here's how I resolved the problem for any other poor schmucks like me that run into this:

1) I ran Exmerge and got .pst backups of all mailboxes saved onto another server.
2) I then dismounted the mailbox store, stopped all exchange services and made copies of the mail store database files and transaction logs to a location on another server.  
3) Rename the folder(s) that your database files and transaction logs are located in and then recreate a folder with the original name. For instance my database files and transaction logs were in a folder named MDBDATA I renamed that folder to MDBDATA.OLD and created a new MDBDATA folder.
4) Restart all exchange services and go into exchange system manager. Attempt to remount your mail box store. System manager will give you a message along the lines of (One or more files for this store are missing, forcing a mount with cause a creation of a new database. Do you want to continue?) Click yes.
5) You now have a new, clean, empty databse. Now run exmerge again only this time you will be importing the pst files to the server. Give it your info and wait for the files to import.

Thats pretty much all I did. I got a few peoples outlook that didn't like it and gave them a weird message but by just removing their exchange mail account out of outlook and re-adding it it fixed this problem and they still have all of their mail. Everything is running smooth now and I got my first successful backup of the exchange database in months. Thanks to everyone for their help and suggestions!!
Sorry to give you any "bad" news.  When you exported using exmerge and then imported back in, you lost your Single Instance Storage, one of the benefits of Exchange.

If you have the disk space, you can still attempt a repair of your database.  Backup the old database files, and then run the isinteg and eseutil programs with the appropriate flags.  That may repair your database and allow you to mount it. This will preserve the SIS.  Then, if you want, you can Exmerge out of your currently live database anything new, and import that into your repaired database.

Also, the reason you should use NTBackup or another Exchange aware backup utility is because it will alert you if the database is corrupt.

In the end, happy you have things up and running.

JReacherAuthor Commented:
Would this only effect the SIS on the messages that were previosuly on the server? Would any new messages still take advantage of SIS? My database was at about 85 GB before I performed this fix and it is now at 13GB. I would love to keep it how it is now without going through any more hassles. And if I understand correctly wouldn't anyone in a pop3 through exchange not take advantage of SIS? I worked for a company that had 4x the amount of users we do that had pop3 set up through exchange and never had any disk space problems. I currently have 450gb free on the drivet hat the mail box store database is stored on and only 75 email users so whata re your thoughts on operating without SIS?
Wow!  85 Gig to 13 Gig?  Leave it alone.

I'm not sure of the Pop3 not being able to take advantage of SIS.

I would say leave Exchange as is.  Going forward, just make sure you are doing Exchange Aware backups, and check the jobs daily to see if they complete.

JReacherAuthor Commented:
Will do. I cant thank you enough for all of your help Rob. You rock!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.