Exchange Logs inconsistent, causing backups to fail

A client of ours is running Exchange 2010 and has a mailbox store of 450GB.  The storage on the server has been filling up and they were deleting log files for the past couple of months.  
When asked to perform a backup, the backup fails when checking the consistency of the Exchange logs.  
We need to unmount the store and run a check on it, and I'm assuming there would be failures on it resulting in the need for repair.  The problem is that the store is so large that it would create a long amount of downtime for the client.

Can anyone suggest a method for fixing this issue that wouldn't result in several days of downtime?
averylnAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

suriyaehnopCommented:
I would like to suggest to create a new database then move the user over to new DB instead of dismount the original DB
Ibrahim BennaTechnology LeadCommented:
I second that motion - As suriyaehnop suggested, create a new database and move mailboxes to it. In fact, I would create multiple databases and split up the mailboxes between them. I do not see a reason for keeping all the mailboxes in a single database when you can divide them into smaller manageable databases.
Simon Butler (Sembee)ConsultantCommented:
Enable circular logging. Dismount and remount the database. The logs will then flush out.
When complete, disable circular logging and run a backup.

I would then look at splitting the database up as already advised. As you can have four databases on Exchange 2010 (five if no public folders) it is more efficient to have multiple databases.

Simon.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

averylnAuthor Commented:
With circular logging, once that is enabled and the database is mounted and dismounted, will the logs flush out even if there have been problems with consistency due to some files being deleted?  Can anyone speak to the amount of time to perform this operation?

Splitting the database up is definitely something we'd like to do.  The store is just way too large.
Ibrahim BennaTechnology LeadCommented:
Here's the problem how I see it  and from assumptions based on what you have mentioned - Log files have been deleted to allocate more disk space but it is not known (or is it??) if the log files that were deleted are required to make the database consistent. That means there is a possibility when you come to dismount the database that it may crash when it doesn't find the log files in the "logs required" range - This would mean the database must be repaired and this could take plenty of time (5-10GB an hour for repair then 7-10GB an hour for offline defragmentation).

In my opinion, I would just create new databases and start moving mailboxes as soon as possible. its Exchange 2010 so moves are online and users will not be affected until the move is completed - when they will be prompted to restart Outlook to reconfigure their profile settings.

How far back do the log files currently go? Were they randomly\individually deleted or were some deleted in a group chunk?
averylnAuthor Commented:
Yes, log files were deleted to make more space as the drive was filling up.  I do not know if the log files deleted were required to make the database consistent.  (I do know that the logs were deleted without even dismounting the store.)  However, when attempting to backup the database with Backup Exec it fails, reporting that the logs are not consistent.  Is this indicative that the files deleted were in the "logs required" range?  If I'm going to be faced with a situation where I'll need to do a repair after unmounting, before I can mount again, I'll definitely need to know ahead of time!  
It does seem that the safest thing to do would be to create a new database and start moving mailboxes, but given that there is an issue with space, I don't know that this would even be possible.  
The log files go back to the beginning of Feb. from what I can see so far, and when I asked how they were deleted I was told that the oldest files were selected and "shift deleted".
The Exchange folder moves so slow, when I can report back with more concrete numbers for the number and age of the logs I will!
averylnAuthor Commented:
Ok, here is some additional information:

450GB Mailbox Store
355,016 files in the Exchange Folder
Roughly 400GB in log files dating back to the beginning of Feb.
Simon Butler (Sembee)ConsultantCommented:
As long as you don't restart the services, the database should mount with circular logging enabled. Has the server never been rebooted?

Do you have the space to split the database up? I doubt if you do.

The number of logs that you have is too high, unless this is a fresh installation. It could be that you have another problem that is causing high log generation - usually a mobile device.

The Exchange product team have an article on troubleshooting this issue.

http://blogs.technet.com/b/exchange/archive/2013/04/18/troubleshooting-rapid-growth-in-databases-and-transaction-log-files-in-exchange-server-2007-and-2010.aspx

Basically it looks like you have two issues.

1. The logs have been manually deleted.
2. The logs are growing at a rate that is faster than normal.

Simon.
averylnAuthor Commented:
The server was rebooted multiple times a few weeks ago when Windows Updates were done.  This is not a fresh installation.  Definitely don't have space to split the database up.
When enabling circular logging, do you need to dismount the database first, enable it, then remount?  Is that when the logs start to clear?

Yes, the log files are too many and add up to too much space.  They haven't had a good backup in a long time which contributes to the log files not clearing.
Simon Butler (Sembee)ConsultantCommented:
Enable, dismount, remount. It doesn't take effect until the database has been through the dismount/remount cycle.

If there are 400gb of logs but they only date back to February, that is too much. If they dated back to the point the server was installed then I could understand it, but 400gb for a 450gb database since February is simply excessive and indication of a problem elsewhere.

Simon.
averylnAuthor Commented:
Ok, good to know.  Given the size of the store and the logs, is this a process that will take a long time?
Also, should I backup after remouting and the logs are cleared out (with circular logging still enabled), or should I disable circular logging, dismount, and remount again before trying to backup?

Yes, definitely will be looking into excessive logging once I have a good backup!
Simon Butler (Sembee)ConsultantCommented:
I would normally do the backup AFTER disabling circular logging.
It will take a while to flush the logs - maybe an hour or so with that many.

Simon.
averylnAuthor Commented:
Ok, final thing I'm trying to determine, as it will make a huge difference in the expected downtime...

When I turn on circular logging and unmount the database, will I be able to remount it, or will the database need to be repaired? (If so it could take 50+ hours)

I wish there was a way to know for certain if I'll be able to remount it or not.  Otherwise I'm bringing this uncertainty to the client. (It could take 50+ hours, or it could take up to 3 hours)
Simon Butler (Sembee)ConsultantCommented:
You shouldn't need to do anything with the database, dismount and remount.
The log clearing doesn't require any further downtime - so your downtime is actually about 30 seconds.

However I have made that recommendation based on what you have said about the server being rebooted since the log deletions have been taking place - so the database has mounted successfully.

Simon.
averylnAuthor Commented:
Thank you!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.