Exchange Server 2003 logs won't clear up
I am working with an Exchange Server 2003, Standard SP2 situation in which the Exchange logs have continually increased over the course of several months and now add up to 18GB. There is still ample disk space, however I definitely would like to find a way to have the logs decreased back to a much smaller number.
This is a single Exchange server installation with a Microsoft Windows Server 2003 R2, Standard edition operating system. There are 20 users and the total size of the combined priv1.edb and priv1.stm is only 35GB so having log files the size of 18GB needs to be corrected.
I have performed several full backups of NTBackup making sure to utilize the Microsoft Exchange Server option within NTBackup however the logs have never refreshed and decreased in size. The last backup shows as today within System Manager as that is the last time I ran a full backup of Exchange and then restarted the server, but to no avail as the logs still show 18GB.
I realize the logs could be manually removed, but would prefer to find the cause of this problem first. Thanks in advance for any good ideas!
This is a single Exchange server installation with a Microsoft Windows Server 2003 R2, Standard edition operating system. There are 20 users and the total size of the combined priv1.edb and priv1.stm is only 35GB so having log files the size of 18GB needs to be corrected.
I have performed several full backups of NTBackup making sure to utilize the Microsoft Exchange Server option within NTBackup however the logs have never refreshed and decreased in size. The last backup shows as today within System Manager as that is the last time I ran a full backup of Exchange and then restarted the server, but to no avail as the logs still show 18GB.
I realize the logs could be manually removed, but would prefer to find the cause of this problem first. Thanks in advance for any good ideas!
Or if you want to find the cause first...
http://msexchangetips.blogspot.com/2006/08/exchange-transaction-log-files-growing.html
http://msexchangetips.blogspot.com/2006/08/exchange-transaction-log-files-growing.html
For the"type" of backup, are you selecting Normal or Copy? A Copy backup will not flush the transaction logs whereas a Normal backup should.
Also, check the security on the user you are running ntbackup under. If you are not already doing it, try running the job as a user that has the exchange administrator role as well as is a member of the domain admins.
Also, check the security on the user you are running ntbackup under. If you are not already doing it, try running the job as a user that has the exchange administrator role as well as is a member of the domain admins.
ASKER
Thanks dfxdeimos for your input! I have found the same article regarding manually deleteting the logs, but as I have hard drive space available right now, I would prefer finding out what is causing the problem otherwise I will be doing this again.
I looked at the link you provided and checked into both suggestions with no luck yet.
Some additional background on this Exchange server may help....this Exchange server 2003 was installed 2 1/2 years ago as a result of a conversion from a POP3 ISP email service so there was no conversion from a previous version of Exchange. Since then Exchange has run well.
The only "new" thing I can find is the logs have not been flushed since August which is about the same time I began using MozyPro for an off-site data backup solution. I've been troubleshooting with their support folks for awhile and no luck yet but will continue down that path. May be wrong but am not sure MozyPro would have anything to do with Exchange logs not clearing.
Thanks zmagick for your question. I have run the Normal backup in NTBackup just last night and it completed successfully which is why this is very confusing as to why the logs are not clearing. The user account I am using to do this is the network administrator account. Also, within System Manager the mailbox store's properties show the backup done just last night.....
I looked at the link you provided and checked into both suggestions with no luck yet.
Some additional background on this Exchange server may help....this Exchange server 2003 was installed 2 1/2 years ago as a result of a conversion from a POP3 ISP email service so there was no conversion from a previous version of Exchange. Since then Exchange has run well.
The only "new" thing I can find is the logs have not been flushed since August which is about the same time I began using MozyPro for an off-site data backup solution. I've been troubleshooting with their support folks for awhile and no luck yet but will continue down that path. May be wrong but am not sure MozyPro would have anything to do with Exchange logs not clearing.
Thanks zmagick for your question. I have run the Normal backup in NTBackup just last night and it completed successfully which is why this is very confusing as to why the logs are not clearing. The user account I am using to do this is the network administrator account. Also, within System Manager the mailbox store's properties show the backup done just last night.....
Usually when Exchange 2003 backups up completely but does not flush the logs it's usually to do with a previous log file being corrupt due to one reason or another- dirty shutdown, anti virus etc.
Firsly, If you cleanly dismount the store so it's in a 'successful' stop state, all needed logs will be written to the database, and Exchange then detaches itself from its logs. Only then can you move (I NEVER delete to start with) all log file older than the latest *.chk file. The name of the checkpoint file will always be [prefix].chk, for example, E00.chk.
This checkpoint file marks the position in the log files at which the database is in a consistent state. Anything older then this file is considered 'inactive' and everything newer then the E00.chk is active.
You can read this file by using Eseutil.exe, only when the database is in a dismounted state. This utilty is kept in \exchsrvr\bin directory and to read it you use the /MK command line switch. For example:
"C:\Program Files\exchsrvr\bin\eseutil
This command will produce some output, but one of the lines in the output will be the checkpoint field, like:
Checkpoint (0x21EE,11B0,9A)
There are three values separated by commas, but you only have to pay attention to the first one. This first number 0x21EE lists the sequence number of the oldest log file needed to recover the database.
If your checkpoint prefix is E00 (E00.chk), then that means you need log file E00021EE.log, along with all newer log files, in order to recover the database, and you can remove log file E02021ED.log, and all older log files, without affecting the recoverability of the database.
I recently had this problem in Exchange 2007 and I left E00.chk, E00.log, E00tmp.log, res1.jrs, and res2.jrs in the logs directory, and MOVED all the other logs files to another partition and remounted the database. It should start creating fresh new logs, and hopefully will flush the logs when you perform a full backup.
If reading the checkfile is too difficult you safe to do the above but remember NEVER delete the files first, move them and make sure your Exchange start flushing its logs correctly before you can safely delete them.
more info her: http://support.microsoft.com/kb/240145
Firsly, If you cleanly dismount the store so it's in a 'successful' stop state, all needed logs will be written to the database, and Exchange then detaches itself from its logs. Only then can you move (I NEVER delete to start with) all log file older than the latest *.chk file. The name of the checkpoint file will always be [prefix].chk, for example, E00.chk.
This checkpoint file marks the position in the log files at which the database is in a consistent state. Anything older then this file is considered 'inactive' and everything newer then the E00.chk is active.
You can read this file by using Eseutil.exe, only when the database is in a dismounted state. This utilty is kept in \exchsrvr\bin directory and to read it you use the /MK command line switch. For example:
"C:\Program Files\exchsrvr\bin\eseutil
This command will produce some output, but one of the lines in the output will be the checkpoint field, like:
Checkpoint (0x21EE,11B0,9A)
There are three values separated by commas, but you only have to pay attention to the first one. This first number 0x21EE lists the sequence number of the oldest log file needed to recover the database.
If your checkpoint prefix is E00 (E00.chk), then that means you need log file E00021EE.log, along with all newer log files, in order to recover the database, and you can remove log file E02021ED.log, and all older log files, without affecting the recoverability of the database.
I recently had this problem in Exchange 2007 and I left E00.chk, E00.log, E00tmp.log, res1.jrs, and res2.jrs in the logs directory, and MOVED all the other logs files to another partition and remounted the database. It should start creating fresh new logs, and hopefully will flush the logs when you perform a full backup.
If reading the checkfile is too difficult you safe to do the above but remember NEVER delete the files first, move them and make sure your Exchange start flushing its logs correctly before you can safely delete them.
more info her: http://support.microsoft.com/kb/240145
ASKER
Thanks orphanc! I am looking in the Exchange log file directory now and don't see any file w/ the extension of .chk and am wondering if that may be part of the problem? Perhaps a dismount of the store after hours may create this file as you talk about above?
Rainman13
Rainman13
Hi Rainman13,
The checkpoint file is used to track which transactions have been committed to the database . The name of the file is EX0.chk (X stands for the storage group) and its size is 8KB.
So in your case it seems no trnx has been commited to your Exchange database- i.e, Exchange has not been cleanly detached from it's log files.
What are the first 4 files you see in your log directory?E00.log, E00tmp.log, res1.jrs, and res2.jrs or thereabouts?
Take a note. When you get time, dismount your database. Make sure is stops cleanly. You should see a.chk file then.
In it's dismounted state, I would then MOVE (not delete) all the *.log files to another drive. Move even the resx.jrs files (these are just used by the storage group if it ever runs out of space) and re-mount the store. Take note of the new log file created.
Run a ntbackup.exe of your database- if it works the problem was with a previously corrupt .log file.
If not, it's something with the database. In 2007 having 'LCR enabled' can cause this to occur (I had this). On the oldest date of the logs- you mention August, does that date coincide with anything that happened or changed on Exchange? That's how I started to solve my issue, by looking back at events that occured on that date. Check eventvwr on your exchange server. Some people with Exchange 2003 enable circular logging which in short recycles the logs but also prevents DR of your database, so don't do this. First try the above and see where you go. You have no choice but to dismount the store at some point soon, and move your logs. Don't leave it too late.
Hope this helps.
The checkpoint file is used to track which transactions have been committed to the database . The name of the file is EX0.chk (X stands for the storage group) and its size is 8KB.
So in your case it seems no trnx has been commited to your Exchange database- i.e, Exchange has not been cleanly detached from it's log files.
What are the first 4 files you see in your log directory?E00.log, E00tmp.log, res1.jrs, and res2.jrs or thereabouts?
Take a note. When you get time, dismount your database. Make sure is stops cleanly. You should see a.chk file then.
In it's dismounted state, I would then MOVE (not delete) all the *.log files to another drive. Move even the resx.jrs files (these are just used by the storage group if it ever runs out of space) and re-mount the store. Take note of the new log file created.
Run a ntbackup.exe of your database- if it works the problem was with a previously corrupt .log file.
If not, it's something with the database. In 2007 having 'LCR enabled' can cause this to occur (I had this). On the oldest date of the logs- you mention August, does that date coincide with anything that happened or changed on Exchange? That's how I started to solve my issue, by looking back at events that occured on that date. Check eventvwr on your exchange server. Some people with Exchange 2003 enable circular logging which in short recycles the logs but also prevents DR of your database, so don't do this. First try the above and see where you go. You have no choice but to dismount the store at some point soon, and move your logs. Don't leave it too late.
Hope this helps.
ASKER
Thank you orphanc! I will start this process tonight and will advise -
ASKER
Orphanc - I dismounted the mailbox store but no luck with a .chk file being generated.
The first 4 files in my log directory are: E00.log, E00077F3.log, E00077F2.log and E00077F1.log
I'm going to keep digging...
The first 4 files in my log directory are: E00.log, E00077F3.log, E00077F2.log and E00077F1.log
I'm going to keep digging...
ASKER
Hi orphanc!
Before I did anything further tonight, I completed a successful NTBackup of the Microsoft Information Store.
Thought I'd then try dismounting the mailbox store and checking to see if it is clean or not, but no luck. Perhaps I am dense, but the Exchange Store files are located on the d: drive of this server and as such am not having any luck running the eseutil.exe /MH command on the d: drive. When trying to run the following command
d:\Exchange Store\eseutil /MH priv1.edb
I receive the following error message:
This application has failed to start because the ESE.dll was not found. Re-installing the application may fix the problem".
As the eseutil.exe is on the c: drive in Program Files, I evidently didn't copy all of the correct eseutil files to the d: drive so this can run on the respective mailbox store....
You are probably right as there are most likely corrupt log files in play here which means if I move the log files and the mailbox store is indeed in a dirty state, I could create a bigger problem for myself such as needing to do a database restore to get email backup and running in the interim.
I know you don't recommend it, but I think I am going to turn on circular logging for a day or so while keeping an eye on things, running nightly Exchange backups in NTBackup and see if this can flush the logs and clear up this problem.
Before I did anything further tonight, I completed a successful NTBackup of the Microsoft Information Store.
Thought I'd then try dismounting the mailbox store and checking to see if it is clean or not, but no luck. Perhaps I am dense, but the Exchange Store files are located on the d: drive of this server and as such am not having any luck running the eseutil.exe /MH command on the d: drive. When trying to run the following command
d:\Exchange Store\eseutil /MH priv1.edb
I receive the following error message:
This application has failed to start because the ESE.dll was not found. Re-installing the application may fix the problem".
As the eseutil.exe is on the c: drive in Program Files, I evidently didn't copy all of the correct eseutil files to the d: drive so this can run on the respective mailbox store....
You are probably right as there are most likely corrupt log files in play here which means if I move the log files and the mailbox store is indeed in a dirty state, I could create a bigger problem for myself such as needing to do a database restore to get email backup and running in the interim.
I know you don't recommend it, but I think I am going to turn on circular logging for a day or so while keeping an eye on things, running nightly Exchange backups in NTBackup and see if this can flush the logs and clear up this problem.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Morning!
I was able to get to this screen, but had no luck running the /MH or /MK commands....when I entered a / to initiate the /MH or /MK commands the the command line interface would disappear - what is the protocol for initiating these commands from here?
I agree on pros / cons of circular logging but wanted to get this fixed pronto so turned it on for one day and it cleared up the problem. The NTBackup works great and I kicked off a MozyPro backup to do a complete Exchange backup - as it is 35~GB it will take a few days to get that first full Exchange backup to see if that goes OK.
Yes thanks for the thought on digging into EViewer....that did pop up some issues that could've caused the log file corruption to begin with. Before we started using MozyPro for offsite data backup, Backup Exec was used on this server. I've noticed other posts that talk about this same type of problem with BE and Exchange so am guessing there is probably an issue there which may have caused this to begin with.
Thanks again for all the good info!
I was able to get to this screen, but had no luck running the /MH or /MK commands....when I entered a / to initiate the /MH or /MK commands the the command line interface would disappear - what is the protocol for initiating these commands from here?
I agree on pros / cons of circular logging but wanted to get this fixed pronto so turned it on for one day and it cleared up the problem. The NTBackup works great and I kicked off a MozyPro backup to do a complete Exchange backup - as it is 35~GB it will take a few days to get that first full Exchange backup to see if that goes OK.
Yes thanks for the thought on digging into EViewer....that did pop up some issues that could've caused the log file corruption to begin with. Before we started using MozyPro for offsite data backup, Backup Exec was used on this server. I've noticed other posts that talk about this same type of problem with BE and Exchange so am guessing there is probably an issue there which may have caused this to begin with.
Thanks again for all the good info!
http://support.microsoft.com/kb/240145