jbobst
asked on
Exchange backups - understanding the log files
I have a single Exchange 2003 server, running on Windows Server 2003. For that past year, we have been using Veritas Backup Exec along with a DLT tape drive for our daily backups. We have also been using a product from Symantec called Live State Recovery as an additional method to backup our Exchange Server (and Domain controller server). The Live State Recovery product does server imaging...where it takes a snapshop of the entire server hard drives and creates a big image file that we copy over to an external USB hard drive. We want to get rid of the tape backups and move exclusively to server images stored on external hard drive. Hard drives are cheap, and restoring a single file or an entire server from these images are VERY easy. Tape restores have always been an ugly thing to do...especially when you are trying to recover the entire server from a disater.
My understanding of Exchange backups and transaction logs is that you need to do a full backup often, so that Exchange will clear it's transaction logs. Otherwise, the transaction logs will build up. With the Veritas Backup Exec product, the Exchange Agent somehow alerted the Exchange server that a full backup was performed and the transaction logs were then purged. However, I am not sure the our imaging backup does this. I am in the process of purchasing a support contract from Symantec to ask them if their software is capable of the same thing, but I was hoping someone could clarify the whole transaction log files and the relationship to the daily backups. For example if someone had an exchange server that they NEVER backed up, would their log files grow and grow until they filled up the hard disk? Is there a manual way of clearing the log files? If I understand it correctly, the transaction logs contain all the changes to the server since the last time they were "committed" to the database...is this correct? Other than using Veritas, how would someone "commit" the changes in the transaction logs to the database manually?
Thanks for the help!
Jeff
My understanding of Exchange backups and transaction logs is that you need to do a full backup often, so that Exchange will clear it's transaction logs. Otherwise, the transaction logs will build up. With the Veritas Backup Exec product, the Exchange Agent somehow alerted the Exchange server that a full backup was performed and the transaction logs were then purged. However, I am not sure the our imaging backup does this. I am in the process of purchasing a support contract from Symantec to ask them if their software is capable of the same thing, but I was hoping someone could clarify the whole transaction log files and the relationship to the daily backups. For example if someone had an exchange server that they NEVER backed up, would their log files grow and grow until they filled up the hard disk? Is there a manual way of clearing the log files? If I understand it correctly, the transaction logs contain all the changes to the server since the last time they were "committed" to the database...is this correct? Other than using Veritas, how would someone "commit" the changes in the transaction logs to the database manually?
Thanks for the help!
Jeff
ASKER
I do a full image of the server each night. And when we had the tape backup system, I did a full tape backup every night, so the exposure is a maximum of losing the last 24hrs of work with either backup method. Is sounds like the circular logging might be the way to go. Would you recommend that?
Am I right that when the transaction logs are flushed, all the changes they hold are committed to the database? So, in the circular logging mode write any new data directly to the database?
Am I right that when the transaction logs are flushed, all the changes they hold are committed to the database? So, in the circular logging mode write any new data directly to the database?
You could use veritas (since you already have it) or NT backup (comes with windows) to just backup exchange to a file every night. That file could be saved to another server or drive or something. It would have the benefit of clearing the transaction logs and it never hurts to have an extra backup. You could also use it to rebuild the DB instead of the entire server if needed.
I never recommend circular logging on a backend server for any reason.
Circular logging gives you no chance of data recovery. The data is written and the logs flush immediately.
The thing to remember is that your recovery method basically wipes the machine. Therefore you would have to retrieve the transaction logs off the machine in order to replay them. You would be doing a backup just to flush the logs.
Simon.
Circular logging gives you no chance of data recovery. The data is written and the logs flush immediately.
The thing to remember is that your recovery method basically wipes the machine. Therefore you would have to retrieve the transaction logs off the machine in order to replay them. You would be doing a backup just to flush the logs.
Simon.
ASKER
I think I am still confused on HOW the logs work. I have done a little more reading about transaction logs, and I think I was incorrect on my initial understanding. Apparently, the logs are written, or committed, to the database as soon as the server can do it without sacrificing performance. So, at any given point during the day, transaction logs can be committed to the database during server idle times. When an exchange aware backup software, like our old Veritas Backup Exec, does a backup, it tells the server to delete all the old logs. My original thought was that the backup software told the Exchange server to COMMIT and DELETE all at the same time....meaning until you did an exchange aware backup, non of the changes had been committed up until that point. Can someone verify this?
Since I backup every night to a hard disk now, and since circular logging is not recommended, is there way to visually tell which logs have been committed? Then, is there a way to set a time threshold for exchange to keep the logs? If my current understanding is accurate, I would like to analyze the server for a few days and get an average time on how long it takes the logs to be committed to the database. For example, if a transaction log typically takes 5 hours to become committed to the database, I could set a 48 hour threshold to keep the log files and be pretty sure that I have given plenty of time for the server to commit the logs. This of course is dependant on the ability for Exchange to delete the logs on it's own, which I am unsure about.
Why can't exchange do some sort of simple verification that if a log file has been committed to the database, verify the data is there, and then just delete the file?
Thanks
Jeff
Since I backup every night to a hard disk now, and since circular logging is not recommended, is there way to visually tell which logs have been committed? Then, is there a way to set a time threshold for exchange to keep the logs? If my current understanding is accurate, I would like to analyze the server for a few days and get an average time on how long it takes the logs to be committed to the database. For example, if a transaction log typically takes 5 hours to become committed to the database, I could set a 48 hour threshold to keep the log files and be pretty sure that I have given plenty of time for the server to commit the logs. This of course is dependant on the ability for Exchange to delete the logs on it's own, which I am unsure about.
Why can't exchange do some sort of simple verification that if a log file has been committed to the database, verify the data is there, and then just delete the file?
Thanks
Jeff
Your understanding is not quite correct.
The transaction logs are written as Exchange writes to its database. There is no lag between them. If there was, then under performing Exchange servers simply wouldn't write their logs and you would be at risk from data loss in the event of a system failure or power loss.
When you do a backup, the changes are then committed to the database. At that point the transaction logs are not required and are flushed.
The only way that Exchange can delete its own logs is by using circular logging. Otherwise it is down to the Exchange aware backup application.
The logs that are present in the folder are the logs that are uncommitted and those will be flushed after the next backup.
Simon.
The transaction logs are written as Exchange writes to its database. There is no lag between them. If there was, then under performing Exchange servers simply wouldn't write their logs and you would be at risk from data loss in the event of a system failure or power loss.
When you do a backup, the changes are then committed to the database. At that point the transaction logs are not required and are flushed.
The only way that Exchange can delete its own logs is by using circular logging. Otherwise it is down to the Exchange aware backup application.
The logs that are present in the folder are the logs that are uncommitted and those will be flushed after the next backup.
Simon.
ASKER
Simon,
If the logs are written as Exchange writes to it's database, then does Exchange write to its database? Only during a backup? If someone only backed up their server once a week, you could potentially loose a week's worth of data...right?
If the logs are written as Exchange writes to it's database, then does Exchange write to its database? Only during a backup? If someone only backed up their server once a week, you could potentially loose a week's worth of data...right?
It writes to both its database and the logs at the same time.
If you only backed up the server once a week, then in the event of a total failure (including all hard disks) you would lose the week's data.
That is one of the reasons why we recommend that Exchange is installed with the databases and logs on separate drives.
Simon.
If you only backed up the server once a week, then in the event of a total failure (including all hard disks) you would lose the week's data.
That is one of the reasons why we recommend that Exchange is installed with the databases and logs on separate drives.
Simon.
ASKER
Simon,
I had a typo in my previous post...I meant to say "...WHEN does Exchange write to its database?..." However, it seems from your responses that the the WHEN is only when it's backed up. Well, if that's my only option, then I suppose I'll have to add some other sort of backup routine that is exchange aware.
Thanks.
Jeff
I had a typo in my previous post...I meant to say "...WHEN does Exchange write to its database?..." However, it seems from your responses that the the WHEN is only when it's backed up. Well, if that's my only option, then I suppose I'll have to add some other sort of backup routine that is exchange aware.
Thanks.
Jeff
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Sorry for such a delay in responding to this question...I am trying to things a little cleaned up in my account now.
I still don't think I fully understand the purpose of the log files and how they are flushed, but in my situation with my server imaging, I now just run a nightly exchange backup using the Windows backup utility. This is done before the server is imaged each night and seems to take care of any problems. The next day, after the image has been taken, I have a scheduled task that deletes the exchange backup file before the next backup is run. That way the backup files won't build up (as they are fairly large).
Thanks for the help.
Jeff
I still don't think I fully understand the purpose of the log files and how they are flushed, but in my situation with my server imaging, I now just run a nightly exchange backup using the Windows backup utility. This is done before the server is imaged each night and seems to take care of any problems. The next day, after the image has been taken, I have a scheduled task that deletes the exchange backup file before the next backup is run. That way the backup files won't build up (as they are fairly large).
Thanks for the help.
Jeff
uh, that was exactly my suggestion.
and, to try to explain log files briefly: The log files hold a copy of all changes that have been made to the database since it's last backup so, if the database is corrupted or lost, you can restore the last backup and use the log files to get up to date.
You can also configure windows backup to overwrite the last backup file. You shouldn't need to run another task to delete it.
and, to try to explain log files briefly: The log files hold a copy of all changes that have been made to the database since it's last backup so, if the database is corrupted or lost, you can restore the last backup and use the log files to get up to date.
You can also configure windows backup to overwrite the last backup file. You shouldn't need to run another task to delete it.
ASKER
brakk0,
You did suggest that, and that is why I set it up that way...my apologies for at least not splitting the points with you or not accepting your answer. I let this thread sit for awhile and didn't think about your answer yesterday when I went to close out this question.
Thank you for the further explanation on log files...I think I finally get it. You say the log files hold a COPY of all changes...(I think Simon had said the same thing in an earlier post). I was confused, because everytime I hear about the log files, I got the impression that the log files were NOT committed to the database UNTIL the backup takes places. Simon was explaning this correctly and I just wasn't getting it for some reason. Thanks for the clarification...sorry about the points!
Jeff
You did suggest that, and that is why I set it up that way...my apologies for at least not splitting the points with you or not accepting your answer. I let this thread sit for awhile and didn't think about your answer yesterday when I went to close out this question.
Thank you for the further explanation on log files...I think I finally get it. You say the log files hold a COPY of all changes...(I think Simon had said the same thing in an earlier post). I was confused, because everytime I hear about the log files, I got the impression that the log files were NOT committed to the database UNTIL the backup takes places. Simon was explaning this correctly and I just wasn't getting it for some reason. Thanks for the clarification...sorry about the points!
Jeff
Therefore the logs will never be flushed.
It isn't just logs that are flushed - there are settings within Exchange where data is dropped once it has been backed up.
Veritas has nothing to do with the transaction logs.
Veritas Backup Exec tells Exchange that a full Exchange aware backup has been completed, the logs are flushed.
In the event of a failure of the database, you would restore the last full backup and then replay the transaction logs. The replaying of the transaction logs is a Microsoft function.
If you don't do an Exchange aware backup, then the transaction logs continue to build.
There is a function in Exchange called circular logging. That stops the build up of the transaction logs. However you will get no logs at all, so you will be exposed between the backups taken from the imaging software.
Simon.