Solved

Exchange backups - understanding the log files

Posted on 2006-07-14
13
478 Views
Last Modified: 2010-03-06
I have a single Exchange 2003 server, running on Windows Server 2003.  For that past year, we have been using Veritas Backup Exec along with a DLT tape drive for our daily backups.  We have also been using a product from Symantec called Live State Recovery as an additional method to backup our Exchange Server (and Domain controller server).  The Live State Recovery product does server imaging...where it takes a snapshop of the entire server hard drives and creates a big image file that we copy over to an external USB hard drive.  We want to get rid of the tape backups and move exclusively to server images stored on external hard drive.  Hard drives are cheap, and restoring a single file or an entire server from these images are VERY easy.  Tape restores have always been an ugly thing to do...especially when you are trying to recover the entire server from a disater.

My understanding of Exchange backups and transaction logs is that you need to do a full backup often, so that Exchange will clear it's transaction logs.  Otherwise, the transaction logs will build up.  With the Veritas Backup Exec product, the Exchange Agent somehow alerted the Exchange server that a full backup was performed and the transaction logs were then purged.  However, I am not sure the our imaging backup does this.  I am in the process of purchasing a support contract from Symantec to ask them if their software is capable of the same thing, but I was hoping someone could clarify the whole transaction log files and the relationship to the daily backups.  For example if someone had an exchange server that they NEVER backed up, would their log files grow and grow until they filled up the hard disk?  Is there a manual way of clearing the log files?  If I understand it correctly, the transaction logs contain all the changes to the server since the last time they were "committed" to the database...is this correct?  Other than using Veritas, how would someone "commit" the changes in the transaction logs to the database manually?

Thanks for the help!
Jeff
0
Comment
Question by:jbobst
  • 6
  • 5
  • 2
13 Comments
 
LVL 104

Expert Comment

by:Sembee
ID: 17110706
Your imaging backup software will not be an Exchange aware backup.
Therefore the logs will never be flushed.
It isn't just logs that are flushed - there are settings within Exchange where data is dropped once it has been backed up.

Veritas has nothing to do with the transaction logs.
Veritas Backup Exec tells Exchange that a full Exchange aware backup has been completed, the logs are flushed.
In the event of a failure of the database, you would restore the last full backup and then replay the transaction logs. The replaying of the transaction logs is a Microsoft function.

If you don't do an Exchange aware backup, then the transaction logs continue to build.
There is a function in Exchange called circular logging. That stops the build up of the transaction logs. However you will get no logs at all, so you will be exposed between the backups taken from the imaging software.

Simon.
0
 
LVL 1

Author Comment

by:jbobst
ID: 17110825
I do a full image of the server each night.  And when we had the tape backup system, I did a full tape backup every night, so the exposure is a maximum of losing the last 24hrs of work with either backup method.  Is sounds like the circular logging might be the way to go.  Would you recommend that?

Am I right that when the transaction logs are flushed, all the changes they hold are committed to the database?  So, in the circular logging mode write any new data directly to the database?
0
 
LVL 10

Expert Comment

by:brakk0
ID: 17110869
You could use veritas (since you already have it) or NT backup (comes with windows) to just backup exchange to a file every night. That file could be saved to another server or drive or something. It would have the benefit of clearing the transaction logs and it never hurts to have an extra backup. You could also use it to rebuild the DB instead of the entire server if needed.
0
 
LVL 104

Expert Comment

by:Sembee
ID: 17112525
I never recommend circular logging on a backend server for any reason.
Circular logging gives you no chance of data recovery. The data is written and the logs flush immediately.

The thing to remember is that your recovery method basically wipes the machine. Therefore you would have to retrieve the transaction logs off the machine in order to replay them. You would be doing a backup just to flush the logs.

Simon.
0
 
LVL 1

Author Comment

by:jbobst
ID: 17140263
I think I am still confused on HOW the logs work.  I have done a little more reading about transaction logs, and I think I was incorrect on my initial understanding.  Apparently, the logs are written, or committed, to the database as soon as the server can do it without sacrificing performance.  So, at any given point during the day, transaction logs can be committed to the database during server idle times.  When an exchange aware backup software, like our old Veritas Backup Exec, does a backup, it tells the server to delete all the old logs.  My original thought was that the backup software told the Exchange server to COMMIT and DELETE all at the same time....meaning until you did an exchange aware backup, non of the changes had been committed up until that point.  Can someone verify this?

Since I backup every night to a hard disk now, and since circular logging is not recommended, is there way to visually tell which logs have been committed?  Then, is there a way to set a time threshold for exchange to keep the logs?  If my current understanding is accurate, I would like to analyze the server for a few days and get an average time on how long it takes the logs to be committed to the database.  For example, if a transaction log typically takes 5 hours to become committed to the database, I could set a 48 hour threshold to keep the log files and be pretty sure that I have given plenty of time for the server to commit the logs.  This of course is dependant on the ability for Exchange to delete the logs on it's own, which I am unsure about.

Why can't exchange do some sort of simple verification that if a log file has been committed to the database, verify the data is there, and then just delete the file?

Thanks
Jeff
0
 
LVL 104

Expert Comment

by:Sembee
ID: 17141215
Your understanding is not quite correct.

The transaction logs are written as Exchange writes to its database. There is no lag between them. If there was, then under performing Exchange servers simply wouldn't write their logs and you would be at risk from data loss in the event of a system failure or power loss.

When you do a backup, the changes are then committed to the database. At that point the transaction logs are not required and are flushed.

The only way that Exchange can delete its own logs is by using circular logging. Otherwise it is down to the Exchange aware backup application.

The logs that are present in the folder are the logs that are uncommitted and those will be flushed after the next backup.

Simon.
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 1

Author Comment

by:jbobst
ID: 17141815
Simon,

If the logs are written as Exchange writes to it's database, then does Exchange write to its database?  Only during a backup?  If someone only backed up their server once a week, you could potentially loose a week's worth of data...right?
0
 
LVL 104

Expert Comment

by:Sembee
ID: 17141961
It writes to both its database and the logs at the same time.
If you only backed up the server once a week, then in the event of a total failure (including all hard disks) you would lose the week's data.

That is one of the reasons why we recommend that Exchange is installed with the databases and logs on separate drives.

Simon.
0
 
LVL 1

Author Comment

by:jbobst
ID: 17142485
Simon,

I had a typo in my previous post...I meant to say "...WHEN does Exchange write to its database?..."  However, it seems from your responses that the the WHEN is only when it's backed up.  Well, if that's my only option, then I suppose I'll have to add some other sort of backup routine that is exchange aware.

Thanks.
Jeff
0
 
LVL 104

Accepted Solution

by:
Sembee earned 250 total points
ID: 17150398
You could run an Exchange server and NEVER back it up. It would work correctly.
Eventually the server will die as the transaction logs would fill up the drive, but until that point you could keep going with Exchange.

ALL that the backups do is tell the database that it has been backed up to that point and that the transaction logs can be flushed. They are no longer required.

The idea being that in the event of a failure, you restore from the last backup and then replay the transaction logs to fill in the gap between the last backup and the the point of failure.

Simon.
0
 
LVL 1

Author Comment

by:jbobst
ID: 17274884
Sorry for such a delay in responding to this question...I am trying to things a little cleaned up in my account now.

I still don't think I fully understand the purpose of the log files and how they are flushed, but in my situation with my server imaging, I now just run a nightly exchange backup using the Windows backup utility.  This is done before the server is imaged each night and seems to take care of any problems.  The next day, after the image has been taken, I have a scheduled task that deletes the exchange backup file before the next backup is run.  That way the backup files won't build up (as they are fairly large).

Thanks for the help.
Jeff
0
 
LVL 10

Expert Comment

by:brakk0
ID: 17278887
uh, that was exactly my suggestion.

and, to try to explain log files briefly: The log files hold a copy of all changes that have been made to the database since it's last backup so, if the database is corrupted or lost, you can restore the last backup and use the log files to get up to date.

You can also configure windows backup to overwrite the last backup file. You shouldn't need to run another task to delete it.

0
 
LVL 1

Author Comment

by:jbobst
ID: 17280790
brakk0,

You did suggest that, and that is why I set it up that way...my apologies for at least not splitting the points with you or not accepting your answer.  I let this thread sit for awhile and didn't think about your answer yesterday when I went to close out this question.

Thank you for the further explanation on log files...I think I finally get it.  You say the log files hold a COPY of all changes...(I think Simon had said the same thing in an earlier post).  I was confused, because everytime I hear about the log files, I got the impression that the log files were NOT committed to the database UNTIL the backup takes places.  Simon was explaning this correctly and I just wasn't getting it for some reason.  Thanks for the clarification...sorry about the points!

Jeff
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Disabling the Directory Sync Service Account in Office 365 will stop directory synchronization from working.
Local Continuous Replication is a cost effective and quick way of backing up Exchange server data. The following article describes the steps required to configure Local Continuous Replication. Also, the article tells you how to restore from a backup…
In this video we show how to create a Contact in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Contact ta…
The video tutorial explains the basics of the Exchange server Database Availability groups. The components of this video include: 1. Automatic Failover 2. Failover Clustering 3. Active Manager

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now