Types of Defragmentation in Exchange Server databases

What is the difference between an Exchange Server Online and Offline Defragmentation of the Exchange databases?

Many Exchange Administrators do not know the answer to this common question, probably because they have never had the need to think about defragmenting their Exchange databases.  The truth is, the difference is actually a lot; the two tasks are executed at different times, by different routines, and are used in different situations.

Before we delve into the two methods of defragmentation, we must examine how the Exchange databases work.  

For each Exchange Server mail store, Exchange creates a .EDB file and .STM file on the Hard Disk Volume of the Microsoft Exchange Server computer.  These files store the email mailboxes of all users with mailboxes residing in that particular mail store, and their collective size (size of EDB file + size of STM file) determines the total size of the Exchange stores.  When a vigilant email administrator is comparing the size of the database files on disk with the total size of mailboxes as listed in the Exchange System Manager, or Exchange Management Console for Microsoft Exchange Server 2007, they usually show up here on Experts Exchange and ask:

Why are my database files on disk much larger than the total size of my mailboxes in the Exchange Management Utility?

The behaviour they are seeing is intentional.  In a nutshell, Exchange Server makes use of empty space in its mail store databases - essentially database pages which do not store anything. Nothing.  It is blank space within the database file, which appears on disk to be used by the database file (since the space is reserved on the disk).  

The reason behind the use of this white space all comes down to efficiency and performance of the Exchange Server.  If Exchange needed to dynamically resize the database every time someone sent a message to the server, or shrink it every time someone deleted a message... well, you can see the problems which could unfold with Disk I/O.  Thus, the blank space in the database is space which Exchange foresees it may need to use, so it expands the database before it actually needs that space; the whitespace is also created when something is purged from the database, since Exchange does not automatically shrink the database files.

So how does this relate to defragmentation methods?

Online Defragmentation in Exchange Server is a process which, by default, is performed automatically by the server, usually in the middle of the night when the server is under little to no load.  The process is essentially a routine which scans the database, purging any data which is no longer required.  No, it doesn't decide to purge your precious email.  It purges only deleted email.  

When you permanently delete an email from your Deleted Items folder, it usually remains on the server for a period of between 7 - 30 days (can be longer) to enable quick recovery if necessary; once an email has been deleted for longer than this retention period, Online Defrag will properly purge the message from the database.  The same concept happens for deleted mailboxes, and these are also handled by this process.

When Online Defragmentation performs this operation, it frees up whitespace in the database, since, as I mentioned earlier, the database size will not shrink.  Normally, this is not a problem with small volumes of data being deleted; the white space generated will quickly be used up by more incoming mail, or data which is otherwise added to a user's mailbox (in Calendar items, Tasks, Sent Items and so on).  The problem occurs when you have a large volume of whitespace being created quickly in the database - for example, if you delete a lot of users who had large mailboxes, or if you move a lot of user mailboxes to another mail store or server.

This is the time when Offline Defragmentation is required.  An Offline Defrag, performed using the Eseutil.exe (Exchange Server Database Utilities) tool, essentially creates a new temporary mail store and copies all data from the existing mail store into the new mail store file.  It then purges the old mail store from disk, then performs a rename on the new mail store so its name matches the name of the old mail store.  This reorganises the data in the store and reduces the size of the store if a large amount of whitespace was present.  

This is worthwhile if you notice your mailstores are significantly larger than the size of your mailboxes added together (remember it is the size of the EDB and the STM file, added together), since an offline defrag could enable you to reclaim precious disk space.  However, the only downside to it is due to the method by which it works, the mail stores must be dismounted - sometimes for several hours - resulting in disrupted mail flow and the inability for users to access their email.

It is only recommended you do OffLine Defrag if you need the disk space, or if there is several gigabytes of whitespace in the stores which you do not anticipate being used very quickly.  Furthermore, it is only critical to perform the procedure if you are running Exchange 2000/2003 Standard Edition and it is the only Exchange Server on the network.  If you run Exchange 2000/2003 Enterprise, or any edition of Exchange 2007, you can create a new mailbox store, move mailboxes to this store, then drop the old store.  The end result is zero risk and zero downtime.

To read more about the eseutil utility, and all its switches and their function, please see http://technet.microsoft.com/en-us/library/aa998249.aspx.

-Matt