Exchange Transaction logs slowly eating up free space after a backup completes
We are running Exchange 2010.
we started noticing a slow and steady decrease in free space as backups complete and the transaction logs get truncated.
each time they get truncated, a little less free space is freed up after each backup completes.
we have been hitting our heads against the wall trying to figure out what is it that is causing this issue.
to illustrate whats going on please see the graph below
As you can see, the free space starts at the very top and starts to drop as the day goes by with transanction logs. Once backups complete, the graph jumps back up to the top but if you notice, the bar is lowered slightly with each completion of Backups
So are there any good tools to use to find out what exactly is causing this behavior? please bear with me, I am new to exchange.
When i run a get-mailboxdatabasestatus command everything looks healthy.
It appears so far that not all the logs are getting truncated and thats whats causing the issue, but we dont know why this is happening.
we started noticing a slow and steady decrease in free space as backups complete and the transaction logs get truncated.
each time they get truncated, a little less free space is freed up after each backup completes.
we have been hitting our heads against the wall trying to figure out what is it that is causing this issue.
to illustrate whats going on please see the graph below
As you can see, the free space starts at the very top and starts to drop as the day goes by with transanction logs. Once backups complete, the graph jumps back up to the top but if you notice, the bar is lowered slightly with each completion of Backups
So are there any good tools to use to find out what exactly is causing this behavior? please bear with me, I am new to exchange.
When i run a get-mailboxdatabasestatus command everything looks healthy.
It appears so far that not all the logs are getting truncated and thats whats causing the issue, but we dont know why this is happening.
Amit
Can you tell me what is drive size and db size and percent free?
Do you have DAG enabled on your Exchange servers? Exchange 2010 is designed to have very small transaction logs for better performance when shipping the logs to other DAG members.
Also, do you have Anti-virus installed on your Exchange Servers? Are the drives showing hidden files and folders? Are there other hidden objects that might be taking up space?
Will.
Also, do you have Anti-virus installed on your Exchange Servers? Are the drives showing hidden files and folders? Are there other hidden objects that might be taking up space?
Will.
ASKER
sorry yes we have a DAG enabled i forgot to mention this. sorry about that.
the Drive Size is 159GB total. 103GB FreeSpace
Please see the Screeen Shot below for the DB info
Yes we do have anti-virus installed and the drives are showing HIdden files/folders.
the Drive Size is 159GB total. 103GB FreeSpace
Please see the Screeen Shot below for the DB info
Yes we do have anti-virus installed and the drives are showing HIdden files/folders.
The first thing to do is run the following commands:
get-databaseavailabilitygr
then
get-mailboxdatabase | get-mailboxdatabasecopysta
Ensure that it comes back healthy/mounted on all servers in the DAG and all members are listed.
This is a dedicated drive for the transaction logs?
Anything else stored on the drive?
Are shadow copies enabled on the drive?
Simon.
get-databaseavailabilitygr
then
get-mailboxdatabase | get-mailboxdatabasecopysta
Ensure that it comes back healthy/mounted on all servers in the DAG and all members are listed.
This is a dedicated drive for the transaction logs?
Anything else stored on the drive?
Are shadow copies enabled on the drive?
Simon.
In the Application event log, are there any errors or messages showing up for the log truncation? Any errors saying "no log files can be truncated"?
ASKER
yes this is the dedicated drive for transaction logs.
Shadow Copies are disabled on the drive
Everything looks healthy/mounted and all members are listed.
Shadow Copies are disabled on the drive
Everything looks healthy/mounted and all members are listed.
ASKER
looking at the event viewer now bare with me
ASKER
not finding anything in the even viewer logs
Which drive these DB's are located? C:?
ASKER
these DB's reside on Drives G, H, I, and J
Can you paste the screeshot from disk management.
Just out of curiosity, which type of backup are you doing on the database? Full, incremental, etc.?
ASKER
we are doing incremental's mon - thursday and on friday, fulls
each backup runs at 11:30pm we are using commvault v 9 for backups
each backup runs at 11:30pm we are using commvault v 9 for backups
ASKER
I see you have different drives for logs. That is e drive. Also, if you do rough calculation, db's are using 67% diskspace
257 GB DB size total.
385 GB total size disk size (g+h+i+j)
80 GB Free space disk free (g+h+i+j)
I can see you have a very low disk allocated to Exchange db's. Just 100GB max. Which is very low. I would suggest to increase them to 200 GB.
I personally don't see any issue related to space. It is more design issue to me. You use Exchange 2010 calculator and redesign this server again.
Remember, you need to plan server for next 4-5 years requirement. It is always good to have 20% free space, if it is below that, that means you need to increase disk space.
Also, check drives for any hidden folder.
257 GB DB size total.
385 GB total size disk size (g+h+i+j)
80 GB Free space disk free (g+h+i+j)
I can see you have a very low disk allocated to Exchange db's. Just 100GB max. Which is very low. I would suggest to increase them to 200 GB.
I personally don't see any issue related to space. It is more design issue to me. You use Exchange 2010 calculator and redesign this server again.
Remember, you need to plan server for next 4-5 years requirement. It is always good to have 20% free space, if it is below that, that means you need to increase disk space.
Also, check drives for any hidden folder.
ASKER
this just started happening like a week or 2 ago. before we would have a steady amount of freespace that would always be the same after backups completed. But now we are seeing a slow steady decrease each time.
so no, we should not have to redesign anything.
but it is definitely somthing to think about moving forward. However this is not the culprit causing the issue.
i need help determining what is causing this
so no, we should not have to redesign anything.
but it is definitely somthing to think about moving forward. However this is not the culprit causing the issue.
i need help determining what is causing this
I'd check your Commvault logs, see if anything sticks out. If there's an error, some log somewhere is going to mention something about it (provided logging is enabled to the right level). The log is extidbbackup.log. I don't know much about Commvault, but I got some ideas reading up here:
http://sso.forum.commvault.com/forums/15946/ShowThread.aspx
http://sso.forum.commvault.com/forums/15946/ShowThread.aspx
If it is a physical server, you restart it and then check the space again. Some time, RAID caches are stored on drives. Which are not visible by default. With reboot it get cleared.
Why are you doing daily backups, and incremental at that, when you have a DAG?
I only ever do full backups, I cannot be bothered with incremental at all - for a successful recovery you need all of the incremental backups to be successful. One doesn't work and you are stuffed (and backups not being good are very common).
Furthermore what do they do for you? You aren't going to use them for recovery - that is what the DAG is for.
If you need them for data retention, then do a backup once a week or a couple of times a week. A full backup, not incremental. Increase the DIR time on the databases to 14 days with the option to not remove the content until backup is complete and you have covered yourself for every eventuality.
The only reason I can see for this issue with regards to Exchange would be if the DAG is running on a delay replication rather than live. The logs are flushed after BOTH the backup has completed and the DAG has reported successful replication. If you have any issues with replication that could explain the build up.
Have you looked at the time stamps of the log files? The oldest one should be somewhere around the start time of the last backup. If that is the case then the problem is almost certainly outside of Exchange.
Simon.
I only ever do full backups, I cannot be bothered with incremental at all - for a successful recovery you need all of the incremental backups to be successful. One doesn't work and you are stuffed (and backups not being good are very common).
Furthermore what do they do for you? You aren't going to use them for recovery - that is what the DAG is for.
If you need them for data retention, then do a backup once a week or a couple of times a week. A full backup, not incremental. Increase the DIR time on the databases to 14 days with the option to not remove the content until backup is complete and you have covered yourself for every eventuality.
The only reason I can see for this issue with regards to Exchange would be if the DAG is running on a delay replication rather than live. The logs are flushed after BOTH the backup has completed and the DAG has reported successful replication. If you have any issues with replication that could explain the build up.
Have you looked at the time stamps of the log files? The oldest one should be somewhere around the start time of the last backup. If that is the case then the problem is almost certainly outside of Exchange.
Simon.
ASKER
but if incrementals dont happen then how do the transaction logs get truncated???
ASKER
the oldest log file i can see is 5-26-2015 @ 10:25PM (backups start at 11:30pm)
Logs are truncated by doing a full backup.
Are you doing full backups as well?
The log time is not far off what I would expect, the email flow is probably lower at that time of night and therefore a new log wasn't generated afterwards.
Simon.
Are you doing full backups as well?
The log time is not far off what I would expect, the email flow is probably lower at that time of night and therefore a new log wasn't generated afterwards.
Simon.
ASKER
yes we are doing full backups every friday. But after every incremental the logs truncate as well.
we are leaning towards it being Active Sync. We have used a tool called Log Parser that allows us to view the hits to certain users and we can see 1 or 2 user with 70,000 hits or more. We are thinking that this is whats causing the increase in transaction logs.
we are leaning towards it being Active Sync. We have used a tool called Log Parser that allows us to view the hits to certain users and we can see 1 or 2 user with 70,000 hits or more. We are thinking that this is whats causing the increase in transaction logs.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You should look into IIS log folders. There you will see the log consumed disk space. I created a script which removes last 90 days logs or can be customized according to the need. Can you post IIS log folder size screen shot.
Or you can use treesize tool to check folder sizes.
Or you can use treesize tool to check folder sizes.
ASKER
This is more than likely what is happening.Backups cleaned up the logs this morning and now we are seeing the free space jump back to where it should be. we will use the script from the link Simon Butler posted and i will get Amit the screen shot he asked for going forward. Thanks!