Exchange Transaction logs slowly eating up free space after a backup completes

We are running Exchange 2010.

we started noticing a slow and steady decrease in free space as backups complete and the transaction logs get truncated.

each time they get truncated, a little less free space is freed up after each backup completes.

we have been hitting our heads against the wall trying to figure out what is it that is causing this issue.

to illustrate whats going on please see the graph below
Transaction Logs
As you can see, the free space starts at the very top and starts to drop as the day goes by with transanction logs.  Once backups complete, the graph jumps back up to the top but if you notice, the bar is lowered slightly with each completion of Backups

So are there any good tools to use to find out what exactly is causing this behavior?  please bear with me, I am new to exchange.

When i run a get-mailboxdatabasestatus command everything looks healthy.  

It appears so far that not all the logs are getting truncated and thats whats causing the issue, but we dont know why this is happening.
JB BlancoSr Systems EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AmitIT ArchitectCommented:
Can you tell me what is drive size and db size and percent free?
0
Will SzymkowskiSenior Solution ArchitectCommented:
Do you have DAG enabled on your Exchange servers? Exchange 2010 is designed to have very small transaction logs for better performance when shipping the logs to other DAG members.

Also, do you have Anti-virus installed on your Exchange Servers? Are the drives showing hidden files and folders? Are there other hidden objects that might be taking up space?

Will.
0
JB BlancoSr Systems EngineerAuthor Commented:
sorry yes we have a DAG enabled i forgot to mention this.  sorry about that.

the Drive Size is 159GB total.   103GB FreeSpace

Please see the Screeen Shot below for the DB info

Exchange DB info
Yes we do have anti-virus installed and the drives are showing HIdden files/folders.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Simon Butler (Sembee)ConsultantCommented:
The first thing to do is run the following commands:

get-databaseavailabilitygroup -Status

then

get-mailboxdatabase | get-mailboxdatabasecopystatus

Ensure that it comes back healthy/mounted on all servers in the DAG and all members are listed.

This is a dedicated drive for the transaction logs?
Anything else stored on the drive?
Are shadow copies enabled on the drive?

Simon.
0
R TCommented:
In the Application event log, are there any errors or messages showing up for the log truncation?  Any errors saying "no log files can be truncated"?
0
JB BlancoSr Systems EngineerAuthor Commented:
yes this is the dedicated drive for transaction logs.

Shadow Copies are disabled on the drive

Everything looks healthy/mounted and all members are listed.
0
JB BlancoSr Systems EngineerAuthor Commented:
looking at the event viewer now bare with me
0
JB BlancoSr Systems EngineerAuthor Commented:
not finding anything in the even viewer logs
0
AmitIT ArchitectCommented:
Which drive these DB's are located? C:?
0
JB BlancoSr Systems EngineerAuthor Commented:
these DB's reside on Drives G, H, I, and J
0
AmitIT ArchitectCommented:
Can you paste the screeshot from disk management.
0
R TCommented:
Just out of curiosity, which type of backup are you doing on the database?  Full, incremental, etc.?
0
JB BlancoSr Systems EngineerAuthor Commented:
we are doing incremental's mon - thursday and on friday, fulls

each backup runs at 11:30pm  we are using commvault v 9 for backups
0
JB BlancoSr Systems EngineerAuthor Commented:
disk manager
0
AmitIT ArchitectCommented:
I see you have different drives for logs. That is e drive. Also, if you do rough calculation, db's are using 67% diskspace

257 GB DB size total.

385 GB total size disk size (g+h+i+j)

80 GB Free space disk free (g+h+i+j)

I can see you have a very low disk allocated to Exchange db's. Just 100GB max. Which is very low. I would suggest to increase them to 200 GB.

I personally don't see any issue related to space. It is more design issue to me. You use Exchange 2010 calculator and redesign this server again.

Remember, you need to plan server for next 4-5 years requirement. It is always good to have 20% free space, if it is below that, that means you need to increase disk space.

Also, check drives for any hidden folder.
0
JB BlancoSr Systems EngineerAuthor Commented:
this just started happening like a week or 2 ago.  before we would have a steady amount of freespace that would always be the same after backups completed.  But now we are seeing a slow steady decrease each time.

so no, we should not have to redesign anything.

but it is definitely somthing to think about moving forward.  However this is not the culprit causing the issue.

i need help determining what is causing this
0
R TCommented:
I'd check your Commvault logs, see if anything sticks out.  If there's an error, some log somewhere is going to mention something about it (provided logging is enabled to the right level).  The log is extidbbackup.log.  I don't know much about Commvault, but I got some ideas reading up here:

http://sso.forum.commvault.com/forums/15946/ShowThread.aspx
0
AmitIT ArchitectCommented:
If it is a physical server,  you restart it and then check the space again. Some time, RAID caches are stored on drives. Which are not visible by default. With reboot it get cleared.
0
Simon Butler (Sembee)ConsultantCommented:
Why are you doing daily backups, and incremental at that, when you have a DAG?
I only ever do full backups, I cannot be bothered with incremental at all - for a successful recovery you need all of the incremental backups to be successful. One doesn't work and you are stuffed (and backups not being good are very common).
Furthermore what do they do for you? You aren't going to use them for recovery - that is what the DAG is for.

If you need them for data retention, then do a backup once a week or a couple of times a week. A full backup, not incremental. Increase the DIR time on the databases to 14 days with the option to not remove the content until backup is complete and you have covered yourself for every eventuality.

The only reason I can see for this issue with regards to Exchange would be if the DAG is running on a delay replication rather than live. The logs are flushed after BOTH the backup has completed and the DAG has reported successful replication. If you have any issues with replication that could explain the build up.
Have you looked at the time stamps of the log files? The oldest one should be somewhere around the start time of the last backup. If that is the case then the problem is almost certainly outside of Exchange.

Simon.
0
JB BlancoSr Systems EngineerAuthor Commented:
but if incrementals dont happen then how do the transaction logs get truncated???
0
JB BlancoSr Systems EngineerAuthor Commented:
the oldest log file i can see is 5-26-2015 @ 10:25PM   (backups start at 11:30pm)
Log Files
0
Simon Butler (Sembee)ConsultantCommented:
Logs are truncated by doing a full backup.
Are you doing full backups as well?

The log time is not far off what I would expect, the email flow is probably lower at that time of night and therefore a new log wasn't generated afterwards.

Simon.
0
JB BlancoSr Systems EngineerAuthor Commented:
yes we are doing full backups every friday.  But after every incremental the logs truncate as well.

we are leaning towards it being Active Sync.   We have used a tool called Log Parser that allows us to view the hits to certain users and we can see 1 or 2 user with 70,000 hits or more.  We are thinking that this is whats causing the increase in transaction logs.
0
Simon Butler (Sembee)ConsultantCommented:
Have you been using this technique?
http://blogs.technet.com/b/exchange/archive/2012/01/31/a-script-to-troubleshoot-issues-with-exchange-activesync.aspx

ActiveSync is the most common cause of rapid log growth.

I know incremental flushes the logs - but to restore the backup you would need to restore the last full, then the incremental. If you are trying to get back to Thursday, you would be hoping that the four incremental backups are successful. Although you shouldn't need to do a restore - that is what the DAG is for.

Simon.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AmitIT ArchitectCommented:
You should look into IIS log folders. There you will see the log consumed disk space. I created a script which removes last 90 days logs or can be customized according to the need. Can you post IIS log folder size screen shot.

Or you can use treesize tool to check folder sizes.
0
JB BlancoSr Systems EngineerAuthor Commented:
5-28-2015-2-11-47-PM.pngThis is more than likely what is happening.

Backups cleaned up the logs this morning and now we are seeing the free space jump back to where it should be.  we will use the script from the link Simon Butler posted and i will get Amit  the screen shot he asked for going forward.  Thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.