Exchange 2013 logs 1 TB

I have - 1 TB in exchange logs for one database.  About 4 x day ago the LOG volume ran out of storage.  I added Storage and ran a LOG Exchange backup job, however, the LOGs are still 1 TB in size...  What should I do?  Also..  Will I need 1 TB free on the database store one these get committed?
gopher_49Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gopher_49Author Commented:
Also..  It keeps on growing...  Around 10 GBs every hour...
0
timgreen7077Exchange EngineerCommented:
the first thing I would suggest is the enable circular logging on the database. that will clear the logs and not allow it to grow like that. then look at why your backup job isn't clearing the logs once the backup job completes. circular logging is safe and you can keep it enabled if you choose or you can resolve your backup issue and then disable it. enabling it will quickly clear the log space.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
gopher_49Author Commented:
I started a VEEAM VSS truncate backup job.  My offsite backup job was supposed to be doing the truncating and VEEAM was copy only.  For some reason this one particular DB isn't truncating..  When the current VEEAM VSS job is done we'll see if it truncates..  If not I at least have a current copy of the Exchange server prior to enabling circular logging.  If I resort to circular logging where do I need to look in regards to the backup job not truncating?  I assume the System or application logs?
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

gopher_49Author Commented:
Question.  Don't I need whatever Storage size is being used in the log folders available on my database drive?
0
timgreen7077Exchange EngineerCommented:
application and system logs is where I always look for any issues. Also look at logs from the back server or offsite to see if it shows anything.

The log file size doesn't need to be the same size of your db. if you have a 1 TB db, you should be fine with a 300 GB log file size as long as the backup are clearing the logs afterwards. if the logs aren't being cleared then it doesn't matter what size the log drive size is because it will eventually run out of space. enable circular logging until then.

I would suggest removing the backup job for that DB and
 re-creating it to see if that helps.
0
gopher_49Author Commented:
What I meant was this...  Drive E: has 800GBs in log files not committed to the database.  Does this mean the drive D: that the database resides on needs 800GBs available?
0
timgreen7077Exchange EngineerCommented:
No. the reason the log size is so large is because they are not being cleared but the email has already been written to the database. how much capacity and free space does your db show.
0
Lee W, MVPTechnology and Business Process AdvisorCommented:
In my opinion, Circular Logging is bad advice - the logs are there for a reason - to restore the emails lost in the event the exchange database is lost.  They should be cleared properly if you backup properly.  I don't use Exchange 2013 (went from 2010 to 2016) and I don't use Veeam (I use Altaro) but I would expect if you setup the backup job properly it will clear the logs.

As for why they are growing so fast, look at your mail inflow - could it be spam?  NDRs?  or do you just have a lot of users sending and receiving a lot of mail?
0
timgreen7077Exchange EngineerCommented:
circular logging isn't bad advice. it's recommended advice if your logs are growing exponentially and you don't know why. you can't keep adding space no stop. circular logging is a suggested method to keep from killing your exchange server until you can resolve issue, that's why it's there.
0
Lee W, MVPTechnology and Business Process AdvisorCommented:
It's bad advice in my opinion.  You don't IGNORE the problem and enable circular logging - you FIND the problem and FIX it.  Maybe they need to put a spam filter in place?  Maybe a virus is sending out spam and you don't know it.  Don't ignore the cause - find it and fix it.
0
timgreen7077Exchange EngineerCommented:
Who says ignore the problem. circular allows you time to find the problem without crashing exchange due to logs taking all your drive. hint hint that's why it's there.
0
Lee W, MVPTechnology and Business Process AdvisorCommented:
Really?  Is that why we're here?
If the log files are growing at 10GB/hour then that's 1 TB every 4 days.  A lot, sure.  But not unmanageable especially when the OP indicates there's already 1 TB there...  cleared with a proper nightly backup.  There's no need for circular logging at this point.  Before advising someone does something that puts their recovery at risk, take into consideration if, given the information provided, the advice is really good advice.

https://blogs.technet.microsoft.com/exchange/2010/08/18/exchange-circular-logging-and-vss-backups/

It's clear you disagree with me.  So be it... I've made my argument.  And my recommendations - find out what's going on and fix it as well as ensuring the backups are running properly.
0
Jose Gabriel Ortega CastroCEOCommented:
I recommend you 3 steps:

step 1. Do a Full backup of your server.

Step 2. Check the sizes of the logs and DB, if EDB file is slightly smaller than LOG size reported by the script
https://gallery.technet.microsoft.com/office/What-is-the-size-of-my-0e1993a7

step 3. Enable Circular logging on your database.
https://gallery.technet.microsoft.com/office/Enable-CircularLogging-In-beaaa3bd

After reading, I think lee has right, there's no point in growing at that rate, there must be some kind of issue, check in event viewer, before trying this steps.
0
Adam BrownSr Solutions ArchitectCommented:
Just so you know, Exchange Transaction logs are "committed" to the DB almost immediately after being written. http://acbrownit.com/2014/03/20/exchange-transaction-logs-reducing-the-confusion goes over it in more detail, as well as explaining the purpose of the files, and what they're for (TL;DR, entirely for point-in-time recovery). If you're generating 10GB of logs per hour, I'd recommend expanding to a multi-server deployment.
0
gopher_49Author Commented:
The VEEAM VSS job finished.  It cleared up 700 GBs of logs, however, there are currently 280 GBs of logs for that database.  I'll watch to see why it keeps on growing.  We don't have a lot of email traffic.  Maybe an internal loop?  We are constantly changing email rules.  I thought exchange had protection from now...  Now..  I have a product called GFI MailArchiver.  It reads each sub folder of every user and makes note of where the archived email is to be located.  It preserves Outlook folders in the mail archived...  This results in large IIS files but wouldn't think it would mess with transaction logs.
0
timgreen7077Exchange EngineerCommented:
how does your transport queue look? does it seem to get backed up or over whelmed with mail. are users compalining about spam or anything. are you using any type of spam and malware filtering. its good the veeam  backup cleared the logs. I would run it one more time to see if it completely clears the remaining logs it of some logs remained.
0
gopher_49Author Commented:
We have close to no spam slipping through on busy days...  Over the last few days Email traffic is close to nothing.  I'll check transport rules queue shortly, however, the growth of the log folder has slowed down a lot. I'll reply soon with more updates
0
timgreen7077Exchange EngineerCommented:
good to hear.
0
MB ShaikhCommented:
Hi
Everyone the same issue i am also facing with the VEEAM Backup Application, if you run the backup with native windows backup tools, it will purge the logs successfully, so do a full backup of all the databases by windows backup, it will purge all the logs then you start using your regular VEEAM backup tool, this happen to me and MS Technical Support Team also involved but noting wrong found in exchange.

In fact may there are some logs missed during earlier truncation etc. so make sure all your VSS Writes status. by the below command

From command promt type the below

VSSAdmin List Providers

To display all provider

VSSAdmin List Writers

To display all Writers and their Status

Note :- In most of the cases in our issue the Microsft Exchange Writer was failed and retryable so the solution for this

Just restart the service that the writer was associated with and/or fix whatever configuration issue is causing the failures.  For example, given the above output I would restart the
Exchange Replication Service in an attempt to return the writer to a Stable No Error state.  

Note :- you can restart Micorsoft Exchange Replication service without any interuption,
but Information Store Service will have interuption so take care which is even not needed here.

(If it would have been the Microsoft Exchange Writer I would have restarted the Exchange Information Store Service).
0
gopher_49Author Commented:
How do you run an Exchange only back via the windows backup tool?  

Also,

The DB that won't purge is 1.7 TBs and I don't have the Database Size Limit reg keys in place.  I just added them.  What do I need to do for it take affect?  Default limit is 1 TB.  Maybe that's why my logs have grown so large for this DB?  It's too big to commit the logs?
0
timgreen7077Exchange EngineerCommented:
This will guide you in backing up Exchange using Windows backup server. make sure the windows backup server feature is installed first.

https://technet.microsoft.com/en-us/library/dd876854(v=exchg.150).aspx


You can right click the DB and look at the properties to see how much storage capacity you have and what's used.
0
gopher_49Author Commented:
If my database has reached it allowed Database Size Limit won't it stop allowing logs to be committed and result in logs building up?
0
timgreen7077Exchange EngineerCommented:
Yes and it will dismount the DB.
0
gopher_49Author Commented:
So..  I added the 'Database Size Limit' REGWORD key.. I set it to 3000 GBs..  How do I make this take affect while DB is still mounted?  I followed the guide at https://support.microsoft.com/en-us/help/3059008/exchange-server-2013-standard-edition-can-t-mount-databases-that-are-l
0
timgreen7077Exchange EngineerCommented:
most registry changes take affect after computer reboot.
0
gopher_49Author Commented:
So,

VEEAM VSS Truncate job is done...  All but 200 GBs is purged in the logs...  I guess the only way to get rid of these logs is to enable circular logging?  No matter what there is always 200 GBs left over...
0
Jose Gabriel Ortega CastroCEOCommented:
Yes, it is. then you can disable it again.
0
gopher_49Author Commented:
Assuming logs don't grow really large each day I'll go ahead and hold off until Friday evening.. That way I have more time if something goes wrong and I execute my back out plan...  Previously logs where growing so quick that even a work shift was a problem..
0
Jose Gabriel Ortega CastroCEOCommented:
You need to find out the reason why they were growing.
You need to check the event logs and find out why it is growing like that.
Enabling the Circular logging is a workaround but the real issue is not that one.
0
gopher_49Author Commented:
The only thing I see is that the affected DB was out of logical space.  It's been going on for a long time.  I utilized the reg key to increase allowed Exchange DB size.  It seems it was blocking log files from being committed.  Not sure to be honest... Thats the only error in the event log aside from cert errors
0
timgreen7077Exchange EngineerCommented:
Yes I suggest enabling circular logging to clear those log files Nd then re-enable it
0
gopher_49Author Commented:
The problem is this...  there are so many files to purge.  How long will it be offline if there are 100,000+ files to purge?
0
timgreen7077Exchange EngineerCommented:
once you enable circular logging it will immediately start cleaning up the logs. only a few minutes
0
gopher_49Author Commented:
Ok.  I might do it at the end of tomorrow's work shift
0
gopher_49Author Commented:
I enabled circular logging..  I had to dismount and Mount.  But theirs are still there?  Do I have to manually delete them?
0
gopher_49Author Commented:
Never mind...  They are now all purged...  So...  Tomorrow after I take a full backup with truncate enabled should I re-enable it?
0
Jose Gabriel Ortega CastroCEOCommented:
Well, circular logging is not a thing to play with.
if you want to leave it enabled it, just do it, it's not something that you should be enabling and disabling every day because you can get corruption in your database and then you will have to pay 1000$USD software to recover PST from EDB, if you were unable to find what is the source of the problem, hire a freelancer.
0
gopher_49Author Commented:
Jose,

For now I enabled circular logging just until I troubleshoot further..  It was to the point I had to.. I was simply running out of storage to allocate to the VM..  I had no choice.  I'm considering opening a ticket with Microsoft..  I have around 2 TBs of DB storage being used, however, my mailbox report only shows 800 GBs of mailbox usage?  This does not include logs...  That alone is an issue..  In regards to the logs.. So..  I'm not sure to why this particular DB was growing so large...  I tried two different backup utilities and they never could fully purge the logs...  What do you recommend my next step is?
0
gopher_49Author Commented:
Now my DB is getting even bigger..  something is wrong
0
timgreen7077Exchange EngineerCommented:
how many mailboxes do you have on that DB. also what are the sizes of those mailboxes.
0
gopher_49Author Commented:
The total size equals right at 1TB but I'm showing 2.3TBs of usage...  It's growing as I type this...
0
timgreen7077Exchange EngineerCommented:
Question is how many mailboxes do you have on that DB and how large are those mailboxes? If you need assistance finding this info out let me know.
0
gopher_49Author Commented:
I generated a report .  I have a total of 140 mailboxes across all DBs.  They total 800 GBs total ...  On this DB I have a mailbox that is 80 GBs on that DB...  And a few that are 50 GBs...  Like two more.  Send command to generate stat for specified DB and I'll run it...
0
timgreen7077Exchange EngineerCommented:
Run the following:

Get-Mailbox -Database "DB Name" | Get-MailboxStatistics | ft displayname, totalitemsize
0
timgreen7077Exchange EngineerCommented:
Run the cmd just for the DB with the issue.
0
gopher_49Author Commented:
I ran it...  One mailbox is huge but all total way under the 2.1 TB data file size.  The .edb is 2.1 TBs
0
gopher_49Author Commented:
I don't know where all of the extra storage is coming from.  The .edb alone is 2.1 TBs...  All mailboxes across the entire server is 800 GBs
0
timgreen7077Exchange EngineerCommented:
well you aren't answering my question.  look into moving the mailboxes to a different DB if the other DBs have capacity. that way you can remove that DB and recreate.
0
gopher_49Author Commented:
I don't have enough storage...  Ill have to mount a different iSCSI appliance..
0
gopher_49Author Commented:
I'm driving but I would say there are 500 GBs of mailboxes on that DB
0
gopher_49Author Commented:
I can use my log volume.  It will be slow for cache is disabled but there is a lot of space there.  Question.  When moving mailboxes to a new DB how much storage in logging is needed ?  Can I use circular logging on the new target DB to void huge logs?
0
timgreen7077Exchange EngineerCommented:
Yes you can use circular logging. enable it before starting the moves. if the mailboxes are large I would recommend moving only 1 or 2 at a time because moves can be really taxing on exchange. I would recommend moving after business hours. also pause the moves during your backup hours and resume afterwards.
0
gopher_49Author Commented:
Sounds like a plan.  Reads will be slow but writes are fine.  Do you know if a reason that I might have a problem since caching is turned off on the target volume?
0
timgreen7077Exchange EngineerCommented:
I'm not a storage guy but it shouldn't matter. just move large mailboxes 1 or 2 at a time only. also make sure that there are no hard limits set on the DB that may cause the move to fail. those DB limits are set in exchange so make sure there are no limits since you have really large mailboxes.
0
gopher_49Author Commented:
I'll disable size limits and increase allowed errors.  I'm making a current backup now...  It takes forever since I'm using reverse incremental and now that the virtual disks are so big...  But..  I'll add 100 GBs of Storage to the DB volume.  That way tonight I can sleep knowing that it's good on space...  Then I'll move tomorrow after hours.  Some I can do during work hours.  I'll try a small one to see if it can handle it...
0
timgreen7077Exchange EngineerCommented:
Good luck.
0
gopher_49Author Commented:
I'm onto something .  The process MSExchangeDelivery.exe is writing constantly...  What does this tell us?
0
gopher_49Author Commented:
IIS is constantly writing also
0
timgreen7077Exchange EngineerCommented:
MSExchangeDelivery.exe accepts email messages from the Transport service and delivers the email message to the mailbox.
0
gopher_49Author Commented:
I have OpManager that monitors via PowerShell.  I see thousands of transactions from it..  I turned it off for now...
0
timgreen7077Exchange EngineerCommented:
How is your transport queues. I wonder if you are being slammed with spam.
0
timgreen7077Exchange EngineerCommented:
how many exchange servers do you have?
0
gopher_49Author Commented:
Only one...  Tomorrow morning when backups are done I'm moving mailboxes...  This DB seems to be messed up...  Not sure if I'll have enough time..  The most important mailbox is 80 GBs and will take forever to migrate
0
timgreen7077Exchange EngineerCommented:
Why did you allow that mailbox to grow to that size. Its hard to migrate and restore if ever needed. Yep move the mailboxes and scrap that DB. That is the best option at this point. How are the system resources on your exchange server. Whats the CPUs and RAM.
0
gopher_49Author Commented:
It just started to expand this large over the last week.  I didn't realize. It's constantly growing.  How can I instantly free up space after mailboxes migrate?  Also..  Will it use up space on the source DB when migrating to new target?
0
timgreen7077Exchange EngineerCommented:
you can enable the archive mailbox for that large mailbox and move some mail from the primary mailbox to the archive.
0
gopher_49Author Commented:
What if I run an Outlook rule and purge?  I have a ton of emails I can purge.. Like 30 GBs worth..  I can run a permanent delete rule?  How can I force that storage to be released?  I set the DB to keep deleted items for 0 days.  Does this mean one deleted from deleted items the mailbox will get smaller?  Or..  Is the white space still there and simply means I can migrate quicker?
0
timgreen7077Exchange EngineerCommented:
Storage will not show to be free any more, but exchange will reuse that free space regardless.
Yes purge whatever you don't need in order to shrink the mailbox size.  Once you purge the mail, run the belowcommand so that you can clean out the recovery dumpster so that when you migrate the mailbox, the recovery dumpster won't migrate with 30GB of mail that you deleted. Remember once you clear mail from the deleted items folder it goes to the recovery dumpster for a set amount of days before exchange purges that dumpster, and those days are set on the exchange server. run this command to clear the recovery dumpster after your purge:

Search-Mailbox -Identity username -SearchDumpsterOnly -DeleteContent -Confirm:$false -Force
0
gopher_49Author Commented:
I set keep deleted items for 0 days for the entire DB.  Do I still need to run that command ?
0
timgreen7077Exchange EngineerCommented:
nope. no need to run it then.
0
gopher_49Author Commented:
ok..  I'm starting to purge now..  VEEAM VM level ESXi backups are running..  I guess purging will slow the backups down but need to head a head start.
0
gopher_49Author Commented:
I think I found the problem.. I'm running an Outlook that permanently deletes any email older than 3 x months from the 83 GB mailbox.  I noticed after running it for about 20 mins by storage quit getting eaten up..  And the .edb didn't increase in size...  It's held the same size for almost 30 mins now..  At minimum it's growing much slower..  At this rate it will easily buy me enough time to migrate all mission critical mailboxes..  I've learned my lesson.. I'm getting cap to 50 GBs on the 3 x users that need large mailboxes.. I'm 99% sure this is where the corruption was caused...  As soon as the mailbox was trimmed down all the sudden by DB growth issues went away...

I have a few questions..  

As I understand the migration process of Exchange 2013 is more of a sync..  Once it catches up it finalizes... Only during the finalization process is the mailbox unavailable, correct?

In regards to my other two DBs.. I plan to move them to a different drive letter so I can purge the messed up DB and the old drive.  Should I enable circular logging before moving these two DBs to void a bunch of logs being generated?
0
timgreen7077Exchange EngineerCommented:
That is correct, the finalization process is cutover to Exchange 2013 is actually happening. The user can actually be in outlook the entire time and outlook will prompt them to restart outlook. I'm currently migrating Exchange 2010 users to Exchange 2016 and I auto suspend the mailbox at 95% and during non business hours I complete the move so that users don't have to get prompted to restart outlook during the cutover process and finalization process. You can do either because once they restart outlook after the prompt they are generally fine.

Why do you want to move the DBs to a different drive letter. If you do this they will dismount because Exchange will be looking for the old path. I wouldn't suggest moving them to a different drive unless you know to move your DBs and TLogs to a different and fix it in Exchange.  If you are comfortable doing it fine, but be careful. No actual need to change drive letter. I would just move the mailboxes to different DB and scrap the problem DB and recreate it.
0
gopher_49Author Commented:
After creating new DBs and moving mailboxes the symptoms came back for one of the DBs.  Just like before.  It ended up being a corrupt journal mailbox.  I had Microsoft working on it with me...  They used the Exchange monitoring tool and determined where all of the high transactions where.  It was a shot in the dark for that mailbox should have a lot of transactions.  I suspected my GFI MailArchiver so I disabled polling, however, I didn't think about the journal mailbox itself.  Anyway.  That ended up being the problem.  In 24 hours a lone it generated 200GBs of unneeded space.  I'm migrating that DB to a new one again. I want to free up that space for it's not even white space.  Who knows what it actually is..  Also..  I have 12 GBs of RAM allocated to the Exchange server VM.  It used to stay in the high 90%..  Now it's 54%.  I guess whatever loop it was in was using up a lot of memory.
0
Seth SimmonsSr. Systems AdministratorCommented:
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.

I have recommended this question be closed as follows:

Split:
-- timgreen7077 (https:#a42465011)
-- Adam Brown (https:#a42465088)
-- timgreen7077 (https:#a42468180)
-- timgreen7077 (https:#a42468383)


If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.

seth2740
Experts-Exchange Cleanup Volunteer
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.