Exchange 2013 logs 1 TB

Also..  It keeps on growing...  Around 10 GBs every hour...

I started a VEEAM VSS truncate backup job.  My offsite backup job was supposed to be doing the truncating and VEEAM was copy only.  For some reason this one particular DB isn't truncating..  When the current VEEAM VSS job is done we'll see if it truncates..  If not I at least have a current copy of the Exchange server prior to enabling circular logging.  If I resort to circular logging where do I need to look in regards to the backup job not truncating?  I assume the System or application logs?

Question.  Don't I need whatever Storage size is being used in the log folders available on my database drive?

application and system logs is where I always look for any issues. Also look at logs from the back server or offsite to see if it shows anything.

The log file size doesn't need to be the same size of your db. if you have a 1 TB db, you should be fine with a 300 GB log file size as long as the backup are clearing the logs afterwards. if the logs aren't being cleared then it doesn't matter what size the log drive size is because it will eventually run out of space. enable circular logging until then.

I would suggest removing the backup job for that DB and
 re-creating it to see if that helps.

What I meant was this...  Drive E: has 800GBs in log files not committed to the database.  Does this mean the drive D: that the database resides on needs 800GBs available?

No. the reason the log size is so large is because they are not being cleared but the email has already been written to the database. how much capacity and free space does your db show.

In my opinion, Circular Logging is bad advice - the logs are there for a reason - to restore the emails lost in the event the exchange database is lost.  They should be cleared properly if you backup properly.  I don't use Exchange 2013 (went from 2010 to 2016) and I don't use Veeam (I use Altaro) but I would expect if you setup the backup job properly it will clear the logs.

As for why they are growing so fast, look at your mail inflow - could it be spam?  NDRs?  or do you just have a lot of users sending and receiving a lot of mail?

circular logging isn't bad advice. it's recommended advice if your logs are growing exponentially and you don't know why. you can't keep adding space no stop. circular logging is a suggested method to keep from killing your exchange server until you can resolve issue, that's why it's there.

It's bad advice in my opinion.  You don't IGNORE the problem and enable circular logging - you FIND the problem and FIX it.  Maybe they need to put a spam filter in place?  Maybe a virus is sending out spam and you don't know it.  Don't ignore the cause - find it and fix it.

Who says ignore the problem. circular allows you time to find the problem without crashing exchange due to logs taking all your drive. hint hint that's why it's there.

Really?  Is that why we're here?
If the log files are growing at 10GB/hour then that's 1 TB every 4 days.  A lot, sure.  But not unmanageable especially when the OP indicates there's already 1 TB there...  cleared with a proper nightly backup.  There's no need for circular logging at this point.  Before advising someone does something that puts their recovery at risk, take into consideration if, given the information provided, the advice is really good advice.

https://blogs.technet.microsoft.com/exchange/2010/08/18/exchange-circular-logging-and-vss-backups/

It's clear you disagree with me.  So be it... I've made my argument.  And my recommendations - find out what's going on and fix it as well as ensuring the backups are running properly.

I recommend you 3 steps:

step 1. Do a Full backup of your server.

Step 2. Check the sizes of the logs and DB, if EDB file is slightly smaller than LOG size reported by the script
https://gallery.technet.microsoft.com/office/What-is-the-size-of-my-0e1993a7

step 3. Enable Circular logging on your database.
https://gallery.technet.microsoft.com/office/Enable-CircularLogging-In-beaaa3bd

After reading, I think lee has right, there's no point in growing at that rate, there must be some kind of issue, check in event viewer, before trying this steps.

The VEEAM VSS job finished.  It cleared up 700 GBs of logs, however, there are currently 280 GBs of logs for that database.  I'll watch to see why it keeps on growing.  We don't have a lot of email traffic.  Maybe an internal loop?  We are constantly changing email rules.  I thought exchange had protection from now...  Now..  I have a product called GFI MailArchiver.  It reads each sub folder of every user and makes note of where the archived email is to be located.  It preserves Outlook folders in the mail archived...  This results in large IIS files but wouldn't think it would mess with transaction logs.

how does your transport queue look? does it seem to get backed up or over whelmed with mail. are users compalining about spam or anything. are you using any type of spam and malware filtering. its good the veeam  backup cleared the logs. I would run it one more time to see if it completely clears the remaining logs it of some logs remained.

We have close to no spam slipping through on busy days...  Over the last few days Email traffic is close to nothing.  I'll check transport rules queue shortly, however, the growth of the log folder has slowed down a lot. I'll reply soon with more updates

good to hear.

Hi
Everyone the same issue i am also facing with the VEEAM Backup Application, if you run the backup with native windows backup tools, it will purge the logs successfully, so do a full backup of all the databases by windows backup, it will purge all the logs then you start using your regular VEEAM backup tool, this happen to me and MS Technical Support Team also involved but noting wrong found in exchange.

In fact may there are some logs missed during earlier truncation etc. so make sure all your VSS Writes status. by the below command

From command promt type the below

VSSAdmin List Providers

To display all provider

VSSAdmin List Writers

To display all Writers and their Status

Note :- In most of the cases in our issue the Microsft Exchange Writer was failed and retryable so the solution for this

Just restart the service that the writer was associated with and/or fix whatever configuration issue is causing the failures.  For example, given the above output I would restart the
Exchange Replication Service in an attempt to return the writer to a Stable No Error state.  

Note :- you can restart Micorsoft Exchange Replication service without any interuption,
but Information Store Service will have interuption so take care which is even not needed here.

(If it would have been the Microsoft Exchange Writer I would have restarted the Exchange Information Store Service).

How do you run an Exchange only back via the windows backup tool?  

Also,

The DB that won't purge is 1.7 TBs and I don't have the Database Size Limit reg keys in place.  I just added them.  What do I need to do for it take affect?  Default limit is 1 TB.  Maybe that's why my logs have grown so large for this DB?  It's too big to commit the logs?

This will guide you in backing up Exchange using Windows backup server. make sure the windows backup server feature is installed first.

https://technet.microsoft.com/en-us/library/dd876854(v=exchg.150).aspx


You can right click the DB and look at the properties to see how much storage capacity you have and what's used.

If my database has reached it allowed Database Size Limit won't it stop allowing logs to be committed and result in logs building up?

Yes and it will dismount the DB.

So..  I added the 'Database Size Limit' REGWORD key.. I set it to 3000 GBs..  How do I make this take affect while DB is still mounted?  I followed the guide at https://support.microsoft.com/en-us/help/3059008/exchange-server-2013-standard-edition-can-t-mount-databases-that-are-l

most registry changes take affect after computer reboot.

So,

VEEAM VSS Truncate job is done...  All but 200 GBs is purged in the logs...  I guess the only way to get rid of these logs is to enable circular logging?  No matter what there is always 200 GBs left over...

Yes, it is. then you can disable it again.

Assuming logs don't grow really large each day I'll go ahead and hold off until Friday evening.. That way I have more time if something goes wrong and I execute my back out plan...  Previously logs where growing so quick that even a work shift was a problem..

You need to find out the reason why they were growing.
You need to check the event logs and find out why it is growing like that.
Enabling the Circular logging is a workaround but the real issue is not that one.

The only thing I see is that the affected DB was out of logical space.  It's been going on for a long time.  I utilized the reg key to increase allowed Exchange DB size.  It seems it was blocking log files from being committed.  Not sure to be honest... Thats the only error in the event log aside from cert errors

The problem is this...  there are so many files to purge.  How long will it be offline if there are 100,000+ files to purge?

Ok.  I might do it at the end of tomorrow's work shift

I enabled circular logging..  I had to dismount and Mount.  But theirs are still there?  Do I have to manually delete them?

Never mind...  They are now all purged...  So...  Tomorrow after I take a full backup with truncate enabled should I re-enable it?

Well, circular logging is not a thing to play with.
if you want to leave it enabled it, just do it, it's not something that you should be enabling and disabling every day because you can get corruption in your database and then you will have to pay 1000$USD software to recover PST from EDB, if you were unable to find what is the source of the problem, hire a freelancer.

Jose,

For now I enabled circular logging just until I troubleshoot further..  It was to the point I had to.. I was simply running out of storage to allocate to the VM..  I had no choice.  I'm considering opening a ticket with Microsoft..  I have around 2 TBs of DB storage being used, however, my mailbox report only shows 800 GBs of mailbox usage?  This does not include logs...  That alone is an issue..  In regards to the logs.. So..  I'm not sure to why this particular DB was growing so large...  I tried two different backup utilities and they never could fully purge the logs...  What do you recommend my next step is?

Now my DB is getting even bigger..  something is wrong

how many mailboxes do you have on that DB. also what are the sizes of those mailboxes.

The total size equals right at 1TB but I'm showing 2.3TBs of usage...  It's growing as I type this...

Question is how many mailboxes do you have on that DB and how large are those mailboxes? If you need assistance finding this info out let me know.

I generated a report .  I have a total of 140 mailboxes across all DBs.  They total 800 GBs total ...  On this DB I have a mailbox that is 80 GBs on that DB...  And a few that are 50 GBs...  Like two more.  Send command to generate stat for specified DB and I'll run it...

Run the following:

Get-Mailbox -Database "DB Name" | Get-MailboxStatistics | ft displayname, totalitemsize

Run the cmd just for the DB with the issue.

I ran it...  One mailbox is huge but all total way under the 2.1 TB data file size.  The .edb is 2.1 TBs

I don't know where all of the extra storage is coming from.  The .edb alone is 2.1 TBs...  All mailboxes across the entire server is 800 GBs

well you aren't answering my question.  look into moving the mailboxes to a different DB if the other DBs have capacity. that way you can remove that DB and recreate.

I don't have enough storage...  Ill have to mount a different iSCSI appliance..

I'm driving but I would say there are 500 GBs of mailboxes on that DB

I can use my log volume.  It will be slow for cache is disabled but there is a lot of space there.  Question.  When moving mailboxes to a new DB how much storage in logging is needed ?  Can I use circular logging on the new target DB to void huge logs?

Yes you can use circular logging. enable it before starting the moves. if the mailboxes are large I would recommend moving only 1 or 2 at a time because moves can be really taxing on exchange. I would recommend moving after business hours. also pause the moves during your backup hours and resume afterwards.

Sounds like a plan.  Reads will be slow but writes are fine.  Do you know if a reason that I might have a problem since caching is turned off on the target volume?

I'm not a storage guy but it shouldn't matter. just move large mailboxes 1 or 2 at a time only. also make sure that there are no hard limits set on the DB that may cause the move to fail. those DB limits are set in exchange so make sure there are no limits since you have really large mailboxes.

I'll disable size limits and increase allowed errors.  I'm making a current backup now...  It takes forever since I'm using reverse incremental and now that the virtual disks are so big...  But..  I'll add 100 GBs of Storage to the DB volume.  That way tonight I can sleep knowing that it's good on space...  Then I'll move tomorrow after hours.  Some I can do during work hours.  I'll try a small one to see if it can handle it...

Good luck.

I'm onto something .  The process MSExchangeDelivery.exe is writing constantly...  What does this tell us?

IIS is constantly writing also

MSExchangeDelivery.exe accepts email messages from the Transport service and delivers the email message to the mailbox.

I have OpManager that monitors via PowerShell.  I see thousands of transactions from it..  I turned it off for now...

How is your transport queues. I wonder if you are being slammed with spam.

how many exchange servers do you have?

Only one...  Tomorrow morning when backups are done I'm moving mailboxes...  This DB seems to be messed up...  Not sure if I'll have enough time..  The most important mailbox is 80 GBs and will take forever to migrate

Why did you allow that mailbox to grow to that size. Its hard to migrate and restore if ever needed. Yep move the mailboxes and scrap that DB. That is the best option at this point. How are the system resources on your exchange server. Whats the CPUs and RAM.

It just started to expand this large over the last week.  I didn't realize. It's constantly growing.  How can I instantly free up space after mailboxes migrate?  Also..  Will it use up space on the source DB when migrating to new target?

you can enable the archive mailbox for that large mailbox and move some mail from the primary mailbox to the archive.

What if I run an Outlook rule and purge?  I have a ton of emails I can purge.. Like 30 GBs worth..  I can run a permanent delete rule?  How can I force that storage to be released?  I set the DB to keep deleted items for 0 days.  Does this mean one deleted from deleted items the mailbox will get smaller?  Or..  Is the white space still there and simply means I can migrate quicker?

Storage will not show to be free any more, but exchange will reuse that free space regardless.
Yes purge whatever you don't need in order to shrink the mailbox size.  Once you purge the mail, run the belowcommand so that you can clean out the recovery dumpster so that when you migrate the mailbox, the recovery dumpster won't migrate with 30GB of mail that you deleted. Remember once you clear mail from the deleted items folder it goes to the recovery dumpster for a set amount of days before exchange purges that dumpster, and those days are set on the exchange server. run this command to clear the recovery dumpster after your purge:

Search-Mailbox -Identity username -SearchDumpsterOnly -DeleteContent -Confirm:$false -Force

I set keep deleted items for 0 days for the entire DB.  Do I still need to run that command ?

nope. no need to run it then.

ok..  I'm starting to purge now..  VEEAM VM level ESXi backups are running..  I guess purging will slow the backups down but need to head a head start.

I think I found the problem.. I'm running an Outlook that permanently deletes any email older than 3 x months from the 83 GB mailbox.  I noticed after running it for about 20 mins by storage quit getting eaten up..  And the .edb didn't increase in size...  It's held the same size for almost 30 mins now..  At minimum it's growing much slower..  At this rate it will easily buy me enough time to migrate all mission critical mailboxes..  I've learned my lesson.. I'm getting cap to 50 GBs on the 3 x users that need large mailboxes.. I'm 99% sure this is where the corruption was caused...  As soon as the mailbox was trimmed down all the sudden by DB growth issues went away...

I have a few questions..  

As I understand the migration process of Exchange 2013 is more of a sync..  Once it catches up it finalizes... Only during the finalization process is the mailbox unavailable, correct?

In regards to my other two DBs.. I plan to move them to a different drive letter so I can purge the messed up DB and the old drive.  Should I enable circular logging before moving these two DBs to void a bunch of logs being generated?

That is correct, the finalization process is cutover to Exchange 2013 is actually happening. The user can actually be in outlook the entire time and outlook will prompt them to restart outlook. I'm currently migrating Exchange 2010 users to Exchange 2016 and I auto suspend the mailbox at 95% and during non business hours I complete the move so that users don't have to get prompted to restart outlook during the cutover process and finalization process. You can do either because once they restart outlook after the prompt they are generally fine.

Why do you want to move the DBs to a different drive letter. If you do this they will dismount because Exchange will be looking for the old path. I wouldn't suggest moving them to a different drive unless you know to move your DBs and TLogs to a different and fix it in Exchange.  If you are comfortable doing it fine, but be careful. No actual need to change drive letter. I would just move the mailboxes to different DB and scrap the problem DB and recreate it.

After creating new DBs and moving mailboxes the symptoms came back for one of the DBs.  Just like before.  It ended up being a corrupt journal mailbox.  I had Microsoft working on it with me...  They used the Exchange monitoring tool and determined where all of the high transactions where.  It was a shot in the dark for that mailbox should have a lot of transactions.  I suspected my GFI MailArchiver so I disabled polling, however, I didn't think about the journal mailbox itself.  Anyway.  That ended up being the problem.  In 24 hours a lone it generated 200GBs of unneeded space.  I'm migrating that DB to a new one again. I want to free up that space for it's not even white space.  Who knows what it actually is..  Also..  I have 12 GBs of RAM allocated to the Exchange server VM.  It used to stay in the high 90%..  Now it's 54%.  I guess whatever loop it was in was using up a lot of memory.

No comment has been added to this question in more than 21 days, so it is now classified as abandoned.

I have recommended this question be closed as follows:

Split:
-- timgreen7077 (https:#a42465011)
-- Adam Brown (https:#a42465088)
-- timgreen7077 (https:#a42468180)
-- timgreen7077 (https:#a42468383)


If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.

seth2740
Experts-Exchange Cleanup Volunteer