Exchange 2010 - Excessive Transaction Log generation - Backup is OK.

lapavoni
lapavoni used Ask the Experts™
on
Our Exchange 2010 SP2 server continues to generate thousands of transaction logs daily.  The database is 148GB and we have about 75 active users.  We're using Windows Server Backup currently, backing up daily to an external drive.  The backup completes fine and truncates logs as it is supposed to do.  However, by the time the backup completes, another 400-500 logs are generated (this is at 11:00pm to midnight, when there are few active users). Over the course of a day, between 3,000 and 4,000 log files are generated.

The two suspected causes of this are ActiveSync and accompanying iOS devices.  Other things I've read are mailboxes with messages stuck in the Outbox.

I've cleared all Outboxes and the problem persists.

I've removed all partnerships for iOS devices and recreated them and the problem persists (we have 30+ iPhone users).

I've run ExMON many times.  The "Log Bytes" and percent CPU does not stand out for any one (or few) users.  However, there is an entity under the "User" column with a question mark (?) that seems to be generating many megs worth of log bytes.  I presume this is a system account or process, but I don't know.

I've tried throttling EAS, to no avail.

I'm leaning toward ActiveSync being the cause because I've tried two things to mitigate the log generation.

1. I stopped the MSExchangeSyncApp pool and log generation slows down.
2. I set recycle of the pool to a one-minute increment and that seems to have marginally slowed things down.

I'm hoping there's more knowledge about this issue out there than I've found so far.  Ideas ? Suggestions ?  Thank you !!

ExMON showing excessive log bye generation by user "?"ExMON showing excessive log bye generation by user "?"ExMON showing excessive log bye generation by user "?"
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Commented:
Hm. Those ExMon images show only an IPV6 address? Any way to find out what that address belongs to?

As a side note and a longshot, I had a corrupted database in Exchange 2003 (which ultimately failed), and my transaction log generation with that broken DB mounted was dramatically higher than after it "went away," presumably because Exchange kept trying to file messages, failing, filing, failing.

Author

Commented:
The IPV6 is the Exchange server itself.  That threw me at first :)  I have heard about corrupted databases.  Ours seems ok, but I haven't run any checked other than automatic background maintenance. The logs don't show anything unusual there, either.
Commented:
Just one DB, 148GB, 75 users? You can rule out a corrupt database by creating a new one (or ones) and moving all the mailboxes to it, then delete the original one. This will also get you the benefit of recovering white space (what you used to have to offline defrag for), and eliminating corrupt items from mailboxes. Unless you have a DAG, then I think the servers in the DAG compare their mailbox databases with each other to verify DB health, and you'd be able to tell from that.

But that's kind of getting off track. I think you might be right about AS being a problem.

I see that stopping the app pool is how you disable AS:

http://technet.microsoft.com/en-us/library/bb124502.aspx

So when you do that, does the log generation slow way down, or just a little down? And how long do you leave that app pool disabled?

I've also found a thread about Exchange 2010 and AS and iOS5 with Cloud and other device things:

http://social.technet.microsoft.com/Forums/en/exchange2010/thread/64a0eb7f-658e-421c-86d9-6f0cf6896c51

And one about corrupted items:

http://www.terminal.com/blog/bid/64845/Exchange-2010-Transaction-Logs-Rapidly-Filling-Disk-Space
Success in ‘20 With a Profitable Pricing Strategy

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Shreedhar EtteTechnical Manager
Top Expert 2010

Commented:
- After installing Exmon have you merged the registry file present under Exmon folder.

- Under Exmon also check By Verion and By Clinetmon tab.

Author

Commented:
shreedhar, yes I merged the .reg file.  The Version tab shows most cycles used by the Exchange server (14.2.298.4).  The Clientmon tab varies, but on some captures shows most Sessions owned by the Client Process "edgetransport.exe" under Username "?".

Author

Commented:
ckratsch, yes a single database (no DAG) ... simple setup.  I tested today with the App pool disable for about 30 minutes.  I would describe the slowdown as considerable.  At this time of day, where I observed maybe 15-20 log files generated every minute, it slowed down to maybe 3-5 per minute.

I've read through the second thread, but I just read it again. I'll be trying the Exchange PerfWiz script and see what I get.
EMC->Toolbox->Performance Troubleshooter.

Author

Commented:
Stopping the ActiveSync pool temporarily fixes the issue, but that is not a long term solution. As the backups are completing nightly, it's not a showstopper. ckratsch posted the best suggestions and I do intend to create a new database (or two) and move all mailboxes to it.

Commented:
Not sure if this has been resolved, but I had similar problems.  Turned out to be a corrupt calendar item that was repeatedly sending the invitation to a user and then autodeleting straight into recovery items.  It was generating 1.5Mb a minute.  That soon adds up.
Running exmon with a refresh time of 1 minute shows the person at the top of the logbytes list.  Go to that users recovery items in Outlook and you see all that item, repeated ad infinitum.  Delete the referenced calendar item and then get rid of the recovery items.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial