eightballfreek
asked on
Disk space on Exchange server drops to near-zero, no matter what is deleted.
The disk space on our Exchange server has plummeted to ~100KB this morning, which has unmounted our Exchange store. The contents of the /Exchsrvr/MDBDATA folder come to a total of 59GB out of a total amount of space on the C drive of ~137GB.
I found some unnecessary files on the C drive (Windows XP SP1, Windows 2000 SP4 installation files) and moved them to another server, ostensibly freeing up well over 600MB. I then ran Windows' Disk Defragmenter.
Defrag shows 90% fragmentation - the whole map was red. This is obviously a bad thing, but worse was the fact that the amount of free disk space shown was dropping like a stone, around 4-5MB per second. It fell from around 400MB to eventually level out around 11MB, and then dropped to a few hundred KB and the store dismounted.
I am the only moderately tech-savvy individual in our office (primarily made up of designers and video editors). I am a competent programmer but *not* an Exchange engineer (very little experience with Server 2003, in fact).
Any help or pointers would be much appreciated; Google Search isn't much help as I'm coming at this from a symptoms-only point of view, I need to know what's actually causing this.
I've spotted a few Google entries which mention Veritas issues (we are running Veritas on a separate server, but it does back up the Exchange store) but they tend to point towards purging VSS snapshots, and there doesn't seem to be any on our Exchange server.
Please help!
I found some unnecessary files on the C drive (Windows XP SP1, Windows 2000 SP4 installation files) and moved them to another server, ostensibly freeing up well over 600MB. I then ran Windows' Disk Defragmenter.
Defrag shows 90% fragmentation - the whole map was red. This is obviously a bad thing, but worse was the fact that the amount of free disk space shown was dropping like a stone, around 4-5MB per second. It fell from around 400MB to eventually level out around 11MB, and then dropped to a few hundred KB and the store dismounted.
I am the only moderately tech-savvy individual in our office (primarily made up of designers and video editors). I am a competent programmer but *not* an Exchange engineer (very little experience with Server 2003, in fact).
Any help or pointers would be much appreciated; Google Search isn't much help as I'm coming at this from a symptoms-only point of view, I need to know what's actually causing this.
I've spotted a few Google entries which mention Veritas issues (we are running Veritas on a separate server, but it does back up the Exchange store) but they tend to point towards purging VSS snapshots, and there doesn't seem to be any on our Exchange server.
Please help!
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Yes this seems the transaction logfile. Each transaction logfile occupy 5MB of disk space. Please let me know total how many files are there? Did you check the transaction logfiles path in ESM as I mentioned earlier.
You should not delete them untill they are written into the database file. You can enable circular logging in the ESM to automatically getting overwritten once commits to the database if you have sufficient Disaster Recovery option.
Sekhar
You should not delete them untill they are written into the database file. You can enable circular logging in the ESM to automatically getting overwritten once commits to the database if you have sufficient Disaster Recovery option.
Sekhar
This is a helpful article also: http://msexchangetips.blogspot.com/2006/08/exchange-transaction-log-files-growing.html
ASKER
Some of the log files go as far back as October 2007 (the 23rd, to be precise). I have backed a few of them up to a separate server and then removed them; this freed up ~260MB and then BAM! - my disk space sunk like a stone back to ~30MB. But the only file to have been altered on my C drive (I did a search) was an entry in the EXCHANGE.log folder, and it was just 13MB. I just can't account for this mass loss of free space.
I will try having a look for circular logs. I did mess about with the server a little while ago as it came a cropper of the pre-Service Pack 17GB limit. Backups etc. were all performed then, and (as this was about February time) it didn't purge the logs then - will it this time?
I will try having a look for circular logs. I did mess about with the server a little while ago as it came a cropper of the pre-Service Pack 17GB limit. Backups etc. were all performed then, and (as this was about February time) it didn't purge the logs then - will it this time?
ASKER
Sekhar,
I'm sorry to be dense but you must understand I am a total, utter and complete n00b when it comes to Exchange. Precisely how do I check the log files' path, enable/disable circular logging, etc.?
I'm sorry to be dense but you must understand I am a total, utter and complete n00b when it comes to Exchange. Precisely how do I check the log files' path, enable/disable circular logging, etc.?
Right click on the storage group, select Properties, and click on 'enable circular logging'
ASKER
Ah, I believe I might have found it.
The path for "Transaction log location" is C:\Program Files\Exchsrvr\mdbdata. This is also the path for the next variable, "System path location". The checkboxes for "Zero out deleted database pages" and "Enable circular logging" are both unchecked.
What's the best step-by-step here to ensure I don't break anything vital?
The path for "Transaction log location" is C:\Program Files\Exchsrvr\mdbdata. This is also the path for the next variable, "System path location". The checkboxes for "Zero out deleted database pages" and "Enable circular logging" are both unchecked.
What's the best step-by-step here to ensure I don't break anything vital?
what errors are you getting in the application & system logs?
Refer the attachment and from Exchange System Manager (ESM), you can find the options. Let me know for any help.
Sekhar
SGP.jpg
Sekhar
SGP.jpg
Eightballfreek the first thing you should do is check 'enable circular logging' to arrest the massive disk usage you are seeing. Once you have that under control you can then assess whats causing the large growth in Log Files (This article suggests a few scenarios: http://msexchangetips.blogspot.com/2006/08/exchange-transaction-log-files-growing.html)
ASKER
nwnsltd:
This is the first Exchange-related error we received this morning:
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 428
Date: 08/07/2008
Time: 09:23:43
User: N/A
Computer: EXCHANGE
Description:
Information Store (4680) First Storage Group: The database engine is rejecting update operations due to low free disk space on the log disk.
For more information, click http://www.microsoft.com/contentredirect.asp.
There are now dozens and dozens of Application-type error events in the Event Viewer, all of them along the lines of " because there is not enough disk space."
This is the first Exchange-related error we received this morning:
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 428
Date: 08/07/2008
Time: 09:23:43
User: N/A
Computer: EXCHANGE
Description:
Information Store (4680) First Storage Group: The database engine is rejecting update operations due to low free disk space on the log disk.
For more information, click http://www.microsoft.com/contentredirect.asp.
There are now dozens and dozens of Application-type error events in the Event Viewer, all of them along the lines of " because there is not enough disk space."
ASKER
Sekhar, nwnsltd,
Excellent guys, found Circular Logging and have enabled it. My disk space is currently fluctuating between 20MB and 45MB, it drops to 20MB and then shoots up to ~45MB, only to drop down to 20MB again gradually. Each "cycle" seems to take only about 5 seconds.
How should I proceed in getting disk space back? Or will this happen automatically now that circular logging is enabled?
Excellent guys, found Circular Logging and have enabled it. My disk space is currently fluctuating between 20MB and 45MB, it drops to 20MB and then shoots up to ~45MB, only to drop down to 20MB again gradually. Each "cycle" seems to take only about 5 seconds.
How should I proceed in getting disk space back? Or will this happen automatically now that circular logging is enabled?
yep. Enable Circular logging. This will clear the log files and free up disk space allowing yo9u to mount your store etc.
We can then look at what is causing the increase in transaction logs, or at least why they are not being truncated.
We can then look at what is causing the increase in transaction logs, or at least why they are not being truncated.
can you mount your store so that the logs can be played into the Database.
ASKER
OK - all is looking good - the free disk space is up to 2.4GB and growing. The database hasn't told me it's mounted yet, but I assume this is because it's dealing with the backlog of log files (which go back to October of last year).
Moving forward, should I leave circular logging enabled? Or can I edit which things are logged by Exchange? What would be best practice here?
Our Veritas tape backup system is currently running only weekly full backups (so no grandfather/father/son system in place just yet, though that is the plan).
Moving forward, should I leave circular logging enabled? Or can I edit which things are logged by Exchange? What would be best practice here?
Our Veritas tape backup system is currently running only weekly full backups (so no grandfather/father/son system in place just yet, though that is the plan).
No, don't leave circular logging enabled, once you're down to a safe level with your logs make sure you do a full backup of exchange every night using an exchange certified backup program. Even if you use the built in NTBackup and copy it off to your veritas box it will clear them down for you.
If however you are still seeing massive growth then we'll need to find the cause.
If however you are still seeing massive growth then we'll need to find the cause.
With Circular Logging Enabled you wont be able to do incremental backups. Although Full backups will be fine.
Depending ont he size of the partition your log files are on, you shoudlnt have to enable circular logging as a weekly full backup should truncate the logs. However it sounds like this is not happening with your system at present.
If it were my system i would disable circular logging the day of my full backup, check the logs before the backup and then check to see if they have been removed after the backup. If the full backup is removing the logs as it should i would keep circular logging disabled from then on.
Depending ont he size of the partition your log files are on, you shoudlnt have to enable circular logging as a weekly full backup should truncate the logs. However it sounds like this is not happening with your system at present.
If it were my system i would disable circular logging the day of my full backup, check the logs before the backup and then check to see if they have been removed after the backup. If the full backup is removing the logs as it should i would keep circular logging disabled from then on.
finding whats causing the massive transaction log growth would be my next priority. again, i would advicse reading this article: http://msexchangetips.blogspot.com/2006/08/exchange-transaction-log-files-growing.html which does give some good ideas for what might be causing the large growth in transaction logs.
eightballfreek: If I am not wrong, you mentioned earlier the Exchsrvr\MDBDATA folder occupies 59 GB outof 137GB. Then which are the other files/folders that occupies more space? Are you using Veritas Netbackup as the server backup software? Whether you are experiencing any backup failures from some days?
If yes, check for any hidden files on the root C:\ drive. Let me know for further help.
Sekhar
If yes, check for any hidden files on the root C:\ drive. Let me know for further help.
Sekhar
ASKER
Sekhar, I'm using Veritas Backup Exec 10 on a different server. I believe that Exchange has been told to back up to a folder on this separate server, which is then backed up to tape.
***
Thanks guys, I will keep circular logging enabled until it's weeklybackup time, and then I will check to see if the logs are beingremoved. However, with unbroken records going back to October, I thinkperhaps they're not being removed with a full backup.
Sekhar and nwnsltd, thank you for your obviously very well-informedexplanations and help. BiPod, thanks for those links, they look likeprecisely the kind of thing I'm after for keeping this problem fromoccurring again.
If anything else occurs to you, please don't hesitate to post it here. Had a family emergency this morning but the downed Exchange server meant I couldn't leave. It's now back up and working perfectly.
***
Thanks guys, I will keep circular logging enabled until it's weeklybackup time, and then I will check to see if the logs are beingremoved. However, with unbroken records going back to October, I thinkperhaps they're not being removed with a full backup.
Sekhar and nwnsltd, thank you for your obviously very well-informedexplanations and help. BiPod, thanks for those links, they look likeprecisely the kind of thing I'm after for keeping this problem fromoccurring again.
If anything else occurs to you, please don't hesitate to post it here. Had a family emergency this morning but the downed Exchange server meant I couldn't leave. It's now back up and working perfectly.
ASKER
Thanks guys, you made my day. Very good job. {:¬)
ASKER
Thanks for your swift reply. Would these be the "E00041C0.log" files I'm seeing so many of in the /Exchsrvr/MDBDATA directory? They are all 5,120KB in size and about 15-20 of them have been created just this morning!
Are these the ones to which you're referring, can they be deleted safely, and how on earth can I slow down their creation?! They are indeed being stored on the C drive (in Program Files/Exchsrvr/MDBDATA).
Thanks!