bwhorton
asked on
Problems with Exchange database recovery
Guys,
I'm looking for a little help. Last Wednesday, we had a UPS failure that triggered a dirty shutdown of Exchange. I run two 64-bit servers on the backend for mailboxes (one for Students and one for Faculty/Staff). The Fac/Staff box came up fine with no issues. The student server is another story. By the way, our backups were not valid so I can't go that direction (another story another day).
When it came back up, there were two storage groups with databases that would not mount. My storage groups and DB's are arranged by year, so the 2010-11 and 2011-12 mailstores were the ones affected. After following several threads and links here, I was finally able to recover and restore the 2011-12 SG and mailstore. It is fine. A couple of passes with eseutil to repair and defrag.
Attempted the same process on the 2010-11 box. At first, all seemed OK. But we started getting reports from students that messages in their accounts were empty or they couldn't access it at all after logging in via OWA. I ran the database recovery tool under EMC and it fixed a handful, but others remained problematic. Now it appears that it hasn't completed the recovery due to lack of disk space. I now have three .edb files with a Temp#### prefix. The current 2010_11.edb file is 42GB in it's current storage group. The temp edb files are scattered randomly across other drives.
Question: Are all 3 needed for a full recovery? They are 40GB, 30GB, and 7.5GB in size. I'm assuming that they are all from the same 2010_12.edb file, but stopped in various stages of repair/defrag due to lack of drive space. Should I re-run eseutil /d against them all or what do you advise? I know I need 110% of free space to run the tool through it's completion.
I'm heading out for a few hours but will check back in later tonight. Thanks in Advance!
Ben
I'm looking for a little help. Last Wednesday, we had a UPS failure that triggered a dirty shutdown of Exchange. I run two 64-bit servers on the backend for mailboxes (one for Students and one for Faculty/Staff). The Fac/Staff box came up fine with no issues. The student server is another story. By the way, our backups were not valid so I can't go that direction (another story another day).
When it came back up, there were two storage groups with databases that would not mount. My storage groups and DB's are arranged by year, so the 2010-11 and 2011-12 mailstores were the ones affected. After following several threads and links here, I was finally able to recover and restore the 2011-12 SG and mailstore. It is fine. A couple of passes with eseutil to repair and defrag.
Attempted the same process on the 2010-11 box. At first, all seemed OK. But we started getting reports from students that messages in their accounts were empty or they couldn't access it at all after logging in via OWA. I ran the database recovery tool under EMC and it fixed a handful, but others remained problematic. Now it appears that it hasn't completed the recovery due to lack of disk space. I now have three .edb files with a Temp#### prefix. The current 2010_11.edb file is 42GB in it's current storage group. The temp edb files are scattered randomly across other drives.
Question: Are all 3 needed for a full recovery? They are 40GB, 30GB, and 7.5GB in size. I'm assuming that they are all from the same 2010_12.edb file, but stopped in various stages of repair/defrag due to lack of drive space. Should I re-run eseutil /d against them all or what do you advise? I know I need 110% of free space to run the tool through it's completion.
I'm heading out for a few hours but will check back in later tonight. Thanks in Advance!
Ben
Sorry, just saw it's Exchange 2007.
The tool that might help you is ISINTEG, not New-mailboxrepairrequest.
The tool that might help you is ISINTEG, not New-mailboxrepairrequest.
ASKER
Thanks Wonko. I'm off-site ATM but will attempt a point of clarification. The students could access it (well most of them) when I remounted it to test access and usability. Of course I cant run eseutil or isinteg while it's mounted. Sorry if my description above was incomplete. -ben
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Running a full backup now. Will check in the morning when I get in and see if there is any physical corruption. I made a copy of the db and log files last Thursday when all of this started. I'll copy the most current db and log files after this back up finishes tomorrow a.m. and then check. Will re-run eseutil and isinteg, then post the results here. It may be tomorrow afternoon before I post another update, so stay tuned.
Thanks guys,
Ben
Thanks guys,
Ben
Welcome bwhorton :)
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I haven't forgot about you but the backup is pushing 18 hours now. Not sure why that is; most days it is done within 4-6 hours. Still on track to perform the above suggestions and will post the results here.
Thanks,
Ben
Thanks,
Ben
1. Are you doing just a DB based backup of the exchange server or are you doing a mailbox level backup as well?
2. How big is the log files folder?
3. Any chance you are having hardware problems with the disk etc? assume you have checked the event viewer?
2. How big is the log files folder?
3. Any chance you are having hardware problems with the disk etc? assume you have checked the event viewer?
ASKER
Backup complete. It was a full backup, complete with mb level.
Log files for that storage group is only 7MB
Not seeing any hardware issues. It's on a RAID5 EMC SAN.
Ready to run isinteg after school is out. I'm going to dismount the affected db; however, there are a number of other db's on this server that are not affected. Looking at the syntax (isinteg -s servername -fix - test alltests), it appears to run against the entire server??? Just looking for a point of clarification. Please confirm before I proceed.
thanks
Log files for that storage group is only 7MB
Not seeing any hardware issues. It's on a RAID5 EMC SAN.
Ready to run isinteg after school is out. I'm going to dismount the affected db; however, there are a number of other db's on this server that are not affected. Looking at the syntax (isinteg -s servername -fix - test alltests), it appears to run against the entire server??? Just looking for a point of clarification. Please confirm before I proceed.
thanks
No it only runs against the specified db
The backup took so long because you did a mb level backup as well and those take allot more time and resources
The backup took so long because you did a mb level backup as well and those take allot more time and resources
It will detect which DBs are available and ask which one you want to fix. Since it cannot touch mounted DBs there is no reason to worry, just dismount the problem DB first.
ASKER
Ok, since mid-terms are ongoing at the moment, I'm going to do this tonight when I get home. I will update as I make progress. Thank you all for your patience and assistance thus far.
Ben
Ben
Excellent
ASKER
All appears to be good. A EE subscription is the best money one can spend. Kudos to Wonko the Sane, Exchange9, and lucid8 for their combined solution and clarification.
Thanks,
Ben
Thanks,
Ben
Glad it all worked out and thanks for the points
Glad it all worked out and thanks for the points
You can more than likely get rid of the temp edbs, but if you can keep them somewhere I'd suggest to do thta, even though they are probably useless.
I don't understand right now how your students could access this store at all if the recovery/repair never completed, because then the DB wouldn't mount at all. If your database doesn't mount because it's not in a clean shutdown state and you don't have the logs then you might have to run eseutil /p (repair). If it does mount however and data is missing then you may be out of luck already - the advice here is to restore from backup, but if you don't have one you're in trouble.
You can try the "New-MailboxRepairRequest"
http://technet.microsoft.com/en-us/library/ff625226.aspx
Also check the event logs for any clues on what might be wrong.