A bad page link (error -327) has been detected in a B-Tree (Event ID: 447)

Hello all,

I have an issue with our Exchange server, the application log keeps throwing the following errors:

Source: ESE
Category: Database Corruption
Event ID: 447

Information Store (2692) First Storage Group: A bad page link (error -327) has been detected in a B-Tree (ObjectId: 172, PgnoRoot: 896) of database D:\Program Files\Exchsrvr\MDBData\Public Folder\pub1.edb (9393 => 10230, 4158633983).

Source: ESE
Category: Database Corruption
Event ID: 447

Information Store (2692) First Storage Group: A bad page link (error -327) has been detected in a B-Tree (ObjectId: 14443, PgnoRoot: 2267585) of database D:\Program Files\Exchsrvr\MDBData\Mailbox store\priv1.edb (3182694 => 3182689, 4158633983).

Notice that the error is occuring for both the priv1.edb and pub1.edb.  These errors are found right after the system tries to run an online defrag, but the online defrag fails and these errors come next.

We had an issue with a hard drive failure about 2-3 months ago, but that hard drive was replaced immediately with a new one.  There was also anti-virus software running on the server that does automatic scanning, but I have since disabled the software incase that is what is causing the problem.

This issue is also causing the overall SBS backup that runs every night to fail with the error "Microsoft Information Store\First Storage Group is not a valid drive, or you do not have access."

I feel that if I fix the database corruption issue that it should fix the SBS backup issue since the SBS backup is failing when it tries to backup exchange.  Unfortunately I have no backup available from before this error started occuring, which leads me to believe that I need to do a hard repair on the database.

So I've found the Microsoft KB article that addresses this issue here: http://support.microsoft.com/kb/810190

The process seems pretty straight foward but I'm a little weary of losing tons of data, or losing the database completely if the hard repair doesn't work.  Also the second part of that article that says the Exmerge utility might have to be used if certain issues occur after the hard repair causes some concern for me.  

I'd like to know if the process outlined in KB 810190 is the route I should take, or are there other steps I should try before? Or are there steps to perform before KB 810190 to ensure a lesser chance of running into issues?  

Also I found this software that seems to address this issue as well but in an easier manner, but wanted some input on it: http://www.recover-computerdata.com/exchange-server-recovery.html  Maybe there is some other 3rd party software that can also take care of this that someone could shed some light on?

Any help would be greatly appreciated, thank you!
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Good to be cautious and I would recommend that

A: you can of course install ExMerge and attempt to Export the Production data to PST first but lI would personally be wary of putting too much stress on that system since it sounds like the disk is questionable and reading or writing a bulk amount of data could send you to a bad place.

B: You said you had SBS, correct?  if so do you also have all mailboxes configured with cashed OST files? if yes then I would go to each machine and export all the contents to PST so that you have a good backup of the data

C: Now in terms of the Database itself I would;

1. Check your event logs to see if there is anything else that will help pinpoint the issue

2. Copy that database EDB + STM to another offline source ASAP

3. Check to ensure all the hard drives and connectors are seated properly

4. Have you recently done any firmware or other hardware disk subsystem adjustments?  If so that could be the cause OR it could be that you need a firmware update form the manufacture so I would look at that from both sides

5. If you cannot pinpoint the problem I would be a bit wary of continuing to run on that hardware and would look to build a new machine and migrate users ASAP.

6. As far as tools to rescue data from and offline database you can do a search on Google for "Export Mailboxes from Offline Database"

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
printmediaAuthor Commented:
The hard drive that failed on this machine was replaced and all firmware was updated, the drives were reseated and any cables around the RAID controller were checked. I feel pretty confident with the hard drives the way they are now.  But this problem could've possibly happened before all that was done when the hard drive first failed.  That's all been taken care of now though so I'm just focusing on the solution.

All clients are configured with Cached Exchange Mode, so there's an .ost file on each machine. I'll make sure to export everyone's mailbox contents to .pst files so there is a backup on everyone's individual computer.

1. The only thing the event log keeps showing are the two Event ID 447 errors mentioned above for both the priv1.edb and pub1.edb. Along with the fail of the online defrag, but that's because of the corruption.  

2. Will copy the EDB and STM files to an external drive.

5. No option to run Exchange on another machine.

So after taking these necessary precautions I should run through the steps in KB 810190, or it there another process that might be better?
Thanks for the update and;

1. I would follow KB 810190

2. If it doesn't fix it then you would need to dial tone the database as they outline in the lower portion, however you wouldn't need to use ExMerge if you have all the OST's backed up to PST since you could use them to re-populate the Exchange Server after creating a dial tone DB.  

NOTE: The only other issue to be aware of is that once you complete the repair /p and defrag /d and isinteg -fix you will be bringing the database back online if all is well.  Once its backup you will have new data flowing in and out of the DB to if the problem does not go away you could then do an ExMerge of just the delta data, i.e. from when you backed up the users OST to PST to today OR you take the DB offline and so another Export to PST at each users mailbox, dial tone the DB and then import the PST's you made from Outlook.

One of the issues with ExMerge is that it only supports ANSI PST files which has a 2GB limit and things actually start to degrade at the 1.8gb mark.  The super special thing here is that ExMerge will not tell you that you are about to hit or go over the 2GB limit instead it just corrupts the PST and you know nothing until you go to open the PST.  Conversely Outlook 2003 and higher use the UNICODE PST format as the default which has a 20GB limit for Outlook 2003 and 2007 and a 50GB limit with Outlook 2010, hence another reason to use Outlook to do the exports from OST :-)   We actually make a tool that can export data from an offline database to PST or to import data directly from the offline EDB into a production server but since its SBS you are only talking a handful of mailboxes so it should be fairly straight forward and simple to handle as outlined above but it will take some time so be patient when you are doing the repair and defrag since it can feel like its doing nothing but it is indeed working on it.  Figure about 2-4GB an hour for process
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

printmediaAuthor Commented:
Alright in regards to KB810190 the first couple of steps are:

3.Run the eseutil /p command against the dismounted database file. To do so:
•Click Start, click Run, type cmd in the Open box, and then click OK.
•Open the Program Files\Exchsrvr\Bin folder.

For example, type cd\program files\exchsrvr\bin, and then press ENTER.
•Type eseutil /p "c:\program files\exchsrvr\mdbdata\priv1.edb" (where c:\program files\exchsrvr\mdbdata is the path of the database file, and where priv1.edb is the name of the database file), and then press ENTER.

My "bin" folder is in "C:\Program Files\Exchsrvr\bin" but my databases are in "D:\Program Files\Exchsrvr\MDBData\Mailbox store" (priv) and "D:\Program Files\Exchsrvr\MDBData\Public Folder" (pub)

I don't understand this part of the instructions that tells you to cd into the "bin" folder but then run the eseutil /p command on the database which is in another folder. (in my case its on another volume)  

Also the eseutil command is something that the command prompt will recognize right?  Or is there something that I need to install from Microsoft for the machine to recognize eseutil.

Also I'm assuming the stores need to be dismounted in order to copy the edb and stm files to an offline source right?
1. Yes you need to dismount the stores for the process as well as to back them up as a flat file to an alternate location
2. once dismounted back them up to an alternate location as discussed before you do anything
3. yes do Start/Run CMD for command prompt
4. if the bin file is pathed properly you should just be able to open the CMD prompt and type in eseutil and it will start but will error out since you didnt give it a command.  if that works then jsut go to the database path and run eseutil /p databasename.edb and you will be off and running.

NOTE If the bin isn't pathed out and therefore when you run eseutil it doesn't run you can also do this
A. go to your database path, i.e. open CMD and change to F:\Mydatabase or whatever the path is
B: Then run C:\Program Files\Exchsrvr\bin\eseutil /p Mydatabase.edb
printmediaAuthor Commented:
Alright, I'm going to go ahead with the repair tomorrow as I was able to find a window to bring the server down.

One question though about the KB article.  It says to run eseutil /p then eseutil /d, then Backup the database files, mount the store, dismount the store, then open the command promp and run "isinteg -s servername -fix -test alltests"

Can I skip mounting the store, and dismounting it? Or is there a certain reason to do that?  Also if I backed up the original database files before starting the eseutil process, is it necessary to back them up again before isinteg?
1.  The premise behind mounting and dismounting the store is to ensure that its in a mountable state since it has to be clean/consistent before you can run an isinteg.  I usually do not mount and dismount before I run isinteg so you can safely skip that step.  If for some reason isinteg has an issue starting then you may need to do so but fairly rare IMO

2. The reason to backup the DB between steps is to protect the work that has been done to the DB so that should something go south you can roll back to the previous successful step vs having to completely start over.  I find that you can never have too many backups.....  and true its a pain but whats worse, taking time to do a backup or having to start the entire process over??   Personally I would suggest that you have the following backups

A:  Before you start ensure that you have made a fresh backup of the database
B: Post /p make a backup of the /p repaired database
C: Post /d make a backup of the defragmented database
D: Post isinteg make a backup of the completely repaired database

If you don't have enough space to do so you can trim this down to have less backups but if you have the space i would do them so that you can keep from having to redo any of your work.

3. The other thing you will want to do is to run the isinteg command multiple times, i.e. first time you are more than certain to correct a certain # of ERRORS which is a good thing.  But you want to repeat this command until you get zero errors corrected and then you will have a clean store.
printmediaAuthor Commented:
Did everything last night and it went without a hitch.  Took long, but everything is fine now.  I checked the event viewer in the morning and it's running online defrag's on pub1 and priv1 without a problem, and no more of these errors.  

Thanks for answering all my questions!
Excellent news, thanks for the points and if you are ever in search for Exchange based products check us out on the web

Thanks again and have a great day!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.