Exchange 2003 - Dirty Shutdown - Eseutil

Hey All,

 I have a SBS 2003 box, running exchange 2003. They had a power outage, and the UPS did not keep the server up long enough to finish the proper shutdown. Mail store will not mount now.

 So I verified I have a proper backup, and am running a eseutil /p.

It was all going good, and this is an excerpt of the output;

Scanning the database.

                     Scanning Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................


Repairing damaged tables.

                     Scanning Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ..
Deleting unicode fixup table.
..............................................


It has been sitting at deleting unicode fixup and sitting around (no movement) for about 20 mins, and hasn't moved any farther.

Is it usual for it to sit there for so long? I have never run this utility before.

Please help!
LVL 2
LindsayColeAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AshwinRaj111Commented:
Yes Deleting Unicode is the one which will take the Longest.
It always depends on the Size of the EDB File.
Please dont close the Command prompt windows as it would further corrupt the Database.
AshwinRaj111Commented:
If you want to monitor the progress and make sure that the process is not stuck do the following.
Open Task Manager.
Click on Processes -> View -> Select Columns.
Choose I/O Read Bites & I/O Write Bytes.
Then check the progress of the eseutil .exe process.
Here you will see the increase in I/O Read Bytes & I/O Write Bytes.
 
LindsayColeAuthor Commented:
Thanks guys, I am still waiting. It shows over 10GB written, and is still chugging along.

With the mail store being 15GB, is it unusual for it to take this long? (Going on almost 3 hours).

Or should I involve MS?
Why Diversity in Tech Matters

Kesha Williams, certified professional and software developer, explores the imbalance of diversity in the world of technology -- especially when it comes to hiring women. She showcases ways she's making a difference through the Colors of STEM program.

AshwinRaj111Commented:
Even if you involve Microsoft it would not change any thing.
They would ask you to do the same.
When it comes to Database Corruption Issues there is 2 ways to get it back.
One is to repair it and the Other is to Restore it.
People don't go with the Repair Option as it is time consuming.
The Eseutil command will process around 3 to 5 GB of Database Per Hour.
So if you have 15 GB then 3 to 5 hours is the Windows we are looking at.
 
LindsayColeAuthor Commented:
Good to know.

It is a little disconcerting since it isn't showing the progress indicator anymore, and this is for a big client.

I will hold tight, and be patient!
AshwinRaj111Commented:
Un fortunately we will have to wait for the process to complete.
LindsayColeAuthor Commented:
The process completed. But it is still showing up as corrupted.
--

C:\Program Files\Exchsrvr\bin>eseutil /g "J:\Exchange\Exchange Databases\priv1.e
db"

Microsoft(R) Exchange Server Database Utilities Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating INTEGRITY mode...
        Database: J:\Exchange\Exchange Databases\priv1.edb
  Streaming File: J:\Exchange\Exchange Databases\priv1.STM
  Temp. Database: TEMPINTEG4072.EDB

Checking database integrity.

                     Scanning Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................


Integrity check completed.  Database is CORRUPTED!

Operation terminated with error -1206 (JET_errDatabaseCorrupted, Non database file or corrupted db) after 1636.437 seconds.

---

I am unsure where to go from here; isinteg will not initialize.

Any suggestions?
AshwinRaj111Commented:
There is not Use of running Isinteg.
Isinteg is just to Check the Logical Corruption.
Try this.
Run eseutil /mh "Entire Location of the Database"
Eg:- eseutil /mh "C:\Program Files\exchsrvr\MDBDATA\priv1.edb"
Check for a Value Call State ->
It should be either Clean Shut Down or Dirty Shut Down.
If it is in Clean Shut Down then you would be able to use the Database and mount it.
If it is in Dirty Shut Down then it would show the missing Transaction Log Files using which you can make it to clean shut down.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
LindsayColeAuthor Commented:
As you can see below, it is showing now that it is clean shutdown. (Previously it was dirty.)
----

C:\Program Files\Exchsrvr\bin>eseutil /mh "J:\Exchange\Exchange Databases\priv1.edb"

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
         Database: J:\Exchange\Exchange Databases\priv1.edb

        File Type: Database
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,11
 Engine ulVersion: 0x620,11
Created ulVersion: 0x620,9
     DB Signature: Create time:10/18/2010 12:54:43 Rand:13076136 Computer:
         cbDbPage: 4096
           dbtime: 129683373 (0x7bacfad)
            State: Clean Shutdown
     Log Required: 0-0 (0x0-0x0)
   Streaming File: Yes
         Shadowed: Yes
       Last Objid: 20320
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 4
      Repair Date: 10/18/2010 12:54:43
 Old Repair Count: 0
  Last Consistent: (0x0,0,0)  10/18/2010 14:54:39
      Last Attach: (0x0,0,0)  10/18/2010 12:54:43
      Last Detach: (0x0,0,0)  10/18/2010 14:54:39
             Dbid: 1
    Log Signature: Create time:00/00/1900 00:00:00 Rand:0 Computer:
       OS Version: (5.2.3790 SP 2)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: found (1)
Last Bad Checksum Error Date: 10/15/2010 15:22:14
Old bad Checksum Error Count: found (1)

Operation completed successfully in 9.640 seconds.

---

It shows bad checksum, and I am unable to mount it. I am getting the same error as before when attempting to mount. ID no: c1041724

I have read so many white papers, and technet articles, I am going crazy.

I assume the bad checksum error is my issue, is there a way for me to correct this?
LindsayColeAuthor Commented:
I ran a /k on pub1.edb.


---
File: J:\Exchange\Exchange Databases\pub1.edb

                     Checksum Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................


8194 pages seen
0 bad checksums
0 correctable checksums
187 uninitialized pages
0 wrong page numbers
0xa503a5 highest dbtime (pgno 0x10f4)

513 reads performed
32 MB read
1 seconds taken
32 MB/second
244580 milliseconds used
476 milliseconds per read
1125 milliseconds for the slowest read
0 milliseconds for the fastest read

File: J:\Exchange\Exchange Databases\pub1.STM
ERROR: database was not shutdown cleanly (dirty shutdown)

Operation terminated with error -550 (JET_errDatabaseDirtyShutdown, Database was
 not shutdown cleanly. Recovery must first be run to properly complete database
operations for the previous shutdown.) after 7.219 seconds.
---

Do I need to run a /P on this as well?
LindsayColeAuthor Commented:
Well, after attempting to run a /P on pub1 I am getting;

"Unable to find the callback library jcb.dll..."

I managed to mount my priv1.edb. I was missing log files, so it wouldn't mount. Renamed the log files folder to -old, created a new empty Logs folder and it mounted.

Now we are trying to mount the public folders file and gettings this jcb.dll error. I have swapped it for another copy, no luck. Added it to the environmental variables. Placed it intto System32, and did a regsrvr32. No luck.

Any ideas?
lucid8Commented:
You need to run the /P against the PUB as well since its dirty and also move the old logs aside and then you should be good to go.  HOWEVER, on both databases I would highly recommend that once you know they are mountable that you carry out the following as a best practice;
  1. Get Backups
  2. Take them back down and run and ESEUTIL /D to defragment and in short rebuild to a new DB
  3. then run ISINTEG - FIX until all errors are removed (Warnings are OK and nothing to worry about)
  4. Bring them back up and take another backup ASAP
  5. lastly IMO you are then better off to schedule some time to MOVE the users to a new database if possible since IME once a DB goes corrupt its never the same even if you do all of the above.
Troy Werelius
Lucid8.com
AshwinRaj111Commented:
Hey Lindsay how is the Public Folder Database going.
Guess you opened a Microsoft Support Incident yesterday. :)
Did you get the Public Folder Database Mounted.
LindsayColeAuthor Commented:
Ash, you were a godsend yesterday.

I did end up opening a support incident, as I couldn't get the JCB error to go away. I found many different 'solutions' online but none of which actually fixed the JCB.

The incident professional gave me a "Beta Hotfix" which references KB 964111. Which made the error go away. After which I ran /p on it, then ran isinteg -fix. Which found 0 errors or warnings.

Both mail stores mounted successfully, so I pulled it all back down- took a backup, and put it back up, and re-enabled mail. Watched as 3000+ messages slowly filtered through.

Thanks for your help Ash, you were amazing!
LindsayColeAuthor Commented:
A God walking among men!
lucid8Commented:
Lindsay, do you have a link to 964111 I don't show any reference to that KB and would like to read up on it.
Thanks in advance
Troy Werelius
AshwinRaj111Commented:
Good to know that it all worked out just fine.
LindsayColeAuthor Commented:
I was unable to find a KB article that referenced that. I surmise it is part of their internal KB.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.