Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2805
  • Last Modified:

Dirty data detected. Unable to write dirty data to drives

Hello Everyone,

I noticed a couple days my backups did not run correctly and people complained of a huge exchange slowdown. When I checked I noticed a bad HD. I have since replaced the bad disk with a factory replacement and my backups still do not run. Same exact issue. The disk that failed is part of a RAID 5. I ran the Diag Utiliy and get the following error Dirty data detected. Unable to write dirty data to drives.

In Event viewer I see the following errors.

Logical drive 3 configured on array controller E200 located in server slot 3 returned a fatal error during a read/write request from/to the volume.

 Logical block address 9461208, block count 128 and command 20 were taken from the failed I/O request.

 Drive bus 1 and drive bay 6 were returned from the host firmware as the last physical drive (associated with this logical request),  to report a fatal condition.

When I open the ACU I see the below. Next to the RAID 5 there is a little clock. I am not sure exactly what the means.

 Smart Array E200 in Slot 3
SAS Array A

 72 GB SAS Drive at Port 1I : Box 1 : Bay 1
 72 GB SAS Drive at Port 1I : Box 1 : Bay 2
 Logical Drive 1 ( 69973 MB, RAID 1+0 )

SAS Array B

 72 GB SAS Drive at Port 1I : Box 1 : Bay 3
 72 GB SAS Drive at Port 1I : Box 1 : Bay 4
 Logical Drive 2 ( 69973 MB, RAID 1+0 )

SAS Array C

 146 GB SAS Drive at Port 2I : Box 1 : Bay 5
 146 GB SAS Drive at Port 2I : Box 1 : Bay 6
 146 GB SAS Drive at Port 2I : Box 1 : Bay 7
 Logical Drive 3 (279964 MB, RAID 5 )

Any idea what can be done? I have already rebooted the server once to see if that would help.

Appreciate the help...
0
gsswho6
Asked:
gsswho6
  • 38
  • 16
  • 9
1 Solution
 
DMTechGrooupCommented:
Now that you have replaced the drives try running a chkdsk on all drives..

chkdsk /f <drive letter>

stop all exchange servers first.. you will probably have to reboot to get the chkdsk to run so this could be time consuming.
0
 
gsswho6Author Commented:
You mean each partition correct? Not each actual drive itself.

Just making sure.

Thanks
0
 
DMTechGrooupCommented:
Right.. C: D: E: etc..
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
gsswho6Author Commented:
Yes it is qiltjr.
0
 
gsswho6Author Commented:
So i have replace the drive twice now and same thing. Rebuild fails. Event log is below. The actual Diag util still says dirty data.

So I assume best bet is to just run chkdsk now correct? Just wanted to exhaust all other possibilities before doing so.

SLOT 3 Smart Array E200 Controller ERROR REPORT:

   Logical drive 3 status = Ready for recovery
   Array accelerator status:
      Valid data found at reset
      Dirty data detected. Unable to write dirty data to drives


Due to an unrecoverable read error, the recovery of logical drive 3 configured on array controller E200 located in server slot 3 was  aborted while rebuilding a physical drive.  

The physical drive which was being rebuilt is located in bay 5 of box 1 which is connected to port 2I of  array controller E200 located in server slot 3 .

The physical drive that reported the read error is located in bay 6 of box 1 which is connected to  port 2I of array controller E200 located in server slot 3.
0
 
giltjrCommented:
Can you check the status of the battery and the Array?

I found a few hits where if the server is shut down improperly and the battery is dead data in the cache is not written to the HD.  

0
 
giltjrCommented:
Opps, that should have been check the battery in the array.
0
 
gsswho6Author Commented:
Is there a way to check the battery without actually opening  the case and pull the card? Wouldn't the diag say the battery was bad ?
0
 
gsswho6Author Commented:
ok some major problems now. I rebooted the server last night to see if that would help with all the errors. But it seems to have gotted worse. Ever so often exchange just stops working. Nobody can connect to their mailboxes. I check the server and the store is running.  Below is the first error I see before many others. I have to stop store and restart store in order for everything to come back up. But I know some time later today it will happen again. Any help please would be great.

nformation Store (9364) First Storage Group: An attempt to read from the file "E:\Program Files\Exchsrvr\MDBDATA\priv1.edb" at offset 14144163840 (0x000000034b0ed000) for 12288 (0x00003000) bytes failed after 1388 seconds with system error 1117 (0x0000045d): "The request could not be performed because of an I/O device error. ".  The read operation will fail with error -1022 (0xfffffc02).  If this error persists then the file may be damaged and may need to be restored from a previous backup.

Does this have anything to do with the fact one of my drive in my RAID5 is still not functioning? Would it be best for me to just yank that drive out for now? It almost seems like that drive being in the server trying to rebuild/etc is causing issues.
0
 
giltjrCommented:
Do you have support for the SAN?  If so, I would call the vendor ASAP.

If you have a bad drive in a RAID5 array you will experience performance problems.  Although you should be able to put a new drive in and have it rebuild on the fly, depending on how much read/write activity is going on and how big the array is, it could take awhile.
0
 
gsswho6Author Commented:
There is no SAN. Just 3 different array on the local server. C = raid 1, D = raid 1 and E = raid 5.
0
 
giltjrCommented:
My fault, I meant support for the E200 controller.  It sounds like it is having problems.
0
 
gsswho6Author Commented:
Got this message on the ACU.

Background parity initialization is currently queued or in progress on logical drive 3 ( RAID 5 in array C). If background parity initialization is queued, it will start when I/O is performed on the drive. When background parity initialization completes, the performance of the logical drive will improve.
0
 
giltjrCommented:
That makes it sound as if it is attempting to rebuild the missing data.

When did you put in the new drive?
0
 
gsswho6Author Commented:
A few days ago. But with all the problems I yanked the drive and put it back in to see if that helped. It scared me at first becuase I got a popup saying all data was lost. Then after about 30 secs it gave that message above. Still getting ton of Echange errors in event viewer but for now all exchange services are running and mail is flowing. HP said to leave it alone and let it finish and then mayeb the exchange error will go away once done. I hope so... At this point I want to throw the server in the ocean.
0
 
giltjrCommented:
Well, not knowing a lot about the performance of these drives, the controller and the load on the server it could take awhile to rebuild the missing data.  As in hours if not a day.  I doubt it would take longer than a day though.
0
 
gsswho6Author Commented:
Well as of now exchange is down completely. server rebooted. Now ACU says no problems. No longer says dirty data. But now exchange is down and the store says database corrupt when trying to mount it. So I have all services shutoff and doing a backup of the store. Since I cannot run a backup at all when mounted I guess this is all I can do atm. I guess when backup is done ill try to start it back online. If it doesnt work I guess I need to eseutil /mh ?

All this from a failed drive :(
0
 
gsswho6Author Commented:
esetil /mh dump. So what do to do now?



Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
         Database: e:/program files/exchsrvr/mdbdata/priv1.edb

        File Type: Database
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,11
 Engine ulVersion: 0x620,11
Created ulVersion: 0x620,11
     DB Signature: Create time:12/07/2007 20:13:31 Rand:1867250 Computer:
         cbDbPage: 4096
           dbtime: 3083065799 (0xb7c3d9c7)
            State: Dirty Shutdown
     Log Required: 169474-169474 (0x29602-0x29602)
   Streaming File: Yes
         Shadowed: Yes
       Last Objid: 1724987
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
 Old Repair Count: 0
  Last Consistent: (0x29602,69,16)  04/08/2011 06:40:00
      Last Attach: (0x29602,70,103)  04/08/2011 07:19:10
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:12/07/2007 20:13:29 Rand:1871123 Computer:
       OS Version: (5.2.3790 SP 2)

Previous Full Backup:
        Log Gen: 164723-164726 (0x28373-0x28376)
           Mark: (0x28373,1755,1D2)
           Mark: 03/11/2011 23:01:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: found (1561)
Last Bad Checksum Error Date: 04/08/2011 06:38:52
Old bad Checksum Error Count: none

Operation completed successfully in 3.109 seconds.
0
 
DMTechGrooupCommented:
This really should be opened as a new question.. but just for quick reference look here:

http://social.technet.microsoft.com/Forums/en/exchangesvravailabilityandisasterrecovery/thread/0a365814-5b03-48e6-bfd2-ce5416cef806
0
 
gsswho6Author Commented:
Ok so since I do not have a full backup from this pre all this mess I should go the Eseutil /R route? I am currently doing a backup now but that wont suffice right since its a backup of "now" which is already problematic. Hope that made sense. Thanks for the help guys... Appreciated much
0
 
DMTechGrooupCommented:
If you want to move quicker just make sure the all exchange services are stopped and backup (like copy / paste) the database and logs in a separate folder if you have room.. then start all services but the information store and run the /r recovery mode.
0
 
gsswho6Author Commented:
When you say all service you mean all exchange services except the store service correct?

thanks
0
 
DMTechGrooupCommented:
yes
0
 
gsswho6Author Commented:
Will do thanks. The backup of the DB is almost done... And I am coping the logs to another location for backup since those logs currently reside on a different partition then the db. Once that is done ill give that a shot. appreciate the help.
0
 
gsswho6Author Commented:
Also... Just to make sure my syntax is correct. It would be the same command I issued above but instead of /mh I use the /r flag correct? I see some posts about /i and also E00 .
0
 
giltjrCommented:
The /r will attempt to repair the log files.

There is a /p that attempt to repair a corrupted data store.

Brief overview:

http://www.computerperformance.co.uk/exchange2003/exchange2003_eseutil.htm#Eseutil%20/mh
0
 
gsswho6Author Commented:
ok so I should be using eseutil /r E00 /i then correct? Since this is telling it to start with log E00 and try to repair from that point in?
0
 
DMTechGrooupCommented:
If your logs are in a different drive / path you need to specify that as well.

http://technet.microsoft.com/en-us/library/bb123479(EXCHG.80).aspx
0
 
gsswho6Author Commented:
ok.. yea my logs are on D:\Program Files\Exchsrvr... DC is on E:\Program Files\Exchsrvr\MDBDATA.

So I am confused lol.
0
 
DMTechGrooupCommented:
Are the *.log files in one directory and the priv.edb in another?
0
 
gsswho6Author Commented:
Yes. logs = D:\Program Files\Exchsrvr   priv1.edb = E:\Program Files\Exchsrvr\MDBDATA
0
 
DMTechGrooupCommented:
Eseutil /R E00 /LD:\Program Files\Exchsrvr
0
 
gsswho6Author Commented:
What does /LD do? Sorry I though i had to point to logs as well?
0
 
DMTechGrooupCommented:
/L (Logs)

The D is the beginning of the directory structure
0
 
gsswho6Author Commented:
oh ok I see that was pointing to logs... So i dont actually point to my database in any way?
0
 
DMTechGrooupCommented:
no it knows where that is
0
 
gsswho6Author Commented:
what am i doing wrong?


C:\Program Files\Exchsrvr\bin>eseutil /R E00 /LD:\program files\exchsrvr

Usage Error: Invalid argument 'files\exchsrvr'. Options must be preceded by '-'
or '/'.

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

DESCRIPTION:  Maintenance utilities for Microsoft(R) Exchange Server databases.

MODES OF OPERATION:
      Defragmentation:  ESEUTIL /d <database name> [options]
             Recovery:  ESEUTIL /r <logfile base name> [options]
            Integrity:  ESEUTIL /g <database name> [options]
             Checksum:  ESEUTIL /k <file name> [options]
               Repair:  ESEUTIL /p <database name> [options]
            File Dump:  ESEUTIL /m[mode-modifier] <filename>
            Copy File:  ESEUTIL /y <source file> [options]
              Restore:  ESEUTIL /c[mode-modifier] <path name> [options]

<<<<<  Press a key for more help  >>>>>
D=Defragmentation, R=Recovery, G=inteGrity, K=checKsum,
P=rePair, M=file duMp, Y=copY file, C=restore
=>
0
 
gsswho6Author Commented:
nevermind i got it... forgot the " ".. says performing soft recovery... crosses finger :(
0
 
giltjrCommented:
Try:

eseutil /R E00 /L"D:\program files\exchsrvr"

It looks like it is getting upset because of the space between "program" and "files".
0
 
gsswho6Author Commented:
Yea thanks... I see in the event logs tons of replaying log messages. so far no errors which is good i assume. lol
0
 
gsswho6Author Commented:
This process take awhile? I assume it would since the log folder is 16gb in size.
0
 
DMTechGrooupCommented:
Be sure if this all works out for you that you give the points to giltjr as he was the main help to your original question.
0
 
gsswho6Author Commented:
Oh no Got this error. what to do now? So stressful :(


C:\Program Files\Exchsrvr\bin>eseutil /R E00 /L"D:\program files\exchsrvr"

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode...
    Logfile base name: E00
            Log files: D:\program files\exchsrvr
         System files: <current directory>

Performing soft recovery...



Operation terminated with error -1018 (JET_errReadVerifyFailure, Checksum error
on a database page) after 885.0 seconds.



0
 
gsswho6Author Commented:
next step is eseutil /p ?
0
 
DMTechGrooupCommented:
0
 
gsswho6Author Commented:
huh? Not sure what that means? How is that possible?
0
 
gsswho6Author Commented:
the /mh worked. How can this be a version problem? sorry not sure what to do at this point.
0
 
DMTechGrooupCommented:
What version of Exchange.. this really should be a new question since it seems the Raid one has been resolved.
0
 
gsswho6Author Commented:
exchange 2003.
0
 
gsswho6Author Commented:
I ran eseutil /k and there are tons and tons of page xxxxx checksum failed messages
0
 
gsswho6Author Commented:
just ran eseutil /p against the DB. Will let you know the results.
0
 
gsswho6Author Commented:
Got about a thousand of the below so far. Scanning status of eseutil /p is currently at 40%.

eseutil (6920) The database engine lost one page of bad data. It is highly recommended that an application-level integrity check of the database be run to ensure application-level data integrity.
0
 
gsswho6Author Commented:
Now its doing Deleting unicode fixup table... I notice that my priv1.edb file is growing in size while its doing this.
0
 
gsswho6Author Commented:
Jeez it takes a long time. I wish it would have some kind of indicator on what % complete it was at or something.
0
 
DMTechGrooupCommented:
Depending on the size of the DB and logs it could take hours.. Be sure to follow that guide step by step.
0
 
gsswho6Author Commented:
Yes I am... The edb file is 16gb. So I assume 2ish hours?

After this is eseutil /d... Then mount store/unmount store and run isinteg -s (SERVER) -fix -test alltests.
0
 
gsswho6Author Commented:
Still going... Almost 4 hours doing the deleting unicode fixup table.
0
 
gsswho6Author Commented:
OK it finished... Now trying to run a defrag and getting the below.

C:\Program Files\Exchsrvr\bin>eseutil /d "e:\program files\exchsrvr\mdbdata\priv
1.edb"

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating DEFRAGMENTATION mode...
            Database: e:\program files\exchsrvr\mdbdata\priv1.edb
      Streaming File: e:\program files\exchsrvr\mdbdata\priv1.STM
      Temp. Database: TEMPDFRG1840.EDB
Temp. Streaming File: TEMPDFRG1840.STM

                  Defragmentation Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|




Operation terminated with error -1605 (JET_errKeyDuplicate, Illegal duplicate ke
y) after 6.0 seconds.
0
 
gsswho6Author Commented:
This was log from the repair. Now nothing runs... cant mount store either


Repair completed. Database corruption has been repaired!

Note:
  It is recommended that you immediately perform a full backup
  of this database. If you restore a backup made before the
  repair, the database will be rolled back to the state
  it was in at the time of that backup.



Operation completed successfully with 595 (JET_wrnDatabaseRepaired, Database cor
ruption has been repaired) after 19190.62 seconds.


C:\Program Files\Exchsrvr\bin>eseutil /d "e:\program files\exchsrvr\mdbdata\priv
1.edb"
0
 
gsswho6Author Commented:
After the repair I try to mount the store and I get the error :

An internal processing error has occurred. Try restarting the ESM or the Store service or both.

I have done both and no luck. Any ideas? reboot maybe?
0

Featured Post

Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

  • 38
  • 16
  • 9
Tackle projects and never again get stuck behind a technical roadblock.
Join Now