Solved

Dirty data detected. Unable to write dirty data to drives

Posted on 2011-03-22
63
2,745 Views
Last Modified: 2012-05-11
Hello Everyone,

I noticed a couple days my backups did not run correctly and people complained of a huge exchange slowdown. When I checked I noticed a bad HD. I have since replaced the bad disk with a factory replacement and my backups still do not run. Same exact issue. The disk that failed is part of a RAID 5. I ran the Diag Utiliy and get the following error Dirty data detected. Unable to write dirty data to drives.

In Event viewer I see the following errors.

Logical drive 3 configured on array controller E200 located in server slot 3 returned a fatal error during a read/write request from/to the volume.

 Logical block address 9461208, block count 128 and command 20 were taken from the failed I/O request.

 Drive bus 1 and drive bay 6 were returned from the host firmware as the last physical drive (associated with this logical request),  to report a fatal condition.

When I open the ACU I see the below. Next to the RAID 5 there is a little clock. I am not sure exactly what the means.

 Smart Array E200 in Slot 3
SAS Array A

 72 GB SAS Drive at Port 1I : Box 1 : Bay 1
 72 GB SAS Drive at Port 1I : Box 1 : Bay 2
 Logical Drive 1 ( 69973 MB, RAID 1+0 )

SAS Array B

 72 GB SAS Drive at Port 1I : Box 1 : Bay 3
 72 GB SAS Drive at Port 1I : Box 1 : Bay 4
 Logical Drive 2 ( 69973 MB, RAID 1+0 )

SAS Array C

 146 GB SAS Drive at Port 2I : Box 1 : Bay 5
 146 GB SAS Drive at Port 2I : Box 1 : Bay 6
 146 GB SAS Drive at Port 2I : Box 1 : Bay 7
 Logical Drive 3 (279964 MB, RAID 5 )

Any idea what can be done? I have already rebooted the server once to see if that would help.

Appreciate the help...
0
Comment
Question by:gsswho6
  • 38
  • 16
  • 9
63 Comments
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35195879
Now that you have replaced the drives try running a chkdsk on all drives..

chkdsk /f <drive letter>

stop all exchange servers first.. you will probably have to reboot to get the chkdsk to run so this could be time consuming.
0
 

Author Comment

by:gsswho6
ID: 35199319
You mean each partition correct? Not each actual drive itself.

Just making sure.

Thanks
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35202509
Right.. C: D: E: etc..
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35332332
0
 

Author Comment

by:gsswho6
ID: 35336299
Yes it is qiltjr.
0
 

Author Comment

by:gsswho6
ID: 35336536
So i have replace the drive twice now and same thing. Rebuild fails. Event log is below. The actual Diag util still says dirty data.

So I assume best bet is to just run chkdsk now correct? Just wanted to exhaust all other possibilities before doing so.

SLOT 3 Smart Array E200 Controller ERROR REPORT:

   Logical drive 3 status = Ready for recovery
   Array accelerator status:
      Valid data found at reset
      Dirty data detected. Unable to write dirty data to drives


Due to an unrecoverable read error, the recovery of logical drive 3 configured on array controller E200 located in server slot 3 was  aborted while rebuilding a physical drive.  

The physical drive which was being rebuilt is located in bay 5 of box 1 which is connected to port 2I of  array controller E200 located in server slot 3 .

The physical drive that reported the read error is located in bay 6 of box 1 which is connected to  port 2I of array controller E200 located in server slot 3.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35338132
Can you check the status of the battery and the Array?

I found a few hits where if the server is shut down improperly and the battery is dead data in the cache is not written to the HD.  

0
 
LVL 57

Expert Comment

by:giltjr
ID: 35338744
Opps, that should have been check the battery in the array.
0
 

Author Comment

by:gsswho6
ID: 35338807
Is there a way to check the battery without actually opening  the case and pull the card? Wouldn't the diag say the battery was bad ?
0
 

Author Comment

by:gsswho6
ID: 35342171
ok some major problems now. I rebooted the server last night to see if that would help with all the errors. But it seems to have gotted worse. Ever so often exchange just stops working. Nobody can connect to their mailboxes. I check the server and the store is running.  Below is the first error I see before many others. I have to stop store and restart store in order for everything to come back up. But I know some time later today it will happen again. Any help please would be great.

nformation Store (9364) First Storage Group: An attempt to read from the file "E:\Program Files\Exchsrvr\MDBDATA\priv1.edb" at offset 14144163840 (0x000000034b0ed000) for 12288 (0x00003000) bytes failed after 1388 seconds with system error 1117 (0x0000045d): "The request could not be performed because of an I/O device error. ".  The read operation will fail with error -1022 (0xfffffc02).  If this error persists then the file may be damaged and may need to be restored from a previous backup.

Does this have anything to do with the fact one of my drive in my RAID5 is still not functioning? Would it be best for me to just yank that drive out for now? It almost seems like that drive being in the server trying to rebuild/etc is causing issues.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35342614
Do you have support for the SAN?  If so, I would call the vendor ASAP.

If you have a bad drive in a RAID5 array you will experience performance problems.  Although you should be able to put a new drive in and have it rebuild on the fly, depending on how much read/write activity is going on and how big the array is, it could take awhile.
0
 

Author Comment

by:gsswho6
ID: 35342760
There is no SAN. Just 3 different array on the local server. C = raid 1, D = raid 1 and E = raid 5.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35342797
My fault, I meant support for the E200 controller.  It sounds like it is having problems.
0
 

Author Comment

by:gsswho6
ID: 35342987
Got this message on the ACU.

Background parity initialization is currently queued or in progress on logical drive 3 ( RAID 5 in array C). If background parity initialization is queued, it will start when I/O is performed on the drive. When background parity initialization completes, the performance of the logical drive will improve.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35344768
That makes it sound as if it is attempting to rebuild the missing data.

When did you put in the new drive?
0
 

Author Comment

by:gsswho6
ID: 35344808
A few days ago. But with all the problems I yanked the drive and put it back in to see if that helped. It scared me at first becuase I got a popup saying all data was lost. Then after about 30 secs it gave that message above. Still getting ton of Echange errors in event viewer but for now all exchange services are running and mail is flowing. HP said to leave it alone and let it finish and then mayeb the exchange error will go away once done. I hope so... At this point I want to throw the server in the ocean.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35345246
Well, not knowing a lot about the performance of these drives, the controller and the load on the server it could take awhile to rebuild the missing data.  As in hours if not a day.  I doubt it would take longer than a day though.
0
 

Author Comment

by:gsswho6
ID: 35351607
Well as of now exchange is down completely. server rebooted. Now ACU says no problems. No longer says dirty data. But now exchange is down and the store says database corrupt when trying to mount it. So I have all services shutoff and doing a backup of the store. Since I cannot run a backup at all when mounted I guess this is all I can do atm. I guess when backup is done ill try to start it back online. If it doesnt work I guess I need to eseutil /mh ?

All this from a failed drive :(
0
 

Author Comment

by:gsswho6
ID: 35351751
esetil /mh dump. So what do to do now?



Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
         Database: e:/program files/exchsrvr/mdbdata/priv1.edb

        File Type: Database
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,11
 Engine ulVersion: 0x620,11
Created ulVersion: 0x620,11
     DB Signature: Create time:12/07/2007 20:13:31 Rand:1867250 Computer:
         cbDbPage: 4096
           dbtime: 3083065799 (0xb7c3d9c7)
            State: Dirty Shutdown
     Log Required: 169474-169474 (0x29602-0x29602)
   Streaming File: Yes
         Shadowed: Yes
       Last Objid: 1724987
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
 Old Repair Count: 0
  Last Consistent: (0x29602,69,16)  04/08/2011 06:40:00
      Last Attach: (0x29602,70,103)  04/08/2011 07:19:10
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:12/07/2007 20:13:29 Rand:1871123 Computer:
       OS Version: (5.2.3790 SP 2)

Previous Full Backup:
        Log Gen: 164723-164726 (0x28373-0x28376)
           Mark: (0x28373,1755,1D2)
           Mark: 03/11/2011 23:01:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: found (1561)
Last Bad Checksum Error Date: 04/08/2011 06:38:52
Old bad Checksum Error Count: none

Operation completed successfully in 3.109 seconds.
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35351789
This really should be opened as a new question.. but just for quick reference look here:

http://social.technet.microsoft.com/Forums/en/exchangesvravailabilityandisasterrecovery/thread/0a365814-5b03-48e6-bfd2-ce5416cef806
0
 

Author Comment

by:gsswho6
ID: 35351935
Ok so since I do not have a full backup from this pre all this mess I should go the Eseutil /R route? I am currently doing a backup now but that wont suffice right since its a backup of "now" which is already problematic. Hope that made sense. Thanks for the help guys... Appreciated much
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35351959
If you want to move quicker just make sure the all exchange services are stopped and backup (like copy / paste) the database and logs in a separate folder if you have room.. then start all services but the information store and run the /r recovery mode.
0
 

Author Comment

by:gsswho6
ID: 35351999
When you say all service you mean all exchange services except the store service correct?

thanks
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352002
yes
0
 

Author Comment

by:gsswho6
ID: 35352036
Will do thanks. The backup of the DB is almost done... And I am coping the logs to another location for backup since those logs currently reside on a different partition then the db. Once that is done ill give that a shot. appreciate the help.
0
 

Author Comment

by:gsswho6
ID: 35352082
Also... Just to make sure my syntax is correct. It would be the same command I issued above but instead of /mh I use the /r flag correct? I see some posts about /i and also E00 .
0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 35352104
The /r will attempt to repair the log files.

There is a /p that attempt to repair a corrupted data store.

Brief overview:

http://www.computerperformance.co.uk/exchange2003/exchange2003_eseutil.htm#Eseutil%20/mh
0
 

Author Comment

by:gsswho6
ID: 35352133
ok so I should be using eseutil /r E00 /i then correct? Since this is telling it to start with log E00 and try to repair from that point in?
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352151
If your logs are in a different drive / path you need to specify that as well.

http://technet.microsoft.com/en-us/library/bb123479(EXCHG.80).aspx
0
 

Author Comment

by:gsswho6
ID: 35352194
ok.. yea my logs are on D:\Program Files\Exchsrvr... DC is on E:\Program Files\Exchsrvr\MDBDATA.

So I am confused lol.
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352209
Are the *.log files in one directory and the priv.edb in another?
0
Promote certifications in your email signature

Has your company recently won an award or achieved a certification? They'll no doubt want to show it off. Email signature images used to promote certifications & awards can instantly establish credibility with a recipient and provide you with numerous benefits.

 

Author Comment

by:gsswho6
ID: 35352237
Yes. logs = D:\Program Files\Exchsrvr   priv1.edb = E:\Program Files\Exchsrvr\MDBDATA
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352248
Eseutil /R E00 /LD:\Program Files\Exchsrvr
0
 

Author Comment

by:gsswho6
ID: 35352251
What does /LD do? Sorry I though i had to point to logs as well?
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352256
/L (Logs)

The D is the beginning of the directory structure
0
 

Author Comment

by:gsswho6
ID: 35352259
oh ok I see that was pointing to logs... So i dont actually point to my database in any way?
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352261
no it knows where that is
0
 

Author Comment

by:gsswho6
ID: 35352289
what am i doing wrong?


C:\Program Files\Exchsrvr\bin>eseutil /R E00 /LD:\program files\exchsrvr

Usage Error: Invalid argument 'files\exchsrvr'. Options must be preceded by '-'
or '/'.

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

DESCRIPTION:  Maintenance utilities for Microsoft(R) Exchange Server databases.

MODES OF OPERATION:
      Defragmentation:  ESEUTIL /d <database name> [options]
             Recovery:  ESEUTIL /r <logfile base name> [options]
            Integrity:  ESEUTIL /g <database name> [options]
             Checksum:  ESEUTIL /k <file name> [options]
               Repair:  ESEUTIL /p <database name> [options]
            File Dump:  ESEUTIL /m[mode-modifier] <filename>
            Copy File:  ESEUTIL /y <source file> [options]
              Restore:  ESEUTIL /c[mode-modifier] <path name> [options]

<<<<<  Press a key for more help  >>>>>
D=Defragmentation, R=Recovery, G=inteGrity, K=checKsum,
P=rePair, M=file duMp, Y=copY file, C=restore
=>
0
 

Author Comment

by:gsswho6
ID: 35352308
nevermind i got it... forgot the " ".. says performing soft recovery... crosses finger :(
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35352326
Try:

eseutil /R E00 /L"D:\program files\exchsrvr"

It looks like it is getting upset because of the space between "program" and "files".
0
 

Author Comment

by:gsswho6
ID: 35352331
Yea thanks... I see in the event logs tons of replaying log messages. so far no errors which is good i assume. lol
0
 

Author Comment

by:gsswho6
ID: 35352374
This process take awhile? I assume it would since the log folder is 16gb in size.
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352432
Be sure if this all works out for you that you give the points to giltjr as he was the main help to your original question.
0
 

Author Comment

by:gsswho6
ID: 35352445
Oh no Got this error. what to do now? So stressful :(


C:\Program Files\Exchsrvr\bin>eseutil /R E00 /L"D:\program files\exchsrvr"

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode...
    Logfile base name: E00
            Log files: D:\program files\exchsrvr
         System files: <current directory>

Performing soft recovery...



Operation terminated with error -1018 (JET_errReadVerifyFailure, Checksum error
on a database page) after 885.0 seconds.



0
 

Author Comment

by:gsswho6
ID: 35352522
next step is eseutil /p ?
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352563
0
 

Author Comment

by:gsswho6
ID: 35352573
huh? Not sure what that means? How is that possible?
0
 

Author Comment

by:gsswho6
ID: 35352589
the /mh worked. How can this be a version problem? sorry not sure what to do at this point.
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352591
What version of Exchange.. this really should be a new question since it seems the Raid one has been resolved.
0
 

Author Comment

by:gsswho6
ID: 35352616
exchange 2003.
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35352637
0
 

Author Comment

by:gsswho6
ID: 35352639
I ran eseutil /k and there are tons and tons of page xxxxx checksum failed messages
0
 

Author Comment

by:gsswho6
ID: 35352738
just ran eseutil /p against the DB. Will let you know the results.
0
 

Author Comment

by:gsswho6
ID: 35352786
Got about a thousand of the below so far. Scanning status of eseutil /p is currently at 40%.

eseutil (6920) The database engine lost one page of bad data. It is highly recommended that an application-level integrity check of the database be run to ensure application-level data integrity.
0
 

Author Comment

by:gsswho6
ID: 35352968
Now its doing Deleting unicode fixup table... I notice that my priv1.edb file is growing in size while its doing this.
0
 

Author Comment

by:gsswho6
ID: 35353518
Jeez it takes a long time. I wish it would have some kind of indicator on what % complete it was at or something.
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35353562
Depending on the size of the DB and logs it could take hours.. Be sure to follow that guide step by step.
0
 

Author Comment

by:gsswho6
ID: 35353577
Yes I am... The edb file is 16gb. So I assume 2ish hours?

After this is eseutil /d... Then mount store/unmount store and run isinteg -s (SERVER) -fix -test alltests.
0
 

Author Comment

by:gsswho6
ID: 35354726
Still going... Almost 4 hours doing the deleting unicode fixup table.
0
 

Author Comment

by:gsswho6
ID: 35355321
OK it finished... Now trying to run a defrag and getting the below.

C:\Program Files\Exchsrvr\bin>eseutil /d "e:\program files\exchsrvr\mdbdata\priv
1.edb"

Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating DEFRAGMENTATION mode...
            Database: e:\program files\exchsrvr\mdbdata\priv1.edb
      Streaming File: e:\program files\exchsrvr\mdbdata\priv1.STM
      Temp. Database: TEMPDFRG1840.EDB
Temp. Streaming File: TEMPDFRG1840.STM

                  Defragmentation Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|




Operation terminated with error -1605 (JET_errKeyDuplicate, Illegal duplicate ke
y) after 6.0 seconds.
0
 

Author Comment

by:gsswho6
ID: 35355379
This was log from the repair. Now nothing runs... cant mount store either


Repair completed. Database corruption has been repaired!

Note:
  It is recommended that you immediately perform a full backup
  of this database. If you restore a backup made before the
  repair, the database will be rolled back to the state
  it was in at the time of that backup.



Operation completed successfully with 595 (JET_wrnDatabaseRepaired, Database cor
ruption has been repaired) after 19190.62 seconds.


C:\Program Files\Exchsrvr\bin>eseutil /d "e:\program files\exchsrvr\mdbdata\priv
1.edb"
0
 

Author Comment

by:gsswho6
ID: 35355402
After the repair I try to mount the store and I get the error :

An internal processing error has occurred. Try restarting the ESM or the Store service or both.

I have done both and no luck. Any ideas? reboot maybe?
0
 
LVL 24

Expert Comment

by:DMTechGrooup
ID: 35355756
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

We are happy to announce a brand new addition to our line of acclaimed email signature management products – CodeTwo Email Signatures for Office 365.
Resolve DNS query failed errors for Exchange
In this video we show how to create a Resource Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: Navigate to the Recipients >> Resources tab.: "Recipients" is our default selection …
how to add IIS SMTP to handle application/Scanner relays into office 365.

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now