Solved

Unable to run fsck on ext4 on a 16 TB drive array

Posted on 2010-08-16
8
1,305 Views
Last Modified: 2013-12-02
Hello I have a major issue with a Logical Volume.
I am running Fedora Core 11 64bit with a 16 TB ext4 volume.
The server had a hardware failure that caused the Volume to HARD crash on three occasions.
 
Now I am receiving These error messages " kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269694"

I looked up these errors and found the solution of these to be to run fsck on the volume.

I have tried to run fsck.ext for  on /dev/VolGroup001/Bigvol  and /dev/dm-3

Once I got the drive completely unmounted via a reboot with commented out of fstab
I ran e2fsck -pf /dev/VolGroup001/Bigvol  

This is the error that was returned
"e2fsck 1.41.9  Error determining size of the physical device: File too large"

My Question is this:First on Fedora Core 11 they don't have a YUM package for 1.14.11 . I I use a RPM from fedora12 or a general RPM to upgrade?

Second : When I am able to run fsck on my 16 TB drive what switches should I use and what will I lose and what will I gain?

Thanks for the help
0
Comment
Question by:pjinlaok
  • 5
  • 3
8 Comments
 
LVL 11

Expert Comment

by:Pieter Jordaan
ID: 33453731

Try fsck.ext4
0
 

Author Comment

by:pjinlaok
ID: 33456435
I have tried the fsck.ext4 command. When I run the command I actually get the result from e2fsck running as fsck.ext4.


It seems that there is a known bug in fsck.ext4 on large volumes for version 1.14.9. The fix is in 1.14.11 but I can't find a version for Fedora Core 11.

It is available on FC12 but I don't want to upgrade.

Any other thoughts?
0
 
LVL 11

Expert Comment

by:Pieter Jordaan
ID: 33456513

I wouldn't take any chances with different tools.

Rather boot with the FC12 CD in recovery mode, and run the repair.
0
 

Author Comment

by:pjinlaok
ID: 33456857
That's a great idea. Have you seen this specific issue before?

Any idea on if fsck fixing the errors that I am receiving?

Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269689
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 5908783
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6291470
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269694
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6312512
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269536
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269689
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 5908783
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6291470
bigvolserver
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 11

Expert Comment

by:Pieter Jordaan
ID: 33457044

I have never used an ext4 partition of over 8TB, so this is a new problem for me.

I have seen many corrupted inodes be delete after a power failure.
ext4 has an increased intolerance to unclean shutdowns.

Rather download Fedora Core 13 for the file system check. Some bugs were fixed on the latest release.

0
 

Author Comment

by:pjinlaok
ID: 33470787
I already have the fedora core 12 64 bit DVD so I tried that first. I was able to boot into the recovery console. So far I can't run fsck on /dev/VolGroup001/Bigvol . Error cannot find superblock.

I could tell that the 16 TB drive = /dev/sda
when I run lvdisplay it shows me /dev/VolGroup001/Bigvol
But ?volGoup001/Bigvol doesn't show up in dev to run the fsck against.

Can you give me the steps to fsck a LVM on a rescue disk?

I am downloading over a torrent fedora core 13 64 bit DVD which will take a few more hours.

I am going to attempt the FSCK with 12 and if it fails I will try the fedora 13 DVD.
0
 

Author Comment

by:pjinlaok
ID: 33471027
I am unable to FSCK the two ext4 LVM's from the Fedora Core 12 rescue disk.
When I boot into Rescue mode off of the FC12 DVD I receive an error when processing the large volumes. I get a message "error processing /de/sda Do you want to re-initialize the drive - Warning DATA LOSS WILL OCCUR when RE_INITIALIZING.

I am going to attempt to upgrade to FC12 and hopefully it won't break anything.

 
0
 

Accepted Solution

by:
pjinlaok earned 0 total points
ID: 33489500
I had attempted to boot off of the Ferdora Core 12 DVD and run fsck.ext4, but I received a error processing /dev/sda Skip - Read only - or Initialize - Warning - INITIALIZATION WILL CAUSE DATA LOSS.

 So when I booted the LVM /dev/VolGrou001/Bigvol is not available to do a fsck on.

I then Upgraded to Ferdora core 12 but it only has e2fsprogs V 1.14-9 and it still did not allow to do a fsck on the 16 TB volume. yum would not upgrade beyond this version.

I finally downloaded the latest source from http://e2fsprogs.sourceforge.net/

ran ./configure
make
make install

Bam! it worked! I had to run e2fsck 4 times before all the errors were fixed, but it fixed all the errors and I am back up and running.


0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Bellevue, WA, USA; Late December, 1980 "OK, folks, we nailed the IBM contract. We're going to have many meetings like this to discuss features and functions...and how to move from 86-DOS, the CP/M clone we just purchased, to our baby, MS-DOS. Thi…
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now