Solved

Unable to run fsck on ext4 on a 16 TB drive array

Posted on 2010-08-16
8
1,321 Views
Last Modified: 2013-12-02
Hello I have a major issue with a Logical Volume.
I am running Fedora Core 11 64bit with a 16 TB ext4 volume.
The server had a hardware failure that caused the Volume to HARD crash on three occasions.
 
Now I am receiving These error messages " kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269694"

I looked up these errors and found the solution of these to be to run fsck on the volume.

I have tried to run fsck.ext for  on /dev/VolGroup001/Bigvol  and /dev/dm-3

Once I got the drive completely unmounted via a reboot with commented out of fstab
I ran e2fsck -pf /dev/VolGroup001/Bigvol  

This is the error that was returned
"e2fsck 1.41.9  Error determining size of the physical device: File too large"

My Question is this:First on Fedora Core 11 they don't have a YUM package for 1.14.11 . I I use a RPM from fedora12 or a general RPM to upgrade?

Second : When I am able to run fsck on my 16 TB drive what switches should I use and what will I lose and what will I gain?

Thanks for the help
0
Comment
Question by:pjinlaok
  • 5
  • 3
8 Comments
 
LVL 11

Expert Comment

by:Pieter Jordaan
ID: 33453731

Try fsck.ext4
0
 

Author Comment

by:pjinlaok
ID: 33456435
I have tried the fsck.ext4 command. When I run the command I actually get the result from e2fsck running as fsck.ext4.


It seems that there is a known bug in fsck.ext4 on large volumes for version 1.14.9. The fix is in 1.14.11 but I can't find a version for Fedora Core 11.

It is available on FC12 but I don't want to upgrade.

Any other thoughts?
0
 
LVL 11

Expert Comment

by:Pieter Jordaan
ID: 33456513

I wouldn't take any chances with different tools.

Rather boot with the FC12 CD in recovery mode, and run the repair.
0
Free Webinar: AWS Backup & DR

Join our upcoming webinar with experts from AWS, CloudBerry Lab, and the Town of Edgartown IT to discuss best practices for simplifying online backup management and cutting costs.

 

Author Comment

by:pjinlaok
ID: 33456857
That's a great idea. Have you seen this specific issue before?

Any idea on if fsck fixing the errors that I am receiving?

Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269689
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 5908783
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6291470
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269694
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6312512
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269536
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6269689
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 5908783
Aug 17 10:21:02 bigvolserver kernel: EXT4-fs error (device dm-3): ext4_lookup: deleted inode referenced: 6291470
bigvolserver
0
 
LVL 11

Expert Comment

by:Pieter Jordaan
ID: 33457044

I have never used an ext4 partition of over 8TB, so this is a new problem for me.

I have seen many corrupted inodes be delete after a power failure.
ext4 has an increased intolerance to unclean shutdowns.

Rather download Fedora Core 13 for the file system check. Some bugs were fixed on the latest release.

0
 

Author Comment

by:pjinlaok
ID: 33470787
I already have the fedora core 12 64 bit DVD so I tried that first. I was able to boot into the recovery console. So far I can't run fsck on /dev/VolGroup001/Bigvol . Error cannot find superblock.

I could tell that the 16 TB drive = /dev/sda
when I run lvdisplay it shows me /dev/VolGroup001/Bigvol
But ?volGoup001/Bigvol doesn't show up in dev to run the fsck against.

Can you give me the steps to fsck a LVM on a rescue disk?

I am downloading over a torrent fedora core 13 64 bit DVD which will take a few more hours.

I am going to attempt the FSCK with 12 and if it fails I will try the fedora 13 DVD.
0
 

Author Comment

by:pjinlaok
ID: 33471027
I am unable to FSCK the two ext4 LVM's from the Fedora Core 12 rescue disk.
When I boot into Rescue mode off of the FC12 DVD I receive an error when processing the large volumes. I get a message "error processing /de/sda Do you want to re-initialize the drive - Warning DATA LOSS WILL OCCUR when RE_INITIALIZING.

I am going to attempt to upgrade to FC12 and hopefully it won't break anything.

 
0
 

Accepted Solution

by:
pjinlaok earned 0 total points
ID: 33489500
I had attempted to boot off of the Ferdora Core 12 DVD and run fsck.ext4, but I received a error processing /dev/sda Skip - Read only - or Initialize - Warning - INITIALIZATION WILL CAUSE DATA LOSS.

 So when I booted the LVM /dev/VolGrou001/Bigvol is not available to do a fsck on.

I then Upgraded to Ferdora core 12 but it only has e2fsprogs V 1.14-9 and it still did not allow to do a fsck on the 16 TB volume. yum would not upgrade beyond this version.

I finally downloaded the latest source from http://e2fsprogs.sourceforge.net/

ran ./configure
make
make install

Bam! it worked! I had to run e2fsck 4 times before all the errors were fixed, but it fixed all the errors and I am back up and running.


0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
We look at whether swapping a controller board on a failed hard drive is likely to solve the problem.
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…

679 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question