Link to home
Start Free TrialLog in
Avatar of shootbox
shootboxFlag for Italy

asked on

how to recover file names in lost+found folder under CENTOS?

Dear experts,
I've came across a tricky situation and your advice will be very much appreciated,
I have an Elastix (Centos + Asterisk + FREEPBX) server running at work.
the server froze today due to some failure (I believe it to be a hardware issue related to the raid card) , command line did not react nor "ALT + CTRL + DEL" so I had to force the power off.

On boot I've noticed the server couldn't mount the raid ext3 partition called "/record",
Next I've ran fsck on the raid path, when fsck.ext3 finished it left me with 210 GB of files under lost+found folder with broken file names.

all of the files are *.wav files (call recording) and I must correct the file names in order they can be of any use (the file name linked to a database entry with call information).

I've tried searching the web for a solution but it appears I'm f***ed, I must restore the original file names.

Thanks in advance.
Avatar of David
Flag of United States of America image

If you want to know for sure if the filenames are hidden somewhere on the HDD so they can potentially be recovered, then I would use the strings command on /dev/md0  (assuming this is md0) and grep for one of the longer, more obscure filenames.  If the strings | grep fails, then you are totally screwed.  If you get the string, and then try a few more files and get the string, then you know a forensic recovery is possible.

R-studio is relatively inexpensive and it MAY get your data back, but if we're talking thousands of dollars worth of lost data, or tens of thousands of dollars, just shut everything down now and contact somebody like ontrack.   I personally wouldn't trust anybody else with ext3/ext4 recovery.

If you are using HARDWARE RAID, then the file names could very well be hidden in a stale parity block, and then you need to attack it differently, but you are still out of your realm and need to hire a pro.

Expect professional recovery for this type of data to cost $5000+, but ontrack will give free estimate.
Avatar of shootbox


we're talking about Hardware raid 1 based on 2 SAMSUNG 2TB Drives, data worth is somewhere near $10,000.
is there any way to revert back the fcsk command? (I guees not...)
No way to turn a 0 into a 1 or vice-versa.  Turn it all off, and make a bit-level copy with dd onto a scratch drive.  Then at least you can get a demo version of the r-studio software and let it do what it can do with the copy.  In such cases you do not want to touch the original data.

A hardware RAID controller will typically put metadata starting at physical block 0, for a few MB,but it is all vendor/product specific. but the rstudio software will have no problems figuring that out.
@dlethe you are basically saying there is a good chance R-Studio will be able to recover the real file names? or just use it to clone the data? the data is quite useless without correct file names.
I am saying that this software is best chance at a $200 solution before trying a $5000 solution.  But if it was me, I would write a quick shell script to grep for some filenames on the raw device just to see if they show up.  If they don't, you are absolutely screwed because the strings corresponding to file names just aren't there.    If the strings do show up, then you have a shot.

Be sure to use grep with -i option for case insensitivity and depending on your flavor of linux there may be a grep option for binary files.

Please make a bit-level copy NOW, and take the HDD offline, you must not mount it, unless the mount is read-only.
Where Filenames are stored? Is it part of the file or should I look in another place?
Consider we're discussing hardware raid
I know for a fact that WAV files do NOT contain the file name as any part of the data structure.  File names will be backed up in the ext3 journal. That is why I told you to use the strings command on the raw physical drive to see if any filenames appear.

When you did the fsck without addressing root cause you absolutely affected any journal files and could have destroyed or rolled things back and overwrote the data that you need.  So before you spend any money trying to get file names back, see if those filenames even exist on the HDD.   If they don't, then the data is gone forever.  You could very well see the file names show up more than one, which would be an excellent indicator that professional recovery will get you what you need, or even the Rstudio software for $200 or so would work..

So just see if the filenames appear on the HDD. Good luck.
How do i use strings command on the raw drive?
Avatar of David
Flag of United States of America image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
wait, hardware RAID???  then you have another alternative.  The data could be in the parity XOR data that is hidden from view.  That requires professional data recovery.  But this test is safe, as long as you mount the RAID READ-ONLY.

If the info doesn't show up, then be prepared for $5000+ (but free estimate), and call on track and talk to a human there and tell them what is going on.
Thanks for the advice, 5000$ IS NOT AN OPTION.
I'll check the strings command any perhaps it will help.
I've purchased Rstudio and a friend is helping me with prosoft data rescue, I'm not optimistic about that but I'll give that a chance.

Thanks for the help I'll update on progress.
Please do, but remember, if the strings command doesn't show the file names, then the data isn't there any more.  

Specifically, what RAID controller, RAID chunk size, RAID level & number of disks do you have?  (The data could be in XOR parity chunks, but that depends on specifics of the RAID, and that info wouldn't show up on the strings, even if it was there).
it's a PCI sil Image controller for 4 SATA Drives, Currently we have 2 HD's attached (2TB each)
working on RAID 1 (mirror), not sure about the chunk size tho'.

RStudio is scanning the disks right now, I've choose the best scanning option - 6 hrs to go...
will update then.

thank you for you help so far, I'll keep posting on progress.
RAID1 is all i needed to know.  The chunk size won't matter. There isn't any new information one could get from the RAID1 that would be hidden away.   crossing fingers for you
Rstudio finished scanning the raid and it seems it shows up same as listing the folder regularly.
I've noticed there is another folder called metafiles with names similar to the filenames, does it help?

I'm currently restoring the data to another HD.
Rstudio looks at raw disk and tries to piece together what it can by using orphaned "chunks" and piecing together deleted files and fragments.   But like I said, if the filenames you are looking for do not appear when doing that strings / grep exercise on the raw physical device, then they are gone forever.

Now with restored data you can do checksums of individual files and use that information to at least see what files are on the backup that are not on the munged image.  Then you at least know what files are unaccounted for and minimize the lost data.  Maybe it is a small number.
what about the metadata?

check out the capture
YOu killed it??  No, you have to let it run to completion or you pretty much only get some previously deleted files.  You must let it run for 1-2 days or however as long as it takes.  Note, more RAM and 64-bit processor helps. It is well worth temporarily taking RAM from another machine to let it run.  Also, 64-bit O/S makes big difference also.
It's runs on a i7 2.8 ghz with 4 GB ram which it barely uses.

Do I have to restore all of it? because it sees lots of partitions (from old uses of the HD's)
each of the HD were in diffrent uses before and it shows me 32TB of DATA LOL...

I haven't killed it, I'm recovering only the ext3 partition right now...

btw.. I've getting lots of errors trying to restore the meta files... see attachment
Just let it run 100%. Errors are expected, you already know the filesystem has damage. That is why you are running the software to begin with.    Memory utilization can change significantly over time, depending on what phase it is in and exactly what is going on.  Granted you have a small filesystem and you will need less RAM then if you had a 2 TB filesystem, as example.
100% means 32TB of data from 2x 2TB HD?
do you think it's possible to recover the sound files with the names from another partition (from ages ago) lets say a NTFS?

I appreciate your help a lot, I'm working on a backup plan right now... If the recovery will fail I'll have to write some kind of script that will create the file names from the date stamp of the file (which seems to be correct)
You only need to have the software scan the partition(s) that held the data, but even if that represents 10TB, then it is your only hope of recovery.