Solved

how to recover file names in lost+found folder under CENTOS?

Posted on 2012-12-27
22
1,423 Views
Last Modified: 2013-01-03
Dear experts,
I've came across a tricky situation and your advice will be very much appreciated,
I have an Elastix (Centos + Asterisk + FREEPBX) server running at work.
the server froze today due to some failure (I believe it to be a hardware issue related to the raid card) , command line did not react nor "ALT + CTRL + DEL" so I had to force the power off.

On boot I've noticed the server couldn't mount the raid ext3 partition called "/record",
Next I've ran fsck on the raid path, when fsck.ext3 finished it left me with 210 GB of files under lost+found folder with broken file names.

all of the files are *.wav files (call recording) and I must correct the file names in order they can be of any use (the file name linked to a database entry with call information).

I've tried searching the web for a solution but it appears I'm f***ed, I must restore the original file names.

Thanks in advance.
0
Comment
Question by:shootbox
  • 12
  • 10
22 Comments
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
If you want to know for sure if the filenames are hidden somewhere on the HDD so they can potentially be recovered, then I would use the strings command on /dev/md0  (assuming this is md0) and grep for one of the longer, more obscure filenames.  If the strings | grep fails, then you are totally screwed.  If you get the string, and then try a few more files and get the string, then you know a forensic recovery is possible.

R-studio is relatively inexpensive and it MAY get your data back, but if we're talking thousands of dollars worth of lost data, or tens of thousands of dollars, just shut everything down now and contact somebody like ontrack.   I personally wouldn't trust anybody else with ext3/ext4 recovery.

If you are using HARDWARE RAID, then the file names could very well be hidden in a stale parity block, and then you need to attack it differently, but you are still out of your realm and need to hire a pro.

Expect professional recovery for this type of data to cost $5000+, but ontrack will give free estimate.
0
 

Author Comment

by:shootbox
Comment Utility
we're talking about Hardware raid 1 based on 2 SAMSUNG 2TB Drives, data worth is somewhere near $10,000.
is there any way to revert back the fcsk command? (I guees not...)
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
No way to turn a 0 into a 1 or vice-versa.  Turn it all off, and make a bit-level copy with dd onto a scratch drive.  Then at least you can get a demo version of the r-studio software and let it do what it can do with the copy.  In such cases you do not want to touch the original data.

A hardware RAID controller will typically put metadata starting at physical block 0, for a few MB,but it is all vendor/product specific. but the rstudio software will have no problems figuring that out.
0
 

Author Comment

by:shootbox
Comment Utility
@dlethe you are basically saying there is a good chance R-Studio will be able to recover the real file names? or just use it to clone the data? the data is quite useless without correct file names.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
I am saying that this software is best chance at a $200 solution before trying a $5000 solution.  But if it was me, I would write a quick shell script to grep for some filenames on the raw device just to see if they show up.  If they don't, you are absolutely screwed because the strings corresponding to file names just aren't there.    If the strings do show up, then you have a shot.

Be sure to use grep with -i option for case insensitivity and depending on your flavor of linux there may be a grep option for binary files.

Please make a bit-level copy NOW, and take the HDD offline, you must not mount it, unless the mount is read-only.
0
 

Author Comment

by:shootbox
Comment Utility
Where Filenames are stored? Is it part of the file or should I look in another place?
Consider we're discussing hardware raid
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
I know for a fact that WAV files do NOT contain the file name as any part of the data structure.  File names will be backed up in the ext3 journal. That is why I told you to use the strings command on the raw physical drive to see if any filenames appear.

When you did the fsck without addressing root cause you absolutely affected any journal files and could have destroyed or rolled things back and overwrote the data that you need.  So before you spend any money trying to get file names back, see if those filenames even exist on the HDD.   If they don't, then the data is gone forever.  You could very well see the file names show up more than one, which would be an excellent indicator that professional recovery will get you what you need, or even the Rstudio software for $200 or so would work..

So just see if the filenames appear on the HDD. Good luck.
0
 

Author Comment

by:shootbox
Comment Utility
How do i use strings command on the raw drive?
0
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
Comment Utility
strings -a /dev/hda | grep -i 'filename'   (don't put in the .wav part)  so if you know you had a file called santa_got_run_over_by_a_reindeer.wav,  do

strings -a /dev/hda | grep -i 'santa_got_run_over_by_a_reindeer'

Note this tests contents of the files as well,  as a sanity check,  do a strings -a /dev/hda | grep -i 'boot'  and you will surely get some output
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
wait, hardware RAID???  then you have another alternative.  The data could be in the parity XOR data that is hidden from view.  That requires professional data recovery.  But this test is safe, as long as you mount the RAID READ-ONLY.

If the info doesn't show up, then be prepared for $5000+ (but free estimate), and call on track and talk to a human there and tell them what is going on.
0
 

Author Comment

by:shootbox
Comment Utility
Thanks for the advice, 5000$ IS NOT AN OPTION.
I'll check the strings command any perhaps it will help.
I've purchased Rstudio and a friend is helping me with prosoft data rescue, I'm not optimistic about that but I'll give that a chance.

Thanks for the help I'll update on progress.
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Please do, but remember, if the strings command doesn't show the file names, then the data isn't there any more.  

Specifically, what RAID controller, RAID chunk size, RAID level & number of disks do you have?  (The data could be in XOR parity chunks, but that depends on specifics of the RAID, and that info wouldn't show up on the strings, even if it was there).
0
 

Author Comment

by:shootbox
Comment Utility
it's a PCI sil Image controller for 4 SATA Drives, Currently we have 2 HD's attached (2TB each)
working on RAID 1 (mirror), not sure about the chunk size tho'.

RStudio is scanning the disks right now, I've choose the best scanning option - 6 hrs to go...
will update then.

thank you for you help so far, I'll keep posting on progress.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
RAID1 is all i needed to know.  The chunk size won't matter. There isn't any new information one could get from the RAID1 that would be hidden away.   crossing fingers for you
0
 

Author Comment

by:shootbox
Comment Utility
Rstudio finished scanning the raid and it seems it shows up same as listing the folder regularly.
I've noticed there is another folder called metafiles with names similar to the filenames, does it help?

I'm currently restoring the data to another HD.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Rstudio looks at raw disk and tries to piece together what it can by using orphaned "chunks" and piecing together deleted files and fragments.   But like I said, if the filenames you are looking for do not appear when doing that strings / grep exercise on the raw physical device, then they are gone forever.

Now with restored data you can do checksums of individual files and use that information to at least see what files are on the backup that are not on the munged image.  Then you at least know what files are unaccounted for and minimize the lost data.  Maybe it is a small number.
0
 

Author Comment

by:shootbox
Comment Utility
what about the metadata?

check out the capture
Capture.JPG
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
YOu killed it??  No, you have to let it run to completion or you pretty much only get some previously deleted files.  You must let it run for 1-2 days or however as long as it takes.  Note, more RAM and 64-bit processor helps. It is well worth temporarily taking RAM from another machine to let it run.  Also, 64-bit O/S makes big difference also.
0
 

Author Comment

by:shootbox
Comment Utility
It's runs on a i7 2.8 ghz with 4 GB ram which it barely uses.

Do I have to restore all of it? because it sees lots of partitions (from old uses of the HD's)
each of the HD were in diffrent uses before and it shows me 32TB of DATA LOL...

I haven't killed it, I'm recovering only the ext3 partition right now...

btw.. I've getting lots of errors trying to restore the meta files... see attachment
Capture.JPG
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Just let it run 100%. Errors are expected, you already know the filesystem has damage. That is why you are running the software to begin with.    Memory utilization can change significantly over time, depending on what phase it is in and exactly what is going on.  Granted you have a small filesystem and you will need less RAM then if you had a 2 TB filesystem, as example.
0
 

Author Comment

by:shootbox
Comment Utility
100% means 32TB of data from 2x 2TB HD?
do you think it's possible to recover the sound files with the names from another partition (from ages ago) lets say a NTFS?

I appreciate your help a lot, I'm working on a backup plan right now... If the recovery will fail I'll have to write some kind of script that will create the file names from the date stamp of the file (which seems to be correct)
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
You only need to have the software scan the partition(s) that held the data, but even if that represents 10TB, then it is your only hope of recovery.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

This comprehensive conference-networking guide will help you prep, practice and pack for success, reach out with purpose and confidence, capitalize on connections, and turn all those new leads into long-term connections.
A Bare Metal Image backup allows for the restore of an entire system to a similar or dissimilar hardware. They are highly useful for migrations and disaster recovery. Bare Metal Image backups support Full and Incremental backups. Differential backup…
In this Micro Tutorial viewers will learn how they can get their files copied out from their unbootable system without need to use recovery services. As an example non-bootable Windows 2012R2 installation is used which has boot problems.
In this Micro Tutorial viewers will learn how to restore single file or folder from Bare Metal backup image of their system. Tutorial shows how to restore files and folders from system backup. Often it is not needed to restore entire system when onl…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now