Solved

Huge file systems on Linux (Experienced Admin Please)

Posted on 2010-11-10
16
601 Views
Last Modified: 2012-05-10
My company just had a terriable experience with an XFS file system with more than 13 TB of data.  I need the advise of the most experienced Admins, which file system should I use?  We probably will have to recreate the filesystem so it is time to find the options, pros and cons.
0
Comment
Question by:farzanj
  • 4
  • 4
  • 3
  • +5
16 Comments
 
LVL 4

Assisted Solution

by:zgiuffria
zgiuffria earned 25 total points
Comment Utility
This may help you.  I like ext3.  Has been the most reliable for me...
http://en.wikipedia.org/wiki/Comparison_of_file_systems
0
 
LVL 4

Assisted Solution

by:torque_200bc
torque_200bc earned 25 total points
Comment Utility
ext3: too slow for such a huge FS and quit deprecated. Must not use an inode based fs. Better try another B-tree based FS (like your old XFS). I would try the "new" Btrfs
0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 200 total points
Comment Utility
This is not an easy question, it depends on your workload. What are these 13 TB of date comprised of?

Small files,
lots of directories,
Big files,
what is the read/write ratio,
how many concurrent access,
....?

I also have had bad experiences with XFS, currently I am testing BTRFS, but it is not really stable, i guess it will need some few more month.

You have to experiment which filesystem is right for your workload.

if you provide us with more information we can better advice you.


for most people ext3 or ext4 is an excellent choice, very stable.

if you really need a superstable store for 13 TB you might want to look at (open)solaris and ZFS, which is like BTRFS but superstable, but not nativly available on linux.


best regards
Ray
0
 
LVL 31

Author Comment

by:farzanj
Comment Utility
We have lots of files and directories.  Files are not large at all.  Not whole lot of concurrent access.

Read/Write ratio is almost equal.  We are creating backups in and out of this filesystem so the ratios are equal.
0
 
LVL 14

Assisted Solution

by:small_student
small_student earned 100 total points
Comment Utility
As long as it is for Backups that does not mean a lot of access , so I would go for ext4 because

1- Its stable
2- It has encryption features that are suitable for backups
3- It has a journal so in case while long writes are happening and something went wrong the FS can cope with such incidents without data loss
0
 
LVL 7

Accepted Solution

by:
Hatrix76 earned 200 total points
Comment Utility
Well, in this case I would recommend ext4, as you have really lots of files and directories you should take care of creating lot's of inodes to be sure to not run out of inodes later on.

As you do not have a lot of access the overhead should be insignificant. But increased inodes mean increased overhead.

If you want even better performance put the journal for the ext4 filesystem on an extra harddrive (should be a fast drive, e.g. 7200 rpms for IDE/SATA, 10.000 rpms for SCSI/SAS, 15.000 rpms for SCSI).

Another benefit ext4 will provide is that when BTFS is production-ready, you can make a online migration from ext4 to BTRFS, and assuming you have enough disk-space, you can even rollback to ext4 if you find any issues with BTRFS.

It would be your best bet for now on Linux.

best regards
Ray
0
 
LVL 16

Expert Comment

by:Gerald Connolly
Comment Utility
what problems did you have with xfs?
0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 150 total points
Comment Utility
I'm still trying to think of a good reason to have a single 14TB filesystem in the first place, esp. given the sizes of the files (or: Often a crazy solution is a result of a crazy premise)

Cheers,
-Jon
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 31

Author Comment

by:farzanj
Comment Utility
Problem with XFS:  It was unstable and got corrupted.  Moreover, we tried to recover it but it did not.  Initially it was not even showing the file system.  When e.g. I did ls -l and it would show question marks.  Then we tried to run file system repair command and it stopped every time in the middle with segmentation fault.  We just lost confidence on XFS and wanted a stable and recoverable and stable filesystem.
0
 
LVL 31

Author Comment

by:farzanj
Comment Utility
Reason for a large filesystem:  The server is used for backups and also for transfer of backups.  I am new in the organization and there is little in my control.  It appears that they have NetApps which has snapshots.  This system takes local backups--it is located in the same data center as the NetApps and other source computers.  Then the data is moved to another location -- remote backup.  That is done through this computer too.  Things are not done the ideal way and so is the situation with most organizations that I have consulted with.  But I want to learn the correct ways and strive to work on ideal practices that is why I pick brains of experts like all of you.  I really appreciate participation of all of you!!
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
go with solaris and ZFS INSTEAD
0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 150 total points
Comment Utility
The reason I was asking if there exists a good reason for such a large filesystem (and it appears that there does not) is that *now* is the chance to fix such organizational errors (while you're rebuilding it from scratch) rather than down the road when it's next to impossible to reconfigure.  Reducing this bloated storage system into several, more well-organized and smaller filesystems will certainly lessen the demands of each, and would obviously greatly reduce the chance of a corruption of this scope in the future.

In other words, your filesystem does not need to be over-the-top if your demands are not over-the-top.  Or, to put it another way, anything will eventually melt down if you demand it exceed its capabilities - why not reduce the demand?

The solution is not to implement the superman of filesystems.  The solution is to render the superman of filesystems unneeded.

Cheers,
-Jon
0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 200 total points
Comment Utility
Look Jon, he stated that he is the new guy at a company and that he has not much control over the situation. It is difficult to propose a change if you are the new guy.

Also, it why complicate matters, I knew a lots of clients of ours who have multi-terrabyte extfs filesystems on filersystems which are doing just fine, I do not think changing the filesystem-structure to something different would help much there.

I personally had never an EXTFS(3/4) crashed on me without recoverability in my 12 yeas of linux experience. I had unusable raiser and xfs filesystems. And I am not sure if I had terrible crashes with ext2, but i guess so.

best, ray


0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 150 total points
Comment Utility
"Look Jon, he stated that he is the new guy at a company and that he has not much control over the situation. It is difficult to propose a change if you are the new guy."

Look, Hatrix, he said the old filesystem was broken beyond repair, and he also said he was looking to change the root cause of the problem.  It is an indisputable fact that having a large corruption (say, 17TB worth of corrupted data) is worse than having a small corruption (say 3-4TB of data).  Regardless of which filesystem type is eventually implemented, splitting a huge storage system into several smaller ones achieves this.  I don't know why you'd try to argue with that.  

Also, the fact that you have sold this idea (i.e. a huge unorganized filesystem) to some number of people doesn't mean its right.  Instead of arguing with me based on tradition (i.e. "We've been doing it this way for years"), can you try debating the technical superiority of your methodology?  Isn't that what the author wants?

Cheers,
-Jon




0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 200 total points
Comment Utility
Look again, Jon,

I won't argue with you. I just personally do not see 15 Terabytes as a really huge number anymore. 5-10 years ago it was huge, today it's moderate at best. Huge for today's standards are petabyte storage systems IMHO.

I would be with you if this filesystem would be provided from a Linux server, with hardware raid and a bunch of disks through'n into it, but he said they are using NetApps for storage, so, I really do not see your point, a NetApp is designed to handle large storage. EXT4 is a good and reliable filesystem.

best,
Ray
0
 
LVL 31

Author Closing Comment

by:farzanj
Comment Utility
Thank you all, seniors.  I appreciate everyone's help.  There may be small difference of opinions but you were all trying to help me and I am grateful for that.
0

Featured Post

Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

Join & Write a Comment

How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now