Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Huge file systems on Linux (Experienced Admin Please)

Posted on 2010-11-10
16
Medium Priority
?
610 Views
Last Modified: 2012-05-10
My company just had a terriable experience with an XFS file system with more than 13 TB of data.  I need the advise of the most experienced Admins, which file system should I use?  We probably will have to recreate the filesystem so it is time to find the options, pros and cons.
0
Comment
Question by:farzanj
  • 4
  • 4
  • 3
  • +5
16 Comments
 
LVL 4

Assisted Solution

by:zgiuffria
zgiuffria earned 100 total points
ID: 34104525
This may help you.  I like ext3.  Has been the most reliable for me...
http://en.wikipedia.org/wiki/Comparison_of_file_systems
0
 
LVL 4

Assisted Solution

by:torque_200bc
torque_200bc earned 100 total points
ID: 34105232
ext3: too slow for such a huge FS and quit deprecated. Must not use an inode based fs. Better try another B-tree based FS (like your old XFS). I would try the "new" Btrfs
0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 800 total points
ID: 34105817
This is not an easy question, it depends on your workload. What are these 13 TB of date comprised of?

Small files,
lots of directories,
Big files,
what is the read/write ratio,
how many concurrent access,
....?

I also have had bad experiences with XFS, currently I am testing BTRFS, but it is not really stable, i guess it will need some few more month.

You have to experiment which filesystem is right for your workload.

if you provide us with more information we can better advice you.


for most people ext3 or ext4 is an excellent choice, very stable.

if you really need a superstable store for 13 TB you might want to look at (open)solaris and ZFS, which is like BTRFS but superstable, but not nativly available on linux.


best regards
Ray
0
Nothing ever in the clear!

This technical paper will help you implement VMware’s VM encryption as well as implement Veeam encryption which together will achieve the nothing ever in the clear goal. If a bad guy steals VMs, backups or traffic they get nothing.

 
LVL 31

Author Comment

by:farzanj
ID: 34106306
We have lots of files and directories.  Files are not large at all.  Not whole lot of concurrent access.

Read/Write ratio is almost equal.  We are creating backups in and out of this filesystem so the ratios are equal.
0
 
LVL 14

Assisted Solution

by:Monis Monther
Monis Monther earned 400 total points
ID: 34106385
As long as it is for Backups that does not mean a lot of access , so I would go for ext4 because

1- Its stable
2- It has encryption features that are suitable for backups
3- It has a journal so in case while long writes are happening and something went wrong the FS can cope with such incidents without data loss
0
 
LVL 7

Accepted Solution

by:
Hatrix76 earned 800 total points
ID: 34106449
Well, in this case I would recommend ext4, as you have really lots of files and directories you should take care of creating lot's of inodes to be sure to not run out of inodes later on.

As you do not have a lot of access the overhead should be insignificant. But increased inodes mean increased overhead.

If you want even better performance put the journal for the ext4 filesystem on an extra harddrive (should be a fast drive, e.g. 7200 rpms for IDE/SATA, 10.000 rpms for SCSI/SAS, 15.000 rpms for SCSI).

Another benefit ext4 will provide is that when BTFS is production-ready, you can make a online migration from ext4 to BTRFS, and assuming you have enough disk-space, you can even rollback to ext4 if you find any issues with BTRFS.

It would be your best bet for now on Linux.

best regards
Ray
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 34115226
what problems did you have with xfs?
0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 600 total points
ID: 34125476
I'm still trying to think of a good reason to have a single 14TB filesystem in the first place, esp. given the sizes of the files (or: Often a crazy solution is a result of a crazy premise)

Cheers,
-Jon
0
 
LVL 31

Author Comment

by:farzanj
ID: 34129168
Problem with XFS:  It was unstable and got corrupted.  Moreover, we tried to recover it but it did not.  Initially it was not even showing the file system.  When e.g. I did ls -l and it would show question marks.  Then we tried to run file system repair command and it stopped every time in the middle with segmentation fault.  We just lost confidence on XFS and wanted a stable and recoverable and stable filesystem.
0
 
LVL 31

Author Comment

by:farzanj
ID: 34129179
Reason for a large filesystem:  The server is used for backups and also for transfer of backups.  I am new in the organization and there is little in my control.  It appears that they have NetApps which has snapshots.  This system takes local backups--it is located in the same data center as the NetApps and other source computers.  Then the data is moved to another location -- remote backup.  That is done through this computer too.  Things are not done the ideal way and so is the situation with most organizations that I have consulted with.  But I want to learn the correct ways and strive to work on ideal practices that is why I pick brains of experts like all of you.  I really appreciate participation of all of you!!
0
 
LVL 47

Expert Comment

by:David
ID: 34129246
go with solaris and ZFS INSTEAD
0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 600 total points
ID: 34130037
The reason I was asking if there exists a good reason for such a large filesystem (and it appears that there does not) is that *now* is the chance to fix such organizational errors (while you're rebuilding it from scratch) rather than down the road when it's next to impossible to reconfigure.  Reducing this bloated storage system into several, more well-organized and smaller filesystems will certainly lessen the demands of each, and would obviously greatly reduce the chance of a corruption of this scope in the future.

In other words, your filesystem does not need to be over-the-top if your demands are not over-the-top.  Or, to put it another way, anything will eventually melt down if you demand it exceed its capabilities - why not reduce the demand?

The solution is not to implement the superman of filesystems.  The solution is to render the superman of filesystems unneeded.

Cheers,
-Jon
0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 800 total points
ID: 34130647
Look Jon, he stated that he is the new guy at a company and that he has not much control over the situation. It is difficult to propose a change if you are the new guy.

Also, it why complicate matters, I knew a lots of clients of ours who have multi-terrabyte extfs filesystems on filersystems which are doing just fine, I do not think changing the filesystem-structure to something different would help much there.

I personally had never an EXTFS(3/4) crashed on me without recoverability in my 12 yeas of linux experience. I had unusable raiser and xfs filesystems. And I am not sure if I had terrible crashes with ext2, but i guess so.

best, ray


0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 600 total points
ID: 34134011
"Look Jon, he stated that he is the new guy at a company and that he has not much control over the situation. It is difficult to propose a change if you are the new guy."

Look, Hatrix, he said the old filesystem was broken beyond repair, and he also said he was looking to change the root cause of the problem.  It is an indisputable fact that having a large corruption (say, 17TB worth of corrupted data) is worse than having a small corruption (say 3-4TB of data).  Regardless of which filesystem type is eventually implemented, splitting a huge storage system into several smaller ones achieves this.  I don't know why you'd try to argue with that.  

Also, the fact that you have sold this idea (i.e. a huge unorganized filesystem) to some number of people doesn't mean its right.  Instead of arguing with me based on tradition (i.e. "We've been doing it this way for years"), can you try debating the technical superiority of your methodology?  Isn't that what the author wants?

Cheers,
-Jon




0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 800 total points
ID: 34134468
Look again, Jon,

I won't argue with you. I just personally do not see 15 Terabytes as a really huge number anymore. 5-10 years ago it was huge, today it's moderate at best. Huge for today's standards are petabyte storage systems IMHO.

I would be with you if this filesystem would be provided from a Linux server, with hardware raid and a bunch of disks through'n into it, but he said they are using NetApps for storage, so, I really do not see your point, a NetApp is designed to handle large storage. EXT4 is a good and reliable filesystem.

best,
Ray
0
 
LVL 31

Author Closing Comment

by:farzanj
ID: 34136140
Thank you all, seniors.  I appreciate everyone's help.  There may be small difference of opinions but you were all trying to help me and I am grateful for that.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
How much do you know about the future of data centers? If you're like 50% of organizations, then it's probably not enough. Read on to get up to speed on this emerging field.
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses
Course of the Month9 days, 20 hours left to enroll

926 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question