Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Huge file systems on Linux (Experienced Admin Please)

Posted on 2010-11-10
16
Medium Priority
?
609 Views
Last Modified: 2012-05-10
My company just had a terriable experience with an XFS file system with more than 13 TB of data.  I need the advise of the most experienced Admins, which file system should I use?  We probably will have to recreate the filesystem so it is time to find the options, pros and cons.
0
Comment
Question by:farzanj
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
  • 3
  • +5
16 Comments
 
LVL 4

Assisted Solution

by:zgiuffria
zgiuffria earned 100 total points
ID: 34104525
This may help you.  I like ext3.  Has been the most reliable for me...
http://en.wikipedia.org/wiki/Comparison_of_file_systems
0
 
LVL 4

Assisted Solution

by:torque_200bc
torque_200bc earned 100 total points
ID: 34105232
ext3: too slow for such a huge FS and quit deprecated. Must not use an inode based fs. Better try another B-tree based FS (like your old XFS). I would try the "new" Btrfs
0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 800 total points
ID: 34105817
This is not an easy question, it depends on your workload. What are these 13 TB of date comprised of?

Small files,
lots of directories,
Big files,
what is the read/write ratio,
how many concurrent access,
....?

I also have had bad experiences with XFS, currently I am testing BTRFS, but it is not really stable, i guess it will need some few more month.

You have to experiment which filesystem is right for your workload.

if you provide us with more information we can better advice you.


for most people ext3 or ext4 is an excellent choice, very stable.

if you really need a superstable store for 13 TB you might want to look at (open)solaris and ZFS, which is like BTRFS but superstable, but not nativly available on linux.


best regards
Ray
0
How To Reduce Deployment Times With Pre-Baked AMIs

Even if we can't include all the files in the base image, we can sometimes include some of the larger files that we would otherwise have to download, and we can also sometimes remove the most time-consuming steps. This can help a lot with reducing deployment times.

 
LVL 31

Author Comment

by:farzanj
ID: 34106306
We have lots of files and directories.  Files are not large at all.  Not whole lot of concurrent access.

Read/Write ratio is almost equal.  We are creating backups in and out of this filesystem so the ratios are equal.
0
 
LVL 14

Assisted Solution

by:Monis Monther
Monis Monther earned 400 total points
ID: 34106385
As long as it is for Backups that does not mean a lot of access , so I would go for ext4 because

1- Its stable
2- It has encryption features that are suitable for backups
3- It has a journal so in case while long writes are happening and something went wrong the FS can cope with such incidents without data loss
0
 
LVL 7

Accepted Solution

by:
Hatrix76 earned 800 total points
ID: 34106449
Well, in this case I would recommend ext4, as you have really lots of files and directories you should take care of creating lot's of inodes to be sure to not run out of inodes later on.

As you do not have a lot of access the overhead should be insignificant. But increased inodes mean increased overhead.

If you want even better performance put the journal for the ext4 filesystem on an extra harddrive (should be a fast drive, e.g. 7200 rpms for IDE/SATA, 10.000 rpms for SCSI/SAS, 15.000 rpms for SCSI).

Another benefit ext4 will provide is that when BTFS is production-ready, you can make a online migration from ext4 to BTRFS, and assuming you have enough disk-space, you can even rollback to ext4 if you find any issues with BTRFS.

It would be your best bet for now on Linux.

best regards
Ray
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 34115226
what problems did you have with xfs?
0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 600 total points
ID: 34125476
I'm still trying to think of a good reason to have a single 14TB filesystem in the first place, esp. given the sizes of the files (or: Often a crazy solution is a result of a crazy premise)

Cheers,
-Jon
0
 
LVL 31

Author Comment

by:farzanj
ID: 34129168
Problem with XFS:  It was unstable and got corrupted.  Moreover, we tried to recover it but it did not.  Initially it was not even showing the file system.  When e.g. I did ls -l and it would show question marks.  Then we tried to run file system repair command and it stopped every time in the middle with segmentation fault.  We just lost confidence on XFS and wanted a stable and recoverable and stable filesystem.
0
 
LVL 31

Author Comment

by:farzanj
ID: 34129179
Reason for a large filesystem:  The server is used for backups and also for transfer of backups.  I am new in the organization and there is little in my control.  It appears that they have NetApps which has snapshots.  This system takes local backups--it is located in the same data center as the NetApps and other source computers.  Then the data is moved to another location -- remote backup.  That is done through this computer too.  Things are not done the ideal way and so is the situation with most organizations that I have consulted with.  But I want to learn the correct ways and strive to work on ideal practices that is why I pick brains of experts like all of you.  I really appreciate participation of all of you!!
0
 
LVL 47

Expert Comment

by:David
ID: 34129246
go with solaris and ZFS INSTEAD
0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 600 total points
ID: 34130037
The reason I was asking if there exists a good reason for such a large filesystem (and it appears that there does not) is that *now* is the chance to fix such organizational errors (while you're rebuilding it from scratch) rather than down the road when it's next to impossible to reconfigure.  Reducing this bloated storage system into several, more well-organized and smaller filesystems will certainly lessen the demands of each, and would obviously greatly reduce the chance of a corruption of this scope in the future.

In other words, your filesystem does not need to be over-the-top if your demands are not over-the-top.  Or, to put it another way, anything will eventually melt down if you demand it exceed its capabilities - why not reduce the demand?

The solution is not to implement the superman of filesystems.  The solution is to render the superman of filesystems unneeded.

Cheers,
-Jon
0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 800 total points
ID: 34130647
Look Jon, he stated that he is the new guy at a company and that he has not much control over the situation. It is difficult to propose a change if you are the new guy.

Also, it why complicate matters, I knew a lots of clients of ours who have multi-terrabyte extfs filesystems on filersystems which are doing just fine, I do not think changing the filesystem-structure to something different would help much there.

I personally had never an EXTFS(3/4) crashed on me without recoverability in my 12 yeas of linux experience. I had unusable raiser and xfs filesystems. And I am not sure if I had terrible crashes with ext2, but i guess so.

best, ray


0
 
LVL 16

Assisted Solution

by:The--Captain
The--Captain earned 600 total points
ID: 34134011
"Look Jon, he stated that he is the new guy at a company and that he has not much control over the situation. It is difficult to propose a change if you are the new guy."

Look, Hatrix, he said the old filesystem was broken beyond repair, and he also said he was looking to change the root cause of the problem.  It is an indisputable fact that having a large corruption (say, 17TB worth of corrupted data) is worse than having a small corruption (say 3-4TB of data).  Regardless of which filesystem type is eventually implemented, splitting a huge storage system into several smaller ones achieves this.  I don't know why you'd try to argue with that.  

Also, the fact that you have sold this idea (i.e. a huge unorganized filesystem) to some number of people doesn't mean its right.  Instead of arguing with me based on tradition (i.e. "We've been doing it this way for years"), can you try debating the technical superiority of your methodology?  Isn't that what the author wants?

Cheers,
-Jon




0
 
LVL 7

Assisted Solution

by:Hatrix76
Hatrix76 earned 800 total points
ID: 34134468
Look again, Jon,

I won't argue with you. I just personally do not see 15 Terabytes as a really huge number anymore. 5-10 years ago it was huge, today it's moderate at best. Huge for today's standards are petabyte storage systems IMHO.

I would be with you if this filesystem would be provided from a Linux server, with hardware raid and a bunch of disks through'n into it, but he said they are using NetApps for storage, so, I really do not see your point, a NetApp is designed to handle large storage. EXT4 is a good and reliable filesystem.

best,
Ray
0
 
LVL 31

Author Closing Comment

by:farzanj
ID: 34136140
Thank you all, seniors.  I appreciate everyone's help.  There may be small difference of opinions but you were all trying to help me and I am grateful for that.
0

Featured Post

Learn how to optimize MySQL for your business need

With the increasing importance of apps & networks in both business & personal interconnections, perfor. has become one of the key metrics of successful communication. This ebook is a hands-on business-case-driven guide to understanding MySQL query parameter tuning & database perf

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
When we purchase storage, we typically are advertised storage of 500GB, 1TB, 2TB and so on. However, when you actually install it into your computer, your 500GB HDD will actually show up as 465GB. Why? It has to do with the way people and computers…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question