?
Solved

centos - large data LVM questions

Posted on 2014-10-08
13
Medium Priority
?
319 Views
Last Modified: 2014-11-17
I have to put together a storage system for 20TB - 50TB data with a CentOS head server.

Can someone explain the best way to provision and configure the storage?

The data will be almost exclusively read-only, with growth of 3-5TB/yr.

I will have a SAS-connected SAN with either 1TB or 2TB disks.  The SAN has a max of 16 disks in a RAID5 (or RAID6) array, so I assume I'll need more than 1 array at the storage level.

This means several LUNS presented to the OS.

Assuming I use the 1TB disks, and create 3 different RAID 5 arrays, I'll have 45TB of usable space.

Do I create smaller LUNS to present to the OS, and LVM to create a single large storage pool?

Does the entire Volume Group max out at 16TB?  or is each LV a max of 16TB?
0
Comment
Question by:snowdog_2112
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
  • 3
  • +1
13 Comments
 
LVL 35

Expert Comment

by:Seth Simmons
ID: 40369965
...I'll have 45TB of usable space

how do you figure?
if that san supports up to 16 disks, you will only have some 14tb usable with 1tb drives; about 28tb with 2tb drives and a bit less of both if raid 6

Does the entire Volume Group max out at 16TB?

depends on what version and what file system
RHEL/CentOS 6 has a 16tb file system limit for ext3/4 (due to version of e2fsprogs package)
you can format xfs beyond 16tb which should be very good for mostly static data

the more you break up your raid groups, the less usable space you will have since you will accumulate more parity drives
0
 

Author Comment

by:snowdog_2112
ID: 40370533
Thanks for the response - let me clarify my questions:

The san supports 16 disks in a *single* raid5 array, not 16 disks total.  I will need expansion shelves to accommodate  48 (or more with hot spares), but can configure 3 arrays of 16 disks.

I'm more concerned with how to address the disk space from the OS.

Assuming I have 45TB of usable disk space - how do I best configure that in the OS?

Will I need several LVM's smaller than 16TB and break up the data?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40372305
Best is not to allocate all at the same time. You can add LUN of say your data +50%, then extend it AS/IF needed.
Why dont you get something like FreeNAS with de-duplicating filesystem, and NFS support so you get maximum kick from your storage?
0
Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

 

Author Comment

by:snowdog_2112
ID: 40375920
Same question applies....it doesn't matter *when* I break the 16TB barrier.  I *will* break it.

The initial data seed will be >16TB (didn't mention in OP, sorry), so I need more than 16TB right away.

The question is - *HOW* do I extend it past 16TB, not AS/IF I will need to.

Thanks.
0
 
LVL 2

Assisted Solution

by:pitoren
pitoren earned 2000 total points
ID: 40376073
Your question is not entirely clear.

As long as you are on 64 bit CPU and recent OS/software versions you are not going to hit lvm limits.

If you had

3x  (16x1TB disks in RAID5), you could create 3x 15TB PVs, create a single VG, and a single LV. Then you need a file system and XFS  would be OK.

I would probably personally prefer 8 disks per RAID5 LUN. And multiple dual port HBAs for both multi patching and load-balancing. For this to work best, you need to check how your SAN storage supports LUN failover.

And none of this will work that great if you have zillions of tiny files.  You need files of a decent size.

P
0
 
LVL 62

Expert Comment

by:gheist
ID: 40376335
CentOS 7 will default to XFS, that solves all your concerns.
0
 

Author Comment

by:snowdog_2112
ID: 40406876
Thanks for the replies - you may have answered my question without knowing.

In short - I need *more than 16TB* usable space in a single "directory" - which may contain a single DB larger than 16TB or a "zillion" smaller files.

pitoren: I can create 3 PV's (i.e. the storage presents the LUN's to the OS) of 15TB, then a single volume group from the PV's, and a single Logical Volume of 45TB, but I need to format the partition with XFS.
(the storage is a dual-controller SAN with multi-pathing to the OS, and the SAN supports up to 16 disks per RAID-5 LUN).

gheist: you're saying if my OS is CentOS 7,  it will default to XFS for such a LV.

Please confirm - I'll break out points
0
 
LVL 62

Expert Comment

by:gheist
ID: 40407894
CentOS7 defaults to XFS for any install. You can still choose ext2 if you think filesystem log is a virtue, or ext4 to stay in stone age.
0
 
LVL 2

Accepted Solution

by:
pitoren earned 2000 total points
ID: 40408065
Hi

I think ghesit got his ext2 and ext4 mixed up there.

In RHEL7 the 16TB limitation has been removed for ext4, so you can use xfs or ext4 for RHEL7.

RHE6 your only options is xfs.

I'd strongly suggest to go with xfs - Redhat support it to 500 TB or something. In RHEL7 its their default filesystem.

Be real careful if you have a single directory with zillions of small files - every time a file is created or deleted the directory inode has to be updated. There's a limit how fast things can go in that scenario.

good luck
P
0
 
LVL 62

Expert Comment

by:gheist
ID: 40408069
He wants to exceed 50TB, so RHEL7 EXT4 will not suffice.
0
 
LVL 2

Expert Comment

by:pitoren
ID: 40408100
gheist: That's not written in the post, which starts with

"I have to put together a storage system for 20TB - 50TB data with a CentOS head server."

and in another post the OP suggest he\ll use 3x15TB LUNs.

But as I said, I'd suggest to use xfs, with the caveats written. You can obviously grow both xfs and ext4, to their respective limits, if more storage becomes availalble needed.

And think about how you are going to back the data up.  (If you think "I don't need backups" I don't want to be in your shoes if something goes wrong).

P
0
 
LVL 62

Expert Comment

by:gheist
ID: 40408168
Anyway they need to partition data.. About data locality, map-reduce and all that stuff.
0
 

Author Closing Comment

by:snowdog_2112
ID: 40448068
Thanks for the replies - sorry for my delays.  This has been a back-burner/need-it-now/back-burner case.

THANKS!
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When we purchase storage, we typically are advertised storage of 500GB, 1TB, 2TB and so on. However, when you actually install it into your computer, your 500GB HDD will actually show up as 465GB. Why? It has to do with the way people and computers…
Many businesses neglect disaster recovery and treat it as an after-thought. I can tell you first hand that data will be lost, hard drives die, servers will be hacked, and careless (or malicious) employees can ruin your data.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
Suggested Courses
Course of the Month11 days, 14 hours left to enroll

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question