centos - large data LVM questions
I have to put together a storage system for 20TB - 50TB data with a CentOS head server.
Can someone explain the best way to provision and configure the storage?
The data will be almost exclusively read-only, with growth of 3-5TB/yr.
I will have a SAS-connected SAN with either 1TB or 2TB disks. The SAN has a max of 16 disks in a RAID5 (or RAID6) array, so I assume I'll need more than 1 array at the storage level.
This means several LUNS presented to the OS.
Assuming I use the 1TB disks, and create 3 different RAID 5 arrays, I'll have 45TB of usable space.
Do I create smaller LUNS to present to the OS, and LVM to create a single large storage pool?
Does the entire Volume Group max out at 16TB? or is each LV a max of 16TB?
Can someone explain the best way to provision and configure the storage?
The data will be almost exclusively read-only, with growth of 3-5TB/yr.
I will have a SAS-connected SAN with either 1TB or 2TB disks. The SAN has a max of 16 disks in a RAID5 (or RAID6) array, so I assume I'll need more than 1 array at the storage level.
This means several LUNS presented to the OS.
Assuming I use the 1TB disks, and create 3 different RAID 5 arrays, I'll have 45TB of usable space.
Do I create smaller LUNS to present to the OS, and LVM to create a single large storage pool?
Does the entire Volume Group max out at 16TB? or is each LV a max of 16TB?
ASKER
Thanks for the response - let me clarify my questions:
The san supports 16 disks in a *single* raid5 array, not 16 disks total. I will need expansion shelves to accommodate 48 (or more with hot spares), but can configure 3 arrays of 16 disks.
I'm more concerned with how to address the disk space from the OS.
Assuming I have 45TB of usable disk space - how do I best configure that in the OS?
Will I need several LVM's smaller than 16TB and break up the data?
The san supports 16 disks in a *single* raid5 array, not 16 disks total. I will need expansion shelves to accommodate 48 (or more with hot spares), but can configure 3 arrays of 16 disks.
I'm more concerned with how to address the disk space from the OS.
Assuming I have 45TB of usable disk space - how do I best configure that in the OS?
Will I need several LVM's smaller than 16TB and break up the data?
Best is not to allocate all at the same time. You can add LUN of say your data +50%, then extend it AS/IF needed.
Why dont you get something like FreeNAS with de-duplicating filesystem, and NFS support so you get maximum kick from your storage?
Why dont you get something like FreeNAS with de-duplicating filesystem, and NFS support so you get maximum kick from your storage?
ASKER
Same question applies....it doesn't matter *when* I break the 16TB barrier. I *will* break it.
The initial data seed will be >16TB (didn't mention in OP, sorry), so I need more than 16TB right away.
The question is - *HOW* do I extend it past 16TB, not AS/IF I will need to.
Thanks.
The initial data seed will be >16TB (didn't mention in OP, sorry), so I need more than 16TB right away.
The question is - *HOW* do I extend it past 16TB, not AS/IF I will need to.
Thanks.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
CentOS 7 will default to XFS, that solves all your concerns.
ASKER
Thanks for the replies - you may have answered my question without knowing.
In short - I need *more than 16TB* usable space in a single "directory" - which may contain a single DB larger than 16TB or a "zillion" smaller files.
pitoren: I can create 3 PV's (i.e. the storage presents the LUN's to the OS) of 15TB, then a single volume group from the PV's, and a single Logical Volume of 45TB, but I need to format the partition with XFS.
(the storage is a dual-controller SAN with multi-pathing to the OS, and the SAN supports up to 16 disks per RAID-5 LUN).
gheist: you're saying if my OS is CentOS 7, it will default to XFS for such a LV.
Please confirm - I'll break out points
In short - I need *more than 16TB* usable space in a single "directory" - which may contain a single DB larger than 16TB or a "zillion" smaller files.
pitoren: I can create 3 PV's (i.e. the storage presents the LUN's to the OS) of 15TB, then a single volume group from the PV's, and a single Logical Volume of 45TB, but I need to format the partition with XFS.
(the storage is a dual-controller SAN with multi-pathing to the OS, and the SAN supports up to 16 disks per RAID-5 LUN).
gheist: you're saying if my OS is CentOS 7, it will default to XFS for such a LV.
Please confirm - I'll break out points
CentOS7 defaults to XFS for any install. You can still choose ext2 if you think filesystem log is a virtue, or ext4 to stay in stone age.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
He wants to exceed 50TB, so RHEL7 EXT4 will not suffice.
gheist: That's not written in the post, which starts with
"I have to put together a storage system for 20TB - 50TB data with a CentOS head server."
and in another post the OP suggest he\ll use 3x15TB LUNs.
But as I said, I'd suggest to use xfs, with the caveats written. You can obviously grow both xfs and ext4, to their respective limits, if more storage becomes availalble needed.
And think about how you are going to back the data up. (If you think "I don't need backups" I don't want to be in your shoes if something goes wrong).
P
"I have to put together a storage system for 20TB - 50TB data with a CentOS head server."
and in another post the OP suggest he\ll use 3x15TB LUNs.
But as I said, I'd suggest to use xfs, with the caveats written. You can obviously grow both xfs and ext4, to their respective limits, if more storage becomes availalble needed.
And think about how you are going to back the data up. (If you think "I don't need backups" I don't want to be in your shoes if something goes wrong).
P
Anyway they need to partition data.. About data locality, map-reduce and all that stuff.
ASKER
Thanks for the replies - sorry for my delays. This has been a back-burner/need-it-now/ba
THANKS!
THANKS!
how do you figure?
if that san supports up to 16 disks, you will only have some 14tb usable with 1tb drives; about 28tb with 2tb drives and a bit less of both if raid 6
depends on what version and what file system
RHEL/CentOS 6 has a 16tb file system limit for ext3/4 (due to version of e2fsprogs package)
you can format xfs beyond 16tb which should be very good for mostly static data
the more you break up your raid groups, the less usable space you will have since you will accumulate more parity drives