designing a scalable Linux Storage System!


what is the best reasonable solution to build a scalable storage system using Linux with the following characters:

1- Open-Source based solution if possible (yet to be very trustful).
2- High availability (load balancing).
3- Good and secured Backup System.
4- Scalable and easy to add storage when needed.

the way it will work is like:
1- users will have access to the files through a special programmed client that we made (ssh based)
2- when the program upload and updates the files on the server it will keep revisions of the files.
3- each user will have a limited disk space.

my questions to help understand the case:
1- we need a centralized users server (user name and passwords)? how it will deal with multi servers?
2- Clustering?
3- ZFS (using Open Solaris)? for keeping file revisions?

we just need to have a full understanding first, and then decide.
it seems there is allot of ways to achieve the goal, and we look for the secure and realistic one for enterprise business grade,

Who is Participating?
MarkSlingsbyConnect With a Mentor Commented:
well.. thats quite a list of requirements!

Ok. From experience with these things and what we have been testing internally (as well as using) you have some options, but not all will meet our requirements 100%.

Solution 1: (cheap)
2 x File Server Systems running Nexenta with HA. I must admit that we havent tested the HA. But Nexenta runs a full web based GUI, has HA support as an optional addon.

Solution 2: (expensive)
Netapp with dual controller and CIFS license
The NetApp fully supports versioning, High Availability, snapshots and much more. It really is THE solution, but costly. (How much is your data worth)

Solution 3: (really cheap)
ZFS with rsync - Ubuntu has a port from OpenSolaris for Ubuntu. But dont think your versioning is going to work. ZFS is one of the best/most efficient file systems we have come across, its very similar to NetApp's WAFL file system. It gives you easy flexibility to add new disk to the pool and provision easily.

It sounds to me like you need a frontend machine with backend flexible storage. Use a Linux box with quotas and map that to a NetApp via CIFS. OR you can give your users direct CIFS access on the NetApp (CIFS is Samba/Windows file sharing)

ZFS & WAFL give you snapshotting facilities and are great for backups.
Gerald ConnollyCommented:
You havent told us what kind of performance you want!
But for ultimate scaleable performance, have you looked at things like Panasas (proprietary) or Lustre (Opensource (ish))
Have a look at this collection of file-systems to really blow your mind.
Data-BaseAuthor Commented:
Good info that we are going to test soon :-)

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.