Need LOTS-O-STORAGE space. Can you help design it?
We are putting together a business plan for a new venture and there is one part that I need to get some clarity on. Online storage. This project might take up 500-800 TB of space. I am a LAN network guy and none of my networks has anything over 2 TB so my questions is, what is the best way to create an environment that can handle that amount of data? Most of the files will be 500-600MB. I have 5Gbps bandwidth that we can easily increase and plenty or physical storage space and power.
Direct questions:
1) I am most familiar with Windows but the 2TB volume size limit is a problem. Or is it?
2) I am not afraid of LINUX but what limitations would I run into with that?
3) What would the hardware setup on a large storage system like this look like? I.E. what components would I need to put together?
4) I have a pretty strong feeling that in the end, we will have to hire a company to design/build something like this like IBM or someone but I want some info so I am not just accepting the recommendation blindly.
Direct questions:
1) I am most familiar with Windows but the 2TB volume size limit is a problem. Or is it?
2) I am not afraid of LINUX but what limitations would I run into with that?
3) What would the hardware setup on a large storage system like this look like? I.E. what components would I need to put together?
4) I have a pretty strong feeling that in the end, we will have to hire a company to design/build something like this like IBM or someone but I want some info so I am not just accepting the recommendation blindly.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Dude you are going to need a serious SAN array
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Call EMC for some help on this.
I hope this helps !
I hope this helps !
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
For really huge filesystems like that you don't really want to put it on Windows.
If this is not just for an intellectual exercise but for an enterprise-class filesystem you probably should take the advice of the Experts that have suggested working with a major player in the storage field, to architect a solution for you.
If money is tight, and you have time and energy, you could possibly "roll your own" - but you should put it on Linux, using one of the industrial-strength filesystems like JFS, or if proprietary works for you, AIX, which has JFS as its native filesystem IIRC. It will regardless need large array cages.
If this is not just for an intellectual exercise but for an enterprise-class filesystem you probably should take the advice of the Experts that have suggested working with a major player in the storage field, to architect a solution for you.
If money is tight, and you have time and energy, you could possibly "roll your own" - but you should put it on Linux, using one of the industrial-strength filesystems like JFS, or if proprietary works for you, AIX, which has JFS as its native filesystem IIRC. It will regardless need large array cages.
Expanding on the filesystem thing, and why I suggested JFS - if you google "filesystems" you will come up with several pages that show a chart of all the filesystems developed so far (up to the date of the chart, of course) and what the limitations are.
JFS is the only one, to my knowledge, that has a theoretical *volume* limit in the multi-petabyte range, and if you want to grow the volume, will accept additional extents.
I think Sun's newly-opened-up filesystem might be a good alternative but I don't think it scales quite to that degree, unless I'm mistaken.
JFS is the only one, to my knowledge, that has a theoretical *volume* limit in the multi-petabyte range, and if you want to grow the volume, will accept additional extents.
I think Sun's newly-opened-up filesystem might be a good alternative but I don't think it scales quite to that degree, unless I'm mistaken.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
the emc Symmetrix scales to over a petabyte in a single system... may be the way to go for hardware instead of aggregating SAN systems.
ShineOn - yes, the EMC Symmetrix DMX-3 can do this in one frame (multiple cabinets). This would make scaling up easier and possibly more cost-effective than having to buy multiple storage 'heads' (backplanes, controllers and cache memory)...although, more backend controllers and cache have to be added as more disk is added at certain thresholds. Still, the operating firmware/software would only have to be purchased once, which could be a significant number if one had to buy say, 10 of another type of array. It would be interesting to see what the numbers looked like in that kind of comparison.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.