Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 748
  • Last Modified:

Need High-performance multi-read/write hard drive

I have a big data solution I am working on and need to store a vast trove of data on disk to be served via a complementary API hosting solution. For the initial deployment, the total amount of data might exceed 2TB, but grow to be twice that or more. High speed read/write times, and preferably, multi-read/write capabilities and built-in redundancy, will be essential for steady and reliable performance. Cost is also somewhat of a concern.

As such, I am looking for suggestions as to which equipment might best be suited for this purpose. Any ideas?

Thanks
0
jdannemann
Asked:
jdannemann
  • 4
  • 2
2 Solutions
 
tmoore1962Commented:
Is the data just files or a DB?  If just files you could get a NAS device that uses dual gigabit NIC's (supports teaming) and uses SAS drives if you want the speed I'd go with something like the HP AG652A and put in 15K SAS drives.  Team the NICs for best transfer speed.
0
 
Duncan MeyersCommented:
Low cost. Reliability. Performance. Choose any two.

Can you put some number around the expected read/write performance? Are you likely to need to perform frequent ETL (Extract, Transform, Load) operations from a separate database? What is the expected read profile?

2TB to 4TB of data is easy to store and there's no shortage of high-performance options out in the marketplace, includung all flash storage arrays and dedicated big data hardware solutions.

Speaking of which, take a look at the Greenplum Community Edition. You can download it here: http://gopivotal.com/pivotal-products/data/pivotal-greenplum-database
0
 
jdannemannAuthor Commented:
@meyersd and @tmoore1962

To answer your questions, the drives will only store text files and the "database server" is actually a program I am writing that will utilize the information and provide an API to it. Mostly, the drives just need to deliver the requested files in the most timely fashion possible. Since the servers will be requesting information from the drive constantly, and with some level of concurrency involved, my guess is that purchasing hard drives with multi- read/write capabilities could help prevent typical performance bottlenecks. Where possible, I will also use caching.

Anyway, thank you both for the suggestions. If you have any more, I'm all ears. I will look into what suggestions you have made so far and get back to this thread later.

Cheers :)
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
Duncan MeyersCommented:
What's your budget?
0
 
jdannemannAuthor Commented:
@meyersd At or around $500
0
 
Duncan MeyersCommented:
A couple of SSDs and a couple SAS or SATA drives with ZFS running on them will give you good performance and acceptable reliability.
0
 
Duncan MeyersCommented:
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 4
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now