Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Need High-performance multi-read/write hard drive

Posted on 2013-10-22
7
Medium Priority
?
740 Views
Last Modified: 2016-03-23
I have a big data solution I am working on and need to store a vast trove of data on disk to be served via a complementary API hosting solution. For the initial deployment, the total amount of data might exceed 2TB, but grow to be twice that or more. High speed read/write times, and preferably, multi-read/write capabilities and built-in redundancy, will be essential for steady and reliable performance. Cost is also somewhat of a concern.

As such, I am looking for suggestions as to which equipment might best be suited for this purpose. Any ideas?

Thanks
0
Comment
Question by:jdannemann
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
7 Comments
 
LVL 10

Assisted Solution

by:tmoore1962
tmoore1962 earned 1000 total points
ID: 39591822
Is the data just files or a DB?  If just files you could get a NAS device that uses dual gigabit NIC's (supports teaming) and uses SAS drives if you want the speed I'd go with something like the HP AG652A and put in 15K SAS drives.  Team the NICs for best transfer speed.
0
 
LVL 30

Accepted Solution

by:
Duncan Meyers earned 1000 total points
ID: 39592683
Low cost. Reliability. Performance. Choose any two.

Can you put some number around the expected read/write performance? Are you likely to need to perform frequent ETL (Extract, Transform, Load) operations from a separate database? What is the expected read profile?

2TB to 4TB of data is easy to store and there's no shortage of high-performance options out in the marketplace, includung all flash storage arrays and dedicated big data hardware solutions.

Speaking of which, take a look at the Greenplum Community Edition. You can download it here: http://gopivotal.com/pivotal-products/data/pivotal-greenplum-database
0
 
LVL 1

Author Comment

by:jdannemann
ID: 39594848
@meyersd and @tmoore1962

To answer your questions, the drives will only store text files and the "database server" is actually a program I am writing that will utilize the information and provide an API to it. Mostly, the drives just need to deliver the requested files in the most timely fashion possible. Since the servers will be requesting information from the drive constantly, and with some level of concurrency involved, my guess is that purchasing hard drives with multi- read/write capabilities could help prevent typical performance bottlenecks. Where possible, I will also use caching.

Anyway, thank you both for the suggestions. If you have any more, I'm all ears. I will look into what suggestions you have made so far and get back to this thread later.

Cheers :)
0
Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 39595477
What's your budget?
0
 
LVL 1

Author Comment

by:jdannemann
ID: 39601539
@meyersd At or around $500
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 39601798
A couple of SSDs and a couple SAS or SATA drives with ZFS running on them will give you good performance and acceptable reliability.
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 39601801
0

Featured Post

The top UI technologies you need to be aware of

An important part of the job as a front-end developer is to stay up to date and in contact with new tools, trends and workflows. That’s why you cannot miss this upcoming webinar to explore the latest trends in UI technologies!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
Backups and Disaster RecoveryIn this post, we’ll look at strategies for backups and disaster recovery.
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

661 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question