Need recommendation for the right storage solution for my needs (financial tick data history)

I wish to purchase hard disks for purpose of storing financial tick-data. As per my calculations, 6 TB will be required for 3 months of history. Beyond that I may later decide to keep deleting if I don't need or to buy additional storage if I certainly must store longer history.

I do not want to risk losing any of the latest 3 month historical data due to hard disk failure, that is why I intended to set-up RAID 5.

My question is: Which hard disks should I buy?
My constraints/doubts are:
1)  I don't wish to spend money on SDDs, it's hefty price is not worth the additional read-write speed for my purpose.
2)  I immediately want to buy just one hard disk to take one time data load from someone, but want to make sure I can use this hard disk later (in a matter of  weeks)  when I do end up buying other hard disks, and whatever else hardware is required to set-up RAID etc.
3) Will setting up RAID 5, using say four 2 TB HDDs give me 6 TB of storage?
4) Will this RAID system necessarily need a container (rack) and cooling ? Any recommendations for a cheap but reliable rack if so?  
4) I need the hard disks to be easily removable and mobile (not crash due to speed bumps on road), because I might need to, say once a month, take one out and transport it locally to get it loaded etc.  (can one even do that with a RAID 5? i.e. take out one disk, get more data on it and put it back in the rack and expect all disks to magically "get that data"?)
5) I am not sure whether I will use this on linux or windows yet, so are there hard disks and RAID that will work on either linux or windows? Or is that a decision I must make now?
6) is NAS something I need here? Or that's an overkill for my purpose? (A cursory look at NAS prices for 4 HDD racks, sounded expensive to me)
 
Regards,
Siddharth
sidvermAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AlanConsultantCommented:
Hi Siddharth,

1)  I don't wish to spend money on SDDs, it's hefty price is not worth the additional read-write speed for my purpose.

I agree - stick with spinning disks for now.  I would do a $ / TB calculation and purchase whatever drives are currently in the sweet spot wherever you are.  That might be 2TB or 4TB drives for example.  Be wary of 'run out' deals - see below.  

2)  I immediately want to buy just one hard disk to take one time data load from someone, but want to make sure I can use this hard disk later (in a matter of  weeks)  when I do end up buying other hard disks, and whatever else hardware is required to set-up RAID etc.

Makes perfect sense to me.  I would make sure that whatever you buy is 'generally available' so that it should remain so for the foreseeable future.  No guarantees of course, but if you buy an end of line drive, it might be out of stock in a few months.

3) Will setting up RAID 5, using say four 2 TB HDDs give me 6 TB of storage?

Yes - you lose one drive to the parity bits, so 4 x 2TB = (4-1) x 2TB = 6TB usable.

4A) Will this RAID system necessarily need a container (rack) and cooling ? Any recommendations for a cheap but reliable rack if so?  

I have called this one 4A.  Not at all - you could purchase a four-bay housing for your RAID array, and site it on your desk or wherever you like.  You don't need a rack at all, but a rack does help with keeping things neat, tidy, safe, and importantly, cool.

4B) I need the hard disks to be easily removable and mobile (not crash due to speed bumps on road), because I might need to, say once a month, take one out and transport it locally to get it loaded etc.  (can one even do that with a RAID 5? i.e. take out one disk, get more data on it and put it back in the rack and expect all disks to magically "get that data"?)

No - it won't generally work like that.  The drives are an array, and whilst you can remove one (to replace it), when you do, that one won't have readable data on it, and the array will now be exposed to loss of a drive (and all the data).  You could go to RAID6 which means you can stand the loss of any two drives for greater redundancy (but you lose the capacity of two drives, not just one).

5) I am not sure whether I will use this on linux or windows yet, so are there hard disks and RAID that will work on either linux or windows? Or is that a decision I must make now?

In my experience, the drives don't care, but the physical device that houses your array may do.  For me, I would use Linux for this, but if you are more comfortable with Windows, then that is fine too.

6) is NAS something I need here? Or that's an overkill for my purpose? (A cursory look at NAS prices for 4 HDD racks, sounded expensive to me)

As I mentioned above, you could house your array in an external device such as a NAS, and this is 'easy'.

You could also repurpose an old server with sufficient drive bays if you have one lying around.


Hope that helps,

Alan.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tiborg GuaranaIT beginsCommented:
I agree with the gentleman above me. But I would recommend NAS here, and for disk take enterprise models with 5years warranty.  Dont save money on items that protect your data which is critical and cant be lost :)
0
nobusCommented:
did you consider 4TB drives?  that would give you more space
i'd say, look at the wd PRO and gold ones :  https://www.deskdecode.com/top-best-4tb-hard-disk-drive-monthly-updated/
there are also 8 tb drives :  https://www.deskdecode.com/top-best-8tb-hdd-hard-disk-drive-monthly-updated/
all these have reasonable prices
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

sidvermAuthor Commented:
@nobus, The reason I didn't consider 4TB drives is that I perhaps may not require beyond 6 TB storage. But then I thought I cannot make a RAID 5 set-up with one 4TB and two 2TB drives, can I? So, I  end-up spending money on at least three 4 TB drives for RAID, getting 8 TB of storage, which I don't know I will need, and wasting 4 TB of space (and money). Correct me if there is a way another way to plan for 6 TB using 4 TB drives with RAID?
0
AlanConsultantCommented:
Hi,

You are correct.

If you use 4TB drives, you will 'lose' at least 4TB out of 12TB.

However, if you went with a solution such as Drobo, it works differently, so you can mix and match drive sizes.

Alan.
0
sidvermAuthor Commented:
@Alan, thanks a lot. In your first answer to point 4B, You said I won't be able to plug out, and plug in a hard disk from Raid 5 array. And then you recommended to use RAID 6. Is that all ? So, after I set-up RAID 6, will I simply be able to plug out a disk, transport it to my data vendor, load it there, and plug back in and the whole array will get that data? If not, then can you, or someone else here, tell me how I can achieve mobility along with RAID?
0
AlanConsultantCommented:
Hi,

No - RAID5 and RAID6 are the same with respect to individual disks.  They will not contain a complete viable data set that you can take elsewhere.

The difference between the two is that with RAID5, your 'array' can cope with the loss of one, and only one drive, without losing any date.

With RAID6, you can lose two, but no more than two disks, and still maintain your date.

To move your data you would need to copy it elsewhere.  For example, you might maintain a RAID5 array with four 2TB drives giving 6TB capacity, and have one (or more) separate 6TB drives that you take offsite with a second copy of the date.  That separate drive would 'stand alone'.

Another option would be to 'sync' the data over a net connection to a remote site.  The initial sync might take some time, but once done, it would probably be incremental so much faster.

Alan.
0
sidvermAuthor Commented:
ok, so I browsed for best 2TB hard drives from the same website as @nobus cited,
https://www.deskdecode.com/top-best-2tb-hard-disk-drive-monthly-updated/

The best among top four they recommend is what I searched for on amazon
https://www.amazon.in/Seagate-ST2000NM0033-Constellation-7200RPM-3-5quot/dp/B00A47FS92

But, at least on Amazon India, I got 0 reviews for this product. Instead in "compare with similar items" section in amazon below this product, I found another very similar spec HDD which has the most (140+) reviews, 4.0+ rating, *and* is one third as cheap as the product recommended above!

https://www.amazon.in/dp/B01IEKG402/ref=psdc_1375369031_t2_B00A47FS92

The only noticeable difference  in specs to me 64 MB cache vs 128 MB cache. I can't think of why the price should go up from INR 5000 to INR 14000 just for that?
I am very tempted to go for this cheaper one, unless one of you cautions me against it :)

Is it easily going to fit into any RAID array that I build later on? And can be physically carried to load some data on for my immediate need?  If these two are yes, I will most likely go ahead and buy this today.
0
Tiborg GuaranaIT beginsCommented:
Im affraid this is not the disk you want 14% of reviews gave 1 star. So beware. And when I check for Warranty Details: No Warranty.

I dont really advice you this disk. Go check reviews with 1 star on your own.

On first link you have 3 years warranty which is much better, but I would still go and check if you can get disk with 5 years.

They will fit into array. I usully pick 2 different manufacturers while building raid.
You can load data asap, you only need to own enclousure if you want to use it as external disk, or connect it inside PC.

Good luck with shopping.
0
EirmanChief Operations ManagerCommented:
This type of case will provide extra protection when transporting bare hard drives.
0
andyalderCommented:
The difference between the Seagate Barracuda and Constellation is that the Barracuda is a desktop drive and the Constellation is a server drive. The actual internals are fairly similar but the Constellation has a second processor on the control board for better head stability in multi-disk arrays, workload management etc. If you compare images of the two you can see the Constellation has a bigger PCB for the extra components. , https://www.seagate.com/files/docs/pdf/whitepaper/mb538-drive-selection-guide-us.pdf (the doc is a bit out of date, higher capacities and transfer rates are now available). This plus the fact that they sell a lot more Barracudas explains the price difference.

The Barracuda is not suited to RAID 5/6 on a hardware RAID controller running 24/7, however it is suitable for RAID use in some external enclosures that support desktop drives in a low workload environment and low-end NAS servers.
0
sidvermAuthor Commented:
Thanks a lot everyone, I have ordered one 2TB external hard disk for now for my immediate need. I will also need to use this as a mode of transport of my data once a month as I understand that taking out a disk, loading it, and re-inserting in RAID 5 is not feasible.

@andylander Seagate Constellation is what looks fit for my requirement, as my load will be closer to 24x7 than 5x8 and I do  need higher reliability that Barracuda doesn't seem to offer. Also, I want to use RAID 5, not settle for RAID 0 or 1.  But I will revisit this next week.
0
AlanConsultantCommented:
Question appears to be answered.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.