Best Way to achieve Fast Data Transfer? And, ideal way to deal with Duplicate Data?

I have a lot of data to move, and want to increase the speed of my data transfers.  I have two internal drives (C, D), 3 externally docked drives, and 3 external drives.  Running older AMD Phenom II X4 Processor, with an older mobo (ASUS Pro N78, I think it is…MS Vista), would like to consider building hubs into my system to accommodate all the drives so I can transfer data quicker between them.  eSATA would probably be fastest, but I don’t see many hubs for that available in Canada, nor do I know how I would set it up so I can plug everything in.  USB 3.0?  eSATA?  Hubs?  My system?  

I also believe I have a lot of duplicate data, and would like to clean that up to one original and one backup, and maintain that.  I realize now how I want to set it up/organize it, but what to do to get to that point?  Suggestions welcome to reduce my time requirement on that as well.  Thanks!
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Aaron TomoskyDirector of Solutions ConsultingCommented:
With that many external drives why not combine them into a raid 5 Drobo or similar?
CanadianJeffAuthor Commented:
Okay, that could be a good suggestion, Aaron.  But I haven't the first clue about RAID, what's involved, what a "Drobo" is, or what the benefits of that could be, other than I understand RAID means the drives write faster because they write across each other, somehow... sorry, I'm going to need a bit of an education, on top of any advice.
Aaron TomoskyDirector of Solutions ConsultingCommented:
Drobo is a brand o external storage chassis. The one I use most is a Drobo pro 2nd gen. It has 5 drive bays, usb3, FireWire, and esata. I put 5 2tb drives in and see 1 8tb drive. The reason it's only 8 an not 10 is because it uses raid5. One drive can fail and all I do is replace it and it rebuilds automatically. I lose no data. It also writes faster than a single drive because of the way it stripes data across all the drives. I use slow wd20ears drives and can do a sustained write of 150MBps over esata.
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

CanadianJeffAuthor Commented:
That's rather fascinating!  Sounds a tad expensive, though, for the Drobo?  Recommendations there?  So, a drive fails and it's no big deal... that's pretty cool.  It's like having the ultimate backup, then, isn't it?  The outlay for 5 2tb drives would be a bit steep, but could I do that with (a) my 2x1TB, 2x500GB drives, and (b) what about my closed external drives... would you just take those out of their cases?  

Now, it's intriguing that you suggest you use slower drives (ear?)... like 5400's?  Those are quite a bit cheaper than the 7200's, I might be able to do a few of those.  If I'm hearing you correctly, basically, speed of drive not as important when you use RAID5... 150MBps is very, very fast by any real world standards, I believe.  

Okay, so now that you have my attention... this is great stuff, Aaron;  getting one of these Drobo's, and filling it with drives, is there much to setting that up then?  And, what about the Operating system, should that be a part of the Drobo, or separate?
CanadianJeffAuthor Commented:
Oops... wanted to ask just how much of a speed difference there is between the 7200's and the 5400's when in a Drobo?  And, would it need to be consistent, all 5400's or all 7200's?
Unless you absolutely need the extra storage, I'd recommend RAID 10 over RAID 5. The reasons:

1) RAID 5 and power failures don't mix, so you'll need a UPS
2) When a drive fails in RAID 5, it can take several days to rebuild (depending on size). You have access to your data, but it's slower. Worst still, if a second drive fails during this period, all data is lost. RAID 10 takes time to rebuild, but since it's only writing to a single drive instead of all 4 drives, it's much faster, so the failure window is much better.
3) If two drives fail in RAID 5, the data is gone. If two drives fail in RAID 10, there's a 50% chance you won't lose your data, depending on which drives fail.
4) RAID 10 is less complicated than RAID 5. If the enclosure dies, it will be easier to plug the drives into a PC to get to your data than it would be for RAID 5.
5) RAID 10 will give you better performance, particularly write performance.

Also, keep in mind that RAID is not backup. RAID protects you from downtime in the event of a drive failure; it does not protect you against data loss. It reduces the chances of needing your backups, but it doesn't eliminate the need for backups. Power surges that blow out the controller, system failures, or multiple drive failures can result in data loss with any RAID solution. Always backup your data to a second system, preferably off-site if it's important enough.

As far as interface goes, if you're looking for pure speed, USB 3.0 will be just as fast as eSATA with single-drive enclosures. With multi-drive enclosures in a RAID configuration, it may wind up being a little faster. If you have multiple computers, you might consider a NAS so you can have shared storage. There are some low-cost drives that get near gigabit speeds. They're nowhere near USB 3.0 speeds, but you don't need to leave a computer running for other people to access them.

Concerning your backup problem, you could run tools that will search for duplicates and help your organize things, but it sounds like you have such a big mess that you'll be better off starting over. In a situation like yours, you'll run the risk of an automated solution deleting something that wasn't really a duplicate or winding up with duplicates that the automated solution missed. So, get a new original drive and a backup drive. Setup your backups and get them running regularly. Figure out how you want to organize your data on the new drive.

Once you do that, get the stuff you absolutely need copied over first. Then, each week, empty one of the old drives. It might take an hour or two per drive, but you'll be done in few months, you'll be 100% confident you didn't miss anything important, and things will be organized exactly how you want them.
Aaron TomoskyDirector of Solutions ConsultingCommented:
Last time I bought that Drobo it was under 1300 out the door. Buy the drives separately, use the same ones Drobo sells the wd20ears drives but buy them yourself for much less than anyone sells the loaded Drobo for.

As far as protecting your data goes, here is what I tell people to protect against:
1. Hardware failure. Use a ups and raid. For 5 drives I use raid 5 for more I use raid 6. Some people like raid 10 but they don't do IT for video editors and graphic artists or they could never afford to waste 50% of their space on protecting from a hardware failure
2. Accidentially deletion / virus. Back up everything. If you do this there is no reason to have a raid 10.
3. Building burns down/ other type of disaster. Back it up Offsite.

As you see, this has turned into a larger discussion aout how to devise a storage and backup plan. Maybe you should step back and think about all this for a bit.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
CanadianJeffAuthor Commented:
It has become about devising a storage and backup plan, you're absolutely right about that.  And, I think you're right, I need to step back and look at it.  You've provided me with a ton of information, Aaron, and really useful info.  I know so much more about this than I did yesterday, so thank you.  But, yes, I need to think about it for a little bit.  I'm a guy who believes in going high-end on things to avoid a lot of the pitfalls, I've learned paying extra leaves some of the picky work behind, but I don't think I can fit $1300 into my budget just for a housing for my hard drives, and then start purchasing drives as well.  At this point, I suspect that I might be better off trying to just upgrade what I've got into a faster environment that's not quite so expensive.... I guess transforming my system into USB 3.0 would simply mean upgrading my processor/motherboard to something newer, right?  

I have Vice-Versa, which sounds good and supposedly handles duplicates, but I haven't had a chance to use it yet, and you made a real good point about software making mistakes.  Maybe you're right, about doing a new original drive and a backup drive.  I have a 64GB SSD drive unopened, I could put that in for my original drive with O/S, for fast access, and then purchase/add a 3TB and start putting everything onto there, and then use one or two of my 1TB drives as backups, which I believe Vice-Versa will automatically set those up... any recommendations there?

Also, thinking of upgrading to 64-bit Windows7... will this mean a lot of my software won't run properly if I do a new install and start this whole thing over fresh again?  Sorry, different topic, I should be asking that somewhere else.  Maybe I should take some time and consider my options.  You've given me lots to think about.  I'll ponder this for a bit, a question or two may come up, but I'll award the points as is when I do that, Aaron.  Thank you so much for your expertise!!
Aaron TomoskyDirector of Solutions ConsultingCommented:
I meant it was $1300 with drives, tax, shipping, everything you need for an 8tb raid 5 array. In fact I think I had a $40 esata pciexpress card in there too.  The drobo chassisby itself was a little over $700.
There are cheaper chassis options. You can get 5 drive esata expander chassis for around 300 but it's a put it together yourself kinda thing. I use these for video editing Mac pros and it had to be super easy and quiet.
I've also done sata -> sas chassis with a mid range highpoint rocketraid controller. Even more do it yourself but better speed and almost as cheap as an esata expander chassis.
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
This question has been classified as abandoned and is closed as part of the Cleanup Program. See the recommendation for more details.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Switches / Hubs

From novice to tech pro — start learning today.