Storage and Backup Solution for Video Editing System
Hi Experts,
I been doing some research on the best storage and backup solution for my needs and wanted to seek an experts opinion before I spent my $$.
Up until now I've been backing my system up on an external hard drive, which has recently failed and as I am reaching near capacity with some of my internal drives I'm contemplating a few different options relating to expansion and backup...
Note: I am an amateur when it comes to all of this so please excuse any inaccuracies or lack of knowledge with the information below.
Needs
I'm a videographer and amateur photographer. I work with large HD video files (~100mb-1.2GB) and RAW photography files (~18mb-50mb).
Current Storage Set Up
Internal 1 [60% full] - 1x Crucial M4 256GB SSD
(This is my dedicated Operating System Hard drive)
Internal 2 (Set up in RAID 1) [95% full] - 2 x 1TB SATA 7200RPM 32M
(This drive is used for photos, completed video projects, documents, music etc)
Internal 3 (set up in RAID 0) [35% full] - 2 x Hitachi 2TB Deskstar 7K3000 SATA 6GB 7200RPM / 64MB
(This set up is used purely for rendering and editing large HD Video Files, I'm also using to store video files*)
*No redundancy for this set up - hence I need to find a solution ASAP!
NOTE: I use Acronis software for backing up.
Possible Solutions
1. Buy two external hard drives. Use one for full system back up. Use the other to store additional photos, completed projects, documents etc
Pros: Affordable, Easy to set up
Cons: Not so 'neat' having several external hard drives floating around, not convenient having files stored in separate locations.
2. Buy a new external hard drive for full system back up and take 'Internal 2' out of RAID 1 to create more space for completed projects, photos etc
Pros: Affordable, fairly easy to set up
Cons: Less redundancy, will most likely need more space in 12months time.
3. Buy a NAS and set up in RAID 5 (start with 3 x 2TB drives). Move all of my docs, completed projects, music and photos to the NAS. Use Internal 2 as full system back up (could also perform full system back up on NAS) OR take 'Internal 2' out of RAID 1 and turn into RAID 0 for additional storage for Video editing files and possible editing of RAW photo files.
Pros: Plenty of space for future, Good redundancy
Cons: $$$
4. ???
My Budget?
I would be in a state of despair if I lost all of my files so it's hard to put a price on that.
At this stage if I could find a solution for around ~$500 then I would be happy. However if spending a little more meant meeting my needs for now and into the future then I would consider doing so.
Current System
PSU - Corsair HX-1050 1050W ATX Modular Power Supply
M/B - Asus PX79 Pro
CPU - Intel CORE i7 3820/3.60GHz
GPU - Asus GTX660 TI-DC2-2GD5 2GB GDDR5
RAM - G Skill 16G(4X4G) DDR3 2133MHZ
Case - Antec Twelve Hundred (V3) Full Tower
O/S - Windows 7 Pro (64)
I been doing some research on the best storage and backup solution for my needs and wanted to seek an experts opinion before I spent my $$.
Up until now I've been backing my system up on an external hard drive, which has recently failed and as I am reaching near capacity with some of my internal drives I'm contemplating a few different options relating to expansion and backup...
Note: I am an amateur when it comes to all of this so please excuse any inaccuracies or lack of knowledge with the information below.
Needs
I'm a videographer and amateur photographer. I work with large HD video files (~100mb-1.2GB) and RAW photography files (~18mb-50mb).
Current Storage Set Up
Internal 1 [60% full] - 1x Crucial M4 256GB SSD
(This is my dedicated Operating System Hard drive)
Internal 2 (Set up in RAID 1) [95% full] - 2 x 1TB SATA 7200RPM 32M
(This drive is used for photos, completed video projects, documents, music etc)
Internal 3 (set up in RAID 0) [35% full] - 2 x Hitachi 2TB Deskstar 7K3000 SATA 6GB 7200RPM / 64MB
(This set up is used purely for rendering and editing large HD Video Files, I'm also using to store video files*)
*No redundancy for this set up - hence I need to find a solution ASAP!
NOTE: I use Acronis software for backing up.
Possible Solutions
1. Buy two external hard drives. Use one for full system back up. Use the other to store additional photos, completed projects, documents etc
Pros: Affordable, Easy to set up
Cons: Not so 'neat' having several external hard drives floating around, not convenient having files stored in separate locations.
2. Buy a new external hard drive for full system back up and take 'Internal 2' out of RAID 1 to create more space for completed projects, photos etc
Pros: Affordable, fairly easy to set up
Cons: Less redundancy, will most likely need more space in 12months time.
3. Buy a NAS and set up in RAID 5 (start with 3 x 2TB drives). Move all of my docs, completed projects, music and photos to the NAS. Use Internal 2 as full system back up (could also perform full system back up on NAS) OR take 'Internal 2' out of RAID 1 and turn into RAID 0 for additional storage for Video editing files and possible editing of RAW photo files.
Pros: Plenty of space for future, Good redundancy
Cons: $$$
4. ???
My Budget?
I would be in a state of despair if I lost all of my files so it's hard to put a price on that.
At this stage if I could find a solution for around ~$500 then I would be happy. However if spending a little more meant meeting my needs for now and into the future then I would consider doing so.
Current System
PSU - Corsair HX-1050 1050W ATX Modular Power Supply
M/B - Asus PX79 Pro
CPU - Intel CORE i7 3820/3.60GHz
GPU - Asus GTX660 TI-DC2-2GD5 2GB GDDR5
RAM - G Skill 16G(4X4G) DDR3 2133MHZ
Case - Antec Twelve Hundred (V3) Full Tower
O/S - Windows 7 Pro (64)
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi insidetech,
Thanks for your help!
From what I can see on the MB I only have 1x6Gb sata port free.
6Gb/s Sata Port 1 - Free
6Gb/s Sata Port 2 - SSD drive
3Gb/s Sata Port 1 - DVD Burner
3Gb/s Sata Port 2 - DVD Burner
3Gb/s Sata Port 3 - 1TB in RAID 1
3Gb/s Sata Port 4 - 1TB in RAID 1
6Gb/s Sata Port 1 (Cache Port) - 2TB in RAID 0
6Gb/s Sata Port 2 (Cache Port) - 2TB in RAID 0
???
Thanks for your help!
From what I can see on the MB I only have 1x6Gb sata port free.
6Gb/s Sata Port 1 - Free
6Gb/s Sata Port 2 - SSD drive
3Gb/s Sata Port 1 - DVD Burner
3Gb/s Sata Port 2 - DVD Burner
3Gb/s Sata Port 3 - 1TB in RAID 1
3Gb/s Sata Port 4 - 1TB in RAID 1
6Gb/s Sata Port 1 (Cache Port) - 2TB in RAID 0
6Gb/s Sata Port 2 (Cache Port) - 2TB in RAID 0
???
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Here is the deal with RAID1 when using the native software RAID in Windows.
All data is in two places at once, right? Well the O/S knows this. The O/S also knows what I/Orequests are sent to each drive, and which I/Os are pending at any moment.
So 100% of the time, the O/S can choose what disk can get the data faster. In a perfect world, on reads, each disk gets half of the I/O, meaning you get 2X the performance.
But in RAID0, at best, only 50% of the data is on any one drive. My over simplified summary is that if a program is using RAID0 then it has to wait for the disk that has the data it needs. In RAID1 it never has to wait, because the data is in two places at once.
So if you want READ performance, regardless of the type of I/O use RAID1.
Write intensive stuff should be the SSD, or RAID0, but in grand scheme of things you may find that what you think is write intensive also is read intensive, so RAID1 still may be a better deal.
Note: You better be using the windows software RAID, not any cheap fake raid on a motherboard. They do NOT do read balancing. Expensive RAID controllers do load balancing.
What is make/model of your RAID controller (or are you already setting up AHCI & windows dynamic disks to do the RAID?)
P.S. With windows software RAID, you can partition the same disk to do half RAID1, half RAID0, by the way. Just create 2 partitions. You will still get the RAID1 & RAID0 benefits, but it just won't be as good as if the entire disk was either RAID0 or RAID1.
My deal with human error comment. If the backup is J: and source is C:\ maybe you carelessly "backup" the data from J:\ to c:\ by mistake. Such things happen to the best of us.
Or maybe you fat finger windows explorer and drag something the wrong place or way with the mouse and can't Kill the process fast enough.
All data is in two places at once, right? Well the O/S knows this. The O/S also knows what I/Orequests are sent to each drive, and which I/Os are pending at any moment.
So 100% of the time, the O/S can choose what disk can get the data faster. In a perfect world, on reads, each disk gets half of the I/O, meaning you get 2X the performance.
But in RAID0, at best, only 50% of the data is on any one drive. My over simplified summary is that if a program is using RAID0 then it has to wait for the disk that has the data it needs. In RAID1 it never has to wait, because the data is in two places at once.
So if you want READ performance, regardless of the type of I/O use RAID1.
Write intensive stuff should be the SSD, or RAID0, but in grand scheme of things you may find that what you think is write intensive also is read intensive, so RAID1 still may be a better deal.
Note: You better be using the windows software RAID, not any cheap fake raid on a motherboard. They do NOT do read balancing. Expensive RAID controllers do load balancing.
What is make/model of your RAID controller (or are you already setting up AHCI & windows dynamic disks to do the RAID?)
P.S. With windows software RAID, you can partition the same disk to do half RAID1, half RAID0, by the way. Just create 2 partitions. You will still get the RAID1 & RAID0 benefits, but it just won't be as good as if the entire disk was either RAID0 or RAID1.
My deal with human error comment. If the backup is J: and source is C:\ maybe you carelessly "backup" the data from J:\ to c:\ by mistake. Such things happen to the best of us.
Or maybe you fat finger windows explorer and drag something the wrong place or way with the mouse and can't Kill the process fast enough.
This is what you have... Note the two eSATA (external ports) Also it is Vantec.com for good quality external storage enclosures.
Intel® X79 chipset :
2 x SATA 6Gb/s port(s), white
4 x SATA 3Gb/s port(s), blue
Support Raid 0, 1, 5, 10
Marvell® PCIe 9128 controller :
2 x SATA 6Gb/s port(s), white
ASMedia® ASM1061 controller :
2 x Power eSATA 6Gb/s port(s), green
Intel® X79 chipset :
2 x SATA 6Gb/s port(s), white
4 x SATA 3Gb/s port(s), blue
Support Raid 0, 1, 5, 10
Marvell® PCIe 9128 controller :
2 x SATA 6Gb/s port(s), white
ASMedia® ASM1061 controller :
2 x Power eSATA 6Gb/s port(s), green
For your transition in moving your data unplug one of your CD drives to free up a port.
Forget about the "academics" RAID on Asus hardware works great as long as you know the downside. Also life is too short to move data over USB. ;-)
Forget about the "academics" RAID on Asus hardware works great as long as you know the downside. Also life is too short to move data over USB. ;-)
That Marvell controller does not do read load balancing. It does not have cache for buffering I/Os. You'll get much better performance on your RAID1 if you don't use this controller for it.
If you use this controller for your RAID0, then it will be pretty much a wash in terms of performance if you let this controller do the RAID0.
There is no difference in the 3Gbit or 6Gbit ports, since the best any of your HDDs could do is around 2Gbit/sec, and even then it would be for only a few milliseconds on re-reads.
If you use this controller for your RAID0, then it will be pretty much a wash in terms of performance if you let this controller do the RAID0.
There is no difference in the 3Gbit or 6Gbit ports, since the best any of your HDDs could do is around 2Gbit/sec, and even then it would be for only a few milliseconds on re-reads.
ASKER
dlethe - Yes, I'm using Windows RAID software instead of the Marvell controller on the MB as I was informed that this would be a better solution. Probably because of the point you mentioned in your post above. I don't have a dedicated RAID controller card at this point in time.
Insidetech - Thanks for this. However I don't want to unplug one of the DVD drives, unless you were referring to this as a short term solution?
The Vantex external cases look good and is possible solution. Although I just feel that if I'm going to start building outside my case then I should build a NAS or at least something that is scalable and offers additional uses such as multiple user connectivity. IF I built a NAS though I'm under the impression that performing heavy editing tasks with large I/O will be bottlenecked through the connection from my computer to the NAS (via network cable)? Is this correct? So it would best served for other files not used for video editing and thus freeing up space inside my machine on HDs used for video editing?
dlethe - your simple explanation on RAID1 and RAID0 makes sense to me. However in the link below it would suggest otherwise. Does this have anything to do with running RAID through windows O/S vs a dedicated RAID controller? Or are they wrong?
In the article they suggest to use RAID 3 for video editing - I'm under the impression I would need a dedicated RAID controller for this?
RAID for Video Editing - Adobe Community
Rendering on SSD - Quite a few people on forums have reported that continuous writes on an SSD drive will diminish it's life very quickly and that you're not using it to its full potential. What's your take on this? Because I like the idea of having a dedicated SSD drive for rendering files..
What about...
Assuming I cant set up RAID 3 through Windows O/S, what about getting 2 more 2TB drives (removing the 1TB drives) and setting up RAID 10 - use for source files (photo & captured video)
Installing a single drive (SSD/SAS) in the free 6GB slot - use for rendering, preview files, media cache
Building a 4-bay NAS (buy 2 more 1TB drives) and set up in RAID 5 - music, movies, docs etc / put the 2 existing 1 TB drives in a Vantec external case.
Back up to the cloud when I improve my internet connection.
Insidetech - Thanks for this. However I don't want to unplug one of the DVD drives, unless you were referring to this as a short term solution?
The Vantex external cases look good and is possible solution. Although I just feel that if I'm going to start building outside my case then I should build a NAS or at least something that is scalable and offers additional uses such as multiple user connectivity. IF I built a NAS though I'm under the impression that performing heavy editing tasks with large I/O will be bottlenecked through the connection from my computer to the NAS (via network cable)? Is this correct? So it would best served for other files not used for video editing and thus freeing up space inside my machine on HDs used for video editing?
dlethe - your simple explanation on RAID1 and RAID0 makes sense to me. However in the link below it would suggest otherwise. Does this have anything to do with running RAID through windows O/S vs a dedicated RAID controller? Or are they wrong?
In the article they suggest to use RAID 3 for video editing - I'm under the impression I would need a dedicated RAID controller for this?
RAID for Video Editing - Adobe Community
Rendering on SSD - Quite a few people on forums have reported that continuous writes on an SSD drive will diminish it's life very quickly and that you're not using it to its full potential. What's your take on this? Because I like the idea of having a dedicated SSD drive for rendering files..
What about...
Assuming I cant set up RAID 3 through Windows O/S, what about getting 2 more 2TB drives (removing the 1TB drives) and setting up RAID 10 - use for source files (photo & captured video)
Installing a single drive (SSD/SAS) in the free 6GB slot - use for rendering, preview files, media cache
Building a 4-bay NAS (buy 2 more 1TB drives) and set up in RAID 5 - music, movies, docs etc / put the 2 existing 1 TB drives in a Vantec external case.
Back up to the cloud when I improve my internet connection.
The article is profoundly dated, incomplete, makes flawed generalizations, and is downright wrong on several points.
I know quite a bit about SSD wear leveling as I have developer NDAs with many of the SSD manufacturers and write software to measure and report such things.
Don't worry about "burning up" a modern SSD. I've run tests for military projects where we ran 100% writes, CONTINUOUSLY, in ovens, for over 6 months on hundreds of SSDs of various makes and models. Many survived that process. Now, of course we didn't use $89.95 consumer drives, but we didn't use MIL-spec drives either for our assessment. We used commercial class off-the shelf SSDs.
Just do rendering on a SSD, the best you can afford, minimum size to do the largest project you could possibly need. Also max out the RAM. Some SSDs can do 100,000 random IOPs and 550+ MB/sec sustained. Others do 5,000 random IOPs and 240MB/sec sustained. Both do a heck of a lot better than 100 random IOPS and 120MB/sec sustained like the typical SATA-3 HDD.
I know quite a bit about SSD wear leveling as I have developer NDAs with many of the SSD manufacturers and write software to measure and report such things.
Don't worry about "burning up" a modern SSD. I've run tests for military projects where we ran 100% writes, CONTINUOUSLY, in ovens, for over 6 months on hundreds of SSDs of various makes and models. Many survived that process. Now, of course we didn't use $89.95 consumer drives, but we didn't use MIL-spec drives either for our assessment. We used commercial class off-the shelf SSDs.
Just do rendering on a SSD, the best you can afford, minimum size to do the largest project you could possibly need. Also max out the RAM. Some SSDs can do 100,000 random IOPs and 550+ MB/sec sustained. Others do 5,000 random IOPs and 240MB/sec sustained. Both do a heck of a lot better than 100 random IOPS and 120MB/sec sustained like the typical SATA-3 HDD.
ASKER
Brilliant, I was hoping you would say that.
Which one..
SSD - Corsair 120GB Neutron SSD GTXB with the state of the art LAMD LM87800 processor. R/W: 555/330 MB/s VS Corsair 128GB 2.5" Force Series GS SSD 560MB/s Read 535MB/s Write, fast Toggle NAND
I just need to finalise a solution for additional space (with redundancy). Because if I now convert my 'Internal 3' set up from RAID0 to RAID1 I'll be down 2TB putting my use for this set up close 70%.
Replace the 1TB hard drives?
Which one..
SSD - Corsair 120GB Neutron SSD GTXB with the state of the art LAMD LM87800 processor. R/W: 555/330 MB/s VS Corsair 128GB 2.5" Force Series GS SSD 560MB/s Read 535MB/s Write, fast Toggle NAND
I just need to finalise a solution for additional space (with redundancy). Because if I now convert my 'Internal 3' set up from RAID0 to RAID1 I'll be down 2TB putting my use for this set up close 70%.
Replace the 1TB hard drives?
You had asked about backup. A good, reliable backup needs to be rotated to different media, so you have more than one backup, and you should be able to store those media offsite. For that several single disks are better suited than a NAS. A NAS could be an option if it were in a geographically different location than your PC, so it would also be offsite, which is unlikely in your case because of the limited bandwidth you mentioned. If something happens to your office, your live data and the backup would likely be gone.
ASKER
Thanks Rindi that's good advice. I didn't meant to neglect your previous post, my main priority at the moment is finding a suitable storage solution (ideally with local redundancy). Second to that is finding a better back up solution, which you've identified above and dlethe identified with online back up.
If I removed the 1TB drives I could put them in an external (Vantec) case and use them as means of backing up the system and then storing the drives off site? As you mentioned I should rotate to different media. But that does require a lot more effort than I'm willing to take...at the moment.
If I removed the 1TB drives I could put them in an external (Vantec) case and use them as means of backing up the system and then storing the drives off site? As you mentioned I should rotate to different media. But that does require a lot more effort than I'm willing to take...at the moment.
ASKER
Some good advice, but I'm still wondering what would be a suitable long term storage solution for my needs.
The 'Back up' solution has been answered 100%. But not the storage.
The 'Back up' solution has been answered 100%. But not the storage.
ASKER
Thanks for your response,
I have a very limited monthly data allowance (4GB) with my current internet service provider (due to my location), so internet backup services are probably not going to be suitable for me at this point in time given the upload and downloads that would be required?
Which means for the time being I will have to accept the risks of loss of data due to natural disasters or theft. Although I could use an external hard drive to perform backups and store off site in a secure location (a degree of human error would still be an issue here though).
Q - not sure what you mean by 'human error like copying files in the wrong direction' could you explain this further?
Q - As the for bugs and viruses, I have anti virus software and have my OS on a separate drive so should something fail and I can re-install windows without disrupting other hard drives. I realise this is not full proof, but would it not offer some protection? Apologies if this is a really naive view...
Why is that? Are files not read faster in this set up?
No, can you explain this further? Sorry, but this is beyond my understanding...
With internet back up service potentially on hold I will look to local back up.
My Motherboard only has one dedicated internal port connection for SSD drive and as this is already being used would you suggest I get an external SSD drive for rendering? Sounds like a stupid question but just double checking.
I don't have much more room in my computer for more drives (i.e. a RAID1 set up) and if I change the current RAID 0 to RAID 1 I will lose 2TB of data storage. Looking purely at a local backup (and storage), how would you recommend I proceed?
Thanks for your time dlethe, really appreciate it.