Solved

Storage and Backup Solution for Video Editing System

Posted on 2014-01-11
15
1,887 Views
Last Modified: 2016-10-27
Hi Experts,

I been doing some research on the best storage and backup solution for my needs and wanted to seek an experts opinion before I spent my $$.

Up until now I've been backing my system up on an external hard drive, which has recently failed and as I am reaching near capacity with some of my internal drives I'm contemplating a few different options relating to expansion and backup...

Note: I am an amateur when it comes to all of this so please excuse any inaccuracies or lack of knowledge with the information below.  

Needs
I'm a videographer and amateur photographer. I work with large HD video files (~100mb-1.2GB) and RAW photography files (~18mb-50mb).

Current Storage Set Up  
Internal 1 [60% full] - 1x Crucial M4 256GB SSD
(This is my dedicated Operating System Hard drive)

Internal 2 (Set up in RAID 1) [95% full] - 2 x 1TB SATA 7200RPM 32M
(This drive is used for photos, completed video projects, documents, music etc)

Internal 3 (set up in RAID 0) [35% full] - 2 x Hitachi 2TB Deskstar 7K3000 SATA 6GB 7200RPM / 64MB
(This set up is used purely for rendering and editing large HD Video Files, I'm also using to store video files*)

*No redundancy for this set up - hence I need to find a solution ASAP!

NOTE: I use Acronis software for backing up.

Possible Solutions
1. Buy two external hard drives. Use one for full system back up. Use the other to store additional photos, completed projects, documents etc
Pros: Affordable, Easy to set up
Cons: Not so 'neat' having several external hard drives floating around, not convenient having files stored in separate locations.

2. Buy a new external hard drive for full system back up and take 'Internal 2' out of RAID 1 to create more space for completed projects, photos etc
Pros: Affordable, fairly easy to set up
Cons: Less redundancy, will most likely need more space in 12months time.

3. Buy a NAS and set up in RAID 5 (start with 3 x 2TB drives). Move all of my docs, completed projects, music and photos to the NAS. Use Internal 2 as full system back up (could also perform full system back up on NAS) OR take 'Internal 2' out of RAID 1 and turn into RAID 0 for additional storage for Video editing files and possible editing of RAW photo files.
Pros: Plenty of space for future, Good redundancy
Cons: $$$

4. ???

My Budget?
I would be in a state of despair if I lost all of my files so it's hard to put a price on that.
At this stage if I could find a solution for around ~$500 then I would be happy. However if spending a little more meant meeting my needs for now and into the future then I would consider doing so.  

Current System
PSU - Corsair HX-1050 1050W ATX Modular Power Supply
M/B - Asus PX79 Pro
CPU - Intel CORE i7 3820/3.60GHz
GPU - Asus GTX660 TI-DC2-2GD5 2GB GDDR5
RAM - G Skill 16G(4X4G) DDR3 2133MHZ
Case - Antec Twelve Hundred (V3) Full Tower
O/S - Windows 7 Pro (64)
0
Comment
Question by:BPro2008
  • 6
  • 4
  • 3
  • +1
15 Comments
 
LVL 47

Accepted Solution

by:
dlethe earned 175 total points
Comment Utility
None of the above.  Use a network / internet based backup solution for offsite backup.  Many are out there.  Google for internet backup services and do some research.  They all have strengths and weaknesses for whatever your budget and patience allow.

I will also say RAID0 is totally unacceptable.  Were you aware that windows native software RAID1 does read-load balancing?  100% of all read I/O in RAID1 is twice that of a single disk drive in a perfect world.  Writes are no worse than a single drive.

So use a small SSD for rendering as the temp device, then copy the finished project to a RAID1 somewhere.

RAID1 is more than protection against a HDD failure. It protects against bad blocks and data corruption.

P.S. you have backup & archiving.  People need both.  You have no protection against fire or natural disasters with your setup. You certainly have no protection against human error like copying files in the wrong direction or even windows bugs or a virus.

If the data is important use offsite backup.  Copy files to free space in another RAID1 drive set if you want a local backup.
0
 

Author Comment

by:BPro2008
Comment Utility
Hi dlethe,

Thanks for your response,

I have a very limited monthly data allowance (4GB) with my current internet service provider (due to my location), so internet backup services are probably not going to be suitable for me at this point in time given the upload and downloads that would be required?

Which means for the time being I will have to accept the risks of loss of data due to natural disasters or theft. Although I could use an external hard drive to perform backups and store off site in a secure location (a degree of human error would still be an issue here though).

You certainly have no protection against human error like copying files in the wrong direction or even windows bugs or a virus.

Q - not sure what you mean by 'human error like copying files in the wrong direction' could you explain this further?

Q - As the for bugs and viruses, I have anti virus software and have my OS on a separate drive so should something fail and I can re-install windows without disrupting other hard drives. I realise this is not full proof, but would it not offer some protection? Apologies if this is a really naive view...

I will also say RAID0 is totally unacceptable
Why is that? Are files not read faster in this set up?

Were you aware that windows native software RAID1 does read-load balancing?  100% of all read I/O in RAID1 is twice that of a single disk drive in a perfect world.  Writes are no worse than a single drive.
No, can you explain this further? Sorry, but this is beyond my understanding...

With internet back up service potentially on hold I will look to local back up.

So use a small SSD for rendering as the temp device, then copy the finished project to a RAID1 somewhere.

My Motherboard only has one dedicated internal port connection for SSD drive and as this is already being used would you suggest I get an external SSD drive for rendering? Sounds like a stupid question but just double checking.

Copy files to free space in another RAID1 drive set if you want a local backup.

I don't have much more room in my computer for more drives (i.e. a RAID1 set up) and if I change the current RAID 0 to RAID 1 I will lose 2TB of data storage. Looking purely at a local backup (and storage), how would you recommend I proceed?

Thanks for your time dlethe, really appreciate it.
0
 
LVL 6

Assisted Solution

by:insidetech
insidetech earned 150 total points
Comment Utility
OK... So yo have some options because you do have some awesome hardware.
For starters let's figure out your priorities though.

1. Boot drive is SSD ... That is great, you have fast boot and your programs load fast.
2. Internal 2 is 1 TB RAID 1 Mirror set. Performance not important...
3 internal 3 is 4TB RAID 0 stripe.... For performance... If you did set up the RAID1 on the controller and not in the OS the performance is really there despite what other "experts" say.
Problem you have is that as one drive goes, so does the entire set and 4 TB restore will take a while.
You may not know this, but your MB has two special ports for SSD cached HD.
If you do have the resources, purchase the biggest/fastest SATA Drive with complementary SSD for the cache and you will have a supper fast drive for the rendering functions.
After that, you can re-assign the RAID 0 drives for a data backup use.
Also your MB has eSATA ports so you can attach an external drive with build in RAID. Look at Vantek NextStar MX products.
Look at Acronis products for an excellent software back up solution.
Hope that this helps.
0
 

Author Comment

by:BPro2008
Comment Utility
Hi insidetech,

Thanks for your help!

From what I can see on the MB I only have 1x6Gb sata port free.

6Gb/s Sata Port 1 - Free
6Gb/s Sata Port 2 - SSD drive
3Gb/s Sata Port 1 - DVD Burner
3Gb/s Sata Port 2 - DVD Burner
3Gb/s Sata Port 3 - 1TB in RAID 1
3Gb/s Sata Port 4 - 1TB in RAID 1
6Gb/s Sata Port 1 (Cache Port) - 2TB in RAID 0
6Gb/s Sata Port 2 (Cache Port) - 2TB in RAID 0

???
0
 
LVL 87

Assisted Solution

by:rindi
rindi earned 175 total points
Comment Utility
Buy several disks to backup to, backups should never be done to just one location. You should rather cycle between them. Also always remove the backup disk from the system after it has completed, and then store it off-site.

You can buy standard HD's for your backups, and additionally get a USB dock to drop them in. That keeps the price lower. The first link is to a USB 3 dock, but there are also cheaper USB 2 docks:

http://www.newegg.com/Product/Product.aspx?Item=N82E16817153133
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Here is the deal with RAID1 when using the native software RAID in Windows.
All data is in two places at once, right?  Well the O/S knows this. The O/S also knows what I/Orequests are sent to each drive, and which I/Os are pending at any moment.

So 100% of the time, the O/S can choose what disk can get the data faster.  In a perfect world, on reads, each disk gets half of the I/O, meaning you get 2X the performance.

But in RAID0, at best, only 50% of the data is on any one drive.  My over simplified summary is that if a program is using RAID0 then it has to wait for the disk that has the data it needs.  In RAID1 it never has to wait, because the data is in two places at once.

So if you want READ performance, regardless of the type of I/O use RAID1.  

Write intensive stuff should be the SSD, or RAID0, but in grand scheme of things you may find that what you think is write intensive also is read intensive, so RAID1 still may be a better deal.

Note: You better be using the windows software RAID, not any cheap fake raid on a motherboard. They do NOT do read balancing.   Expensive RAID controllers do load balancing.

What is make/model of your RAID controller (or are you already setting up AHCI & windows dynamic disks to do the RAID?)

P.S.   With windows software RAID, you can partition the same disk to do half RAID1, half RAID0, by the way. Just create 2 partitions.  You will still get the RAID1  & RAID0 benefits, but it just won't be as good as if the entire disk was either RAID0 or RAID1.



My deal with human error comment.   If the backup is J:  and source is C:\   maybe you carelessly "backup" the data from J:\ to c:\   by mistake.   Such things happen to the best of us.

Or maybe you fat finger windows explorer and drag something the wrong place or way with the mouse and can't Kill the process fast enough.
0
 
LVL 6

Expert Comment

by:insidetech
Comment Utility
This is what you have... Note the two eSATA (external ports)  Also it is Vantec.com for good quality external storage enclosures.

Intel® X79 chipset :
2 x SATA 6Gb/s port(s), white
4 x SATA 3Gb/s port(s), blue
Support Raid 0, 1, 5, 10
Marvell® PCIe 9128 controller :
2 x SATA 6Gb/s port(s), white
ASMedia® ASM1061 controller :
2 x Power eSATA 6Gb/s port(s), green
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 
LVL 6

Expert Comment

by:insidetech
Comment Utility
For your transition in moving your data unplug one of your CD drives to free up a port.
Forget about the "academics" RAID on Asus hardware works great as long as you know the downside. Also life is too short to move data over USB. ;-)
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
That Marvell controller does not do read load balancing.  It does not have cache for buffering I/Os. You'll get much better performance on your RAID1 if you don't use this controller for it.  

If you use this controller for your RAID0, then it will be pretty much a wash in terms of performance if you let this controller do the RAID0.

There is no difference in the 3Gbit or 6Gbit ports, since the best any of your HDDs could do is around 2Gbit/sec, and even then it would be for only a few milliseconds on re-reads.
0
 

Author Comment

by:BPro2008
Comment Utility
dlethe - Yes, I'm using Windows RAID software instead of the Marvell controller on the MB as I was informed that this would be a better solution. Probably because of the point you mentioned in your post above. I don't have a dedicated RAID controller card at this point in time.

Insidetech - Thanks for this. However I don't want to unplug one of the DVD drives, unless you were referring to this as a short term solution?

The Vantex external cases look good and is possible solution. Although I just feel that if I'm going to start building outside my case then I should build a NAS or at least something that is scalable and offers additional uses such as multiple user connectivity. IF I built a NAS though I'm under the impression that performing heavy editing tasks with large I/O will be bottlenecked through the connection from my computer to the NAS (via network cable)? Is this correct? So it would best served for other files not used for video editing and thus freeing up space inside my machine on HDs used for video editing?

dlethe - your simple explanation on RAID1 and RAID0 makes sense to me. However in the link below it would suggest otherwise. Does this have anything to do with running RAID through windows O/S vs a dedicated RAID controller? Or are they wrong?

In the article they suggest to use RAID 3 for video editing - I'm under the impression I would need a dedicated RAID controller for this?

RAID for Video Editing - Adobe Community

Rendering on SSD - Quite a few people on forums have reported that continuous writes on an SSD drive will diminish it's life very quickly and that you're not using it to its full potential. What's your take on this? Because I like the idea of having a dedicated SSD drive for rendering files..


What about...
Assuming I cant set up RAID 3 through Windows O/S, what about getting 2 more 2TB drives (removing the 1TB drives) and setting up RAID 10 - use for source files (photo & captured video)

Installing a single drive (SSD/SAS) in the free 6GB slot - use for rendering, preview files, media cache

Building a 4-bay NAS (buy 2 more 1TB drives) and set up in RAID 5 - music, movies, docs etc / put the 2 existing 1 TB drives in a Vantec external case.

Back up to the cloud when I improve my internet connection.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
The article is profoundly dated, incomplete, makes flawed generalizations, and is downright wrong on several points.

I know quite a bit about SSD wear leveling as I have developer NDAs with many of the SSD manufacturers and write software to measure and report such things.

Don't worry about "burning up" a modern SSD.  I've run tests for military projects where we ran 100% writes, CONTINUOUSLY, in ovens, for over 6 months on hundreds of SSDs of various makes and models.  Many survived that process.  Now, of course we didn't use $89.95 consumer drives, but we didn't use MIL-spec drives either for our assessment. We used commercial class off-the shelf SSDs.

Just do rendering on a SSD, the best you can afford, minimum size to do the largest project you could possibly need.   Also max out the RAM.   Some SSDs can do 100,000 random IOPs and 550+ MB/sec sustained.  Others do 5,000 random IOPs and 240MB/sec sustained.   Both do a heck of a lot better than 100 random IOPS and 120MB/sec sustained like the typical SATA-3 HDD.
0
 

Author Comment

by:BPro2008
Comment Utility
Brilliant, I was hoping you would say that.

Which one..
SSD - Corsair 120GB Neutron SSD GTXB with the state of the art LAMD LM87800 processor. R/W: 555/330 MB/s VS Corsair 128GB 2.5" Force Series GS SSD 560MB/s Read 535MB/s Write, fast Toggle NAND

I just need to finalise a solution for additional space (with redundancy). Because if I now convert my 'Internal 3' set up from RAID0 to RAID1 I'll be down 2TB putting my use for this set up close 70%.

Replace the 1TB hard drives?
0
 
LVL 87

Expert Comment

by:rindi
Comment Utility
You had asked about backup. A good, reliable backup needs to be rotated to different media, so you have more than one backup, and you should be able to store those media offsite. For that several single disks are better suited than a NAS. A NAS could be an option if it were in a geographically different location than your PC, so it would also be offsite, which is unlikely in your case because of the limited bandwidth you mentioned. If something happens to your office, your live data and the backup would likely be gone.
0
 

Author Comment

by:BPro2008
Comment Utility
Thanks Rindi that's good advice. I didn't meant to neglect your previous post, my main priority at the moment is finding a suitable storage solution (ideally with local redundancy). Second to that is finding a better back up solution, which you've identified above and dlethe identified with online back up.

If I removed the 1TB drives I could put them in an external (Vantec) case and use them as means of backing up the system and then storing the drives off site? As you mentioned I should rotate to different media. But that does require a lot more effort than I'm willing to take...at the moment.
0
 

Author Closing Comment

by:BPro2008
Comment Utility
Some good advice, but I'm still wondering what would be a suitable long term storage solution for my needs.

The 'Back up' solution has been answered 100%. But not the storage.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Storage devices are generally used to save the data or sometime transfer the data from one computer system to another system. However, sometimes user accidentally erased their important data from the Storage devices. Users have to know how data reco…
Workplace bullying has increased with the use of email and social media. Retain evidence of this with email archiving to protect your employees.
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now