Backing up millions of small files

Computer Helper
Computer Helper used Ask the Experts™
on
I have a doc imaging server that constantly receives files every minute.  These files are things like invoices, packing slips, ect.  They are very small in size but there are millions of them.

We are currently attempting to use DPM 2010 to do this but the replica stays inconsistant and cannot get a complete backup

Does anyone know of a the best way to do this?


Windows server 2008 R2
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Lee W, MVPTechnology and Business Process Advisor
Most Valuable Expert 2013

Commented:
I can't say this is ideal... but Disk2VHD can create a copy of the entire drive and does so fast regardless of file types.  But it's not a "backup" tool per se.  But file access afterwards is easy (mount the VHD) and there is no incremental capability.
Computer HelperWindows Server Admin

Author

Commented:
I am really looking for a turn key solution for this.  Money is not a problem.
Chris HInfrastructure Manager
Commented:
At a "millions" of files level, I'd use a trusted product like Backup exec.  Do an incremental backup and create a catalog.

However, what Lee mentions would probably be (even though it's going to take up much more space) a much faster/more successful backup at the block level.  Recognizing and checking attributes on millions of files can grow cumbersome for the file system.  Doing a disk image on the block level will just copy the entire drive like it's one giant file.

This is really where virtualization wins in my opinion.  This is a simple VCB backup operation that can be performed for free.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Chris HInfrastructure Manager

Commented:
Also, xxcopy would have no problem with these numbers.
IT Systems Director
Top Expert 2014
Commented:
You can do block drive backup (very fast compared with file to file backup) with Drive Snapshot:

http://www.drivesnapshot.de/en/

and it does support differential backups, it also compresses the backup. the backup is performed in a snapshot format that can be mounted to restore files from -fast.
Computer HelperWindows Server Admin

Author

Commented:
Thank you everyone for your answers.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial