Backing up millions of small files

I have a doc imaging server that constantly receives files every minute.  These files are things like invoices, packing slips, ect.  They are very small in size but there are millions of them.

We are currently attempting to use DPM 2010 to do this but the replica stays inconsistant and cannot get a complete backup

Does anyone know of a the best way to do this?


Windows server 2008 R2
Computer HelperWindows Server AdminAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Lee W, MVPTechnology and Business Process AdvisorCommented:
I can't say this is ideal... but Disk2VHD can create a copy of the entire drive and does so fast regardless of file types.  But it's not a "backup" tool per se.  But file access afterwards is easy (mount the VHD) and there is no incremental capability.
Computer HelperWindows Server AdminAuthor Commented:
I am really looking for a turn key solution for this.  Money is not a problem.
Chris HInfrastructure ManagerCommented:
At a "millions" of files level, I'd use a trusted product like Backup exec.  Do an incremental backup and create a catalog.

However, what Lee mentions would probably be (even though it's going to take up much more space) a much faster/more successful backup at the block level.  Recognizing and checking attributes on millions of files can grow cumbersome for the file system.  Doing a disk image on the block level will just copy the entire drive like it's one giant file.

This is really where virtualization wins in my opinion.  This is a simple VCB backup operation that can be performed for free.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Get Your eBook
Chris HInfrastructure ManagerCommented:
Also, xxcopy would have no problem with these numbers.
Carlos IjalbaIT Systems DirectorCommented:
You can do block drive backup (very fast compared with file to file backup) with Drive Snapshot:

http://www.drivesnapshot.de/en/

and it does support differential backups, it also compresses the backup. the backup is performed in a snapshot format that can be mounted to restore files from -fast.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Computer HelperWindows Server AdminAuthor Commented:
Thank you everyone for your answers.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.