Solved

2 Million files, 48 hours to copy

Posted on 2013-12-19
4
245 Views
Last Modified: 2013-12-19
Recently I had to move a large number of files from one 64bit win system to another.  I was moving around 2 million files, 300kb to 16mb in size, most were around 4mb.  About 1.4tb total.  And I was trying to do it with a simple copy and paste over a weekend.  That worked about as well as you might guess...

The reasons for moving the data, were simple, we were out of space on that server.  I moved the data from a number of 600gb 10k drives, to a number of 4tb 7.2k drives.  The rate of growth was such that it just didn't make sense to leave the files on original setup, in the server that they were housed in.  Eventually i'll have to do this again, as we move them to a larger scale SAN.

This process took 3 weekends, because the process failed over and again, then didn't leave me enough time to get in another run at it.  It would get part way through the files (16-30 hours in) and just crash out.  I looked at couple other programs, and eventually used SyncToy to avoid making it 4 weekends, by simply copying what hadn't been copied already.  However even with only copying a lesser amount of the total files, it crashed out as well, and I eventually gave up on trying to do it in one set, and broke up the group of files in to 100k file increments (ish).  And that worked, I could verify that the files came across, that I had the same amount of files on both sides upon completion, and that the size was the same as well (also ran chkdsk against the files).

So getting back to "Eventually i'll have to do this again...", I don't want to break the files up, I want to start the job and it finish on its own.  I don't want to go in every 3 hours and start a new batch of 100k files.  Do you have any recommendations for moving a large amount of files from one 64 bit windows system to another?
0
Comment
Question by:CoffeeBlack
4 Comments
 
LVL 26

Assisted Solution

by:pony10us
pony10us earned 167 total points
ID: 39729547
Question:  Are you backing up all these files? Wouldn't it be easier to do a restore to the new location?
0
 
LVL 69

Assisted Solution

by:Qlemo
Qlemo earned 167 total points
ID: 39729563
Use RoboCopy, a commandline utility coming with recent MS OS. It allows for moving/ copying/mirroring, including attributes and securiy descriptors (if needed), is robust against failures, allows for retry definitions, ...

So you can tailor it to your needs with high probability.

A simple usage would be:
  robocopy c:\ \\server\c$\ /r:2 /w:5
to wait for a max. of 3 x 5 seconds if a file can't get copied, and only copying new(er)/changed files. That does not remove meanwhile deleted files - for having exact duplicates, use the /MIR switch.
0
 
LVL 4

Accepted Solution

by:
amclaughlin01 earned 166 total points
ID: 39729574
A couple options might be to use one of the following programs:

DeltaCopy - http://www.thegeekstuff.com/2012/05/rsync-for-windows-deltacopy/

or

Robocopy - http://sourceforge.net/projects/robocoprobocopy/


Benefits to using one of these is that if you do run into a problem during the copy, you can start it again and it will only copy new or changed files.

It is a great backup program for large directories because it doesn't have to copy the entire directory, but just the changed files.
0
 

Author Closing Comment

by:CoffeeBlack
ID: 39729629
Thx guys, I will get more info on those and get that ready for when the SAN is in place.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How do I merge two partitions in a MacBook Pro running El Capitan? 10 907
Create a image with CloneZilla 14 117
Inventory 1 64
Purchase of laptop with win 7 Professional OS 8 157
Ever notice how you can't use a new drive in Windows without having Windows assigning a Disk Signature?  Ever have a signature collision problem (especially with Virtual Machines?)  This article is intended to help you understand what's going on and…
When you upgrade from Windows 8 to 8.1 or to Windows 10 or if you are like me you are on the Insider Program you may find yourself with many 450MB recovery partitions.  With a traditional disk that may not be a problem but with relatively smaller SS…
Windows 8 comes with a dramatically different user interface known as Metro. Notably missing from the new interface is a Start button and Start Menu. Many users do not like it, much preferring the interface of earlier versions — Windows 7, Windows X…
In this video, we discuss why the need for additional vertical screen space has become more important in recent years, namely, due to the transition in the marketplace of 4x3 computer screens to 16x9 and 16x10 screens (so-called widescreen format). …

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question