[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

What is the best way to copy ~2 million files on Windows 2003?

Posted on 2010-08-24
8
Medium Priority
?
1,035 Views
Last Modified: 2012-05-10
I've recently purchased a duplicate NAS for the sole purpose of mirroring the current one (for redundancy's sake).  They each have a 10TB capacity (they are Netgear ReadyNAS Pros).  The one with all the files on it has 1.8 million files in 64,357 folders taking up 6.42TB of space.  I have tried over the previous two weeks to copy the files over to the new NAS with little success.

First tool i tried was SecondCopy, this crashed a few minutes into the copy, with no error.  Second tool was TeraCopy, this too would just fail about 2-3% into the transfer, no error given (just stopped transferring).  Third tool i tried was the built in rysnc tool on the readynas, it too failed, with a completely non descriptive error in the log.  After that, the fourth tool I've tried is DSynchronize, which seems to work, except about 2 hours into a transfer it starts reporting that files are locked on the source drive, and it too stops in the middle of the job.

Is there a tool that is designed for copying so many files on Windows?  I've even tried moving one folder (at the top level) at a time, but even that seems to fail.
0
Comment
Question by:Andrew_Bella
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
8 Comments
 

Expert Comment

by:xclsys
ID: 33514149
use an offline clone hadrive software to copy the files
0
 

Author Comment

by:Andrew_Bella
ID: 33514241
The volume spans across 6 HDs as a RAID array and taking it down would probably not work out for the users (and it would take too long to do it after hours).

Does synctoy use a different method than Windows copy (forgot to mention i gave that a shot too)?
0
Fill in the form and get your FREE NFR key NOW!

Veeam® is happy to provide a FREE NFR server license to certified engineers, trainers, and bloggers.  It allows for the non‑production use of Veeam Agent for Microsoft Windows. This license is valid for five workstations and two servers.

 
LVL 96

Expert Comment

by:Lee W, MVP
ID: 33514267
Curious... any reason you have used Robocopy or XCOPY?
0
 
LVL 4

Accepted Solution

by:
PcGod718 earned 500 total points
ID: 33514477
my guess is youre either running into memory errors or the folder structure is too deep.  
try using

http://www.xxcopy.com/index.htm
http://technet.microsoft.com/en-gb/magazine/2009.04.utilityspotlight.aspx

they are both reportedly much faster than robocopy while just as stable

you can also look into downloading a Tape backup software like backup exec.  Get the 30 day demo (they are fully functioning) and do a disk to disk backup.
0
 
LVL 6

Expert Comment

by:dax_bad
ID: 33518748
What about Rsync ?

It's an opensource remote sync server tool that can sync the master NAS to the Mirror on the fly.

http://en.wikipedia.org/wiki/Rsync
http://www.samba.org/rsync/

For setting up in windows

http://www.gaztronics.net/rsync.php

Cheers
/Daniel
0
 
LVL 10

Assisted Solution

by:Dextertronic
Dextertronic earned 500 total points
ID: 33528573
Thats a lot of files.
Not surprising that  the low end tools would wipe out trying to enumerate millions of files.
If i were in your position I would do one of two things.
1) break up the file copy into manageable chunks (start with a 100000 files ata a time)
2) Use an enterprise level backup suite like Symantec Backup Exec or CA's Arcserve in to backup to either LTO tape or disk storage, then restore from that media to your target device.
3) Use the copy feature in CA Arcserve to copy the data across (very robust process which is designed to handle these kinds of numbers).
4) Use a copy of Doubletake Backup which has the best sync solution around and would rip through this challenge.
Note that you can download fully functional trial versions of those backup software suites and hire out an LTO tape drive on a short term basis.
In retropsect I am leaning towards the trial version of CA and use the copy function or the Doubletake trial software, both of which I have used in similar scenarios.
 
 
0
 

Author Closing Comment

by:Andrew_Bella
ID: 33543889
I'm not out of the woods yet (it'll take many days for the copy to complete) but RichCopy has been performing great so far (48 hours into the copy now)!

If that fails, I'm going to need to go the route of a tape backup application.  

Thanks everyone!
0

Featured Post

Enroll in October's Free Course of the Month

Do you work with and analyze data? Enroll in October's Course of the Month for 7+ hours of SQL training, allowing you to quickly and efficiently store or retrieve data. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

We look at whether swapping a controller board on a failed hard drive is likely to solve the problem.
A look at what happened in the Verizon cloud breach.
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question