Robocopy not quite working for 1 million files

Robocopy has been one of my favorite tools for about 15 years. However, in this case, it is not quite finishing the job.

I need to copy close to a million files totaling 10 GB from a remote SAMFS share to local Windows folder.
This is what I used:
Robocopy K:\ D:\Target\images\stills\ /E /COPY:DAT /R:0 /W:1 /NP /TEE /Log+:C:\temp\Robustcopy.log

(K: is mapped to remote SAMFS share.)

About half of the files got copied and then the process terminated on its own. The log file only showed the header. Thinking the log file may be getting too bulky, I removed "/NP /TEE /Log+:C:\temp\Robustcopy.log" and now the console only shows the initial screen (Started: Source: Dest: Files: Options:) and no name of any files. Cannot know if there is any progress or not.

How may I see progress of copying? Are these files too many for Robocopy? What other tool may be used? (This is production environment.) Thanks.

AK
LVL 3
AkulshAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Gary PattersonVP Technology / Senior Consultant Commented:
I've had robocopy crash under similar circumstances - rarely, but it happens.  I've had good luck with fastcopy:

http://ipmsg.org/tools/fastcopy.html.en
0
aikimarkCommented:
1. Are all the service patches applied?  What version are you running?
2. Have you tried doing the copy the other way around? (from server to workstation)
3. How much time elapses before the copy fails?
4. Are you working on the workstation during the copy?
5. Have you tried creating the target directory tree in a separate step?
0
aikimarkCommented:
If you aren't going to retry failed copies, why specify a wait time between retries?
0
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

AkulshAuthor Commented:
Gary,

Fastcopy is a good suggestion, but the link you included takes me to a Chinese/Japaness page when I click on download.
0
AkulshAuthor Commented:
Aikimark,

You are right that /W switch is not needed. But I don't think it should come in the way.

1. The version of robocopy is the one that comes with Win2008-R2, or 5.1.10.1027.
2. There is no workstation here. Source is non-Windows (SAMFS share) and target is Win2008-R2.
3. Time elapsed is at least 12 hours, because I see CMD windows gone next day.
4. Win2008-R2
5. Do you mean mapping a drive to the local folder? Are you sure that is something worth trying?

Thanks.

AK
0
aikimarkCommented:
You are currently using a pull model.  I wondered if a push model might be a work-around.

Or is the system you are currently running the Robocopy command on a Windows Server 2008 box?

If you have the bandwidth in your server room, try adding the multi-threading option to your command Example: /MT:12
0
AkulshAuthor Commented:
Yes, I am running the robocopy on Win2008-R2 box. Cannot run on the Source side since that machine is not Windows.

/MT is a good suggestion -- which I have not used before -- but default threads are 8 and maximum is 128. Is there any reason to limit it to 12? Why not more since files are only 10 to 30 KB? The server has plenty of CPU resource. Thanks.
0
aikimarkCommented:
Is there any reason to limit it to 12?
Thought it would be easy math if your current elapsed is 12 hours.
0
Ivano ViolaSystem AdministratorCommented:
Just another suggestion.....

I've used Free File Sync in my environment with any issues. It's portable, free, fast and also copies over NTFS permissions.
About: http://sourceforge.net/projects/freefilesync/
Download: http://www.fosshub.com/FreeFileSync.html

I highly recommend it.

IV
0
AkulshAuthor Commented:
IV,

Thanks for this suggestion. This looks like a worthy tool but in this case we want to keep some extra files in the Target folder. In other words, we don't wish to mirror just overwrite. Thanks.

AK
0
Ivano ViolaSystem AdministratorCommented:
You can just do an update. You don't have to do a mirror. You have a few options.
0
AkulshAuthor Commented:
Thanks for all your suggestion. In the end, Robocopy options /R:1 and /W:1 worked and no log file.
It took about 24 hours to copy all files, and actual copying does not start till after about 12 hours.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AkulshAuthor Commented:
aikimark got me thinking about /R and /W options but he did not suggest that I should increase the value for /R.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows OS

From novice to tech pro — start learning today.