Solved

Robocopy not quite working for 1 million files

Posted on 2014-11-14
13
344 Views
Last Modified: 2014-11-21
Robocopy has been one of my favorite tools for about 15 years. However, in this case, it is not quite finishing the job.

I need to copy close to a million files totaling 10 GB from a remote SAMFS share to local Windows folder.
This is what I used:
Robocopy K:\ D:\Target\images\stills\ /E /COPY:DAT /R:0 /W:1 /NP /TEE /Log+:C:\temp\Robustcopy.log

(K: is mapped to remote SAMFS share.)

About half of the files got copied and then the process terminated on its own. The log file only showed the header. Thinking the log file may be getting too bulky, I removed "/NP /TEE /Log+:C:\temp\Robustcopy.log" and now the console only shows the initial screen (Started: Source: Dest: Files: Options:) and no name of any files. Cannot know if there is any progress or not.

How may I see progress of copying? Are these files too many for Robocopy? What other tool may be used? (This is production environment.) Thanks.

AK
0
Comment
Question by:Akulsh
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
  • 2
  • +1
13 Comments
 
LVL 35

Assisted Solution

by:Gary Patterson
Gary Patterson earned 50 total points
ID: 40443296
I've had robocopy crash under similar circumstances - rarely, but it happens.  I've had good luck with fastcopy:

http://ipmsg.org/tools/fastcopy.html.en
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40443430
1. Are all the service patches applied?  What version are you running?
2. Have you tried doing the copy the other way around? (from server to workstation)
3. How much time elapses before the copy fails?
4. Are you working on the workstation during the copy?
5. Have you tried creating the target directory tree in a separate step?
0
 
LVL 45

Assisted Solution

by:aikimark
aikimark earned 350 total points
ID: 40443446
If you aren't going to retry failed copies, why specify a wait time between retries?
0
Connect further...control easier

With the ATEN CE624, you can now enjoy a high-quality visual experience powered by HDBaseT technology and the convenience of a single Cat6 cable to transmit uncompressed video with zero latency and multi-streaming for dual-view applications where remote access is required.

 
LVL 3

Author Comment

by:Akulsh
ID: 40443517
Gary,

Fastcopy is a good suggestion, but the link you included takes me to a Chinese/Japaness page when I click on download.
0
 
LVL 3

Author Comment

by:Akulsh
ID: 40443559
Aikimark,

You are right that /W switch is not needed. But I don't think it should come in the way.

1. The version of robocopy is the one that comes with Win2008-R2, or 5.1.10.1027.
2. There is no workstation here. Source is non-Windows (SAMFS share) and target is Win2008-R2.
3. Time elapsed is at least 12 hours, because I see CMD windows gone next day.
4. Win2008-R2
5. Do you mean mapping a drive to the local folder? Are you sure that is something worth trying?

Thanks.

AK
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40443600
You are currently using a pull model.  I wondered if a push model might be a work-around.

Or is the system you are currently running the Robocopy command on a Windows Server 2008 box?

If you have the bandwidth in your server room, try adding the multi-threading option to your command Example: /MT:12
0
 
LVL 3

Author Comment

by:Akulsh
ID: 40443916
Yes, I am running the robocopy on Win2008-R2 box. Cannot run on the Source side since that machine is not Windows.

/MT is a good suggestion -- which I have not used before -- but default threads are 8 and maximum is 128. Is there any reason to limit it to 12? Why not more since files are only 10 to 30 KB? The server has plenty of CPU resource. Thanks.
0
 
LVL 45

Expert Comment

by:aikimark
ID: 40444019
Is there any reason to limit it to 12?
Thought it would be easy math if your current elapsed is 12 hours.
0
 
LVL 21

Assisted Solution

by:Ivano Viola
Ivano Viola earned 100 total points
ID: 40444831
Just another suggestion.....

I've used Free File Sync in my environment with any issues. It's portable, free, fast and also copies over NTFS permissions.
About: http://sourceforge.net/projects/freefilesync/
Download: http://www.fosshub.com/FreeFileSync.html

I highly recommend it.

IV
0
 
LVL 3

Author Comment

by:Akulsh
ID: 40444993
IV,

Thanks for this suggestion. This looks like a worthy tool but in this case we want to keep some extra files in the Target folder. In other words, we don't wish to mirror just overwrite. Thanks.

AK
0
 
LVL 21

Expert Comment

by:Ivano Viola
ID: 40445004
You can just do an update. You don't have to do a mirror. You have a few options.
0
 
LVL 3

Accepted Solution

by:
Akulsh earned 0 total points
ID: 40446507
Thanks for all your suggestion. In the end, Robocopy options /R:1 and /W:1 worked and no log file.
It took about 24 hours to copy all files, and actual copying does not start till after about 12 hours.
0
 
LVL 3

Author Closing Comment

by:Akulsh
ID: 40456973
aikimark got me thinking about /R and /W options but he did not suggest that I should increase the value for /R.
0

Featured Post

How Do You Stack Up Against Your Peers?

With today’s modern enterprise so dependent on digital infrastructures, the impact of major incidents has increased dramatically. Grab the report now to gain insight into how your organization ranks against your peers and learn best-in-class strategies to resolve incidents.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Possible fixes for Windows 7 and Windows Server 2008 updating problem. Solutions mentioned are from Microsoft themselves. I started a case with them from our Microsoft Silver Partner option to open a case and get direct support from Microsoft. If s…
How to record audio from input sources to your PC – connected devices, connected preamp to record vinyl discs, streaming media, that play through your audio card: Vista, Windows 7, Windows 8, Windows 8.1 and Windows 10 – both 32 bit & 64.
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
This is used to tweak the memory usage for your computer, it is used for servers more so than workstations but just be careful editing registry settings as it may cause irreversible results. I hold no responsibility for anything you do to the regist…

751 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question