Link to home
Start Free TrialLog in
Avatar of Albert Widjaja
Albert WidjajaFlag for Australia

asked on

Modifying Robocopy script to migrate (move) from one drive to NFS share (retryable) ?

Hi People,

I'd like to ask some help from you to assist me in modifying the Robocopy script below:

Robocopy "D:\Passenger Details" "\\\Archive\" /S /E /ZB

Open in new window

So that it can:

1. Move the files safely from D:\ Passenger Details directory into the NFS share \\\Archive\ with no data loss
2. The script is retriable so when I double click the script, it should move the remaining files that were added after I run the script
3. Move the files as fast as possible (because when I test run with the above script default MT:8, it takes 2 days+).
4. log the error code only if exist.

Thanks in advance.
Avatar of Qlemo
Flag of Germany image

You can't speed up copying more than by using the multithreading option (many recommend to use /mt:32) and removing any restart option, that is don't use /zb.
You should be careful with \" as RoboCopy will read that as literal quote in some cases. Best to not end paths with a backslash.
Would the hardware allow for a faster copy? Source drive read speed, network bandwidth and destination drive write speed are limiting factors, as is the (in) efficiency of NFS; the lowest will detemine the maximum available speed.

BTW, you are aware that privileges are not transferred?
Avatar of Albert Widjaja


Yes I'm aware that NTFS permission is not copied over.

Since I've copied over some stuff (1.3 millions files into the destination drive), now when I modify the script to use /MOVE /MT:32 it took ages to move the files over.
Above covers 1-3. 4 is not doable with RoboCopy itself if you want to be able to stop it anytime. All you can do is to append-log to a file without progress indicator, header, summary information, and then​ filter the log fie content for errors. Options are /log+:logfile.txt /njh /njs /np
/zb can slow down the restart.
As I see you did not use /Move - is this for test only? A move removes the need for most of the restart checks.
Initially, I did not use /MOVE but this time in production file server, I am using /MOVE /MT:32 /ZB parameter.

It took very long time to test copy the whole 1.7 TB file servers (2.3 millions) so I stop it and the go with the parameter above.
At full speed (100MB/s) 1.7 TB would need around 4.7 hours. At low speed (10MB/s) it are 47 hours = almost two days. Your times should lay between those.
As said now for the third time, remove the /ZB to speed up processing. It only makes sense if you have very large files, where a restart at nought would be catastrophic. Without, the incomplete file will be transferred again.
OK, I end up using the below script line as you suggest:

Robocopy "D:\Passenger Details" "\\\Archive" /MOVE /R:2 /MT:32 /log+:F:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window

Because since last night, the copy process did not move or progressing at all with the /ZB command.
OK, this is very strange.

Robocopy "D:\Passenger Details" "\\\Archive\" /MOVE /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

with the command above, when I leave it running over the weekend, there is nothing "moved" from the source directory D:\Passenger Details into the shared folder on my NAS "\\\Archive\" ?

I have manually move some of the data about 150 GB which is taking almost one working day myself, but why the Robocopy cannot emulate my Cut & Paste from Windows Explorer to the NAS drive?

I just need to move it off from D: drive by overwriting whatever in the destination NAS drive since my initial copy test.

This is the error logged in the file:

	  *EXTRA File 		 807.8 m	20164190497357-23.89358876.wav
	               1.0 m	D:\Passenger Details
	    New File  		   63694	201661631753136-21.92654663.wav

Open in new window

There are about 3.2 millions of files that I need to Cut & Paste, but it takes too long time for me to wait for each Windows Explorer folder refresh everytime I select 50000+ files in single Cut & Paste session.
Avatar of Qlemo
Flag of Germany image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi Qlemo,

So basically some of the files exist on the destination drives since I do the manual test last week.
But now, what's the best script parameter that I need to run to make sure that the data is Cut & Pasted from Source to the destination ?

Robocopy "D:\Passenger Details" "\\\Archive\" /MOVE /IS /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

Use only one of /IS or /CopyAll. I would try the latter first.

Somehow the files do not moved over to the destination shared network drive.

Robocopy "D:\Passenger Details" "\\\Archive\" /MOVE /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

Or is it because the script is still comparing the directory content ?
Did you look into the log file? Does RoboCopy still run?
It is still running.
If I cut & paste manually in one directory above ~512000 files, the Explorer crashed :-|

Hence I manually copy paste the files by 50000 every 2 hours.
Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi Qlemo,
yes, I prefer to clean up the source directory as long as the data is moved over safely.

So the script become like:

Robocopy "D:\Passenger Details" "\\\Archive\" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

Is that correct ?
I'm not certain for /CopyAll - it might freak out if not able to set permissions correctly on NFS. On the other hand, it should do no harm either.
Hi QLemo,

As at today, the directory that I wanted to migrate size is 435 GB (467,403,264,000 bytes) with the total count of 1,119,401 Files, 1 Folders, I cannot copy paste the whole directory as it crashed my Windows explorer.

Using the suggested Robocopy script below:
Robocopy "D:\Passenger Details" "\\\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:X:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window

There is nothing moved over to the shared network drive over the past 48 hrs ?

However, I have modified the script with your Powershell trick to log the error:

$serverlog = "X:\TEMP\MOVElogfile.txt"
Robocopy "D:\Passenger Details" "\\\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log:X:\TEMP\MOVElogfile.txt
if ($LASTEXITCODE -ge 8)  # fatal / failed
  move-item X:\TEMP\MOVElogfile.txt $serverlog
} else {
  remove-item $serverlog -ErrorAction SilentlyContinue

Open in new window

Would that script above work ?
The script should work.
Are there folders? Those are not moved,, since we forgot to add the /S to traverse subfolders.
I cannot tell what happens, you need to see a difference very soon after starting RoboCopy ...
Can you try the same with a small subset of files, creating a folder plus one or two subfolders, and copy say 10 files into each, and then move those via NFS with RoboCopy?
Thanks !