We help IT Professionals succeed at work.

Modifying Robocopy script to migrate (move) from one drive to NFS share (retryable) ?

1,237 Views
Last Modified: 2017-04-25
Hi People,

I'd like to ask some help from you to assist me in modifying the Robocopy script below:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /S /E /ZB

Open in new window


So that it can:

1. Move the files safely from D:\ Passenger Details directory into the NFS share \\10.1.1.12\Archive\ with no data loss
2. The script is retriable so when I double click the script, it should move the remaining files that were added after I run the script
3. Move the files as fast as possible (because when I test run with the above script default MT:8, it takes 2 days+).
4. log the error code only if exist.

Thanks in advance.
Comment
Watch Question

Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
You can't speed up copying more than by using the multithreading option (many recommend to use /mt:32) and removing any restart option, that is don't use /zb.
You should be careful with \" as RoboCopy will read that as literal quote in some cases. Best to not end paths with a backslash.
Would the hardware allow for a faster copy? Source drive read speed, network bandwidth and destination drive write speed are limiting factors, as is the (in) efficiency of NFS; the lowest will detemine the maximum available speed.

BTW, you are aware that privileges are not transferred?
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Yes I'm aware that NTFS permission is not copied over.

Since I've copied over some stuff (1.3 millions files into the destination drive), now when I modify the script to use /MOVE /MT:32 it took ages to move the files over.
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
Above covers 1-3. 4 is not doable with RoboCopy itself if you want to be able to stop it anytime. All you can do is to append-log to a file without progress indicator, header, summary information, and then​ filter the log fie content for errors. Options are /log+:logfile.txt /njh /njs /np
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
/zb can slow down the restart.
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
As I see you did not use /Move - is this for test only? A move removes the need for most of the restart checks.
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Initially, I did not use /MOVE but this time in production file server, I am using /MOVE /MT:32 /ZB parameter.

It took very long time to test copy the whole 1.7 TB file servers (2.3 millions) so I stop it and the go with the parameter above.
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
At full speed (100MB/s) 1.7 TB would need around 4.7 hours. At low speed (10MB/s) it are 47 hours = almost two days. Your times should lay between those.
As said now for the third time, remove the /ZB to speed up processing. It only makes sense if you have very large files, where a restart at nought would be catastrophic. Without, the incomplete file will be transferred again.
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
OK, I end up using the below script line as you suggest:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /R:2 /MT:32 /log+:F:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window


Because since last night, the copy process did not move or progressing at all with the /ZB command.
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
OK, this is very strange.

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


with the command above, when I leave it running over the weekend, there is nothing "moved" from the source directory D:\Passenger Details into the shared folder on my NAS "\\10.1.1.12\Archive\" ?

I have manually move some of the data about 150 GB which is taking almost one working day myself, but why the Robocopy cannot emulate my Cut & Paste from Windows Explorer to the NAS drive?

I just need to move it off from D: drive by overwriting whatever in the destination NAS drive since my initial copy test.

This is the error logged in the file:

	  *EXTRA File 		 807.8 m	20164190497357-23.89358876.wav
	               1.0 m	D:\Passenger Details
	    New File  		   63694	201661631753136-21.92654663.wav
  0%  
100%  

Open in new window


There are about 3.2 millions of files that I need to Cut & Paste, but it takes too long time for me to wait for each Windows Explorer folder refresh everytime I select 50000+ files in single Cut & Paste session.
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Hi Qlemo,

So basically some of the files exist on the destination drives since I do the manual test last week.
But now, what's the best script parameter that I need to run to make sure that the data is Cut & Pasted from Source to the destination ?

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /IS /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
Use only one of /IS or /CopyAll. I would try the latter first.
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Qlemo,

Somehow the files do not moved over to the destination shared network drive.

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


Or is it because the script is still comparing the directory content ?
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
Did you look into the log file? Does RoboCopy still run?
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
It is still running.
If I cut & paste manually in one directory above ~512000 files, the Explorer crashed :-|

Hence I manually copy paste the files by 50000 every 2 hours.
"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Hi Qlemo,
yes, I prefer to clean up the source directory as long as the data is moved over safely.

So the script become like:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


Is that correct ?
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
I'm not certain for /CopyAll - it might freak out if not able to set permissions correctly on NFS. On the other hand, it should do no harm either.
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Hi QLemo,

As at today, the directory that I wanted to migrate size is 435 GB (467,403,264,000 bytes) with the total count of 1,119,401 Files, 1 Folders, I cannot copy paste the whole directory as it crashed my Windows explorer.

Using the suggested Robocopy script below:
Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:X:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window


There is nothing moved over to the shared network drive over the past 48 hrs ?

However, I have modified the script with your Powershell trick to log the error:

$serverlog = "X:\TEMP\MOVElogfile.txt"
Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log:X:\TEMP\MOVElogfile.txt
if ($LASTEXITCODE -ge 8)  # fatal / failed
{   
  move-item X:\TEMP\MOVElogfile.txt $serverlog
} else {
  remove-item $serverlog -ErrorAction SilentlyContinue
}

Open in new window


Would that script above work ?
Qlemo"Batchelor", Developer and EE Topic Advisor
CERTIFIED EXPERT
Top Expert 2015

Commented:
The script should work.
Are there folders? Those are not moved,, since we forgot to add the /S to traverse subfolders.
I cannot tell what happens, you need to see a difference very soon after starting RoboCopy ...
Can you try the same with a small subset of files, creating a folder plus one or two subfolders, and copy say 10 files into each, and then move those via NFS with RoboCopy?
Senior IT System EngineerSenior Systems Engineer
CERTIFIED EXPERT

Author

Commented:
Thanks !

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions