Avatar of Albert Widjaja
Albert Widjaja
Flag for Australia asked on

Modifying Robocopy script to migrate (move) from one drive to NFS share (retryable) ?

Hi People,

I'd like to ask some help from you to assist me in modifying the Robocopy script below:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /S /E /ZB

Open in new window


So that it can:

1. Move the files safely from D:\ Passenger Details directory into the NFS share \\10.1.1.12\Archive\ with no data loss
2. The script is retriable so when I double click the script, it should move the remaining files that were added after I run the script
3. Move the files as fast as possible (because when I test run with the above script default MT:8, it takes 2 days+).
4. log the error code only if exist.

Thanks in advance.
PowershellWindows BatchMicrosoft DOSActive DirectoryScripting Languages

Avatar of undefined
Last Comment
Albert Widjaja

8/22/2022 - Mon
Qlemo

You can't speed up copying more than by using the multithreading option (many recommend to use /mt:32) and removing any restart option, that is don't use /zb.
You should be careful with \" as RoboCopy will read that as literal quote in some cases. Best to not end paths with a backslash.
Would the hardware allow for a faster copy? Source drive read speed, network bandwidth and destination drive write speed are limiting factors, as is the (in) efficiency of NFS; the lowest will detemine the maximum available speed.

BTW, you are aware that privileges are not transferred?
Albert Widjaja

ASKER
Yes I'm aware that NTFS permission is not copied over.

Since I've copied over some stuff (1.3 millions files into the destination drive), now when I modify the script to use /MOVE /MT:32 it took ages to move the files over.
Qlemo

Above covers 1-3. 4 is not doable with RoboCopy itself if you want to be able to stop it anytime. All you can do is to append-log to a file without progress indicator, header, summary information, and then​ filter the log fie content for errors. Options are /log+:logfile.txt /njh /njs /np
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Qlemo

/zb can slow down the restart.
Qlemo

As I see you did not use /Move - is this for test only? A move removes the need for most of the restart checks.
Albert Widjaja

ASKER
Initially, I did not use /MOVE but this time in production file server, I am using /MOVE /MT:32 /ZB parameter.

It took very long time to test copy the whole 1.7 TB file servers (2.3 millions) so I stop it and the go with the parameter above.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Qlemo

At full speed (100MB/s) 1.7 TB would need around 4.7 hours. At low speed (10MB/s) it are 47 hours = almost two days. Your times should lay between those.
As said now for the third time, remove the /ZB to speed up processing. It only makes sense if you have very large files, where a restart at nought would be catastrophic. Without, the incomplete file will be transferred again.
Albert Widjaja

ASKER
OK, I end up using the below script line as you suggest:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /R:2 /MT:32 /log+:F:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window


Because since last night, the copy process did not move or progressing at all with the /ZB command.
Albert Widjaja

ASKER
OK, this is very strange.

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


with the command above, when I leave it running over the weekend, there is nothing "moved" from the source directory D:\Passenger Details into the shared folder on my NAS "\\10.1.1.12\Archive\" ?

I have manually move some of the data about 150 GB which is taking almost one working day myself, but why the Robocopy cannot emulate my Cut & Paste from Windows Explorer to the NAS drive?

I just need to move it off from D: drive by overwriting whatever in the destination NAS drive since my initial copy test.

This is the error logged in the file:

	  *EXTRA File 		 807.8 m	20164190497357-23.89358876.wav
	               1.0 m	D:\Passenger Details
	    New File  		   63694	201661631753136-21.92654663.wav
  0%  
100%  

Open in new window


There are about 3.2 millions of files that I need to Cut & Paste, but it takes too long time for me to wait for each Windows Explorer folder refresh everytime I select 50000+ files in single Cut & Paste session.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
SOLUTION
Qlemo

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Albert Widjaja

ASKER
Hi Qlemo,

So basically some of the files exist on the destination drives since I do the manual test last week.
But now, what's the best script parameter that I need to run to make sure that the data is Cut & Pasted from Source to the destination ?

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /IS /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

Qlemo

Use only one of /IS or /CopyAll. I would try the latter first.
Albert Widjaja

ASKER
Qlemo,

Somehow the files do not moved over to the destination shared network drive.

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


Or is it because the script is still comparing the directory content ?
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Qlemo

Did you look into the log file? Does RoboCopy still run?
Albert Widjaja

ASKER
It is still running.
If I cut & paste manually in one directory above ~512000 files, the Explorer crashed :-|

Hence I manually copy paste the files by 50000 every 2 hours.
ASKER CERTIFIED SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Albert Widjaja

ASKER
Hi Qlemo,
yes, I prefer to clean up the source directory as long as the data is moved over safely.

So the script become like:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


Is that correct ?
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Qlemo

I'm not certain for /CopyAll - it might freak out if not able to set permissions correctly on NFS. On the other hand, it should do no harm either.
Albert Widjaja

ASKER
Hi QLemo,

As at today, the directory that I wanted to migrate size is 435 GB (467,403,264,000 bytes) with the total count of 1,119,401 Files, 1 Folders, I cannot copy paste the whole directory as it crashed my Windows explorer.

Using the suggested Robocopy script below:
Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:X:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window


There is nothing moved over to the shared network drive over the past 48 hrs ?

However, I have modified the script with your Powershell trick to log the error:

$serverlog = "X:\TEMP\MOVElogfile.txt"
Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log:X:\TEMP\MOVElogfile.txt
if ($LASTEXITCODE -ge 8)  # fatal / failed
{   
  move-item X:\TEMP\MOVElogfile.txt $serverlog
} else {
  remove-item $serverlog -ErrorAction SilentlyContinue
}

Open in new window


Would that script above work ?
Qlemo

The script should work.
Are there folders? Those are not moved,, since we forgot to add the /S to traverse subfolders.
I cannot tell what happens, you need to see a difference very soon after starting RoboCopy ...
Can you try the same with a small subset of files, creating a folder plus one or two subfolders, and copy say 10 files into each, and then move those via NFS with RoboCopy?
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Albert Widjaja

ASKER
Thanks !