Modifying Robocopy script to migrate (move) from one drive to NFS share (retryable) ?

Hi People,

I'd like to ask some help from you to assist me in modifying the Robocopy script below:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /S /E /ZB

Open in new window


So that it can:

1. Move the files safely from D:\ Passenger Details directory into the NFS share \\10.1.1.12\Archive\ with no data loss
2. The script is retriable so when I double click the script, it should move the remaining files that were added after I run the script
3. Move the files as fast as possible (because when I test run with the above script default MT:8, it takes 2 days+).
4. log the error code only if exist.

Thanks in advance.
LVL 9
Senior IT System EngineerIT ProfessionalAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
You can't speed up copying more than by using the multithreading option (many recommend to use /mt:32) and removing any restart option, that is don't use /zb.
You should be careful with \" as RoboCopy will read that as literal quote in some cases. Best to not end paths with a backslash.
Would the hardware allow for a faster copy? Source drive read speed, network bandwidth and destination drive write speed are limiting factors, as is the (in) efficiency of NFS; the lowest will detemine the maximum available speed.

BTW, you are aware that privileges are not transferred?
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
Yes I'm aware that NTFS permission is not copied over.

Since I've copied over some stuff (1.3 millions files into the destination drive), now when I modify the script to use /MOVE /MT:32 it took ages to move the files over.
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Above covers 1-3. 4 is not doable with RoboCopy itself if you want to be able to stop it anytime. All you can do is to append-log to a file without progress indicator, header, summary information, and then​ filter the log fie content for errors. Options are /log+:logfile.txt /njh /njs /np
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
/zb can slow down the restart.
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
As I see you did not use /Move - is this for test only? A move removes the need for most of the restart checks.
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Initially, I did not use /MOVE but this time in production file server, I am using /MOVE /MT:32 /ZB parameter.

It took very long time to test copy the whole 1.7 TB file servers (2.3 millions) so I stop it and the go with the parameter above.
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
At full speed (100MB/s) 1.7 TB would need around 4.7 hours. At low speed (10MB/s) it are 47 hours = almost two days. Your times should lay between those.
As said now for the third time, remove the /ZB to speed up processing. It only makes sense if you have very large files, where a restart at nought would be catastrophic. Without, the incomplete file will be transferred again.
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
OK, I end up using the below script line as you suggest:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /R:2 /MT:32 /log+:F:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window


Because since last night, the copy process did not move or progressing at all with the /ZB command.
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
OK, this is very strange.

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


with the command above, when I leave it running over the weekend, there is nothing "moved" from the source directory D:\Passenger Details into the shared folder on my NAS "\\10.1.1.12\Archive\" ?

I have manually move some of the data about 150 GB which is taking almost one working day myself, but why the Robocopy cannot emulate my Cut & Paste from Windows Explorer to the NAS drive?

I just need to move it off from D: drive by overwriting whatever in the destination NAS drive since my initial copy test.

This is the error logged in the file:

	  *EXTRA File 		 807.8 m	20164190497357-23.89358876.wav
	               1.0 m	D:\Passenger Details
	    New File  		   63694	201661631753136-21.92654663.wav
  0%  
100%  

Open in new window


There are about 3.2 millions of files that I need to Cut & Paste, but it takes too long time for me to wait for each Windows Explorer folder refresh everytime I select 50000+ files in single Cut & Paste session.
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
This cannot be the result of the log file of my suggested command. It shows progress indicators (0% and 100%), though I have inhibited those with /np.
An *EXTRA File is a file only located on the destination.
RoboCopy will only move files which are different between source and destination, or do not exist on the destination. Unchanged files are ignored.
You can try if /IS (include same) helps, but I suppose it wiill restart copying those then. Or add /copyall to also copy security descriptors etc., which are usually not caried over, and hence are a change even for "unchanged" files - this should prevent the files from being copied again, but is change enough to remove them from the source.
0
Senior IT System EngineerIT ProfessionalAuthor Commented:
Hi Qlemo,

So basically some of the files exist on the destination drives since I do the manual test last week.
But now, what's the best script parameter that I need to run to make sure that the data is Cut & Pasted from Source to the destination ?

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /IS /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window

0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Use only one of /IS or /CopyAll. I would try the latter first.
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
Qlemo,

Somehow the files do not moved over to the destination shared network drive.

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /COPYALL /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


Or is it because the script is still comparing the directory content ?
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Did you look into the log file? Does RoboCopy still run?
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
It is still running.
If I cut & paste manually in one directory above ~512000 files, the Explorer crashed :-|

Hence I manually copy paste the files by 50000 every 2 hours.
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
File compare is one by one, that is "check if file exists or has changed, then apply action". Hence you should see a difference almost immediately - unless it doesn't work to move existing and unchanged files over with /CopyAll.
Performing a simple test on SSD showed exactly that. You need to use /IS /XX to perform the move for unchanged files too ... The /XX removes display of *EXTRA, which are files already moved.
Attention: since /Move removes folders too, you will end up loosing your source folder. By using /Mov you keep all folders (with no content), if you prefer that.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Senior IT System EngineerIT ProfessionalAuthor Commented:
Hi Qlemo,
yes, I prefer to clean up the source directory as long as the data is moved over safely.

So the script become like:

Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive\" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:E:\MOVElogfile.txt /njh /njs /np

Open in new window


Is that correct ?
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
I'm not certain for /CopyAll - it might freak out if not able to set permissions correctly on NFS. On the other hand, it should do no harm either.
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
Hi QLemo,

As at today, the directory that I wanted to migrate size is 435 GB (467,403,264,000 bytes) with the total count of 1,119,401 Files, 1 Folders, I cannot copy paste the whole directory as it crashed my Windows explorer.

Using the suggested Robocopy script below:
Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log+:X:\TEMP\MOVElogfile.txt /njh /njs /np

Open in new window


There is nothing moved over to the shared network drive over the past 48 hrs ?

However, I have modified the script with your Powershell trick to log the error:

$serverlog = "X:\TEMP\MOVElogfile.txt"
Robocopy "D:\Passenger Details" "\\10.1.1.12\Archive" /MOVE /COPYALL /IS /XX /R:2 /MT:32 /log:X:\TEMP\MOVElogfile.txt
if ($LASTEXITCODE -ge 8)  # fatal / failed
{   
  move-item X:\TEMP\MOVElogfile.txt $serverlog
} else {
  remove-item $serverlog -ErrorAction SilentlyContinue
}

Open in new window


Would that script above work ?
0
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
The script should work.
Are there folders? Those are not moved,, since we forgot to add the /S to traverse subfolders.
I cannot tell what happens, you need to see a difference very soon after starting RoboCopy ...
Can you try the same with a small subset of files, creating a folder plus one or two subfolders, and copy say 10 files into each, and then move those via NFS with RoboCopy?
1
Senior IT System EngineerIT ProfessionalAuthor Commented:
Thanks !
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Powershell

From novice to tech pro — start learning today.