Solved

Robocopy help - copy last changes

Posted on 2014-12-15
25
283 Views
Last Modified: 2014-12-16
Hi Experts,

for my fileserver migration I used robocopy \\source \\destination /MIR /SEC /SECFIX /COPYALL
Some Shares are very big and it takes hours to migrate them.
Can you help me with the command to just copy the delta of a share to the new location ?
0
Comment
Question by:Eprs_Admin
  • 9
  • 8
  • 7
  • +1
25 Comments
 
LVL 83

Expert Comment

by:oBdA
ID: 40500359
robocopy will by default skip files that are already in the target (same size, same time stamp), unless you specify /IS (include Same), so usually, you just run the same command again.
As to your command line:
* /sec is redundant, it's already included in /copyall.
* If you didn't change the security on any folders already in the target, you don't need /secfix.
* You should add /r:___ (retry, default 1000000) and /w:___ (wait in seconds, default 30) to sensible values (according to your environment), otherwise a single file that can't be copied will basically stop the whole operation (if the connection is stable, /r:0 should do it).
* if you write to a log file, include /np (no progress) as option, otherwise the log file will contain control characters.
So try something like this:
robocopy.exe \\source \\destination /MIR /COPYALL /r:0

Open in new window

0
 
LVL 24

Expert Comment

by:VB ITS
ID: 40500370
Funny that this question popped up on my feed, I'm using robocopy right now to do some file copies..

Did you use the /R:0 and /W:0 switches? If not the copy job will by default retry a million times (!) with 30 seconds between each retry. Also good practice to use the /LOG switch to output the results to a text file as this can increase performance.
0
 

Author Comment

by:Eprs_Admin
ID: 40500386
when I use /MIR again, then at he whole destination will be deleted -> are you joking ???
0
 
LVL 24

Expert Comment

by:VB ITS
ID: 40500388
Also, if you run Robocopy from your 2012 server you can look at using the /MT switch (multi-threading) which will significantly improve copying performance. A good base value to start from would be /MT:32 but I wouldn't go any higher than /MT:64 as there's been reports of network connectivity dropouts when copying large files.
0
 

Author Comment

by:Eprs_Admin
ID: 40500389
once again, I need a command to just copy the delta information to my dest share.
0
 
LVL 24

Expert Comment

by:VB ITS
ID: 40500404
Sorry I just realized what you mean by 'delta copy'. In this case I always opt for the /XO switch as this will leave the files in the destination if it already exists and is the same date or newer than the source.
0
 
LVL 83

Expert Comment

by:oBdA
ID: 40500405
No, I am not joking, because /mir will not "[delete] the whole destination"; where did you get this idea from?
/mir makes sure that the destination is a mirror of the destination; it will (only) delete files/folders in the destination that have been deleted in the target since the initial copy.
0
 
LVL 83

Expert Comment

by:oBdA
ID: 40500452
VB ITS,
using /xo is not required in most cases. Unlike other copy commands, robocopy will by default skip files that are already in the target.
/xo will only apply if a file in the source has been restored to a former date than the one during the first copy had. With a /mir, it can produce erroneous/unintended results; example:
1. User edits file F.
2. The initial copy is run.
3. User isn't happy with his changes in 1, and restores F with an older version.
4. A delta copy with /xo is run: since F now has an older date than the one in the target, it will be skipped, the new file (with the incorrect changes) remains in the target.
5. The user is unhappy when the switch to the new file server is done.
0
 
LVL 24

Expert Comment

by:VB ITS
ID: 40500467
This is all assuming the user has been working out of the old share and not out of the the new share location. Let's wait for further clarification from Eprs Admin before anything else. You make some valid points though.
0
 
LVL 83

Expert Comment

by:oBdA
ID: 40500477
if the user is already working from the new share, a /mir from the old share would be destructive with or without /xo, since any new file or folder created in the new share would be deleted ...
0
 
LVL 24

Expert Comment

by:VB ITS
ID: 40500485
I probably should have been clearer earlier - I meant to say I would opt for the /XO switch over the /MIR switch in this scenario, i.e. no /MIR switch whatsoever.
0
 
LVL 83

Expert Comment

by:oBdA
ID: 40500658
In this case, the result wouldn't be erroneous (since you don't want the folders to be mirrored), but the scenario I described above would still make the user unhappy - his restored file won't be on the new server.
And running a delta copy without the /mir will make even more people unhappy after the switch: all files and folders that they thought they deleted will reappear, and all files and folders they renamed or moved will now be there twice, once with the new name and once with the old name.
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 68

Expert Comment

by:Qlemo
ID: 40500717
Obviously we need more info about what you have done already before being able to provide reasonable advice.  To me it sounds like you started the migration, but were unhappy about the time it took, and stopped it. People are already working with the migrated data, and maybe with the old shares too.
0
 

Author Comment

by:Eprs_Admin
ID: 40502539
Hi obda,
I did /MIR and later on again.
And the the first what robocopy is doing, it deletes the data no the target.
100% sure.
0
 

Author Comment

by:Eprs_Admin
ID: 40502541
so, my first command was :

robocopy \\source \\target /MIR /SEC /SECFIX /COPYALL

Now I need a command to copy the delta of source to target.
Still the share SOURCE is active. Therefore I need to copy all files which have been changed since first command.
0
 
LVL 24

Expert Comment

by:VB ITS
ID: 40502553
Are users still working from the SOURCE data or have they been working from the DESTINATION since the original Robocopy?
0
 

Author Comment

by:Eprs_Admin
ID: 40502587
tonight they do not work anymore on source
0
 
LVL 83

Assisted Solution

by:oBdA
oBdA earned 250 total points
ID: 40502619
I've used "robocopy /mir" in more file server migrations than I can remember, including folders with TBs of data, and it most certainly does not delete data on the target before the delta copy.
It might recopy the data if the target folder doesn't support the NTFS high resolution time stamps (which can happen, for example, with Linux CIFS storages); this can be fixed with /FFT.
If your first copy was done before the DST switch, try /DST.
Note that you can add the /L (List only), which will only "pretend" to copy, but not actually do anything.
Here's a test script that will create a test folder, create a single file in this folder, copy this folder into a target location, copy it a second time, and then delete the test folders again.
@echo off
setlocal
set TestSource=C:\RCTest
set TestTarget=D:\RCTest
set RCOptions=/mir /njh /v /np
if not exist "%TestSource%" md "%TestSource%"
>"%TestSource%\test.txt" echo TestFile
echo ========== Robocopy FIRST run: ==============================
robocopy.exe "%TestSource%" "%TestTarget%" %RCOptions%
echo ========== Robocopy SECOND run: ==============================
robocopy.exe "%TestSource%" "%TestTarget%" %RCOptions%
echo Hit any key to remove the test folders, or ^<Ctrl-C^> to stop.
pause >nul
if exist "%TestSource%" rd /s /q "%TestSource%"
if exist "%TestTarget%" rd /s /q "%TestTarget%"

Open in new window

The output should look something like this:
========== Robocopy FIRST run: ==============================

          New Dir          1    C:\RCTest\
            New File                  10        test.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         1         0         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :        10        10         0         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00


   Speed :               10000 Bytes/sec.
   Speed :               0.572 MegaBytes/min.
   Ended : Tue Dec 16 14:13:53 2014
 
========== Robocopy SECOND run: ==============================

                           1    C:\RCTest\
                  same                10        test.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         0         0         0         0
   Files :         1         0         1         0         0         0
   Bytes :        10         0        10         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00
   Ended : Tue Dec 16 14:13:53 2014

Hit any key to remove the test folders, or <Ctrl-C> to stop.

Open in new window

In the second log, note lines 22, where it classifies the text file found as "same", and line 28, where there is a 1 in the "Skipped" column (which means it was not deleted/recopied), while it was in "Copied" in line 10 during the first run.
0
 

Author Comment

by:Eprs_Admin
ID: 40502665
ok OBDA,
then I try some tests later on with /MIR again.
But in my case I also have to copy all security information /SEC /SECFIX and /COPYALL

When I start this cmd again can I use it like this :
robocopy \\source \\target /MIR /SEC /njh /v /np /COPYALL
0
 

Author Comment

by:Eprs_Admin
ID: 40502691
Please check this ticket, here the guy wrote /MIR deletes all folders on the target dir.
And once I have seen it.
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_Server_2008/Q_28509155.html

Or has it to do with the other switches ?
0
 
LVL 24

Assisted Solution

by:VB ITS
VB ITS earned 250 total points
ID: 40502706
running a delta copy without the /mir will make even more people unhappy after the switch: all files and folders that they thought they deleted will reappear, and all files and folders they renamed or moved will now be there twice, once with the new name and once with the old name.
Good point, I didn't think about this.

@Eprs Admin, oBdA is correct. The /MIR switch doesn't delete the entire contents of the destination. What it does delete though are files/folders in the destination that no longer exist in the source.

Also, no need to run both the /SEC and /COPYALL switches together as the /SEC switch is equivalent to /COPY:DATS
I would also leave out the /SECFIX switch since the /COPYALL switch will also copy over the security permissions, just makes it redundant really.
0
 

Author Comment

by:Eprs_Admin
ID: 40502741
ok now I tested my command and you are right.

With /MIR the files not delted on source, but files which are not on the source but in the target, they will be deleted.
It really creates a mirror of source.

Instead of /SEC /SECFIX /COPYALL I can use just /COPYALL ?
0
 
LVL 24

Assisted Solution

by:VB ITS
VB ITS earned 250 total points
ID: 40502749
That's correct. The /SEC switch is equivalent to /COPY:DATS whereas /COPYALL is equivalent to /COPY:DATSOU

/SECFIX is usually only used to fix security permissions if they go out of whack. We can always run this later without it copying over any files if you run into issues with the security on the destination files/folders.
0
 
LVL 83

Accepted Solution

by:
oBdA earned 250 total points
ID: 40502750
What Svet Paperov was referring to is that if you do the (first) /mir, anything already in the target folder will (probably) be deleted, because it's not in the source and considered "extra". This has nothing to do with robocopy deleting the target folder before a delta copy.
Note that you can always add /L to your switches and rerun the command (stop with Ctrl-C, or use /lev:x to only "copy" the first x levels) to see what it would do. If you add /v as well, you should have lots of "same" classifications.

As to the command line, please read my comments about the command line in http:#a40500359 again; I already mentioned that /sec is included in /copyall.
/sec and /secfix are independent; /secfix will make robocopy check the security of all files again; by default, it does not check for changed security (only for changed size or changed time), so once a file or folder is already there in the same version, a changed ACL will not be copied over without /secfix. Unless you did extensive changes to existing security settings, don't use it, or it will slow down the copy.
I'd use something this for starters (and run the command on either source or target, instead of a third machine):
robocopy.exe \\source \\target /MIR /COPYALL /r:0 /mt /np /tee /log:"C:\Temp\robocopy.log"

Open in new window

Note that /mt is only available since Windows 7/Server 2008 R2.
Add /v at your discretion.
If you want to use a log file with "/log:", you can use /tee to see the console output as well.
/njh in my script is No Job Header, that was just there to keep the output short for the test script.
/v (verbose) will list files classified as "same" (which it will skip); depends on you whether you want that or consider it information overflow. I've used it in the test script to make it obvious that the file wasn't recopied.

Here's a test script for extra, same, newer, and older file classes:
@echo off
setlocal
set TestSource=C:\RCTest
set TestTarget=D:\RCTest
set RCOptions=/copyall /mir /njh /v /np
if not exist "%TestSource%" md "%TestSource%"
>"%TestSource%\test-stays-the-same.txt" echo TestFile
>"%TestSource%\test-will-be-modified.txt" echo TestFile
>"%TestSource%\test-will-be-removed-from-source.txt" echo TestFile
>"%Temp%\test-will-be-replaced-with-older.txt" echo TestFile
ping.exe -n 3 localhost >NUL
>"%TestSource%\test-will-be-replaced-with-older.txt" echo TestFile
echo ========== Robocopy FIRST run: ==============================
robocopy.exe "%TestSource%" "%TestTarget%" %RCOptions%
>>"%TestSource%\test-will-be-modified.txt" echo Content added after second run.
copy "%Temp%\test-will-be-replaced-with-older.txt" "%TestSource%"
del "%TestSource%\test-will-be-removed-from-source.txt" echo TestFile
echo ========== Robocopy SECOND run: ==============================
robocopy.exe "%TestSource%" "%TestTarget%" %RCOptions%
echo Hit any key to remove the test folders, or ^<Ctrl-C^> to stop.
pause >nul
if exist "%TestSource%" rd /s /q "%TestSource%"
if exist "%TestTarget%" rd /s /q "%TestTarget%"

Open in new window

0
 

Author Comment

by:Eprs_Admin
ID: 40502795
ok now I tested some settings.

robocopy \\source \\target /mir /copyall /njh /np /v /log:C:\Temp\log.txt
This command works well and all my security settings also copied.

And of course the List command works well -> Thanks for this:
robocopy \\source \\target /mir /copyall /njh /np /v /L /log:C:\Temp\log.txt

I think we can close now this ticket.
Thanks a lot.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

What to do when Windows Update is not working correctly? What tools can I use to detect the cause of the malfunction problem? What does this numeric error code mean? These and other questions that you have been asking in the past are answered here (…
Resolve DNS query failed errors for Exchange
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now