Link to home
Start Free TrialLog in
Avatar of nobus
nobusFlag for Belgium

asked on

Robocopy questions

i am starting to use robococopy for backing up user data, and  want to know:
1 - is robocopy faster than normal copy operations in windows ?  if yes -  are there any figures on this?
2 - i am looking for the command - or script - or batch file to do this, and it would be nice if i could specify the source and destination
3 - would this script work also on windows 10 ? that would be a bonus
4 - are there other free softwares that do this also ?

my typical setup would be to use a PC to which both source and destination drives are connected by SATA cable, as 2nd and 3rd drive
i guess using  an USB connection would be no problem?
i am using windows 7 64bit
SOLUTION
Avatar of Tomas Valenta
Tomas Valenta
Flag of Czechia image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of bbao
1. if you are talking about ROBOCOPY on W2K8 or higher, the faster performance should be benefited from its multi-thread design which significantly improves COPY performance and network throughput.

FYI - http://blog.fosketts.net/2010/03/25/robocopy-multi-threaded/

2. the basic usage of ROBOCOPY is below, which covers both source and destination.

ROBOCOPY source destination [file [file]...] [options]

3. yes compatible with W10.

4. XCOPY and XXCOPY
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi,

It depends what you want. Robocopy is *not* called that because of it's automation. You have to script it for that. Its full name is Robust Copy. Its purpose is to provide robust, reliable copying over a network or other path. As such most of its switches are filters to include or exclude files and to retry.

The most important switch to me is /z which provides a restartable copy (like ftp resume).

I've found that normal copies using windows explorer appear to work only to discover files are corrupt.

So, if you care about that data use robocopy.

If you want raw speed it may not be the fastest, but then with user data I'm sure you want good copies rather than fast copies that might be corrupt.

Alternatives: I use Directory Opus a lot, but never in a work environment but it's worth looking at. It's a File Management replacement for Explorer and has good filtering and copy progress.
XXCopy is old but good.
Terracopy (someone I knew used it)

Mike
Avatar of nobus

ASKER

Tomas, you said " but faster than copy in Explorer."  my question was HOW MUCH faster
i am aware of it's functionalities and commands  - but they're not easy to use so please, give a specific answer to my Q   i am looking for the command - or script - or batch file to do this
BING - why do you talk about W2k??  i specified on what OS i will use it
tx for mentioning xcopy and xxcopy - and the Multi thread  switch
i know the basic sysntax -  but find it  "not easy" to use - so i'm looking for a batch file or script to run it

Gary i found using Syncback a bit disturbing, because i had to make a backup profile for each folder  - resulting in about 6-10 folders that must be grouped into one profile
But it never occurred to me it was as fast as robocopy
Agree you need to make a profile for each folder in SyncBack ... but that's really not much different than one line in a batch file for each folder, which is what you end up with with RoboCopy.    I've switched to SyncBack because of the GUI interface ... and the very easy ability to have multiple "group" profiles that do my individual backups.

RoboCopy is, however, a very robust utility that does a superb job -- it just takes a bit of effort to set up the command files.
Here is example of robocopy command for mirroring of data (every turn create exact copy of source folder in target location
robocopy "SourceDrive:\" "DestinationDrive:\Backup" /MIR /E /SEC /V /NP /LOG:"c:\homerobocopy.log"  /R:10 /W:30
explanation: /MIR - create mirror include permissions /SE, log operations to the logfile /LOG , retry copy 10x with waits 30seconds between retries
If you use this command without /MIR switch - by default Robocopy will only copy a file if the source and destination have different time stamps or different file sizes.
Avatar of nobus

ASKER

Gary, i also had during the copy of data with syncback some errors on which it stopped, but can't just tell you which one (it's been several months)
so is there a setting to  avoid the process being stopped?

Tomas tx for posting the command line; can you help me with some follow-up questions?
 is the MIR needed for what i want? i will Always copy to an empty folder
and why not use the MT option?
do you mean SE or SEC ?  and what are the V and NP options?
does it copy files with "access denied" problem in explorer?
does it copy active files also?
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of nobus

ASKER

Good info Tomas
what is the command for dcreating a volume snapshot, and what will it backup ?
the easiest method for you is to use free backup SW using Volume Snapshotting. Here is the list freeware
backup SW and most of them has this capability.
https://www.lifewire.com/free-backup-software-tools-2617964
I can recommend Comodo Backup or Cobian Backup, both of them I used in the past.
If you're copying to an empty folder, you don't need /mir.  You also don't really need /z if you're copying lots of small files.  It might actually slow you down.  /z is more useful for large files.

Robocopy will copy quicker than explorer, by several hours or even days, depending on how many files you're copying.  Explorer tries to preload all the files it's copying first, before it starts a copy operation.  The more files it has, the longer it takes before it even starts copying.  Robocopy enumerates the files as it copies, so it will be faster because of that.  The /MT option should be set to the maximimun number of threads your system allows if you want it to go as fast as possible.  If you plan on working on the system or your system is running other tasks, set it to 1 less than the maximum number of threads you have available.  In older Windows, I just ran multiple robocopy processes, up to the number of threads and approximate the folder groupings to get them to copy about equally, to increase.  I seem to recall that you can copy the newer version of robocopy.exe back to older windows and still take advantage of the multithreading feature.

If you use robocopy, I would not set /R and /W to such high values.  Those cause copy delays when you encounter errors, especially if you have lots of open files, because of multiple users.  It will be much quicker to set those to the lowest allowed value, /R:1 /W:1, and just run robocopy again immediately afterwards to catch the open, in-use files when they are closed.  It will be much, much quicker to skip those files after 1 second than to spend 10 retries with 30 second waits.  That's a 5 minute delay for each error, plus the restart of the copy, a very significant delay.  I've always just sent the output to the error logs and restarted the copy to compare the in-use files.  It's a significant time savings to just run another robocopy afterwards.

If you really need to copy in-use files, don't use robocopy.  You'll need hobocopy, a free 3rd party tool.  You'll need to turn on VSS (volume shadow service) and hobocopy will make use of that to copy any files that are in use.  https://candera.github.io/hobocopy/

If you want to do deduplications, I suggest you get HardLinkBackup.  The free version is sufficient to make multiple snapshots, but will suffer from the same problems as robocopy in regards to in-use files.  If you pay for license, you can copy files that are in use.  It will also allow you to schedule your copies.  You can make 1023 deduped copies onto a single disk, using the built in NTFS hard links.  You can rotate out older copies.  http://www.lupinho.net/en/hardlinkbackup/

For both robocopy.exe and hobocopy.ex, the simplest way to schedule them is to place them into a task as a one-liner script.  Prepare and test the command with all the options you need, then just cut and paste that full command into the scheduler.

Robocopy and hobocopy are free and I've used them at places that didn't have funds.
HardLinkBackup is nice too.  I've used this at places that didn't want to spend money on NAS, but were willing to buy cheaper, single external disks.  

With all 3, the first copies take the longest.  All of these copy faster than dragging and dropping from explorer.

robocopy D:\  E:\Backup /copyall /r:1 /w:1 /log:E:\rocopylog.txt

How much data are you copying?  Is it to spinning disk?  RAID? SSD?  Are they small files?  The /mt will help with small files, but they'll still be slower than copying larger files.  300 GB to spinning disk RAID could still take 12 hours or more if it's a lot of small files, versus 300 1 GB files.

If you're doing incremental copies, HardLinkBackup is worth a look.
Avatar of nobus

ASKER

serialband -tx for hopping in - good info
Questions : how do i know how much threads i have? eg on an i-5 cpu ?

what does this do? explain options plse

>>  How much data are you copying?  Is it to spinning disk?  RAID? SSD?  Are they small files? <<  as said it will be used to copy data from user  disk  (normal or SSD) no raid
they can by any size - probably a mix

is there nowhere anything to be found on the speed difference with explorer copy ?
"... i also had during the copy of data with syncback some errors on which it stopped, but can't just tell you which one (it's been several months)
 so is there a setting to  avoid the process being stopped? "  ==>  I've not seen SyncBack ever stop, although it will display the log via HTML (i.e. a pop-up browser windows) if there are any errors.    You can uncheck the box that displays that log and that won't happen;  or you can set it to e-mail the log after a profile is run ... again this can be set to it only happens if there are errors.
ASKER CERTIFIED SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of nobus

ASKER

ok tx for all this
so this is the command to run
 robocopy D:\  E:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
i'll have a couple of test runs with it

then i'll close this Q
Avatar of nobus

ASKER

i tried to copy from E:\Gebruikers\Ianka    -->this is the path shown in explorer (Gebruikers = Users)
to : X:\Backup  -->  Backup is an empty folder  on the X: drive
with the command :

 robocopy E\Gebruikers\Ianka:\  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
it only created the log file, that said source path not found
what am i doing wrong?  i ran the command as admin from a command prompt
this is corrected command:
robocopy E:\Gebruikers\Ianka  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
Avatar of nobus

ASKER

yes - sorry i forgot E: - but that was in the command i ran ok
so - what's the cause then?
Do you have permissions to the source path?  Are you sure the source path is spelled correctly?
Avatar of nobus

ASKER

nobody said anything about permissions for running the robocopy commands - do i need them, or can i use an option?

here's the command i ran :  
 robocopy E:\Gebruikers\Ianka\  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt

2016/10/26 14:32:06 fout 3 (0x00000003) Bronmap lezen E:\Gebruikers\Ianka\
Het systeem kan het opgegeven pad niet vinden.  == system cannot find the  path

here's what explorer shows
User generated image
Robocopy runs with the permissions of the user that runs it.  Most people run it as an Administrator account, but Windows allows you to remove Administrators from the permissions list, denying access to the Administrator account.  One way is to schedule your robocopy command and it should run as System.  The other ways is to add admin account privileges to all files with cacls.
Avatar of nobus

ASKER

ok - tell me how
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of nobus

ASKER

i posted in my question : "i am starting to use robococopy for backing up user data, "
so how do i take ownership of the whole drive?  i suppose that's the easiest
for making backups

in the meantime, ik'll tryout the psexec command
takeown /f Drive_name /r /d y
icacls Drive_name /grant administrators:F /t

takeown /f E:\ /r /d y
icacls E:\ /grant administrators:F /t
Avatar of nobus

ASKER

i ran takeown /f E:\Gebruikers\ianka/r /d y  icacls Drive_name /grant administrators:F /t
and got syntax error
Sorry, that should be 2 separate commands.  It cut and pasted into a single line for some reason, when I cut and pasted the multi-line examples from that site.  You only want to run one of them.  Both should take ownership, so you don't need to run it twice.
Avatar of nobus

ASKER

i run only  takeown /f E:\Gebruikers\ianka/r /d y   and got wrong syntax, value y cannot be used for option d
Avatar of nobus

ASKER

serialband  - can you help?
Sorry, I don't know why that is the wrong syntax for you.  Maybe you should try the icacls command.

TAKEOWN /?
TAKEOWN [/S system [/U username [/P [password]]]]
        /F filename [/A] [/R [/D prompt]]

Description:
    This tool allows an administrator to recover access to a file that
    was denied by re-assigning file ownership.

Parameter List:
    /S           system          Specifies the remote system to
                                 connect to.

    /U           [domain\]user   Specifies the user context under
                                 which the command should execute.

    /P           [password]      Specifies the password for the
                                 given user context.
                                 Prompts for input if omitted.

    /F           filename        Specifies the filename or directory
                                 name pattern. Wildcard "*" can be used
                                 to specify the pattern. Allows
                                 sharename\filename.

    /A                           Gives ownership to the administrators
                                 group instead of the current user.

    /R                           Recurse: instructs tool to operate on
                                 files in specified directory and all
                                 subdirectories.

    /D           prompt          Default answer used when the current user
                                 does not have the "list folder" permission
                                 on a directory.  This occurs while operating
                                 recursively (/R) on sub-directories. Valid
                                 values "Y" to take ownership or "N" to skip.

    /SKIPSL                      Do not follow symbolic links.
                                 Only applicable with /R.

    /?                           Displays this help message.

    NOTE: 1) If /A is not specified, file ownership will be given to the
             current logged on user.

          2) Mixed patterns using "?" and "*" are not supported.

          3) /D is used to suppress the confirmation prompt.

Examples:
    TAKEOWN /?
    TAKEOWN /F lostfile
    TAKEOWN /F \\system\share\lostfile /A
    TAKEOWN /F directory /R /D N
    TAKEOWN /F directory /R /A
    TAKEOWN /F *
    TAKEOWN /F C:\Windows\System32\acme.exe
    TAKEOWN /F %windir%\*.txt
    TAKEOWN /S system /F MyShare\Acme*.doc
    TAKEOWN /S system /U user /F MyShare\MyBinary.dll
    TAKEOWN /S system /U domain\user /P password /F share\filename
    TAKEOWN /S system /U user /P password /F Doc\Report.doc /A
    TAKEOWN /S system /U user /P password /F Myshare\*
    TAKEOWN /S system /U user /P password /F Home\Logon /R
    TAKEOWN /S system /U user /P password /F Myshare\directory /R /A
PS C:\Users\administrator.TOMSAWYER>

Open in new window

@nobus

Put a space after ianka?

takeown /f E:\Gebruikers\ianka /r /d y
Avatar of nobus

ASKER

nvit, that can be, i'll check it
serialband, i don't see an option y for the command?  is it needed?
Line 32 - 36 in the code window shows the option.  You'll need it if it prompts you continuously.
Avatar of nobus

ASKER

ok -thanks for the clarification!
Avatar of nobus

ASKER

i ran takeown /f E:\  /r /d y same problem though
Avatar of nobus

ASKER

he guys  - no more help for me??
How about trying the icacls command?

icacls E:\ /grant administrators:F /t


P.S.  I thought I answered last time, but it seems to not have registered.
Avatar of nobus

ASKER

i'll try that as well - and tx for the support, serialband,
 i assume E: is the drive i want to grant access - but what is the :F  ?
That is the permission you assign to Administrator.  F = full.

You can see the basic help with the /? options in Windows.
takeown /?
icacls /?
Avatar of nobus

ASKER

well, itried the help - but must have missed it - tx for clearing that up
Avatar of nobus

ASKER

iran the grant command, and here's what i got -does this seem ok for you ?
 User generated image
Avatar of nobus

ASKER

sertial band - any comments?
I don't know.
The icacls with the  /T option should have traversed subdirectories.  Your version of takeown is different somehow, and now, your icacls is different.  Those should have worked.
Avatar of nobus

ASKER

not yet a solution, but i'm closing this Q it's getting too long