Robocopy questions

i am starting to use robococopy for backing up user data, and  want to know:
1 - is robocopy faster than normal copy operations in windows ?  if yes -  are there any figures on this?
2 - i am looking for the command - or script - or batch file to do this, and it would be nice if i could specify the source and destination
3 - would this script work also on windows 10 ? that would be a bonus
4 - are there other free softwares that do this also ?

my typical setup would be to use a PC to which both source and destination drives are connected by SATA cable, as 2nd and 3rd drive
i guess using  an USB connection would be no problem?
i am using windows 7 64bit
LVL 94
nobusAsked:
Who is Participating?
 
serialbandConnect With a Mentor Commented:
Modern Intel CPUs support 2 threads per core.  You can easily count the number of threads when you launch task manager and look in the performance tab if you've selected One graph per CPU.  That actually shows you one graph per CPU.  If Hyperthreading is enabled, you should see one graph per logical CPU.

You can also run the following command and it will show you your CPUs with the number of cores and processors you have.  Add them up and it will tell you how many threads you have.
WMIC CPU Get DeviceID,NumberOfCores,NumberOfLogicalProcessors

  
is there nowhere anything to be found on the speed difference with explorer copy ?
I don't think anyone has ever run tests and actually benchmarked them.  They're really for different uses.   Explorer is meant for normal daily user use.  You'd never use that to do system wide backups or archives, because you change folder time stamps and permissions.  To copy an entire drive, it spends initial time trying to load the file names to enumerate them for copying.  I've seen junior level admins attempt this, but they'd always have to break them up into smaller chunks.  I remember testing this once to compare, a long time ago, well over a decade back, but just gave up when explorer didn't even start the copy the next morning after almost 24 hours.  It was with a ~300 GB SCSI RAID, that was most certainly not a full, but there were a lot of little files.  I do remember that the first pass of robocopy took a few days to complete and subsequent updates took 0.5-2 hours, depending on activity.

The larger files will copy faster.  Smaller files means more stops and starts and that can delay the copy tremendously.  With older robocopy, I used to just open multiple command prompts and start copies on different branches to improve the copy speed of smaller files.  When those completed I'd run a final robocopy on the full disk.  The newer robocopy does that for you with the /mt (multithreaded) option.

what does this do? explain options plse
I presume you mean this line
   robocopy D:\  E:\Backup /copyall /r:1 /w:1 /log:E:\rocopylog.txt
It copies from drive D:\
 to a backup folder on E:\
/copyall = /copy:DATSOU  ==> D=Data, A=Attributes, T=Timestamps,  S=Security=NTFS ACLs, O=Owner info, U=aUditing info
/r:1 == set retry to one time : default = keep trying
/w:1 == set wait time between retries to 1s.
/log:E:\rocopylog.txt  == create a log file on E:\ drive named rocopylog.txt

You want the /r  and /w options if you have activity and files in use, otherwise your copies will never finish on time.  It's far easier, faster, and better to just skip those files and run robocopy again.  I set them to one because I recall that setting retries  to 0 means infinite retries, and not waiting at all caused problems.

XXCopy used to be more useful when SMB copy speeds were throttled by Microsoft's own CIFS shares.  After they fixed it around Windows 7, it was less necessary.  I used to help graphics shops purchase and install Helios PCShare and Xinet's older product (can't remember the name) to make sure files could copy to Windows shares at full ethernet speeds instead of 1/4 speed.  That was critical when sending 1GB -2 GB files from the client systems to the RIP servers to the large format Printers over 10 MB (15-30minutes versus 1-2 hours) and 100 MB (90-180seconds versus 5-10minutes) ethernet.  All those products became much less necessary once 1GB ethernet became consumer priced, but some companies still needed that extra speed for productivity boosts.  Once Microsoft fixed their own SMB/CIFS, these products weren't really worth the costs, except to a very few.
0
 
Tomas ValentaConnect With a Mentor IT ManagerCommented:
Hello,
copying files from command line - robocopy or xcopy - approx. the same speed, but faster than copy in Explorer.
Robocopy is working also in Windows 10.
Robocopy could be run by several switches modified its functionality:
copy mode - just copy files, can overwrite existing
mirror mode - mirrors files and structure - exact mirror that's mean delete files or folder in target which are not in source
Here is one of how-to's you can use:
http://ss64.com/nt/robocopy.html
You can find several similar tools, mostly named like File Syncing.. Also several NAS (QNAP or Synology)
has its own sync client for backing up data on computers.
There is no problem o type of connectivity HDD's
0
 
bbaoIT ConsultantCommented:
1. if you are talking about ROBOCOPY on W2K8 or higher, the faster performance should be benefited from its multi-thread design which significantly improves COPY performance and network throughput.

FYI - http://blog.fosketts.net/2010/03/25/robocopy-multi-threaded/

2. the basic usage of ROBOCOPY is below, which covers both source and destination.

ROBOCOPY source destination [file [file]...] [options]

3. yes compatible with W10.

4. XCOPY and XXCOPY
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
Gary CaseConnect With a Mentor RetiredCommented:
I've used both XXCopy and RoboCopy for years => both work VERY well for batch files that can do some pretty sophisticated copy operations (there are MANY filter options that let you select files by various criteria).

However, I've changed virtually all of my backup scripts to use SyncBack -- I use the paid "Plus" version; but the free version is nearly as powerful.    http://www.2brightsparks.com/freeware/

You can create as many "profiles" as you want -- each one has a specified source and destination; and you can set MANY options that allow you to make each profile do pretty much anything you'd want.    You can have "group" profiles that execute a collection of individual backup profiles.    Any profile can have a schedule associated with it so it will automatically run at the time(s) you select.   And it's TRIVIAL to set up an icon on your desktop that will automatically run any specific profile with a simple double-click.

And it's VERY fast -- it easily matches the copy speeds of RoboCopy or XXCopy.
0
 
Mike TLeading EngineerCommented:
Hi,

It depends what you want. Robocopy is *not* called that because of it's automation. You have to script it for that. Its full name is Robust Copy. Its purpose is to provide robust, reliable copying over a network or other path. As such most of its switches are filters to include or exclude files and to retry.

The most important switch to me is /z which provides a restartable copy (like ftp resume).

I've found that normal copies using windows explorer appear to work only to discover files are corrupt.

So, if you care about that data use robocopy.

If you want raw speed it may not be the fastest, but then with user data I'm sure you want good copies rather than fast copies that might be corrupt.

Alternatives: I use Directory Opus a lot, but never in a work environment but it's worth looking at. It's a File Management replacement for Explorer and has good filtering and copy progress.
XXCopy is old but good.
Terracopy (someone I knew used it)

Mike
0
 
nobusAuthor Commented:
Tomas, you said " but faster than copy in Explorer."  my question was HOW MUCH faster
i am aware of it's functionalities and commands  - but they're not easy to use so please, give a specific answer to my Q   i am looking for the command - or script - or batch file to do this
BING - why do you talk about W2k??  i specified on what OS i will use it
tx for mentioning xcopy and xxcopy - and the Multi thread  switch
i know the basic sysntax -  but find it  "not easy" to use - so i'm looking for a batch file or script to run it

Gary i found using Syncback a bit disturbing, because i had to make a backup profile for each folder  - resulting in about 6-10 folders that must be grouped into one profile
But it never occurred to me it was as fast as robocopy
0
 
Gary CaseRetiredCommented:
Agree you need to make a profile for each folder in SyncBack ... but that's really not much different than one line in a batch file for each folder, which is what you end up with with RoboCopy.    I've switched to SyncBack because of the GUI interface ... and the very easy ability to have multiple "group" profiles that do my individual backups.

RoboCopy is, however, a very robust utility that does a superb job -- it just takes a bit of effort to set up the command files.
0
 
Tomas ValentaIT ManagerCommented:
Here is example of robocopy command for mirroring of data (every turn create exact copy of source folder in target location
robocopy "SourceDrive:\" "DestinationDrive:\Backup" /MIR /E /SEC /V /NP /LOG:"c:\homerobocopy.log"  /R:10 /W:30
explanation: /MIR - create mirror include permissions /SE, log operations to the logfile /LOG , retry copy 10x with waits 30seconds between retries
If you use this command without /MIR switch - by default Robocopy will only copy a file if the source and destination have different time stamps or different file sizes.
0
 
nobusAuthor Commented:
Gary, i also had during the copy of data with syncback some errors on which it stopped, but can't just tell you which one (it's been several months)
so is there a setting to  avoid the process being stopped?

Tomas tx for posting the command line; can you help me with some follow-up questions?
 is the MIR needed for what i want? i will Always copy to an empty folder
and why not use the MT option?
do you mean SE or SEC ?  and what are the V and NP options?
does it copy files with "access denied" problem in explorer?
does it copy active files also?
0
 
Tomas ValentaConnect With a Mentor IT ManagerCommented:
- /MIR not needed in your case - copy to an empty folder
- you can use /MT (Multithread copy) but you need to test it what number of threads will be best
- I mean /SEC - security permissions of files and folders - if you do not need it do not use it - copy will be faster
Access denied problem is connected with permissions or file in use issue. For file in use issue the switch /R:10 makes 10 retries with  /W:30 seconds between each and this could help you. Also you can find these files in log file. For Access Denied you need to run robocopy command in scheduler under System account which in normal situation has permission everywhere.
If you find important files not backuped and you need to backup them then there is only system snapshots solutions - command for creating volume snapshot, this snapshot you can mount and copy files from it, all done by script.
0
 
nobusAuthor Commented:
Good info Tomas
what is the command for dcreating a volume snapshot, and what will it backup ?
0
 
Tomas ValentaIT ManagerCommented:
the easiest method for you is to use free backup SW using Volume Snapshotting. Here is the list freeware
backup SW and most of them has this capability.
https://www.lifewire.com/free-backup-software-tools-2617964
I can recommend Comodo Backup or Cobian Backup, both of them I used in the past.
0
 
serialbandCommented:
If you're copying to an empty folder, you don't need /mir.  You also don't really need /z if you're copying lots of small files.  It might actually slow you down.  /z is more useful for large files.

Robocopy will copy quicker than explorer, by several hours or even days, depending on how many files you're copying.  Explorer tries to preload all the files it's copying first, before it starts a copy operation.  The more files it has, the longer it takes before it even starts copying.  Robocopy enumerates the files as it copies, so it will be faster because of that.  The /MT option should be set to the maximimun number of threads your system allows if you want it to go as fast as possible.  If you plan on working on the system or your system is running other tasks, set it to 1 less than the maximum number of threads you have available.  In older Windows, I just ran multiple robocopy processes, up to the number of threads and approximate the folder groupings to get them to copy about equally, to increase.  I seem to recall that you can copy the newer version of robocopy.exe back to older windows and still take advantage of the multithreading feature.

If you use robocopy, I would not set /R and /W to such high values.  Those cause copy delays when you encounter errors, especially if you have lots of open files, because of multiple users.  It will be much quicker to set those to the lowest allowed value, /R:1 /W:1, and just run robocopy again immediately afterwards to catch the open, in-use files when they are closed.  It will be much, much quicker to skip those files after 1 second than to spend 10 retries with 30 second waits.  That's a 5 minute delay for each error, plus the restart of the copy, a very significant delay.  I've always just sent the output to the error logs and restarted the copy to compare the in-use files.  It's a significant time savings to just run another robocopy afterwards.

If you really need to copy in-use files, don't use robocopy.  You'll need hobocopy, a free 3rd party tool.  You'll need to turn on VSS (volume shadow service) and hobocopy will make use of that to copy any files that are in use.  https://candera.github.io/hobocopy/

If you want to do deduplications, I suggest you get HardLinkBackup.  The free version is sufficient to make multiple snapshots, but will suffer from the same problems as robocopy in regards to in-use files.  If you pay for license, you can copy files that are in use.  It will also allow you to schedule your copies.  You can make 1023 deduped copies onto a single disk, using the built in NTFS hard links.  You can rotate out older copies.  http://www.lupinho.net/en/hardlinkbackup/

For both robocopy.exe and hobocopy.ex, the simplest way to schedule them is to place them into a task as a one-liner script.  Prepare and test the command with all the options you need, then just cut and paste that full command into the scheduler.

Robocopy and hobocopy are free and I've used them at places that didn't have funds.
HardLinkBackup is nice too.  I've used this at places that didn't want to spend money on NAS, but were willing to buy cheaper, single external disks.  

With all 3, the first copies take the longest.  All of these copy faster than dragging and dropping from explorer.

robocopy D:\  E:\Backup /copyall /r:1 /w:1 /log:E:\rocopylog.txt

How much data are you copying?  Is it to spinning disk?  RAID? SSD?  Are they small files?  The /mt will help with small files, but they'll still be slower than copying larger files.  300 GB to spinning disk RAID could still take 12 hours or more if it's a lot of small files, versus 300 1 GB files.

If you're doing incremental copies, HardLinkBackup is worth a look.
0
 
nobusAuthor Commented:
serialband -tx for hopping in - good info
Questions : how do i know how much threads i have? eg on an i-5 cpu ?

what does this do? explain options plse

>>  How much data are you copying?  Is it to spinning disk?  RAID? SSD?  Are they small files? <<  as said it will be used to copy data from user  disk  (normal or SSD) no raid
they can by any size - probably a mix

is there nowhere anything to be found on the speed difference with explorer copy ?
0
 
Gary CaseRetiredCommented:
"... i also had during the copy of data with syncback some errors on which it stopped, but can't just tell you which one (it's been several months)
 so is there a setting to  avoid the process being stopped? "  ==>  I've not seen SyncBack ever stop, although it will display the log via HTML (i.e. a pop-up browser windows) if there are any errors.    You can uncheck the box that displays that log and that won't happen;  or you can set it to e-mail the log after a profile is run ... again this can be set to it only happens if there are errors.
0
 
nobusAuthor Commented:
ok tx for all this
so this is the command to run
 robocopy D:\  E:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
i'll have a couple of test runs with it

then i'll close this Q
0
 
nobusAuthor Commented:
i tried to copy from E:\Gebruikers\Ianka    -->this is the path shown in explorer (Gebruikers = Users)
to : X:\Backup  -->  Backup is an empty folder  on the X: drive
with the command :

 robocopy E\Gebruikers\Ianka:\  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
it only created the log file, that said source path not found
what am i doing wrong?  i ran the command as admin from a command prompt
0
 
Tomas ValentaIT ManagerCommented:
this is corrected command:
robocopy E:\Gebruikers\Ianka  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
0
 
nobusAuthor Commented:
yes - sorry i forgot E: - but that was in the command i ran ok
so - what's the cause then?
0
 
serialbandCommented:
Do you have permissions to the source path?  Are you sure the source path is spelled correctly?
0
 
nobusAuthor Commented:
nobody said anything about permissions for running the robocopy commands - do i need them, or can i use an option?

here's the command i ran :  
 robocopy E:\Gebruikers\Ianka\  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt

2016/10/26 14:32:06 fout 3 (0x00000003) Bronmap lezen E:\Gebruikers\Ianka\
Het systeem kan het opgegeven pad niet vinden.  == system cannot find the  path

here's what explorer shows
e
0
 
serialbandCommented:
Robocopy runs with the permissions of the user that runs it.  Most people run it as an Administrator account, but Windows allows you to remove Administrators from the permissions list, denying access to the Administrator account.  One way is to schedule your robocopy command and it should run as System.  The other ways is to add admin account privileges to all files with cacls.
0
 
nobusAuthor Commented:
ok - tell me how
0
 
serialbandConnect With a Mentor Commented:
https://technet.microsoft.com/en-us/library/ff404240.aspx
Taking Ownership of Files and Folders
If you are an administrator, an authorized user, or a backup operator, you can take ownership of a file or folder by completing the following steps:
1. In Windows Explorer, open the file or folder’s Properties dialog box by right-clicking the file or folder and then clicking Properties.
2. On the Security tab, click Advanced to display the Advanced Security Settings dialog box.
3. On the Owner tab, click Edit.This opens the Advanced Security Settings dialog box for editing.
4. In the Change Owner To list, select the new owner.If you’re taking ownership of a folder, you can take ownership of all subfolders and files within the folder by selecting the Replace Owner On Subcontainers And Objects option.
5. Click OK twice when you have finished.

http://www.tipandtrick.net/how-to-take-ownership-and-full-control-permissions-of-files-and-folders-in-vista/
takeown /f directory_name /r /d y
icacls directory_name /grant administrators:F /t

There are several ways to run a command as the system account.
https://msdn.microsoft.com/windows/hardware/drivers/taef/runas-system
psexec is probably a much easier way to run a process as the system account.
psexec -i -s robocopy E:\Gebruikers\Ianka\  X:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
0
 
nobusAuthor Commented:
i posted in my question : "i am starting to use robococopy for backing up user data, "
so how do i take ownership of the whole drive?  i suppose that's the easiest
for making backups

in the meantime, ik'll tryout the psexec command
0
 
serialbandCommented:
takeown /f Drive_name /r /d y
icacls Drive_name /grant administrators:F /t

takeown /f E:\ /r /d y
icacls E:\ /grant administrators:F /t
0
 
nobusAuthor Commented:
i ran takeown /f E:\Gebruikers\ianka/r /d y  icacls Drive_name /grant administrators:F /t
and got syntax error
0
 
serialbandCommented:
Sorry, that should be 2 separate commands.  It cut and pasted into a single line for some reason, when I cut and pasted the multi-line examples from that site.  You only want to run one of them.  Both should take ownership, so you don't need to run it twice.
0
 
nobusAuthor Commented:
i run only  takeown /f E:\Gebruikers\ianka/r /d y   and got wrong syntax, value y cannot be used for option d
0
 
nobusAuthor Commented:
serialband  - can you help?
0
 
serialbandCommented:
Sorry, I don't know why that is the wrong syntax for you.  Maybe you should try the icacls command.

TAKEOWN /?
TAKEOWN [/S system [/U username [/P [password]]]]
        /F filename [/A] [/R [/D prompt]]

Description:
    This tool allows an administrator to recover access to a file that
    was denied by re-assigning file ownership.

Parameter List:
    /S           system          Specifies the remote system to
                                 connect to.

    /U           [domain\]user   Specifies the user context under
                                 which the command should execute.

    /P           [password]      Specifies the password for the
                                 given user context.
                                 Prompts for input if omitted.

    /F           filename        Specifies the filename or directory
                                 name pattern. Wildcard "*" can be used
                                 to specify the pattern. Allows
                                 sharename\filename.

    /A                           Gives ownership to the administrators
                                 group instead of the current user.

    /R                           Recurse: instructs tool to operate on
                                 files in specified directory and all
                                 subdirectories.

    /D           prompt          Default answer used when the current user
                                 does not have the "list folder" permission
                                 on a directory.  This occurs while operating
                                 recursively (/R) on sub-directories. Valid
                                 values "Y" to take ownership or "N" to skip.

    /SKIPSL                      Do not follow symbolic links.
                                 Only applicable with /R.

    /?                           Displays this help message.

    NOTE: 1) If /A is not specified, file ownership will be given to the
             current logged on user.

          2) Mixed patterns using "?" and "*" are not supported.

          3) /D is used to suppress the confirmation prompt.

Examples:
    TAKEOWN /?
    TAKEOWN /F lostfile
    TAKEOWN /F \\system\share\lostfile /A
    TAKEOWN /F directory /R /D N
    TAKEOWN /F directory /R /A
    TAKEOWN /F *
    TAKEOWN /F C:\Windows\System32\acme.exe
    TAKEOWN /F %windir%\*.txt
    TAKEOWN /S system /F MyShare\Acme*.doc
    TAKEOWN /S system /U user /F MyShare\MyBinary.dll
    TAKEOWN /S system /U domain\user /P password /F share\filename
    TAKEOWN /S system /U user /P password /F Doc\Report.doc /A
    TAKEOWN /S system /U user /P password /F Myshare\*
    TAKEOWN /S system /U user /P password /F Home\Logon /R
    TAKEOWN /S system /U user /P password /F Myshare\directory /R /A
PS C:\Users\administrator.TOMSAWYER>

Open in new window

0
 
NVITCommented:
@nobus

Put a space after ianka?

takeown /f E:\Gebruikers\ianka /r /d y
0
 
nobusAuthor Commented:
nvit, that can be, i'll check it
serialband, i don't see an option y for the command?  is it needed?
0
 
serialbandCommented:
Line 32 - 36 in the code window shows the option.  You'll need it if it prompts you continuously.
0
 
nobusAuthor Commented:
ok -thanks for the clarification!
0
 
nobusAuthor Commented:
i ran takeown /f E:\  /r /d y same problem though
0
 
nobusAuthor Commented:
he guys  - no more help for me??
0
 
serialbandCommented:
How about trying the icacls command?

icacls E:\ /grant administrators:F /t


P.S.  I thought I answered last time, but it seems to not have registered.
0
 
nobusAuthor Commented:
i'll try that as well - and tx for the support, serialband,
 i assume E: is the drive i want to grant access - but what is the :F  ?
0
 
serialbandCommented:
That is the permission you assign to Administrator.  F = full.

You can see the basic help with the /? options in Windows.
takeown /?
icacls /?
0
 
nobusAuthor Commented:
well, itried the help - but must have missed it - tx for clearing that up
0
 
nobusAuthor Commented:
iran the grant command, and here's what i got -does this seem ok for you ?
 g
0
 
nobusAuthor Commented:
sertial band - any comments?
0
 
serialbandCommented:
I don't know.
The icacls with the  /T option should have traversed subdirectories.  Your version of takeown is different somehow, and now, your icacls is different.  Those should have worked.
0
 
nobusAuthor Commented:
not yet a solution, but i'm closing this Q it's getting too long
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.