nobus
asked on
Robocopy questions
i am starting to use robococopy for backing up user data, and want to know:
1 - is robocopy faster than normal copy operations in windows ? if yes - are there any figures on this?
2 - i am looking for the command - or script - or batch file to do this, and it would be nice if i could specify the source and destination
3 - would this script work also on windows 10 ? that would be a bonus
4 - are there other free softwares that do this also ?
my typical setup would be to use a PC to which both source and destination drives are connected by SATA cable, as 2nd and 3rd drive
i guess using an USB connection would be no problem?
i am using windows 7 64bit
1 - is robocopy faster than normal copy operations in windows ? if yes - are there any figures on this?
2 - i am looking for the command - or script - or batch file to do this, and it would be nice if i could specify the source and destination
3 - would this script work also on windows 10 ? that would be a bonus
4 - are there other free softwares that do this also ?
my typical setup would be to use a PC to which both source and destination drives are connected by SATA cable, as 2nd and 3rd drive
i guess using an USB connection would be no problem?
i am using windows 7 64bit
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Hi,
It depends what you want. Robocopy is *not* called that because of it's automation. You have to script it for that. Its full name is Robust Copy. Its purpose is to provide robust, reliable copying over a network or other path. As such most of its switches are filters to include or exclude files and to retry.
The most important switch to me is /z which provides a restartable copy (like ftp resume).
I've found that normal copies using windows explorer appear to work only to discover files are corrupt.
So, if you care about that data use robocopy.
If you want raw speed it may not be the fastest, but then with user data I'm sure you want good copies rather than fast copies that might be corrupt.
Alternatives: I use Directory Opus a lot, but never in a work environment but it's worth looking at. It's a File Management replacement for Explorer and has good filtering and copy progress.
XXCopy is old but good.
Terracopy (someone I knew used it)
Mike
It depends what you want. Robocopy is *not* called that because of it's automation. You have to script it for that. Its full name is Robust Copy. Its purpose is to provide robust, reliable copying over a network or other path. As such most of its switches are filters to include or exclude files and to retry.
The most important switch to me is /z which provides a restartable copy (like ftp resume).
I've found that normal copies using windows explorer appear to work only to discover files are corrupt.
So, if you care about that data use robocopy.
If you want raw speed it may not be the fastest, but then with user data I'm sure you want good copies rather than fast copies that might be corrupt.
Alternatives: I use Directory Opus a lot, but never in a work environment but it's worth looking at. It's a File Management replacement for Explorer and has good filtering and copy progress.
XXCopy is old but good.
Terracopy (someone I knew used it)
Mike
ASKER
Tomas, you said " but faster than copy in Explorer." my question was HOW MUCH faster
i am aware of it's functionalities and commands - but they're not easy to use so please, give a specific answer to my Q i am looking for the command - or script - or batch file to do this
BING - why do you talk about W2k?? i specified on what OS i will use it
tx for mentioning xcopy and xxcopy - and the Multi thread switch
i know the basic sysntax - but find it "not easy" to use - so i'm looking for a batch file or script to run it
Gary i found using Syncback a bit disturbing, because i had to make a backup profile for each folder - resulting in about 6-10 folders that must be grouped into one profile
But it never occurred to me it was as fast as robocopy
i am aware of it's functionalities and commands - but they're not easy to use so please, give a specific answer to my Q i am looking for the command - or script - or batch file to do this
BING - why do you talk about W2k?? i specified on what OS i will use it
tx for mentioning xcopy and xxcopy - and the Multi thread switch
i know the basic sysntax - but find it "not easy" to use - so i'm looking for a batch file or script to run it
Gary i found using Syncback a bit disturbing, because i had to make a backup profile for each folder - resulting in about 6-10 folders that must be grouped into one profile
But it never occurred to me it was as fast as robocopy
Agree you need to make a profile for each folder in SyncBack ... but that's really not much different than one line in a batch file for each folder, which is what you end up with with RoboCopy. I've switched to SyncBack because of the GUI interface ... and the very easy ability to have multiple "group" profiles that do my individual backups.
RoboCopy is, however, a very robust utility that does a superb job -- it just takes a bit of effort to set up the command files.
RoboCopy is, however, a very robust utility that does a superb job -- it just takes a bit of effort to set up the command files.
Here is example of robocopy command for mirroring of data (every turn create exact copy of source folder in target location
robocopy "SourceDrive:\" "DestinationDrive:\Backup" /MIR /E /SEC /V /NP /LOG:"c:\homerobocopy.log" /R:10 /W:30
explanation: /MIR - create mirror include permissions /SE, log operations to the logfile /LOG , retry copy 10x with waits 30seconds between retries
If you use this command without /MIR switch - by default Robocopy will only copy a file if the source and destination have different time stamps or different file sizes.
robocopy "SourceDrive:\" "DestinationDrive:\Backup"
explanation: /MIR - create mirror include permissions /SE, log operations to the logfile /LOG , retry copy 10x with waits 30seconds between retries
If you use this command without /MIR switch - by default Robocopy will only copy a file if the source and destination have different time stamps or different file sizes.
ASKER
Gary, i also had during the copy of data with syncback some errors on which it stopped, but can't just tell you which one (it's been several months)
so is there a setting to avoid the process being stopped?
Tomas tx for posting the command line; can you help me with some follow-up questions?
is the MIR needed for what i want? i will Always copy to an empty folder
and why not use the MT option?
do you mean SE or SEC ? and what are the V and NP options?
does it copy files with "access denied" problem in explorer?
does it copy active files also?
so is there a setting to avoid the process being stopped?
Tomas tx for posting the command line; can you help me with some follow-up questions?
is the MIR needed for what i want? i will Always copy to an empty folder
and why not use the MT option?
do you mean SE or SEC ? and what are the V and NP options?
does it copy files with "access denied" problem in explorer?
does it copy active files also?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Good info Tomas
what is the command for dcreating a volume snapshot, and what will it backup ?
what is the command for dcreating a volume snapshot, and what will it backup ?
the easiest method for you is to use free backup SW using Volume Snapshotting. Here is the list freeware
backup SW and most of them has this capability.
https://www.lifewire.com/free-backup-software-tools-2617964
I can recommend Comodo Backup or Cobian Backup, both of them I used in the past.
backup SW and most of them has this capability.
https://www.lifewire.com/free-backup-software-tools-2617964
I can recommend Comodo Backup or Cobian Backup, both of them I used in the past.
If you're copying to an empty folder, you don't need /mir. You also don't really need /z if you're copying lots of small files. It might actually slow you down. /z is more useful for large files.
Robocopy will copy quicker than explorer, by several hours or even days, depending on how many files you're copying. Explorer tries to preload all the files it's copying first, before it starts a copy operation. The more files it has, the longer it takes before it even starts copying. Robocopy enumerates the files as it copies, so it will be faster because of that. The /MT option should be set to the maximimun number of threads your system allows if you want it to go as fast as possible. If you plan on working on the system or your system is running other tasks, set it to 1 less than the maximum number of threads you have available. In older Windows, I just ran multiple robocopy processes, up to the number of threads and approximate the folder groupings to get them to copy about equally, to increase. I seem to recall that you can copy the newer version of robocopy.exe back to older windows and still take advantage of the multithreading feature.
If you use robocopy, I would not set /R and /W to such high values. Those cause copy delays when you encounter errors, especially if you have lots of open files, because of multiple users. It will be much quicker to set those to the lowest allowed value, /R:1 /W:1, and just run robocopy again immediately afterwards to catch the open, in-use files when they are closed. It will be much, much quicker to skip those files after 1 second than to spend 10 retries with 30 second waits. That's a 5 minute delay for each error, plus the restart of the copy, a very significant delay. I've always just sent the output to the error logs and restarted the copy to compare the in-use files. It's a significant time savings to just run another robocopy afterwards.
If you really need to copy in-use files, don't use robocopy. You'll need hobocopy, a free 3rd party tool. You'll need to turn on VSS (volume shadow service) and hobocopy will make use of that to copy any files that are in use. https://candera.github.io/hobocopy/
If you want to do deduplications, I suggest you get HardLinkBackup. The free version is sufficient to make multiple snapshots, but will suffer from the same problems as robocopy in regards to in-use files. If you pay for license, you can copy files that are in use. It will also allow you to schedule your copies. You can make 1023 deduped copies onto a single disk, using the built in NTFS hard links. You can rotate out older copies. http://www.lupinho.net/en/hardlinkbackup/
For both robocopy.exe and hobocopy.ex, the simplest way to schedule them is to place them into a task as a one-liner script. Prepare and test the command with all the options you need, then just cut and paste that full command into the scheduler.
Robocopy and hobocopy are free and I've used them at places that didn't have funds.
HardLinkBackup is nice too. I've used this at places that didn't want to spend money on NAS, but were willing to buy cheaper, single external disks.
With all 3, the first copies take the longest. All of these copy faster than dragging and dropping from explorer.
robocopy D:\ E:\Backup /copyall /r:1 /w:1 /log:E:\rocopylog.txt
How much data are you copying? Is it to spinning disk? RAID? SSD? Are they small files? The /mt will help with small files, but they'll still be slower than copying larger files. 300 GB to spinning disk RAID could still take 12 hours or more if it's a lot of small files, versus 300 1 GB files.
If you're doing incremental copies, HardLinkBackup is worth a look.
Robocopy will copy quicker than explorer, by several hours or even days, depending on how many files you're copying. Explorer tries to preload all the files it's copying first, before it starts a copy operation. The more files it has, the longer it takes before it even starts copying. Robocopy enumerates the files as it copies, so it will be faster because of that. The /MT option should be set to the maximimun number of threads your system allows if you want it to go as fast as possible. If you plan on working on the system or your system is running other tasks, set it to 1 less than the maximum number of threads you have available. In older Windows, I just ran multiple robocopy processes, up to the number of threads and approximate the folder groupings to get them to copy about equally, to increase. I seem to recall that you can copy the newer version of robocopy.exe back to older windows and still take advantage of the multithreading feature.
If you use robocopy, I would not set /R and /W to such high values. Those cause copy delays when you encounter errors, especially if you have lots of open files, because of multiple users. It will be much quicker to set those to the lowest allowed value, /R:1 /W:1, and just run robocopy again immediately afterwards to catch the open, in-use files when they are closed. It will be much, much quicker to skip those files after 1 second than to spend 10 retries with 30 second waits. That's a 5 minute delay for each error, plus the restart of the copy, a very significant delay. I've always just sent the output to the error logs and restarted the copy to compare the in-use files. It's a significant time savings to just run another robocopy afterwards.
If you really need to copy in-use files, don't use robocopy. You'll need hobocopy, a free 3rd party tool. You'll need to turn on VSS (volume shadow service) and hobocopy will make use of that to copy any files that are in use. https://candera.github.io/hobocopy/
If you want to do deduplications, I suggest you get HardLinkBackup. The free version is sufficient to make multiple snapshots, but will suffer from the same problems as robocopy in regards to in-use files. If you pay for license, you can copy files that are in use. It will also allow you to schedule your copies. You can make 1023 deduped copies onto a single disk, using the built in NTFS hard links. You can rotate out older copies. http://www.lupinho.net/en/hardlinkbackup/
For both robocopy.exe and hobocopy.ex, the simplest way to schedule them is to place them into a task as a one-liner script. Prepare and test the command with all the options you need, then just cut and paste that full command into the scheduler.
Robocopy and hobocopy are free and I've used them at places that didn't have funds.
HardLinkBackup is nice too. I've used this at places that didn't want to spend money on NAS, but were willing to buy cheaper, single external disks.
With all 3, the first copies take the longest. All of these copy faster than dragging and dropping from explorer.
robocopy D:\ E:\Backup /copyall /r:1 /w:1 /log:E:\rocopylog.txt
How much data are you copying? Is it to spinning disk? RAID? SSD? Are they small files? The /mt will help with small files, but they'll still be slower than copying larger files. 300 GB to spinning disk RAID could still take 12 hours or more if it's a lot of small files, versus 300 1 GB files.
If you're doing incremental copies, HardLinkBackup is worth a look.
ASKER
serialband -tx for hopping in - good info
Questions : how do i know how much threads i have? eg on an i-5 cpu ?
what does this do? explain options plse
>> How much data are you copying? Is it to spinning disk? RAID? SSD? Are they small files? << as said it will be used to copy data from user disk (normal or SSD) no raid
they can by any size - probably a mix
is there nowhere anything to be found on the speed difference with explorer copy ?
Questions : how do i know how much threads i have? eg on an i-5 cpu ?
what does this do? explain options plse
>> How much data are you copying? Is it to spinning disk? RAID? SSD? Are they small files? << as said it will be used to copy data from user disk (normal or SSD) no raid
they can by any size - probably a mix
is there nowhere anything to be found on the speed difference with explorer copy ?
"... i also had during the copy of data with syncback some errors on which it stopped, but can't just tell you which one (it's been several months)
so is there a setting to avoid the process being stopped? " ==> I've not seen SyncBack ever stop, although it will display the log via HTML (i.e. a pop-up browser windows) if there are any errors. You can uncheck the box that displays that log and that won't happen; or you can set it to e-mail the log after a profile is run ... again this can be set to it only happens if there are errors.
so is there a setting to avoid the process being stopped? " ==> I've not seen SyncBack ever stop, although it will display the log via HTML (i.e. a pop-up browser windows) if there are any errors. You can uncheck the box that displays that log and that won't happen; or you can set it to e-mail the log after a profile is run ... again this can be set to it only happens if there are errors.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
ok tx for all this
so this is the command to run
robocopy D:\ E:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
i'll have a couple of test runs with it
then i'll close this Q
so this is the command to run
robocopy D:\ E:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
i'll have a couple of test runs with it
then i'll close this Q
ASKER
i tried to copy from E:\Gebruikers\Ianka -->this is the path shown in explorer (Gebruikers = Users)
to : X:\Backup --> Backup is an empty folder on the X: drive
with the command :
robocopy E\Gebruikers\Ianka:\ X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
it only created the log file, that said source path not found
what am i doing wrong? i ran the command as admin from a command prompt
to : X:\Backup --> Backup is an empty folder on the X: drive
with the command :
robocopy E\Gebruikers\Ianka:\ X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
it only created the log file, that said source path not found
what am i doing wrong? i ran the command as admin from a command prompt
this is corrected command:
robocopy E:\Gebruikers\Ianka X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
robocopy E:\Gebruikers\Ianka X:\Backup /copyall /MT:7 /r:1 /w:1 /log:X:\robocopylog.txt
ASKER
yes - sorry i forgot E: - but that was in the command i ran ok
so - what's the cause then?
so - what's the cause then?
Do you have permissions to the source path? Are you sure the source path is spelled correctly?
ASKER
nobody said anything about permissions for running the robocopy commands - do i need them, or can i use an option?
here's the command i ran :
robocopy E:\Gebruikers\Ianka\ X:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
2016/10/26 14:32:06 fout 3 (0x00000003) Bronmap lezen E:\Gebruikers\Ianka\
Het systeem kan het opgegeven pad niet vinden. == system cannot find the path
here's what explorer shows
here's the command i ran :
robocopy E:\Gebruikers\Ianka\ X:\Backup /copyall /MT:7 /r:1 /w:1 /log:E:\rocopylog.txt
2016/10/26 14:32:06 fout 3 (0x00000003) Bronmap lezen E:\Gebruikers\Ianka\
Het systeem kan het opgegeven pad niet vinden. == system cannot find the path
here's what explorer shows
Robocopy runs with the permissions of the user that runs it. Most people run it as an Administrator account, but Windows allows you to remove Administrators from the permissions list, denying access to the Administrator account. One way is to schedule your robocopy command and it should run as System. The other ways is to add admin account privileges to all files with cacls.
ASKER
ok - tell me how
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
i posted in my question : "i am starting to use robococopy for backing up user data, "
so how do i take ownership of the whole drive? i suppose that's the easiest
for making backups
in the meantime, ik'll tryout the psexec command
so how do i take ownership of the whole drive? i suppose that's the easiest
for making backups
in the meantime, ik'll tryout the psexec command
takeown /f Drive_name /r /d y
icacls Drive_name /grant administrators:F /t
takeown /f E:\ /r /d y
icacls E:\ /grant administrators:F /t
icacls Drive_name /grant administrators:F /t
takeown /f E:\ /r /d y
icacls E:\ /grant administrators:F /t
ASKER
i ran takeown /f E:\Gebruikers\ianka/r /d y icacls Drive_name /grant administrators:F /t
and got syntax error
and got syntax error
Sorry, that should be 2 separate commands. It cut and pasted into a single line for some reason, when I cut and pasted the multi-line examples from that site. You only want to run one of them. Both should take ownership, so you don't need to run it twice.
ASKER
i run only takeown /f E:\Gebruikers\ianka/r /d y and got wrong syntax, value y cannot be used for option d
ASKER
serialband - can you help?
Sorry, I don't know why that is the wrong syntax for you. Maybe you should try the icacls command.
TAKEOWN /?
TAKEOWN [/S system [/U username [/P [password]]]]
/F filename [/A] [/R [/D prompt]]
Description:
This tool allows an administrator to recover access to a file that
was denied by re-assigning file ownership.
Parameter List:
/S system Specifies the remote system to
connect to.
/U [domain\]user Specifies the user context under
which the command should execute.
/P [password] Specifies the password for the
given user context.
Prompts for input if omitted.
/F filename Specifies the filename or directory
name pattern. Wildcard "*" can be used
to specify the pattern. Allows
sharename\filename.
/A Gives ownership to the administrators
group instead of the current user.
/R Recurse: instructs tool to operate on
files in specified directory and all
subdirectories.
/D prompt Default answer used when the current user
does not have the "list folder" permission
on a directory. This occurs while operating
recursively (/R) on sub-directories. Valid
values "Y" to take ownership or "N" to skip.
/SKIPSL Do not follow symbolic links.
Only applicable with /R.
/? Displays this help message.
NOTE: 1) If /A is not specified, file ownership will be given to the
current logged on user.
2) Mixed patterns using "?" and "*" are not supported.
3) /D is used to suppress the confirmation prompt.
Examples:
TAKEOWN /?
TAKEOWN /F lostfile
TAKEOWN /F \\system\share\lostfile /A
TAKEOWN /F directory /R /D N
TAKEOWN /F directory /R /A
TAKEOWN /F *
TAKEOWN /F C:\Windows\System32\acme.exe
TAKEOWN /F %windir%\*.txt
TAKEOWN /S system /F MyShare\Acme*.doc
TAKEOWN /S system /U user /F MyShare\MyBinary.dll
TAKEOWN /S system /U domain\user /P password /F share\filename
TAKEOWN /S system /U user /P password /F Doc\Report.doc /A
TAKEOWN /S system /U user /P password /F Myshare\*
TAKEOWN /S system /U user /P password /F Home\Logon /R
TAKEOWN /S system /U user /P password /F Myshare\directory /R /A
PS C:\Users\administrator.TOMSAWYER>
@nobus
Put a space after ianka?
takeown /f E:\Gebruikers\ianka /r /d y
Put a space after ianka?
takeown /f E:\Gebruikers\ianka /r /d y
ASKER
nvit, that can be, i'll check it
serialband, i don't see an option y for the command? is it needed?
serialband, i don't see an option y for the command? is it needed?
Line 32 - 36 in the code window shows the option. You'll need it if it prompts you continuously.
ASKER
ok -thanks for the clarification!
ASKER
i ran takeown /f E:\ /r /d y same problem though
ASKER
he guys - no more help for me??
How about trying the icacls command?
icacls E:\ /grant administrators:F /t
P.S. I thought I answered last time, but it seems to not have registered.
icacls E:\ /grant administrators:F /t
P.S. I thought I answered last time, but it seems to not have registered.
ASKER
i'll try that as well - and tx for the support, serialband,
i assume E: is the drive i want to grant access - but what is the :F ?
i assume E: is the drive i want to grant access - but what is the :F ?
That is the permission you assign to Administrator. F = full.
You can see the basic help with the /? options in Windows.
takeown /?
icacls /?
You can see the basic help with the /? options in Windows.
takeown /?
icacls /?
ASKER
well, itried the help - but must have missed it - tx for clearing that up
ASKER
sertial band - any comments?
I don't know.
The icacls with the /T option should have traversed subdirectories. Your version of takeown is different somehow, and now, your icacls is different. Those should have worked.
The icacls with the /T option should have traversed subdirectories. Your version of takeown is different somehow, and now, your icacls is different. Those should have worked.
ASKER
not yet a solution, but i'm closing this Q it's getting too long
FYI - http://blog.fosketts.net/2010/03/25/robocopy-multi-threaded/
2. the basic usage of ROBOCOPY is below, which covers both source and destination.
ROBOCOPY source destination [file [file]...] [options]
3. yes compatible with W10.
4. XCOPY and XXCOPY