Link to home
Start Free TrialLog in
Avatar of bobgraf
bobgrafFlag for United States of America

asked on

Robycopy Copy Data \ Disk to Disk (new\changed data)

Greetings,
I'm in the process of moving about 1.3TB of data from Disk3 (G:) to Disk6 (K:) on a Windows 2008R2 virtual server.  Once the data from Disk3 is moved, it will be wiped.  Disk3 is a production disk so many files\folders are changed\added during the work week.  Using the batch below, the data took 134 hours to COPY to the Disk6.  (started 1/29/17, completed 2/4/17).  There is no production during the weekend, so ASAP I'd like to run a Robocopy batch that copies just new or changed data to Disk6.  Then do all the things necessary to shut down Disk3 (Disk6 will be the new production disk.)  Both disks are NTFS.  I can provide more detail as needed.  Any tips appreciated.

 c:\robo\ROBOCOPY.EXE G:\DISK3  K:\DISK6 /SEC /S /E /Z /A-:A /R:2 /W:2 /LOG+:C:\ROBO\GtoK.TXT /tee
Avatar of yo_bee
yo_bee
Flag of United States of America image

c:\robo\ROBOCOPY.EXE G:\DISK3  K:\DISK6 /SEC /S /E /XO /Z /A-:A /R:2 /W:2 /LOG+:C:\ROBO\GtoK.TXT /tee

Just add /XO to exclude older. I would leave out /A switch since you are looking to mirror the drive.

Question is what is the ultimate purpose of this routine?
Also I just saw you are using both /S and /E. Only use one of them.

You probably better off using /E. There are some other questions.  
Why are you running the command from c:\robo ?
This is what I would run


[code]
c:\ROBOCOPY.EXE G:\DISK3  K:\DISK6 /SEC /E /XO /Z  /R:1 /W:1 /LOG+:C:\ROBO\GtoK.TXT /NP
[/code]

When you run /TEE you slow the process down.  
The /NP is No Progress.  Something you want when logging when you run into a large file like a PST or something like that.  Without the /NP your log file will grow very large.
yo_bee, I don't agree to any of  your statements made. The original command is fine, just repeating it should work.

/z might need to be changed to /z to improve perfomance (something you should try by running it for  short time, and compare how far the comparision has processed with both variants).

/XO is default, if not changed by other switches like /secfix.
/E includes /S, but there is no harm in  providing both.
/NP is the only switch I agree should be set if using a logfile.
The first copy takes a very long time, but subsequent copies only copy the changes, so they should be much faster.  When I did these, I would run it a one or two times  afterwards to make sure most of it is synchronized first, then stop sharing the old disk and run robocopy once more to copy any final changes.
/z might need to be changed to /z to improve perfomance (something you should try by running it for  short time, and compare how far the comparision has processed with both variants).
I assume you mean /ZB?

/XO is default, if not changed by other switches like /secfix.
No, it is not http://screencast-o-matic.com/watch/cbVtbmQNok
Yes, it should have been "/z to /zb". And I certainly won't view a long-winded video for disproving that /XO is the default. A simple robocopy without additional options excludes older files.
I agree to serialband, I always copy twice, as I've seen that e.g. fixing the folder timestamp won't work all the time (in particular for folders not already on the target).
Yes, it should have been "/z to /zb". And I certainly won't view a long-winded video for disproving that /XO is the default. A simple robocopy without additional options excludes older files.
@Qlemo: You are wrong. By default, it overwrites newer files in destination. /XO prevents this from happening
Obviously I never had that case, but you are correct, changed files on the destination are replaced by older files from the source, making /XO required. Strange I never had issues ...
It is important to use /XO if the destination files will be accessed before all the files are copied (recopied) OR source and destination accessed simultaneously such as a parallel file server migration
@Qlemo: I recommend not posting statements unless you test them. If you'd watched the 2-minute video by Shaun Vermaak or tested Robocopy (as I just did) you would not have made the erroneous statement, "A simple robocopy without additional options excludes older files."

The /XO option falls under "Advanced options you'll probably never use" here: http://ss64.com/nt/robocopy.html
So it makes sense that it is not the default.

I have a hard time imagining many situations in which the /XO option is needed. For it to have any effect, a file in the source would have to have an modified date that is older than its modified date on the day that the earlier Robocopy operation was run. The only scenario I can think of (mind you, its Monday morning) is that someone restores or syncs an older version of a file at some point between the original Robocopy run and the next Robocopy run. And if they actually wanted to do that for some important reason, that good, older file would be lost when Robocopy is run with /XO.

For example, let's say a user is working on a really important document. They are saving it to the network and, just to be safe, they are saving copies to their C drive. Now the document gets messed up - user changes their mind or the file gets corrupted or encrypted.  Your Robocopy operation runs. The next day, the user realizes the problem and copies the older version of the file from their C drive to the network and deletes it from C. Later, you run a Robocopy operation with /XO to update your destination. The newer, bad version wins.

Ideally, in most any situation, you would have versioning protecting your files. If something goes wrong with your file migration, you have versions in a backup that can save your users from a multitude of problems, self-imposed or otherwise.
ASKER CERTIFIED SOLUTION
Avatar of yo_bee
yo_bee
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of bobgraf

ASKER

Thanks to all for the comments, due to my workload I won't be able to test until tomorrow evening.

As to the ultimate purpose, we've recently added a large amount of storage (HP4500) to our virtual environment.  I'm working with a VM consultant who suggested we create disks based on how often they are snapshotted (tier1, tier2, tier3.)  So basically I have to move the data to the applicable disks.  This routing worked great for al the other folders on disk as they were not as large.  As for the various command switches, I'll be honest, I'm not a robocopy expert, and found an example routine that I used.  Once the data is moved to the new Disk6, that's where it will stay.
No matter which options you use, you can just keep running robocopy to copy the changes.  Do it a few times before your final cutover, then turn off the sharing on your old server, run one last robocopy, then enable the sharing on the new server for all the users.  This will reduce the downtime copying the last bits over.
What are the disks you are reading from/writing to?

1.3TB in 134 hours works out to be 2.7MB/sec which is woefully slow!

If you use the old adage that says 20% of your data changes over time (YMMV) its still going to take 24 hours to copy that much at that speed

Tell us a lot more about your setup please?
Avatar of bobgraf

ASKER

I feel the only way I can award the points fairly is to run the procedure again and compare the data and times.  I started the commands again this evening (it was the first chance I had.)  I will monitor closely and report back as soon as completed.  My apologies for any inconvenience.
Avatar of bobgraf

ASKER

Thanks to all who contributed.  I ran yo_bee's suggested command (see below) several seven times between 2/12 and 2/19, as of today all correct information was transferred.  If anyone is interested in the times per run, I'd be happy to provide that, just let me know.

robocopy G:\DISK3 K:\DISK6 /S /XO /W:1 /SEC /R:1 /Log:C:\ROBO\QSR-GtoK1.TXT /NP

Thank you again.
I am curious to know if the /XO switch did make a difference.  If not I learned something new today.
Avatar of bobgraf

ASKER

Again my apologies for the late, late, late reply.  I've attached the results of each robocopy.  The time does go down dramatically from GtoK and GtoK1 and that's due to the fact (like a fool) I copied over the TXT file so I'm missing 2 robocopy sessions.  But from memory I know GtoK1 was at least a couple hours faster.  I did run into one issue only after finding out from our users.  I needed all the empty files copied to the new drive.  Now I have to figure out how to copy the empty folders from the old G to the new K and not copy over the folders that have added data since the final copy.