Copy files from A to B (Only copy non existent or newer files)

Posted on 2016-10-20
Medium Priority
Last Modified: 2016-10-31
I have a situation where data was copied from a server to a NAS device but the mapped drives were not updated on the workstations. Files were changed throughout the day on the server instead of on the NAS. It also looks like some files did not get copied originally.

The files were originally copied using Roadkill's Unstoppable Copier. One of the directories was over 500GB and I would guess that 50-100GB of data was changed or modified. The server is failing so we are only getting a transfer rate of 5 MB/s and I can't risk trying to copy everything over again.

I need to copy the data from the server to the NAS again, this time only copying files that either do not exist on the target location or files that are older than the files being copied. Also some users did get the drive mapped so some files on the target are newer than the source and vice versa.

I have researched using xcopy, or xxcopy, roadkill using the (Copy Newer Files Only) switch, and a program called beyond compare. I am a little overwhelmed and on a time constraint. Could you guys give me some suggestions that have the smallest risk of data being lost or overwritten? I would prefer an option that offers logging or a UI but not required.

Thank you
Question by:ITIExperts1
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
  • 2
  • +1
LVL 56

Assisted Solution

by:Bill Prew
Bill Prew earned 1000 total points
ID: 41852853
I would think this could be a good job for ROBOCOPY with the MIR option.  It can make a second tree structure mirror a first tree structure, which sounds about like what you want.  If you want to keep newer files in the destination that is an option.

One thing to keep in mind is that deletes will be particularly problematic, since you know have two copies of the same starting tree of data, and updates (adds, deletes, changes) to either could have occurred.  It's going to take some effort and potentially some manual work if you want to get it perfect.

ROBOCOPY is a command line tool but does support logging to a text file.

Beyond Compare is a truly wonderful tool, and could potentially make it easier to see where the out of syncs are, but I would want to do that in GUI mode rather than it's batch mode.  So in that case you start it off comparing the two trees from the base folder, and it will show where things are different.  Then you work through that list on screen and decide how you want to deal with things.  For example when a folder is missing from one side or the other, you have to decide did it get deleted, or added.

There have been some good posts on Experts Exchange about merging two sets of folders when changes have gone on to both, but I haven't been able to track those down yet in searching...


Author Comment

ID: 41852876
Hey Bill,
 Beyond Compare seems pretty tedious, RoboCopy seems like it would work. I am not concerned with deleted files being "Restored" if that is the only issue there, they can deal with that. My only real concern is losing data or having an older file overwrite a newer file. How is RoboCopy comparing changes?

Do you have any experience with Unstoppable Copy and its "Copy Newer Files Only" switch? Documentation is slim.

I have also scoured Expert Exchange, aside from some xcopy commands which seemed to work really well for 1 guy, I have not found much that is "complete".

Thanks for your time!
LVL 56

Assisted Solution

by:Bill Prew
Bill Prew earned 1000 total points
ID: 41853161
Sorry, no experience on Unstoppable.

You can tell ROBOCOPY not to bring over older files with the XO option:

/XO :: eXclude Older files

It has a test mode too, where it will just report what files would have been processed but not update anything.  Just make sure you test thoroughly.

Are You Using the Best Web Development Editor?

The worlds of web hosting and web development are constantly evolving. Every year we see design trends change, coding standards adapt and new frameworks/CMS created. With such a quick pace of change it’s easy to get lost trying to keep up.

See if your editor made the list.

LVL 20

Accepted Solution

marsilies earned 1000 total points
ID: 41854316
In  Roadkill's Unstoppable Copier, go to the settings tab and uncheck "Overwrite Existing Files" . This will cause it to skip any files that already exist in the target folder. Run it.

Then, on a second pass, go to the settings tab and check both  "Overwrite Existing Files" and "Copy Newer Files Only." This will overwrite only files that are newer on the source.

I just ran a test using these settings, and it worked, but you may want to test it yourself with some test folders before running on the directories. The only other things checked in the settings tab were "Keep File Attributes," "Keep File Ownership," "Keep File Time/Date," and "Enable Log File."
LVL 20

Expert Comment

ID: 41854376
Alternatively, you can run once with "Overwrite Existing Files" and "Copy Newer Files Only" both checked. This will copy over both missing files, and files that are newer in the source folder. If the file in the target folder is either newer or the same time stamp, it won't get overwritten.

See my example. "New Text Document" was newer in source, "New Text Document (2)" was the same in both, "New Text Document (3)" was newer in target, and "New Text Document (4)" was in source folder but not target.

Unstoppable Copier Results

Expert Comment

by:Kylo Ren
ID: 41855416
forget about the other solutions. robocopy is king

this is what you do

create two test folders
update files in both locations
run the following command

robocopy source destination /xo

confirm you have the latest files in the test folder

the above command is flawless. have personally used a million times


Author Closing Comment

ID: 41866905
Thanks for the help guys, I was able to do it using unstoppable copier doing what Marsilies described in his comment. Worked like a charm.


Featured Post

Portable, direct connect server access

The ATEN CV211 connects a laptop directly to any server allowing you instant access to perform data maintenance and local operations, for quick troubleshooting, updating, service and repair.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When asking a question in a forum or creating documentation, screenshots are vital tools that can convey a lot more information and save you and your reader a lot of time
Backups and Disaster RecoveryIn this post, we’ll look at strategies for backups and disaster recovery.
This is used to tweak the memory usage for your computer, it is used for servers more so than workstations but just be careful editing registry settings as it may cause irreversible results. I hold no responsibility for anything you do to the regist…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, just open a new email message. In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question