Copy files from A to B (Only copy non existent or newer files)

I have a situation where data was copied from a server to a NAS device but the mapped drives were not updated on the workstations. Files were changed throughout the day on the server instead of on the NAS. It also looks like some files did not get copied originally.

The files were originally copied using Roadkill's Unstoppable Copier. One of the directories was over 500GB and I would guess that 50-100GB of data was changed or modified. The server is failing so we are only getting a transfer rate of 5 MB/s and I can't risk trying to copy everything over again.

I need to copy the data from the server to the NAS again, this time only copying files that either do not exist on the target location or files that are older than the files being copied. Also some users did get the drive mapped so some files on the target are newer than the source and vice versa.

I have researched using xcopy, or xxcopy, roadkill using the (Copy Newer Files Only) switch, and a program called beyond compare. I am a little overwhelmed and on a time constraint. Could you guys give me some suggestions that have the smallest risk of data being lost or overwritten? I would prefer an option that offers logging or a UI but not required.

Thank you
Who is Participating?
marsiliesConnect With a Mentor Commented:
In  Roadkill's Unstoppable Copier, go to the settings tab and uncheck "Overwrite Existing Files" . This will cause it to skip any files that already exist in the target folder. Run it.

Then, on a second pass, go to the settings tab and check both  "Overwrite Existing Files" and "Copy Newer Files Only." This will overwrite only files that are newer on the source.

I just ran a test using these settings, and it worked, but you may want to test it yourself with some test folders before running on the directories. The only other things checked in the settings tab were "Keep File Attributes," "Keep File Ownership," "Keep File Time/Date," and "Enable Log File."
Bill PrewConnect With a Mentor Commented:
I would think this could be a good job for ROBOCOPY with the MIR option.  It can make a second tree structure mirror a first tree structure, which sounds about like what you want.  If you want to keep newer files in the destination that is an option.

One thing to keep in mind is that deletes will be particularly problematic, since you know have two copies of the same starting tree of data, and updates (adds, deletes, changes) to either could have occurred.  It's going to take some effort and potentially some manual work if you want to get it perfect.

ROBOCOPY is a command line tool but does support logging to a text file.

Beyond Compare is a truly wonderful tool, and could potentially make it easier to see where the out of syncs are, but I would want to do that in GUI mode rather than it's batch mode.  So in that case you start it off comparing the two trees from the base folder, and it will show where things are different.  Then you work through that list on screen and decide how you want to deal with things.  For example when a folder is missing from one side or the other, you have to decide did it get deleted, or added.

There have been some good posts on Experts Exchange about merging two sets of folders when changes have gone on to both, but I haven't been able to track those down yet in searching...

ITIExperts1Author Commented:
Hey Bill,
 Beyond Compare seems pretty tedious, RoboCopy seems like it would work. I am not concerned with deleted files being "Restored" if that is the only issue there, they can deal with that. My only real concern is losing data or having an older file overwrite a newer file. How is RoboCopy comparing changes?

Do you have any experience with Unstoppable Copy and its "Copy Newer Files Only" switch? Documentation is slim.

I have also scoured Expert Exchange, aside from some xcopy commands which seemed to work really well for 1 guy, I have not found much that is "complete".

Thanks for your time!
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

Bill PrewConnect With a Mentor Commented:
Sorry, no experience on Unstoppable.

You can tell ROBOCOPY not to bring over older files with the XO option:

/XO :: eXclude Older files

It has a test mode too, where it will just report what files would have been processed but not update anything.  Just make sure you test thoroughly.

Alternatively, you can run once with "Overwrite Existing Files" and "Copy Newer Files Only" both checked. This will copy over both missing files, and files that are newer in the source folder. If the file in the target folder is either newer or the same time stamp, it won't get overwritten.

See my example. "New Text Document" was newer in source, "New Text Document (2)" was the same in both, "New Text Document (3)" was newer in target, and "New Text Document (4)" was in source folder but not target.

Unstoppable Copier Results
Kylo RenSystem EngineerCommented:
forget about the other solutions. robocopy is king

this is what you do

create two test folders
update files in both locations
run the following command

robocopy source destination /xo

confirm you have the latest files in the test folder

the above command is flawless. have personally used a million times

ITIExperts1Author Commented:
Thanks for the help guys, I was able to do it using unstoppable copier doing what Marsilies described in his comment. Worked like a charm.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.