Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions

Copy files from A to B (Only copy non existent or newer files)

Posted on 2016-10-20
  • Storage Software
  • Windows OS
  • Storage Hardware
  • Microsoft DOS
  • Windows Server 2008
  • +1
Last Modified: 2016-10-31
I have a situation where data was copied from a server to a NAS device but the mapped drives were not updated on the workstations. Files were changed throughout the day on the server instead of on the NAS. It also looks like some files did not get copied originally.

The files were originally copied using Roadkill's Unstoppable Copier. One of the directories was over 500GB and I would guess that 50-100GB of data was changed or modified. The server is failing so we are only getting a transfer rate of 5 MB/s and I can't risk trying to copy everything over again.

I need to copy the data from the server to the NAS again, this time only copying files that either do not exist on the target location or files that are older than the files being copied. Also some users did get the drive mapped so some files on the target are newer than the source and vice versa.

I have researched using xcopy, or xxcopy, roadkill using the (Copy Newer Files Only) switch, and a program called beyond compare. I am a little overwhelmed and on a time constraint. Could you guys give me some suggestions that have the smallest risk of data being lost or overwritten? I would prefer an option that offers logging or a UI but not required.

Thank you
Question by:ITIExperts1
  • 2
  • 2
  • 2
  • +1
LVL 53

Assisted Solution

by:Bill Prew
Bill Prew earned 250 total points
ID: 41852853
I would think this could be a good job for ROBOCOPY with the MIR option.  It can make a second tree structure mirror a first tree structure, which sounds about like what you want.  If you want to keep newer files in the destination that is an option.

One thing to keep in mind is that deletes will be particularly problematic, since you know have two copies of the same starting tree of data, and updates (adds, deletes, changes) to either could have occurred.  It's going to take some effort and potentially some manual work if you want to get it perfect.

ROBOCOPY is a command line tool but does support logging to a text file.

Beyond Compare is a truly wonderful tool, and could potentially make it easier to see where the out of syncs are, but I would want to do that in GUI mode rather than it's batch mode.  So in that case you start it off comparing the two trees from the base folder, and it will show where things are different.  Then you work through that list on screen and decide how you want to deal with things.  For example when a folder is missing from one side or the other, you have to decide did it get deleted, or added.

There have been some good posts on Experts Exchange about merging two sets of folders when changes have gone on to both, but I haven't been able to track those down yet in searching...


Author Comment

ID: 41852876
Hey Bill,
 Beyond Compare seems pretty tedious, RoboCopy seems like it would work. I am not concerned with deleted files being "Restored" if that is the only issue there, they can deal with that. My only real concern is losing data or having an older file overwrite a newer file. How is RoboCopy comparing changes?

Do you have any experience with Unstoppable Copy and its "Copy Newer Files Only" switch? Documentation is slim.

I have also scoured Expert Exchange, aside from some xcopy commands which seemed to work really well for 1 guy, I have not found much that is "complete".

Thanks for your time!
LVL 53

Assisted Solution

by:Bill Prew
Bill Prew earned 250 total points
ID: 41853161
Sorry, no experience on Unstoppable.

You can tell ROBOCOPY not to bring over older files with the XO option:

/XO :: eXclude Older files

It has a test mode too, where it will just report what files would have been processed but not update anything.  Just make sure you test thoroughly.

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

LVL 20

Accepted Solution

marsilies earned 250 total points
ID: 41854316
In  Roadkill's Unstoppable Copier, go to the settings tab and uncheck "Overwrite Existing Files" . This will cause it to skip any files that already exist in the target folder. Run it.

Then, on a second pass, go to the settings tab and check both  "Overwrite Existing Files" and "Copy Newer Files Only." This will overwrite only files that are newer on the source.

I just ran a test using these settings, and it worked, but you may want to test it yourself with some test folders before running on the directories. The only other things checked in the settings tab were "Keep File Attributes," "Keep File Ownership," "Keep File Time/Date," and "Enable Log File."
LVL 20

Expert Comment

ID: 41854376
Alternatively, you can run once with "Overwrite Existing Files" and "Copy Newer Files Only" both checked. This will copy over both missing files, and files that are newer in the source folder. If the file in the target folder is either newer or the same time stamp, it won't get overwritten.

See my example. "New Text Document" was newer in source, "New Text Document (2)" was the same in both, "New Text Document (3)" was newer in target, and "New Text Document (4)" was in source folder but not target.

Unstoppable Copier Results

Expert Comment

by:Kylo Ren
ID: 41855416
forget about the other solutions. robocopy is king

this is what you do

create two test folders
update files in both locations
run the following command

robocopy source destination /xo

confirm you have the latest files in the test folder

the above command is flawless. have personally used a million times


Author Closing Comment

ID: 41866905
Thanks for the help guys, I was able to do it using unstoppable copier doing what Marsilies described in his comment. Worked like a charm.


Featured Post

Easy, flexible multimedia distribution & control

Coming soon!  Ideal for large-scale A/V applications, ATEN's VM3200 Modular Matrix Switch is an all-in-one solution that simplifies video wall integration. Easily customize display layouts to see what you want, how you want it in 4k.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
In this article, I will show you HOW TO: Perform a Physical to Virtual (P2V) Conversion the easy way from a computer backup (image).
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…

790 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question