Solved

Copy files from A to B (Only copy non existent or newer files)

Posted on 2016-10-20
  • Storage Software
  • Windows OS
  • Storage Hardware
  • MS DOS
  • Windows Server 2008
  • +1
7
62 Views
Last Modified: 2016-10-31
I have a situation where data was copied from a server to a NAS device but the mapped drives were not updated on the workstations. Files were changed throughout the day on the server instead of on the NAS. It also looks like some files did not get copied originally.

The files were originally copied using Roadkill's Unstoppable Copier. One of the directories was over 500GB and I would guess that 50-100GB of data was changed or modified. The server is failing so we are only getting a transfer rate of 5 MB/s and I can't risk trying to copy everything over again.

I need to copy the data from the server to the NAS again, this time only copying files that either do not exist on the target location or files that are older than the files being copied. Also some users did get the drive mapped so some files on the target are newer than the source and vice versa.

I have researched using xcopy, or xxcopy, roadkill using the (Copy Newer Files Only) switch, and a program called beyond compare. I am a little overwhelmed and on a time constraint. Could you guys give me some suggestions that have the smallest risk of data being lost or overwritten? I would prefer an option that offers logging or a UI but not required.

Thank you
0
Comment
Question by:ITIExperts1
  • 2
  • 2
  • 2
  • +1
7 Comments
 
LVL 51

Assisted Solution

by:Bill Prew
Bill Prew earned 250 total points
Comment Utility
I would think this could be a good job for ROBOCOPY with the MIR option.  It can make a second tree structure mirror a first tree structure, which sounds about like what you want.  If you want to keep newer files in the destination that is an option.

One thing to keep in mind is that deletes will be particularly problematic, since you know have two copies of the same starting tree of data, and updates (adds, deletes, changes) to either could have occurred.  It's going to take some effort and potentially some manual work if you want to get it perfect.

ROBOCOPY is a command line tool but does support logging to a text file.

Beyond Compare is a truly wonderful tool, and could potentially make it easier to see where the out of syncs are, but I would want to do that in GUI mode rather than it's batch mode.  So in that case you start it off comparing the two trees from the base folder, and it will show where things are different.  Then you work through that list on screen and decide how you want to deal with things.  For example when a folder is missing from one side or the other, you have to decide did it get deleted, or added.

There have been some good posts on Experts Exchange about merging two sets of folders when changes have gone on to both, but I haven't been able to track those down yet in searching...

~bp
1
 

Author Comment

by:ITIExperts1
Comment Utility
Hey Bill,
 Beyond Compare seems pretty tedious, RoboCopy seems like it would work. I am not concerned with deleted files being "Restored" if that is the only issue there, they can deal with that. My only real concern is losing data or having an older file overwrite a newer file. How is RoboCopy comparing changes?

Do you have any experience with Unstoppable Copy and its "Copy Newer Files Only" switch? Documentation is slim.

I have also scoured Expert Exchange, aside from some xcopy commands which seemed to work really well for 1 guy, I have not found much that is "complete".

Thanks for your time!
0
 
LVL 51

Assisted Solution

by:Bill Prew
Bill Prew earned 250 total points
Comment Utility
Sorry, no experience on Unstoppable.

You can tell ROBOCOPY not to bring over older files with the XO option:

/XO :: eXclude Older files

It has a test mode too, where it will just report what files would have been processed but not update anything.  Just make sure you test thoroughly.

~bp
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 19

Accepted Solution

by:
marsilies earned 250 total points
Comment Utility
In  Roadkill's Unstoppable Copier, go to the settings tab and uncheck "Overwrite Existing Files" . This will cause it to skip any files that already exist in the target folder. Run it.

Then, on a second pass, go to the settings tab and check both  "Overwrite Existing Files" and "Copy Newer Files Only." This will overwrite only files that are newer on the source.

I just ran a test using these settings, and it worked, but you may want to test it yourself with some test folders before running on the directories. The only other things checked in the settings tab were "Keep File Attributes," "Keep File Ownership," "Keep File Time/Date," and "Enable Log File."
0
 
LVL 19

Expert Comment

by:marsilies
Comment Utility
Alternatively, you can run once with "Overwrite Existing Files" and "Copy Newer Files Only" both checked. This will copy over both missing files, and files that are newer in the source folder. If the file in the target folder is either newer or the same time stamp, it won't get overwritten.

See my example. "New Text Document" was newer in source, "New Text Document (2)" was the same in both, "New Text Document (3)" was newer in target, and "New Text Document (4)" was in source folder but not target.

Unstoppable Copier Results
0
 
LVL 4

Expert Comment

by:Kylo Ren
Comment Utility
forget about the other solutions. robocopy is king

this is what you do

create two test folders
update files in both locations
run the following command

robocopy source destination /xo

confirm you have the latest files in the test folder

the above command is flawless. have personally used a million times

DO NOT USE THE MIRROR SWITCH
0
 

Author Closing Comment

by:ITIExperts1
Comment Utility
Thanks for the help guys, I was able to do it using unstoppable copier doing what Marsilies described in his comment. Worked like a charm.

Thanks!
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Join & Write a Comment

You might have come across a situation when you have Exchange 2013 server in two different sites (Production and DR). After adding the Database copy in ECP console it displays Database copy status unknown for the DR exchange server. Issue is strange…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now