Solved

Copy files from A to B (Only copy non existent or newer files)

Posted on 2016-10-20
7
410 Views
Last Modified: 2016-10-31
I have a situation where data was copied from a server to a NAS device but the mapped drives were not updated on the workstations. Files were changed throughout the day on the server instead of on the NAS. It also looks like some files did not get copied originally.

The files were originally copied using Roadkill's Unstoppable Copier. One of the directories was over 500GB and I would guess that 50-100GB of data was changed or modified. The server is failing so we are only getting a transfer rate of 5 MB/s and I can't risk trying to copy everything over again.

I need to copy the data from the server to the NAS again, this time only copying files that either do not exist on the target location or files that are older than the files being copied. Also some users did get the drive mapped so some files on the target are newer than the source and vice versa.

I have researched using xcopy, or xxcopy, roadkill using the (Copy Newer Files Only) switch, and a program called beyond compare. I am a little overwhelmed and on a time constraint. Could you guys give me some suggestions that have the smallest risk of data being lost or overwritten? I would prefer an option that offers logging or a UI but not required.

Thank you
0
Comment
Question by:ITIExperts1
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
  • 2
  • +1
7 Comments
 
LVL 55

Assisted Solution

by:Bill Prew
Bill Prew earned 250 total points
ID: 41852853
I would think this could be a good job for ROBOCOPY with the MIR option.  It can make a second tree structure mirror a first tree structure, which sounds about like what you want.  If you want to keep newer files in the destination that is an option.

One thing to keep in mind is that deletes will be particularly problematic, since you know have two copies of the same starting tree of data, and updates (adds, deletes, changes) to either could have occurred.  It's going to take some effort and potentially some manual work if you want to get it perfect.

ROBOCOPY is a command line tool but does support logging to a text file.

Beyond Compare is a truly wonderful tool, and could potentially make it easier to see where the out of syncs are, but I would want to do that in GUI mode rather than it's batch mode.  So in that case you start it off comparing the two trees from the base folder, and it will show where things are different.  Then you work through that list on screen and decide how you want to deal with things.  For example when a folder is missing from one side or the other, you have to decide did it get deleted, or added.

There have been some good posts on Experts Exchange about merging two sets of folders when changes have gone on to both, but I haven't been able to track those down yet in searching...

~bp
1
 

Author Comment

by:ITIExperts1
ID: 41852876
Hey Bill,
 Beyond Compare seems pretty tedious, RoboCopy seems like it would work. I am not concerned with deleted files being "Restored" if that is the only issue there, they can deal with that. My only real concern is losing data or having an older file overwrite a newer file. How is RoboCopy comparing changes?

Do you have any experience with Unstoppable Copy and its "Copy Newer Files Only" switch? Documentation is slim.

I have also scoured Expert Exchange, aside from some xcopy commands which seemed to work really well for 1 guy, I have not found much that is "complete".

Thanks for your time!
0
 
LVL 55

Assisted Solution

by:Bill Prew
Bill Prew earned 250 total points
ID: 41853161
Sorry, no experience on Unstoppable.

You can tell ROBOCOPY not to bring over older files with the XO option:

/XO :: eXclude Older files

It has a test mode too, where it will just report what files would have been processed but not update anything.  Just make sure you test thoroughly.

~bp
0
Easy, flexible multimedia distribution & control

Coming soon!  Ideal for large-scale A/V applications, ATEN's VM3200 Modular Matrix Switch is an all-in-one solution that simplifies video wall integration. Easily customize display layouts to see what you want, how you want it in 4k.

 
LVL 20

Accepted Solution

by:
marsilies earned 250 total points
ID: 41854316
In  Roadkill's Unstoppable Copier, go to the settings tab and uncheck "Overwrite Existing Files" . This will cause it to skip any files that already exist in the target folder. Run it.

Then, on a second pass, go to the settings tab and check both  "Overwrite Existing Files" and "Copy Newer Files Only." This will overwrite only files that are newer on the source.

I just ran a test using these settings, and it worked, but you may want to test it yourself with some test folders before running on the directories. The only other things checked in the settings tab were "Keep File Attributes," "Keep File Ownership," "Keep File Time/Date," and "Enable Log File."
0
 
LVL 20

Expert Comment

by:marsilies
ID: 41854376
Alternatively, you can run once with "Overwrite Existing Files" and "Copy Newer Files Only" both checked. This will copy over both missing files, and files that are newer in the source folder. If the file in the target folder is either newer or the same time stamp, it won't get overwritten.

See my example. "New Text Document" was newer in source, "New Text Document (2)" was the same in both, "New Text Document (3)" was newer in target, and "New Text Document (4)" was in source folder but not target.

Unstoppable Copier Results
0
 
LVL 5

Expert Comment

by:Kylo Ren
ID: 41855416
forget about the other solutions. robocopy is king

this is what you do

create two test folders
update files in both locations
run the following command

robocopy source destination /xo

confirm you have the latest files in the test folder

the above command is flawless. have personally used a million times

DO NOT USE THE MIRROR SWITCH
0
 

Author Closing Comment

by:ITIExperts1
ID: 41866905
Thanks for the help guys, I was able to do it using unstoppable copier doing what Marsilies described in his comment. Worked like a charm.

Thanks!
0

Featured Post

Easy, flexible multimedia distribution & control

Coming soon!  Ideal for large-scale A/V applications, ATEN's VM3200 Modular Matrix Switch is an all-in-one solution that simplifies video wall integration. Easily customize display layouts to see what you want, how you want it in 4k.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

There are many software programs on offer that will claim to magically speed up your computer. The best advice I can give you is to avoid them like the plague, because they will often cause far more problems than they solve. Try some of these "do it…
An introduction to the wonderful sport of Scam Baiting.  Learn how to help fight scammers by beating them at their own game. This great pass time helps the world, while providing an endless source of entertainment. Enjoy!
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…

687 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question