Robocopy old/deleted files

I have a production File Server A.  I do a Robocopy /MIR to File Server B every night as a backup.

I'd like to improve upon this by copying all of the files on Server B that Robocopy /MIR will either delete or replace to a 3rd folder as a backup.  This way I'll have a snapshot of the previous state of all of the files that were changed that day.

Does anyone have any ideas for how to perform this operation?

Lee TumblesonAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

NVITEnd-user supportCommented:
Not quite following you. Are you saying you want the same exact copy as A and B?
I don't think robocopy can do that in one script.  Rather, the best option here would probably be performing another instance of robocopy from Server B to say, Server C which you call 3rd folder.

Since Server B is backup, it won't experience activity during the day from Server A or any other server or workstation correct?  If so, then during the day, have Server B run an instance of robocopy that will copy the prior night's backup to Server C, and that would be considered your snapshot.

If Server C is meant to hold all data that was deleted indefinitely instead of just from prior night, then you could just use robocoby from Server B with /XO instead of /MIR.  This would just copy the newer files over to Server C.  Try that out to see if it works for you.

If Server C was just to be a placeholder for the changed files from the day, then you'll still need to use the /XO parameter to copy over just the newer files.  It will still need to have a full backup otherwise it won't know what is considered older files.  So for first time use /MIR to make the full backup from Server B to Server C.  Then from that point on, use /XO instead of /MIR and then write to a log file to see what was copied.

robocopy "C:\source" "D:\path" /XO /FP /LOG+C:\Logpath\copy.log

Open in new window

What are the OSes involved?

AD environment, dfs with dfs-r and using a standard backup process.  The items you are setting up are a hot backup, but as was pointed out going on this route, you might continue adding subsequent server to extend "duration of retention".

An option you might want to consider if not already is a document management system.  There commercial as well as open source ones dealing with controlling and recording access as well as maintain versions when changes are made, etc.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

I had a script to do what you ask a while back.  Will look at work in the morning...
Lionel MMSmall Business IT ConsultantCommented:
You can use these parameters to play around with what you want to copy and copy only those files that a certain number of days old or new
/MAXAGE:n :: MAXimum file AGE - exclude files older than n days/date.
/MINAGE:n :: MINimum file AGE - exclude files newer than n days/date.
/MAXLAD:n :: MAXimum Last Access Date - exclude files unused since n.
/MINLAD:n :: MINimum Last Access Date - exclude files used since n.
(If n < 1900 then n = n days, else n = YYYYMMDD date).
robocopy C:\Data D:\Data /S /MAXAGE:1
will only copy files that are one day or less old (24 hours or less)
Lee TumblesonAuthor Commented:
Thanks for the feedback, but none of this is what I'm looking for.  This is a Windows Server 2012 and 2012R2 environment.

I'm looking to copy files that will change or be deleted.  I don't want a snapshot of all 3TB of data, just the previous state of the 30GB that changed that day.

It could be that someone deletes a file that has a filedate from 2 years ago.  I want to make a copy of that so that I don't have to go back to 2-years-ago backups to retrieve it next week when they need it back.

To further complicate matters, I would only make a snapshot of the previous version of the file if it had a date of more than a month ago.  If a file is being changed frequently, I don't need a snapshot, because I am also performing incremental backups every day, so that file would already have a recent backup.  This should reduce the 30GB that I mentioned to less than 2GB.

File Server C would contain dated folders with these snapshots.

I already create daily backup folders using a robocopy like this:

robocopy [source] [destination]\D%date:~-2,2%%date:~-10,2%%date:~-7,2%\ /s /MAXAGE:2 /copy:datso /FFT

This script creates a folder starting with D (daily) and then the current date.  It only copies files that have a modified date within the past 2 days.

What I want to do is create an S (snapshot) folder, with the current date, and only copy files that are at least a month old and will either be overwritten by a newer file or deleted.
one option is that you will need to run robocopy with the -L flag and then extract/copy the line and file reflected as deleted. will be copied to server C
then the B is synchronized with A.
C will only be a repository for deleted items is that the correct understanding.

robocopy \\servera\src \\serverb\src /MIR /LOG:deleted.txt  /L
then you would need to process the deleted.txt file keeping track of the directory it is in and then looking for the *EXTRA file line at which point $directory\filename is copied to Server C.

Or you can run the opposite comparison with B as the source with A as the destination (do not use /MIR)
here you will be looking for the New File notice.

Replication and its backup does not have negative function.

A document management system would likely be your best option as it gives the admin in some cases to mark the file inaccessible/deleted while it is still there.

to achieve your goal in a single process, you would need  to write the comparator/replacement to robocopy.

myrobocopy(powershell/vbscript/or compiled ) src destination deleted_from_src_repository
Lee TumblesonAuthor Commented:
It might just be easier to write my own /MIR in Visual Studio.  Step through all of the files on both servers.  But, before I initiate the 'copy from A to B', I'll 'copy from B to C'.

This would perform the whole process in one pass.
You could, you have to take several things into consideration:
You can start down this path and ..... never get off the road of continually updating/adding to your application.
filea changed on A. do you need to prior B version copied on the C?
dealing with filea is mistakenly modified.
the test for deleted items, will not be triggered such that filea will be updated to server B.
then you are back where you started when the person realizes that they deleted an entry and need the file back.
if files are text based, windiff is a tool that could extract the differences in the files.
Lee TumblesonAuthor Commented:
I didn't quite follow what you said.  I'm looking to copy, from B to C, files that 'will be' modified or deleted when A is mirrored to B.

This covers the 'mistakenly modified' item that you mentioned.

Not sure what I need to update/add.  I'm thinking that I can write this in less than a day.
So any file that will be copied from A to B will have the B version copied to C and any file that will be deleted from B because it no longer exists on A will be copied to C
Your program will still need to have two passes. One compares B to A to find the files on A that were deleted copy file from B to C. Deleting the file on A.
The second step deals with a pass comparing files from A to B to determine if a newer/modified file exists on A at which point the file from B will be copied to C and then the file from A will be copied to B.
Your next step will be how you deal with file handling from B to C. Will you compare the file you are copying from B to C whether it was modified or deleted?

person A accesses server A file A on Monday. makes an alterations. B copied to C.
On Wednesday, person A is once again accessing Server A file A, and makes another alterations. B copied to C.
On Thursday, Person A was asked a question, accesses Server A file A, and realizes the wrong section of the data was deleted Monday.
Your current setup can only go back to the changes made on Wednesday.

Or you can use the combinations which deals with daily changes (deleted/modified) on server C.

In 2012/R2 using DFS and share snapshots depending on how much space you can allocate on a separate drive/resource to store the snapshots of each member of the DFS.
DFS can be configured in a failover primary/secondary for server A/ServerB setup with replication bi-directional.
Lee TumblesonAuthor Commented:
You missed the part in my previous comment where I do this every night:

robocopy [ServerA]  [ServerC]\D%date:~-2,2%%date:~-10,2%%date:~-7,2%\ /s /MAXAGE:2 /copy:datso /FFT

I'm also utilizing ShadowCopy on these server drives.  I'm doing 'snapshots' every 4 hours (3/day, just during the workday) on A (I found a registry switch that allows up to 512 snapshots, which gives me 3/day snapshots for 170 days).  On B, I'm doing the snapshot once a day.  This gives me a year and half of daily snapshots.

I also do periodic full backups of the 3TB of data, but I'd like to reduce the need to go to those backups.  If I can have these 'snapshot' backups, along with the above MAXAGE backups in folders (I'm storing these on a DROBO, I'm estimating that I can maintain these daily backups for 3-4 years on the DROBO), then I can recover files very easily.
Yes, the range of dates/events to handle is the issue.
The difficulty, how do you locate fileA that has been deleted from ServerA or modified erroneously?
The person coming to you with the issue of the missing file might not be the person who deleted/changed it nor has the person recently accessed the file to help you narrow down when the change occurred for you to locate the file.  Now in addition to what you are already doing, you need a cataloger of deleted/modified files on server C so that when anyone comes saying , I am missing fileA, you can go through the catalog and give them the option to see the file and decide which of the multiple versions is the correct one to be restored.

you could/should incorporate DB backend where you can record the copy out to C/days .....
The more analysis you do, the more features you will need to add to your application. then the backup/consideration of the backend DB ......
We used this to only keep the last 30days in a temporary folder.  (many users dumping stuff in there)
Could be modified to archive instead of delete at the end?

@echo off

SET Clean=G:\Temp
SET Stale=30
SET Staging=G:\share\temp

rem Make an empty location to move old files into
rem =============================================
rmdir %Staging% /S /Q
mkdir %Staging%

rem Move old files into staging area
rem ================================
robocopy %Clean% %Staging% /E /MOVE /MINLAD:%Stale% /LOG:%Staging%\cleanup.txt /TEE

rem Delete staging area
rem ===================
rmdir %Staging% /S /Q

Lee TumblesonAuthor Commented:
I wrote a simple program to do this.  After I run this, then I run the Robocopy /MIR to make ServerB the same as ServerA.  All servers/shares are hard-coded.

Public Class Form1

    Dim RootA As String = "\\ServerA\Share\"
    Dim RootB As String = "\\ServerB\Share\"

    Dim RootS As String = "\\ServerB\Share\Snapshots\" & Format(Now(), "yy") & Format(Now(), "MM") & Format(Now(), "dd") & "\"
    Dim PathS As String

    Public Sub New()

        ' This call is required by the designer.

        ' Add any initialization after the InitializeComponent() call.

        RoboSnapshot(RootB & "Folder1\")
        RoboSnapshot(RootB & "Folder2\")

    End Sub

    Private Sub RoboSnapshot(ByVal PathB As String)
        Dim FolderB As System.IO.DirectoryInfo
        Dim FileInfoA As System.IO.FileInfo
        Dim FileInfoB As System.IO.FileInfo

        Dim PathA As String = RootA & Mid(PathB, Len(RootB) + 1)

        PathS = RootS & Mid(PathB, Len(RootB) + 1)

        FolderB = New IO.DirectoryInfo(PathB)

            For Each FileInfoB In FolderB.GetFiles
                If FileIO.FileSystem.FileExists(PathA & FileInfoB.Name) Then
                    'File exists; check filedate
                    FileInfoA = My.Computer.FileSystem.GetFileInfo(PathA & FileInfoB.Name)
                    If FileInfoB.LastWriteTime < FileInfoA.LastWriteTime Then
                        'File was modified, copy this file to 'Snapshot'
                    End If
                    'File does not exist; it was deleted, copy this file to 'Snapshot'
                End If
            For Each myFolder As System.IO.DirectoryInfo In FolderB.GetDirectories
                RoboSnapshot(myFolder.FullName & "\")
        Catch ex As Exception
        End Try

    End Sub

    Private Sub SnapshotMyFile(ByVal MyFile As System.IO.FileInfo)
        'Check for modification date older than 30 days
        'ELSE it's not old enough to worry about, we already have recent incrementals backup(s)
        If DateDiff(DateInterval.Day, MyFile.LastWriteTime, Now) > 30 Then
            My.Computer.FileSystem.CopyFile(MyFile.FullName, PathS & MyFile.Name)
        End If
    End Sub
End Class

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Lee TumblesonAuthor Commented:
I had to write a program to do what I wanted, nothing existed.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server OS

From novice to tech pro — start learning today.