Robocopy old/deleted files

I have a production File Server A.  I do a Robocopy /MIR to File Server B every night as a backup.

I'd like to improve upon this by copying all of the files on Server B that Robocopy /MIR will either delete or replace to a 3rd folder as a backup.  This way I'll have a snapshot of the previous state of all of the files that were changed that day.

Does anyone have any ideas for how to perform this operation?

Thanks.
Lee TumblesonAsked:
Who is Participating?
 
Lee TumblesonAuthor Commented:
I wrote a simple VB.net program to do this.  After I run this, then I run the Robocopy /MIR to make ServerB the same as ServerA.  All servers/shares are hard-coded.

Public Class Form1

    Dim RootA As String = "\\ServerA\Share\"
    Dim RootB As String = "\\ServerB\Share\"

    'Snapshot
    Dim RootS As String = "\\ServerB\Share\Snapshots\" & Format(Now(), "yy") & Format(Now(), "MM") & Format(Now(), "dd") & "\"
    Dim PathS As String

    Public Sub New()

        ' This call is required by the designer.
        InitializeComponent()

        ' Add any initialization after the InitializeComponent() call.

        RoboSnapshot(RootB & "Folder1\")
        RoboSnapshot(RootB & "Folder2\")

    End Sub

    Private Sub RoboSnapshot(ByVal PathB As String)
        Dim FolderB As System.IO.DirectoryInfo
        Dim FileInfoA As System.IO.FileInfo
        Dim FileInfoB As System.IO.FileInfo

        Dim PathA As String = RootA & Mid(PathB, Len(RootB) + 1)

        PathS = RootS & Mid(PathB, Len(RootB) + 1)

        FolderB = New IO.DirectoryInfo(PathB)

        Try
            For Each FileInfoB In FolderB.GetFiles
                If FileIO.FileSystem.FileExists(PathA & FileInfoB.Name) Then
                    'File exists; check filedate
                    FileInfoA = My.Computer.FileSystem.GetFileInfo(PathA & FileInfoB.Name)
                    If FileInfoB.LastWriteTime < FileInfoA.LastWriteTime Then
                        'File was modified, copy this file to 'Snapshot'
                        SnapshotMyFile(FileInfoB)
                    End If
                Else
                    'File does not exist; it was deleted, copy this file to 'Snapshot'
                    SnapshotMyFile(FileInfoB)
                End If
            Next
            For Each myFolder As System.IO.DirectoryInfo In FolderB.GetDirectories
                RoboSnapshot(myFolder.FullName & "\")
            Next
        Catch ex As Exception
        End Try

    End Sub

    Private Sub SnapshotMyFile(ByVal MyFile As System.IO.FileInfo)
        'Check for modification date older than 30 days
        'ELSE it's not old enough to worry about, we already have recent incrementals backup(s)
        If DateDiff(DateInterval.Day, MyFile.LastWriteTime, Now) > 30 Then
            My.Computer.FileSystem.CopyFile(MyFile.FullName, PathS & MyFile.Name)
        End If
    End Sub
End Class
0
 
NVITCommented:
Not quite following you. Are you saying you want the same exact copy as A and B?
0
 
bigeven2002Commented:
Hello,
I don't think robocopy can do that in one script.  Rather, the best option here would probably be performing another instance of robocopy from Server B to say, Server C which you call 3rd folder.

Since Server B is backup, it won't experience activity during the day from Server A or any other server or workstation correct?  If so, then during the day, have Server B run an instance of robocopy that will copy the prior night's backup to Server C, and that would be considered your snapshot.

If Server C is meant to hold all data that was deleted indefinitely instead of just from prior night, then you could just use robocoby from Server B with /XO instead of /MIR.  This would just copy the newer files over to Server C.  Try that out to see if it works for you.

If Server C was just to be a placeholder for the changed files from the day, then you'll still need to use the /XO parameter to copy over just the newer files.  It will still need to have a full backup otherwise it won't know what is considered older files.  So for first time use /MIR to make the full backup from Server B to Server C.  Then from that point on, use /XO instead of /MIR and then write to a log file to see what was copied.

robocopy "C:\source" "D:\path" /XO /FP /LOG+C:\Logpath\copy.log

Open in new window

0
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
arnoldCommented:
What are the OSes involved?

AD environment, dfs with dfs-r and using a standard backup process.  The items you are setting up are a hot backup, but as was pointed out going on this route, you might continue adding subsequent server to extend "duration of retention".

An option you might want to consider if not already is a document management system.  There commercial as well as open source ones dealing with controlling and recording access as well as maintain versions when changes are made, etc.
0
 
kitmaxCommented:
I had a script to do what you ask a while back.  Will look at work in the morning...
0
 
Lionel MMSmall Business IT ConsultantCommented:
You can use these parameters to play around with what you want to copy and copy only those files that a certain number of days old or new
/MAXAGE:n :: MAXimum file AGE - exclude files older than n days/date.
/MINAGE:n :: MINimum file AGE - exclude files newer than n days/date.
/MAXLAD:n :: MAXimum Last Access Date - exclude files unused since n.
/MINLAD:n :: MINimum Last Access Date - exclude files used since n.
(If n < 1900 then n = n days, else n = YYYYMMDD date).
example
robocopy C:\Data D:\Data /S /MAXAGE:1
will only copy files that are one day or less old (24 hours or less)
0
 
Lee TumblesonAuthor Commented:
Thanks for the feedback, but none of this is what I'm looking for.  This is a Windows Server 2012 and 2012R2 environment.

I'm looking to copy files that will change or be deleted.  I don't want a snapshot of all 3TB of data, just the previous state of the 30GB that changed that day.

It could be that someone deletes a file that has a filedate from 2 years ago.  I want to make a copy of that so that I don't have to go back to 2-years-ago backups to retrieve it next week when they need it back.

To further complicate matters, I would only make a snapshot of the previous version of the file if it had a date of more than a month ago.  If a file is being changed frequently, I don't need a snapshot, because I am also performing incremental backups every day, so that file would already have a recent backup.  This should reduce the 30GB that I mentioned to less than 2GB.

File Server C would contain dated folders with these snapshots.

I already create daily backup folders using a robocopy like this:

robocopy [source] [destination]\D%date:~-2,2%%date:~-10,2%%date:~-7,2%\ /s /MAXAGE:2 /copy:datso /FFT

This script creates a folder starting with D (daily) and then the current date.  It only copies files that have a modified date within the past 2 days.

What I want to do is create an S (snapshot) folder, with the current date, and only copy files that are at least a month old and will either be overwritten by a newer file or deleted.
0
 
arnoldCommented:
one option is that you will need to run robocopy with the -L flag and then extract/copy the line and file reflected as deleted. will be copied to server C
then the B is synchronized with A.
C will only be a repository for deleted items is that the correct understanding.

robocopy \\servera\src \\serverb\src /MIR /LOG:deleted.txt  /L
then you would need to process the deleted.txt file keeping track of the directory it is in and then looking for the *EXTRA file line at which point $directory\filename is copied to Server C.

Or you can run the opposite comparison with B as the source with A as the destination (do not use /MIR)
here you will be looking for the New File notice.

Replication and its backup does not have negative function.

A document management system would likely be your best option as it gives the admin in some cases to mark the file inaccessible/deleted while it is still there.

to achieve your goal in a single process, you would need  to write the comparator/replacement to robocopy.

myrobocopy(powershell/vbscript/or compiled ) src destination deleted_from_src_repository
0
 
Lee TumblesonAuthor Commented:
It might just be easier to write my own /MIR in Visual Studio.  Step through all of the files on both servers.  But, before I initiate the 'copy from A to B', I'll 'copy from B to C'.

This would perform the whole process in one pass.
0
 
arnoldCommented:
You could, you have to take several things into consideration:
You can start down this path and ..... never get off the road of continually updating/adding to your application.
filea changed on A. do you need to prior B version copied on the C?
dealing with filea is mistakenly modified.
the test for deleted items, will not be triggered such that filea will be updated to server B.
then you are back where you started when the person realizes that they deleted an entry and need the file back.
if files are text based, windiff is a tool that could extract the differences in the files.
0
 
Lee TumblesonAuthor Commented:
I didn't quite follow what you said.  I'm looking to copy, from B to C, files that 'will be' modified or deleted when A is mirrored to B.

This covers the 'mistakenly modified' item that you mentioned.

Not sure what I need to update/add.  I'm thinking that I can write this in less than a day.
0
 
arnoldCommented:
So any file that will be copied from A to B will have the B version copied to C and any file that will be deleted from B because it no longer exists on A will be copied to C
Your program will still need to have two passes. One compares B to A to find the files on A that were deleted copy file from B to C. Deleting the file on A.
The second step deals with a pass comparing files from A to B to determine if a newer/modified file exists on A at which point the file from B will be copied to C and then the file from A will be copied to B.
Your next step will be how you deal with file handling from B to C. Will you compare the file you are copying from B to C whether it was modified or deleted?

person A accesses server A file A on Monday. makes an alterations. B copied to C.
On Wednesday, person A is once again accessing Server A file A, and makes another alterations. B copied to C.
On Thursday, Person A was asked a question, accesses Server A file A, and realizes the wrong section of the data was deleted Monday.
Your current setup can only go back to the changes made on Wednesday.

Or you can use the combinations which deals with daily changes (deleted/modified) on server C.

In 2012/R2 using DFS and share snapshots depending on how much space you can allocate on a separate drive/resource to store the snapshots of each member of the DFS.
DFS can be configured in a failover primary/secondary for server A/ServerB setup with replication bi-directional.
0
 
Lee TumblesonAuthor Commented:
You missed the part in my previous comment where I do this every night:

robocopy [ServerA]  [ServerC]\D%date:~-2,2%%date:~-10,2%%date:~-7,2%\ /s /MAXAGE:2 /copy:datso /FFT

I'm also utilizing ShadowCopy on these server drives.  I'm doing 'snapshots' every 4 hours (3/day, just during the workday) on A (I found a registry switch that allows up to 512 snapshots, which gives me 3/day snapshots for 170 days).  On B, I'm doing the snapshot once a day.  This gives me a year and half of daily snapshots.

I also do periodic full backups of the 3TB of data, but I'd like to reduce the need to go to those backups.  If I can have these 'snapshot' backups, along with the above MAXAGE backups in folders (I'm storing these on a DROBO, I'm estimating that I can maintain these daily backups for 3-4 years on the DROBO), then I can recover files very easily.
0
 
arnoldCommented:
Yes, the range of dates/events to handle is the issue.
The difficulty, how do you locate fileA that has been deleted from ServerA or modified erroneously?
The person coming to you with the issue of the missing file might not be the person who deleted/changed it nor has the person recently accessed the file to help you narrow down when the change occurred for you to locate the file.  Now in addition to what you are already doing, you need a cataloger of deleted/modified files on server C so that when anyone comes saying , I am missing fileA, you can go through the catalog and give them the option to see the file and decide which of the multiple versions is the correct one to be restored.

you could/should incorporate DB backend where you can record the copy out to C/days .....
The more analysis you do, the more features you will need to add to your application. then the backup/consideration of the backend DB ......
0
 
kitmaxCommented:
We used this to only keep the last 30days in a temporary folder.  (many users dumping stuff in there)
Could be modified to archive instead of delete at the end?


@echo off

SET Clean=G:\Temp
SET Stale=30
SET Staging=G:\share\temp


rem Make an empty location to move old files into
rem =============================================
rmdir %Staging% /S /Q
mkdir %Staging%

rem Move old files into staging area
rem ================================
robocopy %Clean% %Staging% /E /MOVE /MINLAD:%Stale% /LOG:%Staging%\cleanup.txt /TEE

rem Delete staging area
rem ===================
rmdir %Staging% /S /Q

cls
0
 
Lee TumblesonAuthor Commented:
I had to write a program to do what I wanted, nothing existed.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.