Data Archive Strategies

Posted on 2011-10-18
Last Modified: 2012-06-27
At our company, we have a shared location (K:\) for all of our archived data....basically, jobs that are complete, etc.  This is separate from our project directories (M:\).  What we've typically done in the past is provide the Project Manager a list of folders so that they indicate which jobs can be archived & then the folders are moved to the K:\ drive.  If users need to view or access these files once they're on the K:\ drive, they're supposed to move the folder back to the M:\...especially since the XREF paths in our CAD files won't be correct & won't display properly until moved back to the M:\ drive.  Of course, for other files (ie-.doc, .xls, .pdf), this isn't an problem & they can be viewed without issue when left on the K:\.

The problem is, we have some users that copy the project folder back to the M:\ instead of moving it.  A few months later, when I go to archive completed projects, I don't know if the version that was copied to the M:\ is now different then the version left on the K:\...and because I don't want to loose data, I end up moving a second copy back to the K:\.  Hunting down the user & asking them doesn't always help since it could be months since they touched it & they simply don't remember.

Not only does keeping a second copy of projects eat up a lot more storage space, but it also creates confusion for the next user that needs to look for info in the archived project.

Besides simply reminding our users about the proper procedures for moving archived jobs back to the active jobs folder, is there a way to control this with permissions?  Or at least compare supposed duplicated files between the K:\ & M:\ so that we don't have 2 possibly identical copies of the same archived job on the K:\?
Question by:blugirl
    LVL 17

    Accepted Solution

    There are some useful copy utilities that can merge the folder from M: to K: - for example, Robocopy.  If a user requested a project to be made available to him, what you could do is give him the folder, and when he's finished with it, merge the changes which were made to that folder into the original folder.  An example usage of this program would be:

    Robocopy "M:\Projects\543 - Frozen Taps" "K:\Archive\543 - Frozen Taps" /E /MOVE

    Open in new window

    The above would then merge the data in from the M: drive to the K: drive, and will only copy data which has changed.  It is safe too, because if the user has deleted some documents either deliberately or by accident, those files are not deleted in the target, and are therefore retained.

    You could also help automate this process a bit by creating a "To Archive" folder in your M:\ area, and then running an hourly robocopy synchronise of the To Archive folder to your Archives.  For example, you could do:

    Robocopy "M:\ToArchive" "K:\Archive" /E /MOVE

    Open in new window

    When users move project folders into the To Archive folder, within an hour, that would be resynched with the K: drive, and only overwriting documents which have changed.

    All of this does not help you if you want to retain previous versions.  In an ideal world, documents would be labelled by their name and version number, ensuring that previous versions are not overwritten.  This isn't always guaranteed, and there are two methods we can use to combat this:

    1) Implement Volume Shadow Services on the Archive.  Any documents which are replaced can be retrieved simply by going to the "Previous Versions" tab of the document's properties.

    2) Implement some kind of backup which backs up changes only (thus saving space on the backup volumes), and retains the previous versions.  Microsoft's Data Protection Manager will do a very good job at this.

    The best solution I can advise however is to implement a document management solution like Sharepoint - so that you can get users to check documents out, and you can enable document history so that you can go back in time before changes were made.

    Author Comment

    Hi LesterClayton,

    Thanks for the suggestion.  But I wonder if it would work for us?  When files are copied from the K:\ to the M:\, the modified date of the file & folders change.  Isn't this the only way the copy, or even robocopy commands would determine which files are the latest & greatest?

    Additionally, the "M:\ToArchive" folder defeats the purpose of bringing the project back from the K:\ since the XREF paths in the CAD files would still be off & unable to locate the referenced files' locations.
    LVL 17

    Assisted Solution

    The modified date of the files and folders should not change - when doing a standard copy, the creation date and modified date are as the original file - unless you're using a non Windows API to copy the files.

    Directories unfortunately do not retain the orginal directory date, but in the big scheme of things that isn't too much of an issue.  When merging it back in, the creation date of the folder in K:\ is still what it was before you copied it back to M:\, and will still be unchanged when you sync it back to K:\

    Robocopy automatically ignores all files which have the same "modified" date stamp as the target file, or is the same size.  Robocopy is a great tool, fully supported by Microsoft and also supports the copying of file attributes and permissions.  It defaults to copying only changed stuff, if the same items exist in the source and target.

    Author Comment

    Hey!  I didn't realize that about the files retaining their date even though the folders don't!  Thanks for clarifying!  We'll give it a shot! ^_^

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Organizations create, modify, and maintain huge amounts of data to help their businesses earn money and generally function.  Typically every network user within an organization has a bit of disk space to store in process items and personal files.   …
    The following article will describe how to add/edit a dimension style through AutoCAD VBA. AutoCAD VBA has its quirks and when it comes to dimensions and controlling how they look through VBA.  This is where AutoCAD can be vividly confusing. The…
    how to add IIS SMTP to handle application/Scanner relays into office 365.
    This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor ( If you're looking for how to monitor bandwidth using netflow or packet s…

    760 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    9 Experts available now in Live!

    Get 1:1 Help Now