Solved

What is a good way to retrieve files from large archives overseas and cache them to be used locally?

Posted on 2014-07-28
10
271 Views
Last Modified: 2014-08-11
Hello,

I would like some recommendations a solution I am trying to implement for a company that has two offices: one in Holland and one in the US. The office in the US is new and it still needs to access a network share that has a large archive with many individual files that is located in Holland. The problem is that accessing these files (they're using some program that opens architectural drawings) and making any changes can be slow because of the distance barrier. To solve this I thought about doing a DFS share and creating some shared virtual folders that would sync overnight from the US share to the Holland share but that idea lost steam after realizing the archives are big and the users don't have a preference for a smaller set of data. So what I am looking for is to find out if there are any solutions out there that do this sort of file caching? Kind of like a "fetch as you use" type of deal.
0
Comment
Question by:Samir Saber
  • 6
  • 4
10 Comments
 
LVL 4

Accepted Solution

by:
Mitchell Milligan earned 500 total points
ID: 40224215
You may not want to completely abandon the DFS share idea just yet.  You may want to seed the DFS first, which will reduce that synchronization time and bandwidth utilization.  You may want to have the data copied to an external hard drive and sent to the other location, then you can set up the DFS and the image will be pre-seeded.

DFS Replication supports copying files to a replication group member before the initial replication. This "prestaging" can dramatically reduce the amount of data replicated during the initial replication.

The initial replication does not need to replicate contents when files differ only by real attributes or time stamps. A real attribute is an attribute that can be set by the Win32 function SetFileAttributes. For more information, see SetFileAttributes Function in the MSDN library (http://go.microsoft.com/fwlink/?LinkId=182269). If two files differ by other attributes, such as compression, then the contents of the file are replicated.

To prestage a replication group member, copy the files to the appropriate folder on the destination server(s), create the replication group, and then choose a primary member. Choose the member that has the most up-to-date files that you want to replicate because the primary member's content is considered "authoritative." This means that during initial replication, the primary member's files will always overwrite other versions of the files on other members of the replication group.

For information about pre-seeding and cloning the DFSR database, see DFS Replication Initial Sync in Windows Server 2012 R2: Attack of the Clones.

For more information about the initial replication, see Create a Replication Group.

Source: http://technet.microsoft.com/en-us/library/cc773238%28v=ws.10%29.aspx#BKMK_079
0
 

Author Comment

by:Samir Saber
ID: 40224292
Thanks for the great suggestion Mitchell and you are right - I haven't crossed the DFS idea out yet. It will come down to what is the best option financially because I am not sure yet how big the archive is.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40224307
If you can find out, you may be able to get an external hard drive, copy the data to it, then send it in the mail to your other site.
0
Is Your AD Toolbox Looking More Like a Toybox?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

 

Author Comment

by:Samir Saber
ID: 40224396
Will do. I believe the archive is at least 100 TBs.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40224545
Ok, so we are talking between 10 and 100 external hard drives?
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40224548
That's a lot more than i had thought you would say.  Wow, that's a lot of data.
0
 

Author Comment

by:Samir Saber
ID: 40229010
Just checked with their IT guys and they said it's 3.8TB instead (I was also surprised it was that big.. that's what happens when common users tell me what they think of the size). I will most likely go the DFS route.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40229034
Sounds great, I would still look into seeding it if possible before starting the DFS.  That will save significant time and bandwidth I think.  I am not sure how long or how much it would cost to ship an external hard drive between your sites, but I can only imaging how long it would take to synchronize 3.8 TB.  Good luck, and if this has answered your question, please be sure to close your question.
0
 

Author Comment

by:Samir Saber
ID: 40253118
Thanks again Mitchell.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40253145
No problem, good luck!
0

Featured Post

Are your AD admin tools letting you down?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developer tools in browsers have been around for a while, yet they are still heavily underused by developers. Developers still fix html or CSS then refresh page to see effect, or they put alert or debugger in JavaScript and then try again and again …
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Wufoo.com provides powerful tools for surveying targeted groups, and utilizing data from completed surveys to find trends, discover areas of demand or customer expectation, and make business decisions on products or services.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question