Solved

What is a good way to retrieve files from large archives overseas and cache them to be used locally?

Posted on 2014-07-28
10
274 Views
Last Modified: 2014-08-11
Hello,

I would like some recommendations a solution I am trying to implement for a company that has two offices: one in Holland and one in the US. The office in the US is new and it still needs to access a network share that has a large archive with many individual files that is located in Holland. The problem is that accessing these files (they're using some program that opens architectural drawings) and making any changes can be slow because of the distance barrier. To solve this I thought about doing a DFS share and creating some shared virtual folders that would sync overnight from the US share to the Holland share but that idea lost steam after realizing the archives are big and the users don't have a preference for a smaller set of data. So what I am looking for is to find out if there are any solutions out there that do this sort of file caching? Kind of like a "fetch as you use" type of deal.
0
Comment
Question by:Samir Saber
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
10 Comments
 
LVL 4

Accepted Solution

by:
Mitchell Milligan earned 500 total points
ID: 40224215
You may not want to completely abandon the DFS share idea just yet.  You may want to seed the DFS first, which will reduce that synchronization time and bandwidth utilization.  You may want to have the data copied to an external hard drive and sent to the other location, then you can set up the DFS and the image will be pre-seeded.

DFS Replication supports copying files to a replication group member before the initial replication. This "prestaging" can dramatically reduce the amount of data replicated during the initial replication.

The initial replication does not need to replicate contents when files differ only by real attributes or time stamps. A real attribute is an attribute that can be set by the Win32 function SetFileAttributes. For more information, see SetFileAttributes Function in the MSDN library (http://go.microsoft.com/fwlink/?LinkId=182269). If two files differ by other attributes, such as compression, then the contents of the file are replicated.

To prestage a replication group member, copy the files to the appropriate folder on the destination server(s), create the replication group, and then choose a primary member. Choose the member that has the most up-to-date files that you want to replicate because the primary member's content is considered "authoritative." This means that during initial replication, the primary member's files will always overwrite other versions of the files on other members of the replication group.

For information about pre-seeding and cloning the DFSR database, see DFS Replication Initial Sync in Windows Server 2012 R2: Attack of the Clones.

For more information about the initial replication, see Create a Replication Group.

Source: http://technet.microsoft.com/en-us/library/cc773238%28v=ws.10%29.aspx#BKMK_079
0
 

Author Comment

by:Samir Saber
ID: 40224292
Thanks for the great suggestion Mitchell and you are right - I haven't crossed the DFS idea out yet. It will come down to what is the best option financially because I am not sure yet how big the archive is.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40224307
If you can find out, you may be able to get an external hard drive, copy the data to it, then send it in the mail to your other site.
0
When ransomware hits your clients, what do you do?

MSPs: Endpoint security isn’t enough to prevent ransomware.
As the impact and severity of crypto ransomware attacks has grown, Webroot has fought back, not just by building a next-gen endpoint solution capable of preventing ransomware attacks but also by being a thought leader.

 

Author Comment

by:Samir Saber
ID: 40224396
Will do. I believe the archive is at least 100 TBs.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40224545
Ok, so we are talking between 10 and 100 external hard drives?
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40224548
That's a lot more than i had thought you would say.  Wow, that's a lot of data.
0
 

Author Comment

by:Samir Saber
ID: 40229010
Just checked with their IT guys and they said it's 3.8TB instead (I was also surprised it was that big.. that's what happens when common users tell me what they think of the size). I will most likely go the DFS route.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40229034
Sounds great, I would still look into seeding it if possible before starting the DFS.  That will save significant time and bandwidth I think.  I am not sure how long or how much it would cost to ship an external hard drive between your sites, but I can only imaging how long it would take to synchronize 3.8 TB.  Good luck, and if this has answered your question, please be sure to close your question.
0
 

Author Comment

by:Samir Saber
ID: 40253118
Thanks again Mitchell.
0
 
LVL 4

Expert Comment

by:Mitchell Milligan
ID: 40253145
No problem, good luck!
0

Featured Post

Transaction Monitoring Vs. Real User Monitoring

Synthetic Transaction Monitoring Vs. Real User Monitoring: When To Use Each Approach? In this article, we will discuss two major monitoring approaches: Synthetic Transaction and Real User Monitoring.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

New Relic: Our company recently started researching several products to figure out what were the best ways for us to increase our web page speed and to quickly identify performance problems that we may be having. One of the products we evaluated wa…
New Relic recently released its Synthetics product that allows for the creation of performance monitors that periodically test a site's performance. If you wish to test an interactive workflow New Relic employs Selenium WebDriverJS to run those test…
The purpose of this video is to demonstrate how to set up the WordPress backend so that each page automatically generates a Mailchimp signup form in the sidebar. This will be demonstrated using a Windows 8 PC. Tools Used are Photoshop, Awesome…
Wufoo.com provides powerful tools for surveying targeted groups, and utilizing data from completed surveys to find trends, discover areas of demand or customer expectation, and make business decisions on products or services.

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question