Solved

Archiving plan over 10mb line

Posted on 2015-01-06
14
154 Views
Last Modified: 2015-04-19
Hi,

I have a 10mb line from our regional office to the HQ.
I wanted to plan an archive design strategy to archive something which is more than 25GB into HQ's archive server.
For all our HQ users (100mb line), we have designed the archive for each department and a mapped the network drive assigned using GPO preference for the each department users to connect using a drive.
Because we have only 10mb link, I am not sure just archiving the files to HQ going to help the network utilization of the users in the regional office.
But before thinking about non office hours, I really want to know if anyone done anything similar to our infrastructure, any practical help would be appreciated.

I think maybe running a Robocopy script or sql job may achieve what I am trying to do, not sure, please advise.

Thanks
la
0
Comment
Question by:la-tempestad
  • 7
  • 7
14 Comments
 
LVL 57

Expert Comment

by:giltjr
ID: 40533546
First question is what type of data; text, binary, or mixed?

If you compress the files how much does it reduce the amount of data?

On average how many of the files change daily?
0
 
LVL 1

Author Comment

by:la-tempestad
ID: 40535340
Hi,

Majority of the files are .doc,.docx,.txt,.xlsx,.msg and .pdf
Its one of the users home profile and he is using it as an archive for all the other users shared work files.
Not sure how much average file changes at the moment?

Thanks
la
0
 
LVL 57

Expert Comment

by:giltjr
ID: 40536392
Most of those file can be compressed easily and get over 80% compression, which means your 25MB of data shrinks down to 5 MB.

On a 10Mb WAN link, depending on the latency, during off hours you should be able to get 500KB second.  So even the full 25MB of data would take 50 seconds, less than one minute.  If the files can be compressed down to 5 MB, then 10 seconds or so.
0
 
LVL 1

Author Comment

by:la-tempestad
ID: 40581215
I appreciate your reply, but It's not 25MB,its 25GB.

I have done a test run on a test environment with same 10MB line and it took 20Hrs to finish the whole files using Robocopy. So I don't think it's an ideal solution at all as we disrupt the normal services.

thanks
la
0
 
LVL 57

Expert Comment

by:giltjr
ID: 40581396
Oops sorry about that on the MB vs GB.  However, did you zip the files?  If the 25GB zip's down to say 5GB.

If it took you 10 hours to do the full 25 GB that is about 750Kbps of throughput , so 5GB at the same rate would take about 2 hours plus the time to zip the files.  Since this is for archive purposes you don't need to unzip the files unless you need to recover some of them.  Even then you only need to unzip the files/folders you need to recover.

The only way to reduce the time is to increase bandwidth or decrease the amount of data.  Robocopy vs. Windows copy reduces some of the overhead that Windows copy has, but only slightly.
0
 
LVL 1

Author Comment

by:la-tempestad
ID: 40603184
No I have not zipped the files as it is something which the dept wants to run as a scheduled job, so just zipping the files/folders doesn't get us the required result.

Thanks
la
0
 
LVL 57

Expert Comment

by:giltjr
ID: 40603826
Actually scheduling make it even easier to zip.  Schedule a cmd file that zips the folders you want to process, copies/ftps the zip file to the remote site.  You can include a date in the zip'ed file name so you know when the file was created.
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 1

Author Comment

by:la-tempestad
ID: 40605198
Sorry the whole point of making this archive change is to have an archive location for the end users(from regional office) to see it as a mapped network drive, just making it as .zip doesn't help the users to work with the file.
I hope this helps a bit more about my situation.

Thanks for the inputs,
0
 
LVL 57

Expert Comment

by:giltjr
ID: 40605274
An archive location is normally a "backup" that users do not directly interact with.    

If the users need to interact with the files at the remote location then you just add one more step to the process and that is at the remote location you trigger a task that unzips the file.

What you are trying to do is not new, this type of process has been going on since computers starting talking over networks.

There are only 3 options to get the data from location A to location B in timely manner:

1) Get enough bandwidth between the two locations to meet your requirements.
2) Reduce the amount of data you need to move by means of compression.
3) A combination of #1 and #2.

There are NO other ways.  There is no magic bullet that will allow you to get 25GB of data over a 10mbps link any faster that a 10mbps link will allow.
0
 
LVL 1

Author Comment

by:la-tempestad
ID: 40605332
Yeah I totally understand what you are suggesting, but zipping and unzipping the files on a scheduled basis using cmd's always a tricky thing , I used it in the past using Robocopy and every now and then its failed because of the low b/w.
Also increasing the b/w is not an option at the moment, any help with the cmd's which I can use to copy/mov files of that volume from one site to another?

Thanks.
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 250 total points
ID: 40605392
I'm not sure why you think a zip command is tricky.

Again, there is no magic it is more bandwidth, reduced the volume of data, or a combination.
However, let me ask you some ore questions.

Again, when I hear "archive copy", I think backup.  Meaning a copy of a file that is NOT directly accessed by users.  That if that copy is needed, it is copied back to a "live" location and then accessed from there.

You you eluded to is that this is not a archive copy, but  a copy that is put in a central place for users to access.

How are they accessing it?  Meaning, is the "local" copy edited/updated, and then once a day "archived" for other to access in a read only mode?

Must you keep multiple archive copies?    Meaning if you create a archive copy  today and another one tomorrow do you have to have both copies saved?

If you don't have to keep multiple copies, does the archive have to be ONLY once day?      Or can it be a "live" replication using RSYCN?  RSYCN will copy the local file to a remote location every time the local copy is updated.  You could do RSYCN over a SSH tunnel, and with SSH you can enable compression.  I'm not sure what the compression ratio is with SSH.  

Maybe if I understand what you are really trying to accomplish I can come up with a solution.
0
 
LVL 1

Accepted Solution

by:
la-tempestad earned 0 total points
ID: 40650686
hi,

sorry for the late reply as I was away on a holiday.
The scenario is not complicated at all.
there is a network mapped to user's computer(Z:) and every week (or if possible everyday) we need to copy the files from the live location to an archive location which has been mapped to Z: drive update with the files/folders from the live location, because of the volume of the files(25GB), I think it is going to be not possible to copy everything on  a daily basis.

So what are the solutions for a weekly update of this archive location with volume size of 25GB on a 10MB line?

Worst scenario,I am going to ask the users to update the archive location as they needed,once I have copied all the 25GB files over on  a weekend using robocopy ?

Regards
la
0
 
LVL 57

Expert Comment

by:giltjr
ID: 40650842
I still need some clarification.

1) This is more of a weekly backup, not daily.
2) The users will NOT normally use the files on the Z drive.
3) Do you need to keep the prior backups/archives?
4) If you need keep prior backups, how many do you need to keep?

I personally would run a test on a weekend.  Zip the 25GB up to see how big of a zip file it is.  Then transfer the zip file using FTP.  FTP is way more efficient than Windows copy and I think more efficient than robocopy.
0
 
LVL 1

Author Closing Comment

by:la-tempestad
ID: 40731911
I have created a new archive location at the HQ and robocopied all the files to this new location and asked the users to use this as their live archive location for the future and it was not an ideal situation but users are satisfied with this.
0

Featured Post

Give your grad a cloud of their own!

With up to 8TB of storage, give your favorite graduate their own personal cloud to centralize all their photos, videos and music in one safe place. They can save, sync and share all their stuff, and automatic photo backup helps free up space on their smartphone and tablet.

Join & Write a Comment

Don’t let your business fall victim to the coming apocalypse – use our Survival Guide for the Fax Apocalypse to identify the risks and signs of zombie fax activities at your business.
If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now