Better faster way to backup data across a vpn?

I have a vpn established with Sonicwalls (Tz100) to a remote site.
At the remote site they have a cable connection that runs at 1.5 up and down.
At my main office we have a Windstream T1 that runs at 1.5 up and down.
My backup takes around 8 hours to complete using robocopy with the mirror function which should only copy changed data.
I also am testing syncback pro (set to use a group that includes the paths to be backed up and the mirror function)  to see if I can get the data to come across the pipe faster.
I have tried initializing the backup from the remote site and also the main office but it doesn't seem to be any faster.
The majority of files that are backed up are image files in the tiff format.

The context of my robocopy script is below.

for /f "tokens=1-5 delims=/ " %%d in ("%date%") do rename "v:\ftov.log" %%e-%%f-%%g.log
robocopy F:\smi\index v:\smi\index /MIR /R:2 /W:20 /LOG:c:\backup\ftov.log
robocopy F:\imaging v:\imaging /MIR /R:2 /W:20 /LOG+:c:\backup\ftov.log
robocopy F:\disk1a v:\disk1a /MIR /R:2 /W:20 /LOG+:c:\backup\ftov.log
robocopy F:\disk2a v:\disk2a /MIR /R:2 /W:20 /LOG+:c:\backup\ftov.log
robocopy F:\security v:\security /MIR /R:2 /W:20 /LOG+:c:\backup\ftov.log

I also have attached my robocopy logfile.

Is there a way to determine the amount (size) of data   that is backed up by using robocopy or some other program?
Is there a way to speed up the backup?

Management wants to add other backups but since this one takes so long they are concerned about other backups taking longer.

They are comparing this backup to a bank T1 that supposedly transmitted 4000 images of checks in 15 minutes.

I have already made the argument that we need a bigger pipe at the main office to handle traffic.


Thanks in advance


ftov.log
kallatechAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

SemperWiFiCommented:
Wish I had some good news for you here, but I don't.

You might point out to your management that an image file isn't often the same as another. An image of a check from a bank - low res, small, small, small. So the comparison isn't valid. In order for a benchmark to have validity then there must be a table st for equal comparison. When talking about data transfer then it needs to be done with data numbers in terms of size not amount. I can transfer 1,000,000 text documents in no time, why can't you transfer the same amount of log files? Well because the log files are 400 x times the size.

The issue you have, as it seems you already know, is a bandwidth limitation. There is a way to speed it up, up the bandwidth. Otherwise you are simply trying to push more through the same hole.

A way to explain it to them: If you are attempting to empty a pool of water through a a 1" pipe then you are limited to the amount of water that can flow through a 1" pipe at one time. The method in which you choose to use to deliver the water to the pipe doesn't matter. But if you attach a 4" pipe to the same pool then the water will inherently flow out faster.

This is the main reason why companies get bigger connections.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
theonlyallanCommented:
8 Hours seems very long. How much data are you transferring?

In theory a 1.5Mbit connection can only transmitt: 675Mb/hour
It may be slower because of the overhead required in Windows File Sharing as well.

Can you use backup programs that can compress?
I use Acronis True Image Enterprise. It compresses the data and transmits to a local server using ftp.
kallatechAuthor Commented:
So there is no tweaking that I can do to any of the methods I use to speed it up?
Would zipping it make it faster?
What impact would a 3 meg up and down at our main office do for the backup if we initiated the backup from our main office (download) rather than intializing the backup from the remote location (upload)?

How can I tell the size of the files that are coming across the vpn?
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

BigSchmuhCommented:
SemperFiFi is correct : a T1 1.5Mbs line can not send (or receive) more than a theorical 192KB per second (=675MB per hour = 15.8GB per day)

a T3 line is 28x times a T1 line...

Some compression software may help to downsize the volume.
I would although consider using Win 2008 R2 DFS because it embeds both the compression and replication mechanisms.
theonlyallanCommented:
It sounds like you are using a very simple program to backup..
The verification process across the 2 drives is eating a lot of overhead.
The only way you can tell the transmit size is:
Check the Connection status of your VPN before you leave the office
and check is again in the morning after the backup. - calculate the difference.

Zipping will reduce the file size.
A 3 meg connection will cut your 8 hours to 4 hours... But that's because your current backup method is not efficient. And also, it makes no difference which side you initiate the transfer from.
If it takes  8 hours on a T1 connection, i'm guessing you are transferring 5Gig of data..
kallatechAuthor Commented:
What do I need to do to make this backup efficient?
theonlyallanCommented:
robocopy is an amature program.
there are commercial programs available.  It depends how your network is setup. I need to know a little more like:  
-if data will be modified on both sides?
or is your data only being copied for backup purposes?

as BigSchmuh suggested, DFS replication would be a good idea, as both your shares would be consistently synced.
nappy_dThere are a 1000 ways to skin the technology cat.Commented:
No good news here but, you make want to look at Cisco WAN accellerators. Pricey but can help.
kallatechAuthor Commented:
The data is copied from our offsite location and served out to subscribers on a website at our main office.
Our main office is also where the data is backed up.
The server at the offsite facility  is a 2003 standard server and the server at the main office is a 2008 standard server.

nappy_dThere are a 1000 ways to skin the technology cat.Commented:
Here is the product I mentioned earlier that would improve your robocopy of data http://www.cisco.com/en/US/products/ps6474/index.html
kallatechAuthor Commented:
Would the folder sync function in  syncback pro work better than waiting until after hours to do a backup?
kallatechAuthor Commented:
Thanks to everyone who responded
The backup is taking alot of bandwidth.
When it runs my amount of bandwidth spikes around 1440.
So we definitely need to boost our bandwidth at our main office.

I also started using syncback pro with the fast backup option which has improved my backup time.

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Disaster Recovery

From novice to tech pro — start learning today.