Best solution to sync HUGE QTY of files across slow WAN?

mike2401
mike2401 used Ask the Experts™
on
Best solution to sync HUGE QTY of files across slow WAN?

Hello creative experts!

I have 1.7 million files (4 TB) in one of our remote offices.  

We presently use Vice-Versa to replicate the remote office data to headquarters.

Vice-versa is installed on a server here in HQ.

The initial "comparing source vs. destination" part of the run takes 21 hours over our 50 mbps wan connection.  

The actual file copy of changed files typically takes about 3 hours.

I'm guessing it's soooo slow because it's having a chatty conversation across a slow WAN connection to determine which files got added and deleted.  [This is a total guess as I don't know how the software is written]

What's the best solution?

Is there a solution that perhaps has agent software running on the other side such that each side determines local changes and THEN compares notes?

Is there a solution that works something like OneDrive?  (For example: there's no day long process evaluating local vs cloud before figuring out what to sync.  I presume if you delete a local folder, the agent gives the path of what to delete to the cloud and it's deleted.  Likewise, if a file gets added locally, just that file gets uploaded).

As I type this: I wonder if there's a microsoft solution which could leverage our E3 Office 365 subscription for our 400 users?

We started getting quotes for cloud backup and it was surprisingly expensive: maybe $90,000 per year for 8TB  - that's super approximate but gives me an order of magnitude.

Thanks,
Mike
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
David FavorFractional CTO
Distinguished Expert 2018

Commented:
Seems like the problem relates to Vice-Versa, which... fails to implement sensible file comparison algorithms.

For example, rsync (standard everywhere) checks things like file stamps first + only syncs files which have changed... then rsync only syncs parts of files which have changed, rather than the entire file.

Walking a directory of 1.7M files should be fairly quick. No more than a few minutes.

As an experiment, install one of the many rsync ports for Windows + test time required to do your file sync.
AlanConsultant

Commented:
Plus one for rsync - it would always be my first choice 'go to' for file replication / sync.


Alan.
Fractional CTO
Distinguished Expert 2018
Commented:
Just did a couple of checks on various machines, related to file scan time.

Linux machine - 14 minutes to scan 12,441,746.

Recent iMac - 2 minutes to scan 1,738,534 files.

So fast file comparison of timestamps should be very fast for any OS... even Windows...

If you have small files, sometimes using the rsync whole file option actually saves time, because only the last modified time is checked, then the entire file synced.

Your mention of 50 mbps wan connection suggests a solution. This slow a speed suggests you're running on premise storage, where running remote storage will be better.

OVH Storage Server Pricing is very inexpensive. $100/month USD for 18TB storage or $200/month USD for 60TB RAID storage, from a quick scan of their pricing.
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

David FavorFractional CTO
Distinguished Expert 2018

Commented:
rsync - Unsung hero of the Internet.

Author

Commented:
Thanks David.

Yes it's from a remote office to HQ over our VPLS WAN (50mbps).

Were those impressive times you mentioned rsync across the internet or WAN?  (or local pc to external USB3 hard drive)?

The pricing you mentioned is dirt cheap.  Though I never heard of OVH,  I'll have to check out raw storage from some more familiar names like Amazon :-)

Thanks so much!

Author

Commented:
Thanks!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial