File Sync Across Internet between two Windows File Server

Does anyone know of a good, open source product or very reasonably priced product that can be used as an alternative to DFS?

I have a customer with 500gb of data that we want to sync up on a nightly basis, so that if the main site ever goes down, their branch site can stay operational.

So far we've done it the old fashioned way.  Bought a 2TB USB attached drive that's MAILED back and forth.  However, this only gives us a weekly file sync and we'd like it to be over the Internet.  The amount of data which is updated daily is typically 1gb or less, but for the sake of having additional capacity, let's say we want to copy up to 5gb across the Internet from the Main Site to the Branch Site.  There is no VPN between the two locations and the uplink at the Main Site is 1-3 megabits (DSL) and the Branch Site is also in the 1-5 megabits (Cable Internet).  We've thought about doing something like FileZilla and setting up a secure FTP / SCP site and then using WINSCP to run a script to back up modified files, but from reading about it, it doesn't appear to be the best solution.  Duplicati was something else we've kind of looked in to, but it does actual backups and we want to do the backup from Main to Branch and then restore the changed files to the Branch location.  Seems like a two step process that we'd like to avoid if at all possible.

Thanks in advance :-)
Erika KoelleAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

EirmanChief Operations ManagerCommented:
Believe it or not, BitTorrent can be used for very secure syncing.
BitTorrent Sync works on all platforms, including Windows, Mac, Linux, iOS, and Android.

REVIEW:
https://www.nextofwindows.com/securely-sync-files-between-multiple-computers-with-unlimited-space-using-bittorrent-sync
"It uses AES with 256-bit key created on the base of the secret, which is a random 21-bit key that is unique to every folder you put in your sync network. I think the use of the secret code is the key to putting BitTorrent Sync in a superb place above other File sync and sharing network in terms of security. It even gives you more flexibility to have your own longer key if you are not satisfied by the default 21-bit key."
1
Erika KoelleAuthor Commented:
Interesting article, but the Bit Torrent Sync is now by Resilio and they want $30 monthly for their sync services.

I spent a few hours setting up and testing Filezilla with WinSCP.  Here's a recap of what I ended up doing.

Runs at Main Site:
Filezilla - enabled TLS/SSL, added SSL cert converting PFX file to PEM format, changed default port from 990 to 1990, in passive section added port range 1990-1995, in IP filters added * to deny ALL and added the IP's of 3 public IP's I'd permit in, created a user called SYNCUSER and assigned the S: Drive on the server to that user as the Home Directory, under general setting changed connections timeout to 0 and no transfer timeout to zero as well.  This was important because using the SYNCHRONIZE command it would otherwise timeout and disconnect after 10 minutes because it was still checking which files had changed and wasn't copying files yet.  Copied Filezilla Server.xml off the server and emailed it to myself so I have a copy of the working config and also copies SSL.PEM file and emailed it to myself so I have a copy of it.

Main Site:
Firewall - Opened port range TCP 1990-1995 to IP address of server where Filezilla is running.

Remote Branch:
WinSCP - Created a BAT file that I can run as part of a scheduled task (hopefully daily it it doesn't take too long)  It's been running for an hour now and it's still checking for changed files.  Below is the BAT file I wrote.  I hope someone will find it helpful.  It took a lot of searching around and testing to get the parameters set just right for my project.

FTP_SYNC_FROM_MAIN_TO_BRANCH.BAT
REM FTPS From Main Office to Branch Office
@echo %time%
"c:\program files (x86)\winscp\winscp.com" /command "open ftps://syncuser:Pa$$word@main,mycorp.com:1990" "synchronize local s:\data /data -transfer=binary -criteria=time,size -delete" "exit"
@echo %time%
REM This checks for file changes based on time and size (criteria)
REM Binary transfer mode for FTPS
REM Connects on Port 1990 instead of default port on 990
REM syncuser is username use for FTPS login
REM local means it'll pull files form the SFTP server to the local server (destination) and update items on the local server only, not making any changes to the source
REM -delete will delete files from the destination server if they no longer exist on the source server
REM Open ftps is for TLS/SSL Implicit Encryption (Example is Filezilla servers set up for SSL)
REM Open ftpes is for TLS/SSL Explicit Encryption (Example is GoDaddy FTP servers)

*Now that I know how to do it, I could get this done in 30 minutes from start to finish.
**I noticed the Filezilla FTP client supports compression, so if anyone knows how to do make a simple BAT file like the one above using Filezilla's FTP client and enable compressing on a SYNC to speed things up, by all means please let us know.
0
Shaun VermaakTechnical Specialist IVCommented:
I don't know how much this cost, but I tested this a while ago
https://azure.microsoft.com/en-us/blog/announcing-the-public-preview-for-azure-file-sync/
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Erika KoelleAuthor Commented:
Here's some feedback on using WinSCP with Filezilla.  It worked, but due to needing to copy a total of 150gb of data and that data being in hundreds of thousands of files it takes a long time to run.  I think if I only had a few thousand files and maybe 10gb or less then it would be just fine and complete pretty quickly.

I'm also evaluating a product from GoodSync.com which costs $29 per workstation you want to sync.  It also takes quite some time to run, but it's algorithm is more sophisticated than just plain old FTP.  It knows if files have been moved and makes adjustments on the destination to accommodate that instead of just deleting the old location and copying it to the new location like FTP does.  

I did some reading about RSYNC and have spoken with some work associates about it as well and how it works great for them, but getting it set up seems like a lot of work when I can use GoodSync for $29 instead.  Although I love the idea of open source.  One associate said I should just run CYGWIN on a Windows server and use RSYNC.  If someone has this documented, then I'd love to see it.  It has to be documented, but finding it may not be that easy.

Finally, I did a search of open source deduplication and stumbled across a website called opendedup.org and they've got Linux, Windows and even an appliance based backup.  I'm not sure how smart GoodSync or RSYNC are, but possibly being able to reduce the amount of data transferred from 150gb to 30gb because of Deduplication sounds very appealing to me because of my slow uplink.  

In the meantime, I'm sort of sticking with GoodSync and am in the process of copying 100gb to Amazon's S3 storage.  I'm running it from two systems in different locations at the same time to speed things up.  I figure I already have a copy of the data from my Main location at the Branch location, so why not start at the top of the list for the Main location and copy data up to Amazon and then start 60% of the way down the list from the Branch location and copy things up as well.  At some point they'll catch up.  It was somewhat daunting when the backup to Amazon S3 started because it calculated two weeks for it to complete and the estimate didn't change for 12 hours while it ran.  Needless to say this is an after hours only function presently to not kill Internet access at the two locations during the daytime when people need to use it.

The take away:
1.  Does someone have RSYNC nicely documented so I could install CYGWIN on a Windows computer and backup say c:\backup to c:\backup across the Internet (securely and encrypted) from a Main location to a Branch location?
2.  Does someone have experience with OpenDeDup, possibly even the appliance for me to copy from SERVER2 (let's say IP 192.168.1.10) to a remote appliance across the Internet and have it restore data to SERVER2?

Thanks in advance!!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Rob KnightConsultantCommented:
Hi,

Consider resilio (https://www.resilio.com/) or perhaps TeamViewer VPN, which will allow you to do a copy from one server to the other.

Thanks.
0
PberSolutions ArchitectCommented:
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.

I have recommended this question be closed as follows:

Split:
-- Erika Koelle (https:#a42408300)
-- RobMobility (https:#a42410455)
-- Shaun Vermaak (https:#a42405262)
-- Eirman (https:#a42404810)


If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.

Pber
Experts-Exchange Cleanup Volunteer
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows OS

From novice to tech pro — start learning today.