How to automate file transfer?

Hi,
You can create Visual studio project to copy/transfer files from one place like
\\113.my.private.ip0\shared_path

to another one below.
\\113.my.private.ip1\shared_path

Or you can also involve FTP server to hold the temporary files during the transfer.

You can use a batch file and/or .EXE and call that from Windows Task scheduler.

or were you taking more about the transfer itself, like using FTP?

Jim.

I have a meeting today with my company to discuss the options. It should be a secure transfer so I believe FTP is what we would need

Thanks

<<It should be a secure transfer so I believe FTP is what we would need >>

 Standard FTP is not secure.   You'd need to use sFTP (uses SSH) or FTPs (uses TLS).   Those are secure encrypted connections and while you can piece something together for free, chances are you will want to buy some 3rd party software, especially if you are looking for a MFT (Managed File Transfer) type process where sending/receiving of files is guarenteed.

 https://www.arcesb.com/mft/  offers that type of connectivity, and for one partner, used to be free.   They've changed their licensing quite a bit lately though, so it may be no longer.

Jim.

If you're going for FTP, you should stick to "FTP over SSL" and/or SFTP! Plain FTP may be a lot faster, but lacks security!

Another good option would be IPSec as shown here (example):
https://www.snel.com/support/how-to-set-up-an-l2tp-ipsec-vpn-on-windows-server-2019/

There are tons of scenarios for this out there, but as an alternative it'll surely rock!
As always: it depends on what you REALLY need...

Keep in mind too that you could use something like DropBox, with a local folder on each machine automatically sync'd to the box.

In effect, that would get you a "file transfer".

Jim.

As a side note: unless you have some crazy rocket science coding going on within this transfer process, DO NOT under any circumstances create a "custom" VS project which solely act as some sort of proxy or stub for copying files from one server to another ;-) THAT would be very bad practice plus the totally wrong way!

@Jim: Dropbox??? For server 2 server copying/communication?! In a production environment?? Are you serious?!? Can't believe what I'm reading here :-(
For Heaven's Sake: DO NOT DO THIS!!

Thank you lots of good advice

@Alex,

@Jim: Dropbox??? For server 2 server copying/communication?! In a production environment?? Are you serious?!? Can't believe what I'm reading here :-(
For Heaven's Sake: DO NOT DO THIS!!

  Ah, he didn't say the two servers were on the same network, or even in the same domain/forest.

Jim.

How can you automate the transfer of files from one Windows server to another [..]?

The important questions are about:

- Where are those servers located?
- Can they connect to each other directly?
- Are they in the same AD/forest/tenant?
- Are there security measures active (fire walls)?
- Is that copy task server-to-server without any user interference?

The common way would be imho a VPN and a simple share to share copy via PowerShell or VBS or a command line batch scheduled by the Task Scheduler.

Ah, he didn't say the two servers were on the same network, or even in the same domain/forest.

Doesn't matter either! You just don't use Dropbox for this kind of (business/production) stuff! You really don't want to argue over this, do you?!

I'd also add that based on the question, he's looking for something cheap/free, but with the added requirement of security, then that complicates things.

A dropbox setup would work for a low load situation (which he also didn't mention - is this a file every few seconds or once a day?) and would satisfy everything.  

Jim.

Sorry, but using Dropbox here plays in the same league as using emails with attachments for half-way "wannabe" automated export/import jobs where the email body is being "parsed" and stuff like that.
This is just my opinion, don't want to bother! But I've seen this bad paractice way too often :-(

Dropbox is used everyday in business and it works.   Granted this would not be a typical use case, but it could be workable depending on the requirements.

You should also note that I first pointed him to an Enterprise Level MFT solution.

Jim.

The 2 servers are not on the same network.

Well, whether they are on the same network or not, is only relevant when there are security measures between those networks..

@Jim: As I said: it's my opinion and for me, it's just horrible using tools like Dropbox within production/business environments, sorry ;-)
But let us just not get too off-topic here :-)

Well, whether they are on the same network or not, is only relevant when there are security measures between those networks..                                   

If there were security measures/tools/appliances present, I doubt that you'd have to use an extra level of securing the file transfer ;-)

The original question and follow up's don't say anything about this. And the phrasing could mean two servers in the same rack.

If we use SFTP can it be automated with a script or PowerShell?

Sure.

It depends on the software you use, but in general, most do have command line interfaces.  For example WinSCP:

https://winscp.net/eng/index.php

 or better yet, FileZilla (which offers both a free client and server):

https://filezilla-project.org/

 Note that it does FTPs though (still secure however)

Jim.

What type of access exists between the two systems?

Depending onthe systems, you could install ssh from Windows deployment.

You can then use rsync over ssh tunnel...

Even ROBOCOPY would suffice depending on the connection ;-)

The server we're sending to is at 1 of our customers locations

What type of acces do you have to it?
Remote VPN?

In which direction are the files flowing
Out of or into customer located system?

Nature of data, might it make ore sense if possible to transmit data as it is received?

Soap type transactions?

We will be sending a file once a week from our location to the customer's location

JR,

  Is this a one-time deal, or will it be on-going?   Assuming on-going in that you want to automate it.

  How often would you be transferring files, how many, and how large are they?

  How critical is it that the files make it without issue?

   A form of FTP is probably your answer, but it means that one of you will need to run FTP server software.   If you do that on your server, then it's the remote end that would need the automation to actually send/receive the files.

   But FTP has one major problem; it doesn't lock files when in the process of sending/receiving, so it's possible to get a partial file.   There are ways around that (use of a 'guard' file or using different directories), but that's why many look to a MFT (Managed File Transfer) solution, which has that built-in.

  Some more details would help and then I think we could steer you in a good direction.

Jim.   

Any way to change it to a pull mechanism/process?

If ssh is installed scp could be used to transfer a file.

See if this is an option
At eighter or both ends.

https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse

https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration

Cygwin is a way to get an openssh server on a Windows system.

The file is quite large. I don't know exactly the size.
 It is one file we're currently gonna send it once a week and then possibly once A-day. It is critical that it could there without interruption. It is also critical that it be sent securely.

I don't understand the comment that the remote end would do the automation.

@arnold:

Soap type transactions?                   

Do you mean SOAP?! You don't want to use that "old" stuff, do you?

So, what about setting up a site-to-site connection via IPSec?!
https://it.cornell.edu/managed-servers/secure-windows-traffic-ipsec

JR,

<<I don't understand the comment that the remote end would do the automation. >>

  When you use FTP,  one end has FTP server software that allows FTP clients on the other end to connect to it and send/receive files to/from the FTP server.  The FTP server by itself just sits there waiting for a connection request.

  So if you were to have the the FTP Server software on your side, your customer would use a FTP client to connect to it, see if there is a file to be download, download it if there was, and do whatever with it.   The automation of doing that would be on their end in that case.  That's a "Pull" situation", which is not ideal because they have to constantly connect to see if there is something there to download.

 However, if they can install a FTP server on their side, then it would be you using a FTP client to connect to them, and you could "push" a file to them whenever you had one to send them.   In this case, the automation of the process would be up to you.

If the files are only going one-way, this is the best setup, but it means the client would have to run FTP Server software, which is something they may not be willing to do.

As an aside, when files need to go both ways, then sometimes, both sides will run a FTP server.   Each end then can use a FTP client to push files over to the other side.   This avoids then inefficiency with "pull" type setups.

 What you use and how it is setup comes down to how much your customer is willing to do.  If they are willing to install FTP Server software, great.   You are set then and could go with FTP.

 But now that we have some more details on this, I would also not discount using something like Dropbox as it may be more palatable to the customer and simpler to setup.    If you setup the Dropbox software on each end, it can be set to automatically sync files to a local directory.    In that case, each end would have the ability to place a file in a local folder, it would get sync up to Dropbox, then get synced down to the other end and volia, you have a push setup in both directions.   It doesn't have to be Dropbox either.   You might use OneDrive or Google Drive.  They are all similar in the way they work.

 However to Alex's earlier point, this is not something you'd want to do for hundreds or thousands of files on a regular basis, but for single file per day?   It would be quite workable and the simplest to setup.  It's also something your customer would be more likely willing to setup rather than a FTP server.

Jim.





  

Thank you I think I have all the information I need to make a decision.

Even if it's just one file per day or so: do you really want to exchange potentially sensitive data via DropBox, OneDrive or Google Drive?!?! If we were talking about up- and downloading some lunch menu, it would be "ok" to use these tools IMHO. But imagine having to deal with REAL(LY) sensitive data like e.g. personal data, social insurance stuff, medical information etc... Well then, I don't give a damn if a tool is free and/or easy to use/setup! But, as we don't fully know what you're trying to achieve and how "sensitive" this data might be, we can only lead you the way ;-)

We will not be using any of those.    It has to be totally secure.   The company is Federally regulated

Currently you seem to have a push mechanism a system from your side transfering data into the remote system on customer premises if my read is not right, please correct.

Pulling data on a schedule from the Client side opens up many more options.
powershell with https: URL that can pull a file/archive
uncompress the archive, validate the contents , proceed...

https://www.example-code.com/powershell/sftp.asp

Yes we prefer to use the push option rather than the pull option

We will not be using any of those.    It has to be totally secure.   The company is Federally regulated                             

Then please, for Heaven's Sake, look out for a professional solution, even if that means investing some money!

Thanks

a push meaning you drop a file/s
and then there is a process on the other end that validates, etc. and processes the file?

Nature and type of access available could narrow the options you have.


@alex, sorry did not notice your comment: SOAP yes, transactional it can be within a secure tunnel, much depends on what is being transferred.

Think options to pull things in are vaster as compared to a push process.

Yes we will drop the files and the customer will pick them up

I see, your system is isolated on the periphery at the client site.
In a way, the end result is a pull by the customer, the manner in which the files wind up on the intermediate system is up to you. you can working a push/pull if the file is not retrieved, you push it.

Size of the file, throughput on the network, the slowest end. and when the file has to be there.
I gather, a VPN is a given from the client located system to your distribution system.
That would secure the transfer no matter which method you choose.
you can have a VPN that limits to a specific narrow traffic type.

@Alex,

<<But imagine having to deal with REAL(LY) sensitive data like e.g. personal data, social insurance stuff, medical information etc... Well then, I don't give a damn if a tool is free and/or easy to use/setup! >>
 
 FYI, Dropbox files are encrypted at rest with 256 bit AES encryption and use SSL/TLS 128 AES encryption for all transfers.

 So it is totally secure.   As to the servers on each end, well that's another story and I dare say that with Drop Box, One Drive , etc, your files are probably more secure there than on the servers.  

 These services are used world over and are far from being a Micky Mouse setup.  

Jim.

Jim thanks for the clarification

So it is totally secure.

You'll still leak the information, when new data is available and when the file is downloaded by the other party. Meta data is in almost many cases more worth than the actual data.

https://www.cloudwards.net/dropbox-security/

https://www.cloudwards.net/top-10-secure-dropbox-alternatives/

I did not say using Dropbox results in a "Micky Mouse setup", but I would NOT recommend it for these kind of (business/production) stuff!

Further resources which might be interesting regarding the almighty DropBox ;-) [/irony off]

https://en.wikipedia.org/wiki/Criticism_of_Dropbox

https://www.renovodata.com/assets/pdf/7-risks-dropbox.pdf

https://www.groupsense.io/resources/dropbox-security-concerns

https://globalquestinc.com/7-facts-why-dropbox-isnt-safe-for-your-company/

Well I guess no one should be using Windows then either, because heck, they've had all kinds of security problems over the years as well (I think you'd be hard-pressed to find any vendor that has not).

Maybe we all shouldn't even be using computers.

Jim.

Thanks to all.

We went with SFTP.  I am not sure exactly which software they used because I do not have my security clearance yet, to see the systems.