Avatar of J.R. Sitman
J.R. Sitman
Flag for United States of America asked on

How to automate file transfer?

How can you automate the transfer of files from one Windows server to another without purchasing 3rd party software?

Windows OSSoftware

Avatar of undefined
Last Comment
J.R. Sitman

8/22/2022 - Mon
Peter Chan

Hi,
You can create Visual studio project to copy/transfer files from one place like
\\113.my.private.ip0\shared_path

to another one below.
\\113.my.private.ip1\shared_path

Or you can also involve FTP server to hold the temporary files during the transfer.


Jim Dettman (EE MVE)

You can use a batch file and/or .EXE and call that from Windows Task scheduler.

or were you taking more about the transfer itself, like using FTP?

Jim.
J.R. Sitman

ASKER
I have a meeting today with my company to discuss the options. It should be a secure transfer so I believe FTP is what we would need
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
ASKER CERTIFIED SOLUTION
Peter Chan

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
J.R. Sitman

ASKER
Thanks
Jim Dettman (EE MVE)

<<It should be a secure transfer so I believe FTP is what we would need >>

 Standard FTP is not secure.   You'd need to use sFTP (uses SSH) or FTPs (uses TLS).   Those are secure encrypted connections and while you can piece something together for free, chances are you will want to buy some 3rd party software, especially if you are looking for a MFT (Managed File Transfer) type process where sending/receiving of files is guarenteed.

 https://www.arcesb.com/mft/  offers that type of connectivity, and for one partner, used to be free.   They've changed their licensing quite a bit lately though, so it may be no longer.

Jim.
Alex [***Alex140181***]

If you're going for FTP, you should stick to "FTP over SSL" and/or SFTP! Plain FTP may be a lot faster, but lacks security!

Another good option would be IPSec as shown here (example):
https://www.snel.com/support/how-to-set-up-an-l2tp-ipsec-vpn-on-windows-server-2019/

There are tons of scenarios for this out there, but as an alternative it'll surely rock!
As always: it depends on what you REALLY need...
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Jim Dettman (EE MVE)

Keep in mind too that you could use something like DropBox, with a local folder on each machine automatically sync'd to the box.

In effect, that would get you a "file transfer".

Jim.
Alex [***Alex140181***]

As a side note: unless you have some crazy rocket science coding going on within this transfer process, DO NOT under any circumstances create a "custom" VS project which solely act as some sort of proxy or stub for copying files from one server to another ;-) THAT would be very bad practice plus the totally wrong way!
Alex [***Alex140181***]

@Jim: Dropbox??? For server 2 server copying/communication?! In a production environment?? Are you serious?!? Can't believe what I'm reading here :-(
For Heaven's Sake: DO NOT DO THIS!!
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
J.R. Sitman

ASKER
Thank you lots of good advice
Jim Dettman (EE MVE)

@Alex,


@Jim: Dropbox??? For server 2 server copying/communication?! In a production environment?? Are you serious?!? Can't believe what I'm reading here :-(
For Heaven's Sake: DO NOT DO THIS!!

  Ah, he didn't say the two servers were on the same network, or even in the same domain/forest.

Jim.
ste5an

How can you automate the transfer of files from one Windows server to another [..]?
The important questions are about:

- Where are those servers located?
- Can they connect to each other directly?
- Are they in the same AD/forest/tenant?
- Are there security measures active (fire walls)?
- Is that copy task server-to-server without any user interference?

The common way would be imho a VPN and a simple share to share copy via PowerShell or VBS or a command line batch scheduled by the Task Scheduler.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Alex [***Alex140181***]

Ah, he didn't say the two servers were on the same network, or even in the same domain/forest.
Doesn't matter either! You just don't use Dropbox for this kind of (business/production) stuff! You really don't want to argue over this, do you?!
Jim Dettman (EE MVE)

I'd also add that based on the question, he's looking for something cheap/free, but with the added requirement of security, then that complicates things.

A dropbox setup would work for a low load situation (which he also didn't mention - is this a file every few seconds or once a day?) and would satisfy everything.  

Jim.
Alex [***Alex140181***]

Sorry, but using Dropbox here plays in the same league as using emails with attachments for half-way "wannabe" automated export/import jobs where the email body is being "parsed" and stuff like that.
This is just my opinion, don't want to bother! But I've seen this bad paractice way too often :-(
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Jim Dettman (EE MVE)

Dropbox is used everyday in business and it works.   Granted this would not be a typical use case, but it could be workable depending on the requirements.

You should also note that I first pointed him to an Enterprise Level MFT solution.

Jim.
J.R. Sitman

ASKER
The 2 servers are not on the same network.
ste5an

Well, whether they are on the same network or not, is only relevant when there are security measures between those networks..
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Alex [***Alex140181***]

@Jim: As I said: it's my opinion and for me, it's just horrible using tools like Dropbox within production/business environments, sorry ;-)
But let us just not get too off-topic here :-)
Alex [***Alex140181***]


Well, whether they are on the same network or not, is only relevant when there are security measures between those networks..                                   
If there were security measures/tools/appliances present, I doubt that you'd have to use an extra level of securing the file transfer ;-)
ste5an

The original question and follow up's don't say anything about this. And the phrasing could mean two servers in the same rack.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
J.R. Sitman

ASKER
If we use SFTP can it be automated with a script or PowerShell?
ste5an

Sure.
Jim Dettman (EE MVE)

It depends on the software you use, but in general, most do have command line interfaces.  For example WinSCP:

https://winscp.net/eng/index.php

 or better yet, FileZilla (which offers both a free client and server):

https://filezilla-project.org/

 Note that it does FTPs though (still secure however)

Jim.


Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arnold

What type of access exists between the two systems?

Depending onthe systems, you could install ssh from Windows deployment.

You can then use rsync over ssh tunnel...
Alex [***Alex140181***]

Even ROBOCOPY would suffice depending on the connection ;-)
J.R. Sitman

ASKER
The server we're sending to is at 1 of our customers locations
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
arnold

What type of acces do you have to it?
Remote VPN?

In which direction are the files flowing
Out of or into customer located system?

Nature of data, might it make ore sense if possible to transmit data as it is received?

Soap type transactions?
J.R. Sitman

ASKER
We will be sending a file once a week from our location to the customer's location
Jim Dettman (EE MVE)

JR,

  Is this a one-time deal, or will it be on-going?   Assuming on-going in that you want to automate it.

  How often would you be transferring files, how many, and how large are they?

  How critical is it that the files make it without issue?

   A form of FTP is probably your answer, but it means that one of you will need to run FTP server software.   If you do that on your server, then it's the remote end that would need the automation to actually send/receive the files.

   But FTP has one major problem; it doesn't lock files when in the process of sending/receiving, so it's possible to get a partial file.   There are ways around that (use of a 'guard' file or using different directories), but that's why many look to a MFT (Managed File Transfer) solution, which has that built-in.

  Some more details would help and then I think we could steer you in a good direction.

Jim.   
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arnold

Any way to change it to a pull mechanism/process?

If ssh is installed scp could be used to transfer a file.
arnold

J.R. Sitman

ASKER
The file is quite large. I don't know exactly the size.
 It is one file we're currently gonna send it once a week and then possibly once A-day. It is critical that it could there without interruption. It is also critical that it be sent securely.

I don't understand the comment that the remote end would do the automation.
Your help has saved me hundreds of hours of internet surfing.
fblack61
Alex [***Alex140181***]

@arnold:

Soap type transactions?                   
Do you mean SOAP?! You don't want to use that "old" stuff, do you?

So, what about setting up a site-to-site connection via IPSec?!
https://it.cornell.edu/managed-servers/secure-windows-traffic-ipsec
Jim Dettman (EE MVE)

JR,

<<I don't understand the comment that the remote end would do the automation. >>

  When you use FTP,  one end has FTP server software that allows FTP clients on the other end to connect to it and send/receive files to/from the FTP server.  The FTP server by itself just sits there waiting for a connection request.

  So if you were to have the the FTP Server software on your side, your customer would use a FTP client to connect to it, see if there is a file to be download, download it if there was, and do whatever with it.   The automation of doing that would be on their end in that case.  That's a "Pull" situation", which is not ideal because they have to constantly connect to see if there is something there to download.

 However, if they can install a FTP server on their side, then it would be you using a FTP client to connect to them, and you could "push" a file to them whenever you had one to send them.   In this case, the automation of the process would be up to you.

If the files are only going one-way, this is the best setup, but it means the client would have to run FTP Server software, which is something they may not be willing to do.

As an aside, when files need to go both ways, then sometimes, both sides will run a FTP server.   Each end then can use a FTP client to push files over to the other side.   This avoids then inefficiency with "pull" type setups.

 What you use and how it is setup comes down to how much your customer is willing to do.  If they are willing to install FTP Server software, great.   You are set then and could go with FTP.

 But now that we have some more details on this, I would also not discount using something like Dropbox as it may be more palatable to the customer and simpler to setup.    If you setup the Dropbox software on each end, it can be set to automatically sync files to a local directory.    In that case, each end would have the ability to place a file in a local folder, it would get sync up to Dropbox, then get synced down to the other end and volia, you have a push setup in both directions.   It doesn't have to be Dropbox either.   You might use OneDrive or Google Drive.  They are all similar in the way they work.

 However to Alex's earlier point, this is not something you'd want to do for hundreds or thousands of files on a regular basis, but for single file per day?   It would be quite workable and the simplest to setup.  It's also something your customer would be more likely willing to setup rather than a FTP server.

Jim.





  
J.R. Sitman

ASKER
Thank you I think I have all the information I need to make a decision.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Alex [***Alex140181***]

Even if it's just one file per day or so: do you really want to exchange potentially sensitive data via DropBox, OneDrive or Google Drive?!?! If we were talking about up- and downloading some lunch menu, it would be "ok" to use these tools IMHO. But imagine having to deal with REAL(LY) sensitive data like e.g. personal data, social insurance stuff, medical information etc... Well then, I don't give a damn if a tool is free and/or easy to use/setup! But, as we don't fully know what you're trying to achieve and how "sensitive" this data might be, we can only lead you the way ;-)
J.R. Sitman

ASKER
We will not be using any of those.    It has to be totally secure.   The company is Federally regulated
arnold

Currently you seem to have a push mechanism a system from your side transfering data into the remote system on customer premises if my read is not right, please correct.

Pulling data on a schedule from the Client side opens up many more options.
powershell with https: URL that can pull a file/archive
uncompress the archive, validate the contents , proceed...

https://www.example-code.com/powershell/sftp.asp
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
J.R. Sitman

ASKER
Yes we prefer to use the push option rather than the pull option
Alex [***Alex140181***]


We will not be using any of those.    It has to be totally secure.   The company is Federally regulated                             
Then please, for Heaven's Sake, look out for a professional solution, even if that means investing some money!
J.R. Sitman

ASKER
Thanks
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arnold

a push meaning you drop a file/s
and then there is a process on the other end that validates, etc. and processes the file?

Nature and type of access available could narrow the options you have.


@alex, sorry did not notice your comment: SOAP yes, transactional it can be within a secure tunnel, much depends on what is being transferred.

Think options to pull things in are vaster as compared to a push process.
J.R. Sitman

ASKER
Yes we will drop the files and the customer will pick them up
arnold

I see, your system is isolated on the periphery at the client site.
In a way, the end result is a pull by the customer, the manner in which the files wind up on the intermediate system is up to you. you can working a push/pull if the file is not retrieved, you push it.

Size of the file, throughput on the network, the slowest end. and when the file has to be there.
I gather, a VPN is a given from the client located system to your distribution system.
That would secure the transfer no matter which method you choose.
you can have a VPN that limits to a specific narrow traffic type.
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
Jim Dettman (EE MVE)

@Alex,

<<But imagine having to deal with REAL(LY) sensitive data like e.g. personal data, social insurance stuff, medical information etc... Well then, I don't give a damn if a tool is free and/or easy to use/setup! >>
 
 FYI, Dropbox files are encrypted at rest with 256 bit AES encryption and use SSL/TLS 128 AES encryption for all transfers.

 So it is totally secure.   As to the servers on each end, well that's another story and I dare say that with Drop Box, One Drive , etc, your files are probably more secure there than on the servers.  

 These services are used world over and are far from being a Micky Mouse setup.  

Jim.
J.R. Sitman

ASKER
Jim thanks for the clarification
ste5an

So it is totally secure.
You'll still leak the information, when new data is available and when the file is downloaded by the other party. Meta data is in almost many cases more worth than the actual data.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Alex [***Alex140181***]

https://www.cloudwards.net/dropbox-security/

https://www.cloudwards.net/top-10-secure-dropbox-alternatives/

I did not say using Dropbox results in a "Micky Mouse setup", but I would NOT recommend it for these kind of (business/production) stuff!

Alex [***Alex140181***]

Jim Dettman (EE MVE)

Well I guess no one should be using Windows then either, because heck, they've had all kinds of security problems over the years as well (I think you'd be hard-pressed to find any vendor that has not).

Maybe we all shouldn't even be using computers.

Jim.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
J.R. Sitman

ASKER
Thanks to all.

We went with SFTP.  I am not sure exactly which software they used because I do not have my security clearance yet, to see the systems.