Solved

How to SFTP and pull daily files

Posted on 2013-06-27
16
652 Views
Last Modified: 2013-06-29
I have a need to log in daily from a windows 7 box to a cent0s box and retrieve files that are stored in folders with the current date (2013-06-27). Is there a simple method to do this through powershell or a vbs script that you could share? I have the login down but the logic to dynamically pick a folder and get contents is escaping me.
Thanks
0
Comment
Question by:MRS
  • 5
  • 5
  • 5
  • +1
16 Comments
 
LVL 26

Expert Comment

by:pony10us
ID: 39281510
0
 

Author Comment

by:MRS
ID: 39281520
I have used winscp previously however I would like to have it scripted to just run and handle the date changes on its own. Can winscp be embedded into a script and still handle this dynamic date changing?
0
 

Author Comment

by:MRS
ID: 39281538
as an example the path to my folders for yesterday /var/spool/asterisk/monitor/2013/06/26
so todays would be /var/spool/asterisk/monitor/2013/06/27
0
 
LVL 26

Expert Comment

by:pony10us
ID: 39281541
Yes - take a look at:  http://winscp.net/eng/docs/script_download_most_recent_file

EDIT:

Sorry we posted at the same time.  I understand now that you are looking at the folder changing daily.  Take a look at the script above anyway but not sure if it will do what you are after.

You may need to set environment variables using batch and use those variables in the SCP script.
0
 
LVL 26

Expert Comment

by:pony10us
ID: 39281623
Try setting variables using a batch file:

::-----------------------------------------------------------------------------
::  Set variables for the day abbreviation (DAY), month number (MM), 
::  day number (DD) and 4-digit year (YYYY). 
::-----------------------------------------------------------------------------

:setvar

for /F "tokens=1-4 delims=/ " %%i IN ('date /t') DO (
set DT_DAY=%%i
set DT_MM=%%j
set DT_DD=%%k
set DT_YYYY=%%l)

Open in new window


Then call WinSCP from the batch file with the following "get":

get /var/spool/asterisk/monitor/%DT_YYYY%/%DT_MM%/%DT_DAY%/*.* c:\downloadsfolder <this would be where you want it to go>

Open in new window


I have not fully tested this out yet since my Linux box is offline right now.
0
 
LVL 7

Expert Comment

by:BT15
ID: 39281631
The way i have tackled this in the past is to use WinSCP, and then script around its commandline functionality.

for example:

$command = 'c:\Program Files\WinSCP\WinSCP.com'
$commandargs ='/script=scripttemp.txt' 
& $command $commandargs

Open in new window


where scripttemp.txt is a file containing the ftp commands.

for example:

option batch abort
option confirm off
open ftpsite
cd 2013-06-27
get -delete *.* c:\fromFTP\*.*
exit

Open in new window


in this example, ftpsite is the name of a saved connection in the WinSCP GUI.

Given the issue that you have presented, I would do something like this:

$today = Get-Date -UFormat %Y-%m-%d

$file = @("option batch abort","option confirm off","open ftpsite","cd $today","get -delete *.* c:\fromFTP\*.*","exit")

$file | sc scripttemp.txt

$command = 'c:\Program Files\WinSCP\WinSCP.com'
$commandargs ='/script=scripttemp.txt' 
& $command $commandargs


del scripttemp.txt

Open in new window

0
 
LVL 26

Expert Comment

by:pony10us
ID: 39281671
BT15:

That is a nice solution except that it looks at the date as a whole (2013-06-27) single folder. The author indicated that the folder structure was ../yyyy/mm/dd so you would need to split that out somehow.

I showed one way to do that in the first part of my reply.  If you incorporate that (or something similar) into your code then I think it should work.
0
 
LVL 7

Accepted Solution

by:
BT15 earned 500 total points
ID: 39281695
pony10us

i kept getting pulled away from this as i was typing it, so i didnt see any of the replies until i had posted. the -uFormat option of the get-date commandlet is extremely flexible, so changing the hypens is as simple as :

$today = Get-Date -UFormat %Y/%m/%d

Open in new window


so to revise:

$today = Get-Date -UFormat %Y/%m/%d
$folder = "/var/spool/asterisk/monitor/" + $today

$file = @("option batch abort","option confirm off","open ftpsite","cd $folder","get -delete *.* c:\fromFTP\*.*","exit")

$file | sc scripttemp.txt

$command = 'c:\Program Files\WinSCP\WinSCP.com'
$commandargs ='/script=scripttemp.txt' 
& $command $commandargs


del scripttemp.txt

Open in new window

0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 26

Expert Comment

by:pony10us
ID: 39281740
BT15:

Okay, I have not worked with uFormat before. Something for me to look into and learn.  

On another note.  I see that you delete the files from the site after you "get" them?
0
 
LVL 7

Expert Comment

by:BT15
ID: 39281970
pony10us

I have found uFormat to be so convenient that I rarely get-date without it anymore. you can do anything:

$date = get-date -uformat %YExperts%mExchange%d

Open in new window


The ftp script file can be populated with any valid FTP command, so in that example, yes I am deleting files after getting them. That can certainly be changed.
0
 
LVL 16

Expert Comment

by:AlexPace
ID: 39282159
This Robo-FTP script should do the it if you change the download destination folder, server address and login credentials:

SET today = %date 
SETEXTRACT month = yesterday "-" 1
SETEXTRACT day = yesterday "-" 2
SETEXTRACT year = yesterday "-" 3
SET TodayPath = "/var/spool/asterisk/monitor/20" + year + "/" + day + "/" + day
WORKINGDIR "c:\download\destination\folder"
FTPLOGON "sftp.mydomain.com" /user="UserID" /pw="Secret" /servertype=SFTP /trust=all 
FTPCD TodayPath 
IFERROR GOTO Done
RCVFILE "*"
:Done
FTPLOGOFF
EXIT

Open in new window

0
 

Author Comment

by:MRS
ID: 39282213
BT15
Can you please explain exactly how your recent addition would be implemented? Is this called via single batch script that I could schedule?
0
 
LVL 7

Expert Comment

by:BT15
ID: 39282283
sure.

so as to not leave this up to assumption, make sure you have WINSCP installed on your machine. Also, you will need to launch WINSCP from the GUI, set up a connection to your site, and save it as ftpsite (if you dont want to alter my script at all). be sure to check the option to have the credentials saved. manually connect to your ftp site to validate functionality. This is important so that we can rule this out if the script fails for some reason.

on your windows 7 box, save the script as a .PS1 file (c:\scripts\ftp.ps1). in the same directory, make a .CMD file (c:\scripts\ftp.cmd). inside the CMD file, use the following code:

powershell -ExecutionPolicy unrestricted -command .\ftp.ps1

then go into Windows Task Scheduler and have c:\scripts\ftp.cmd executed at whatever interval you prefer.

i know that is a very wide view of setting up a windows scheduled task, but i wasnt sure how granular you needed me to get there.

i have attached files for your convenience. just remove the .txt extension from them and place them in the folder of your choice (c:\scripts if you dont want to change anything)
ftp.cmd.zip
0
 

Author Comment

by:MRS
ID: 39282442
this is great how about if I wanted my local folders to mirror the remote?
0
 

Author Closing Comment

by:MRS
ID: 39284285
thank you great point to get me moving forward.
0
 
LVL 7

Expert Comment

by:BT15
ID: 39286772
you are welcome.

sorry for the delay on responding.

http://winscp.net/eng/docs/task_synchronize_full
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now