Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

FTP script to not send same file multiple times.

Posted on 2014-12-21
19
Medium Priority
?
170 Views
Last Modified: 2015-01-04
Hi,

I have following FTP script that that send files for everyday,
HOST=199.168.0.105
USER=user
PASS=pass
ftp -inv $HOST <<EOF
user $USER $PASS
bin
cd /destination/
put  "*.gz"
quit
EOF

Open in new window


what I want now is that it should not send same files multiple times, it should check if the files already send it should not send it again. on the same day

im thinking of a file to create our of file send and once it will send another file it will check name of file send and abort if send that? don't know how to do this if any good idea please share.

regards
0
Comment
Question by:hi4ppl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 9
19 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 40511462
You can always have a reference file for last transfer.

e.g. you could touch a file at the end of each transfer and next time you run the script it should find files newer than that reference file and transfer them only:

another way would be to do ls (from ftp connection) on remote folder first, and store locally then exclude these files from transfer
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40511469
Hi,

any working example would be highly appreciate it?
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40511481
you could run something like:

HOST=199.168.0.105
USER=user
PASS=pass
ftp -inv $HOST > /tmp/myremfiles <<EOF
user $USER $PASS
bin
cd /destination/
ls
quit
EOF
ls > /tmp/myfiles
grep -v -f /tmp/myremfile /tmp/myfiles > /tmp/files
files=`cat /tmp/files`
ftp -inv $HOST > /tmp/myremfiles <<EOF
user $USER $PASS
bin
cd /destination/
mput $files
quit
EOF
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
LVL 1

Author Comment

by:hi4ppl
ID: 40511505
but in this one where it will check so it will not send duplicate files? I don't see any condition and also how does it know, to change directory based on system date of machine as for everyday there is new directory 20141216.

regards
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40511507
What the script is supposed to do is:

1- connect via ftp to remote system and get list of files on the remote system and store in /tmp/myremfile

2- list local files and store in /tmp/myfiles

3- exclude remote files from local files and store in /tmp/files

4- push the files in /tmp/files to remote system
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40511514
hi, thanks for explaination but the only reason i'm asking not to send duplicate files is that the file that I send will be process from destination and they will not the their when I send the new file so I can't compare them like that... I have to do it locally that if I send one file it should not be send the second time.

thanks
0
 
LVL 3

Expert Comment

by:carlrjr
ID: 40511605
When I had a similar problem I would move the file(s) into a subfolder immediately after the put into a folder named 'archive' .
0
 
LVL 40

Accepted Solution

by:
omarfarid earned 2000 total points
ID: 40511614
you can do the following:

1- transfer files first time
2- touch local file:
touch /tmp/myref
3- next time you see newer files
find . -newer /tmp/myref -t f -exec ls {} \; > /tmp/files
4- run ftp

HOST=199.168.0.105
USER=user
PASS=pass
files=`cat /tmp/files`
ftp -inv $HOST > /tmp/myremfiles <<EOF
user $USER $PASS
bin
cd /destination/
mput $files
quit
EOF
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40511679
sorry I forget to mentioned that where I pull the files are SFTP and where I push them is ftp
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40511687
please elaborate on requirement as I am now not clear what is required
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512560
Hi,

what I mean is the location where I pick up files is SFTP and where I send the file destination is ftp
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40512573
when I sow your script I assumed that files are available on your local server. You can always pull those files with sftp then push with ftp.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525141
Hi sorry for confusing but this is my requirement,

- I have files stored locally in a system that is collected from SFTP ==Done
- I have to send these files to another FTP which i'm doing now with rsynch
       - but I need to put in place a control here that it should remember the files send and don't send those file back to FTP as where I send the files they get them and process them and removing them if I send one file twice they will process it twice.

regards
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40525148
What you do with the file you send on the local server? can you move it to a different folder?
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525169
I only collect locally because the to move it to another FTP, and if I move them it will rsynch it back from SFTP... so yeah I can have a script that move them to some temp folder ... but I all I want is not to send same file to destination the second time
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40525215
what are the chances that the file is fetched again from sftp or its time stamp is updated?
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525238
the local is like ghost of SFTP if I remove anything it will bring it back from SFTP, so that is why I need to keep some log and send files that have not been send before some kind of IF statement
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525317
i'm actually using this and it does not work the way I wanted like to send only file once and keep back up of file name send and next time it run check what files send and send the new one

function sendfile {
   ftp
}

touch ~/tmp/newsync
if [ -f ~/tmp/lastsync ]; then
   NEWERFLAG="-newer ~/tmp/lastsync"
else
   NEWERFLAG=""
fi

find ~/user/data/ ${NEWERFLAG} ! -newer ~/tmp/newsync | while read file; do
      sendfile ${file}
done
touch -r ~/tmp/newsync ~/tmp/lastsync

Open in new window


don't know what i'm doing wrong
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40529257
why you want files not newer !
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This Windows batch file is useful for organizing image files from a digital camera or other source, but can have many other uses.  It simply renames the file(s) to match their create date.  For example, if you took a picture today at 1:40pm and the …
Over the years I've spent many an hour playing on hardened, DMZ'd servers, with only a sub-set of the usual GNU toy's to keep me company; frequently I've needed to save and send log or data extracts from these server back to my PC, or to others, and…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses

609 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question