Solved

How to fip file with php and not to send same file multiple times

Posted on 2014-12-21
20
108 Views
Last Modified: 2014-12-31
Hello,

I have destination like:

/daily/files/ffff/

under that there is folder name for each day

2014-12-16
2014-12-15

and inside that there is files like

Main_20141216_100100_01.tar
Main_20141216_100456_01.tar
etc

so what my goal is:
- to grab files form this in SFTP and FTP to different server
- not to send same file 2 times
- navigate the directory automatically as that directory will be created for everyday new directory.
- destination path is the same as location of the files only it should create that directory for everyday and push the files with above controls.


i'm open with with any script like that work for me PHP, shell or python.

regards
0
Comment
Question by:hi4ppl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 10
20 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 40512103
You can use rsync where you mention sftp.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512556
Hi,

any example would be great... but when you resync I suppose it require the transferred file to be present their, in my case those file would be moved and processed form destination.
0
 
LVL 62

Expert Comment

by:gheist
ID: 40512596
Given extensive information you provided about hos it should work you sure expect script of 10000 rows that does all you need... In javascript at least.

Can you confirm that rsync works at all for transfer? SCP?
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Author Comment

by:hi4ppl
ID: 40512743
I think I would like to go with different approach as this will create things more complicated:

1- I want to get data from SFTP location automatically and I want the script to identify based on sys date the folder as for each day there will be new folder 2014-12-22 would be folder name so the script should go and grab file from that folder

2- create directory locally for each day like 2014-12-22 and put the file here

3- transfer them to another FTP location and create dynamically the folder name 2014-12-22

then I will put this in crontab to run everyday
0
 
LVL 62

Expert Comment

by:gheist
ID: 40512750
Can you elaborate on my previous question?
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512761
Hi, no SCP is not available in all, as I only have control over my own server not otheres as they are as it's I can't modify or install anything on it as it's third party that is the reason I reduce my request on a script to do that... as for as javascript is concern no I only need php, shell or python

regards
0
 
LVL 62

Expert Comment

by:gheist
ID: 40512904
What server you have? i.e Suse/redhat/debian etc?
rsync usually is 1st party software from your Linux vendor.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40514432
hi, it's centos
0
 
LVL 62

Expert Comment

by:gheist
ID: 40514483
Can you confirm or deny working of INCLUDED rsync(rsync) and scp (openssh-clients)
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40514729
hi, yes when I type rsync/scp command it give the help details in both destination and target servers so I guess it means they are available, but one of them is centos the other is ubuntu.

thanks for help

regards
0
 
LVL 62

Expert Comment

by:gheist
ID: 40515685
1) use TODAY=`date +MM-DD-YYYY`

2) rsync remote:/wherever/today /here/today

Fairly easy and you can gradually transfer new and new files every five minutes if you wish. no data tandferred twice ever.

Probably you want to read "man date" to wrap up yesterday before today starts.
And make sure to examine all parameters in rsync's "-a" option - you dont need most of them, and read through rsync return codes to make sure you dont break something.

Sure there are PHP and python libraries for rsync, but I bet for your current needs a shell script with careful return checking will suffice.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40516091
Hi,

thanks... can you give me example with my current detail I have given?

regards
0
 
LVL 62

Accepted Solution

by:
gheist earned 500 total points
ID: 40516134
TODAY=`date +%Y-%M-%d`
FILENAMES=`date +%Y%M%D`
YESTERDAY=`date -d 'yesterday' +%Y-%M-%d`

rsync remote:/path/${TODAY}/Main_${FILENAMES}_01.tar /path/${TODAY}

exit 0
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525207
Hi Thanks for the help it was great but can I have one more control over this that not to copy the same file over and over again... by that what I mean is in the destination where I transfer the file they remove the file so I have to keep the log of what I have send and next time sending the files it have to check if the files have been send or not
0
 
LVL 62

Expert Comment

by:gheist
ID: 40525222
I have hard time guessing you talk about rsync or scp or sftp now.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525237
The above works but I need to have control check before rsync the file, means I should not send same file two times. as where I synch this they load the data and remove the copy so I have to make sure not to send same file many times only send file for that day that I have not synch it before
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525318
i'm trying with this script with no luck:

function sendfile {
   rsync 
}

touch ~/tmp/newsync
if [ -f ~/tmp/lastsync ]; then
   NEWERFLAG="-newer ~/tmp/lastsync"
else
   NEWERFLAG=""
fi

find ~/user/data/ ${NEWERFLAG} ! -newer ~/tmp/newsync | while read file; do
      sendfile ${file}
done
touch -r ~/tmp/newsync ~/tmp/lastsync

Open in new window


don't know what i'm do which is wrong
0
 
LVL 62

Expert Comment

by:gheist
ID: 40525393
rsync will just transfer differences and report via return code if anything was transferred and if anything failed (like file being written)

There is no need to wrap it like that, you just create time windows where script does not transfer new files areound this script is being ran.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525478
well if the file removed and process by DBA ETL in destination  then I don't want to send that file again ... that is why I wrap it like that to avoid duplicate file transfer
0
 
LVL 62

Expert Comment

by:gheist
ID: 40526271
rsync will not transfer a single bit twice.
it is your miss you did not say ETL in the middle as now your question is fully answered but your problem not solved.
0

Featured Post

Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: Swadhin
Introduction of Lists in Python: There are six built-in types of sequences. Lists and tuples are the most common one. In this article we will see how to use Lists in python and how we can utilize it while doing our own program. In general we can al…
3 proven steps to speed up Magento powered sites. The article focus is on optimizing time to first byte (TTFB), full page caching and configuring server for optimal performance.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to dynamically set the form action using jQuery.

687 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question