asked on
How to fip file with php and not to send same file multiple times
Hello,
I have destination like:
/daily/files/ffff/
under that there is folder name for each day
2014-12-16
2014-12-15
and inside that there is files like
Main_20141216_100100_01.ta
Main_20141216_100456_01.ta
etc
so what my goal is:
- to grab files form this in SFTP and FTP to different server
- not to send same file 2 times
- navigate the directory automatically as that directory will be created for everyday new directory.
- destination path is the same as location of the files only it should create that directory for everyday and push the files with above controls.
i'm open with with any script like that work for me PHP, shell or python.
regards
I have destination like:
/daily/files/ffff/
under that there is folder name for each day
2014-12-16
2014-12-15
and inside that there is files like
Main_20141216_100100_01.ta
Main_20141216_100456_01.ta
etc
so what my goal is:
- to grab files form this in SFTP and FTP to different server
- not to send same file 2 times
- navigate the directory automatically as that directory will be created for everyday new directory.
- destination path is the same as location of the files only it should create that directory for everyday and push the files with above controls.
i'm open with with any script like that work for me PHP, shell or python.
regards
Shell ScriptingPHPPython
Last Comment
gheist
You can use rsync where you mention sftp.
ASKER
Hi,
any example would be great... but when you resync I suppose it require the transferred file to be present their, in my case those file would be moved and processed form destination.
any example would be great... but when you resync I suppose it require the transferred file to be present their, in my case those file would be moved and processed form destination.
Given extensive information you provided about hos it should work you sure expect script of 10000 rows that does all you need... In javascript at least.
Can you confirm that rsync works at all for transfer? SCP?
Can you confirm that rsync works at all for transfer? SCP?
ASKER
I think I would like to go with different approach as this will create things more complicated:
1- I want to get data from SFTP location automatically and I want the script to identify based on sys date the folder as for each day there will be new folder 2014-12-22 would be folder name so the script should go and grab file from that folder
2- create directory locally for each day like 2014-12-22 and put the file here
3- transfer them to another FTP location and create dynamically the folder name 2014-12-22
then I will put this in crontab to run everyday
1- I want to get data from SFTP location automatically and I want the script to identify based on sys date the folder as for each day there will be new folder 2014-12-22 would be folder name so the script should go and grab file from that folder
2- create directory locally for each day like 2014-12-22 and put the file here
3- transfer them to another FTP location and create dynamically the folder name 2014-12-22
then I will put this in crontab to run everyday
Can you elaborate on my previous question?
ASKER
Hi, no SCP is not available in all, as I only have control over my own server not otheres as they are as it's I can't modify or install anything on it as it's third party that is the reason I reduce my request on a script to do that... as for as javascript is concern no I only need php, shell or python
regards
regards
What server you have? i.e Suse/redhat/debian etc?
rsync usually is 1st party software from your Linux vendor.
rsync usually is 1st party software from your Linux vendor.
ASKER
hi, it's centos
Can you confirm or deny working of INCLUDED rsync(rsync) and scp (openssh-clients)
ASKER
hi, yes when I type rsync/scp command it give the help details in both destination and target servers so I guess it means they are available, but one of them is centos the other is ubuntu.
thanks for help
regards
thanks for help
regards
1) use TODAY=`date +MM-DD-YYYY`
2) rsync remote:/wherever/today /here/today
Fairly easy and you can gradually transfer new and new files every five minutes if you wish. no data tandferred twice ever.
Probably you want to read "man date" to wrap up yesterday before today starts.
And make sure to examine all parameters in rsync's "-a" option - you dont need most of them, and read through rsync return codes to make sure you dont break something.
Sure there are PHP and python libraries for rsync, but I bet for your current needs a shell script with careful return checking will suffice.
2) rsync remote:/wherever/today /here/today
Fairly easy and you can gradually transfer new and new files every five minutes if you wish. no data tandferred twice ever.
Probably you want to read "man date" to wrap up yesterday before today starts.
And make sure to examine all parameters in rsync's "-a" option - you dont need most of them, and read through rsync return codes to make sure you dont break something.
Sure there are PHP and python libraries for rsync, but I bet for your current needs a shell script with careful return checking will suffice.
ASKER
Hi,
thanks... can you give me example with my current detail I have given?
regards
thanks... can you give me example with my current detail I have given?
regards
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Hi Thanks for the help it was great but can I have one more control over this that not to copy the same file over and over again... by that what I mean is in the destination where I transfer the file they remove the file so I have to keep the log of what I have send and next time sending the files it have to check if the files have been send or not
I have hard time guessing you talk about rsync or scp or sftp now.
ASKER
The above works but I need to have control check before rsync the file, means I should not send same file two times. as where I synch this they load the data and remove the copy so I have to make sure not to send same file many times only send file for that day that I have not synch it before
ASKER
i'm trying with this script with no luck:
don't know what i'm do which is wrong
function sendfile {
rsync
}
touch ~/tmp/newsync
if [ -f ~/tmp/lastsync ]; then
NEWERFLAG="-newer ~/tmp/lastsync"
else
NEWERFLAG=""
fi
find ~/user/data/ ${NEWERFLAG} ! -newer ~/tmp/newsync | while read file; do
sendfile ${file}
done
touch -r ~/tmp/newsync ~/tmp/lastsync
don't know what i'm do which is wrong
rsync will just transfer differences and report via return code if anything was transferred and if anything failed (like file being written)
There is no need to wrap it like that, you just create time windows where script does not transfer new files areound this script is being ran.
There is no need to wrap it like that, you just create time windows where script does not transfer new files areound this script is being ran.
ASKER
well if the file removed and process by DBA ETL in destination then I don't want to send that file again ... that is why I wrap it like that to avoid duplicate file transfer
rsync will not transfer a single bit twice.
it is your miss you did not say ETL in the middle as now your question is fully answered but your problem not solved.
it is your miss you did not say ETL in the middle as now your question is fully answered but your problem not solved.