?
Solved

How to fip file with php and not to send same file multiple times

Posted on 2014-12-21
20
Medium Priority
?
109 Views
Last Modified: 2014-12-31
Hello,

I have destination like:

/daily/files/ffff/

under that there is folder name for each day

2014-12-16
2014-12-15

and inside that there is files like

Main_20141216_100100_01.tar
Main_20141216_100456_01.tar
etc

so what my goal is:
- to grab files form this in SFTP and FTP to different server
- not to send same file 2 times
- navigate the directory automatically as that directory will be created for everyday new directory.
- destination path is the same as location of the files only it should create that directory for everyday and push the files with above controls.


i'm open with with any script like that work for me PHP, shell or python.

regards
0
Comment
Question by:hi4ppl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 10
20 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 40512103
You can use rsync where you mention sftp.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512556
Hi,

any example would be great... but when you resync I suppose it require the transferred file to be present their, in my case those file would be moved and processed form destination.
0
 
LVL 62

Expert Comment

by:gheist
ID: 40512596
Given extensive information you provided about hos it should work you sure expect script of 10000 rows that does all you need... In javascript at least.

Can you confirm that rsync works at all for transfer? SCP?
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Author Comment

by:hi4ppl
ID: 40512743
I think I would like to go with different approach as this will create things more complicated:

1- I want to get data from SFTP location automatically and I want the script to identify based on sys date the folder as for each day there will be new folder 2014-12-22 would be folder name so the script should go and grab file from that folder

2- create directory locally for each day like 2014-12-22 and put the file here

3- transfer them to another FTP location and create dynamically the folder name 2014-12-22

then I will put this in crontab to run everyday
0
 
LVL 62

Expert Comment

by:gheist
ID: 40512750
Can you elaborate on my previous question?
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512761
Hi, no SCP is not available in all, as I only have control over my own server not otheres as they are as it's I can't modify or install anything on it as it's third party that is the reason I reduce my request on a script to do that... as for as javascript is concern no I only need php, shell or python

regards
0
 
LVL 62

Expert Comment

by:gheist
ID: 40512904
What server you have? i.e Suse/redhat/debian etc?
rsync usually is 1st party software from your Linux vendor.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40514432
hi, it's centos
0
 
LVL 62

Expert Comment

by:gheist
ID: 40514483
Can you confirm or deny working of INCLUDED rsync(rsync) and scp (openssh-clients)
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40514729
hi, yes when I type rsync/scp command it give the help details in both destination and target servers so I guess it means they are available, but one of them is centos the other is ubuntu.

thanks for help

regards
0
 
LVL 62

Expert Comment

by:gheist
ID: 40515685
1) use TODAY=`date +MM-DD-YYYY`

2) rsync remote:/wherever/today /here/today

Fairly easy and you can gradually transfer new and new files every five minutes if you wish. no data tandferred twice ever.

Probably you want to read "man date" to wrap up yesterday before today starts.
And make sure to examine all parameters in rsync's "-a" option - you dont need most of them, and read through rsync return codes to make sure you dont break something.

Sure there are PHP and python libraries for rsync, but I bet for your current needs a shell script with careful return checking will suffice.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40516091
Hi,

thanks... can you give me example with my current detail I have given?

regards
0
 
LVL 62

Accepted Solution

by:
gheist earned 2000 total points
ID: 40516134
TODAY=`date +%Y-%M-%d`
FILENAMES=`date +%Y%M%D`
YESTERDAY=`date -d 'yesterday' +%Y-%M-%d`

rsync remote:/path/${TODAY}/Main_${FILENAMES}_01.tar /path/${TODAY}

exit 0
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525207
Hi Thanks for the help it was great but can I have one more control over this that not to copy the same file over and over again... by that what I mean is in the destination where I transfer the file they remove the file so I have to keep the log of what I have send and next time sending the files it have to check if the files have been send or not
0
 
LVL 62

Expert Comment

by:gheist
ID: 40525222
I have hard time guessing you talk about rsync or scp or sftp now.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525237
The above works but I need to have control check before rsync the file, means I should not send same file two times. as where I synch this they load the data and remove the copy so I have to make sure not to send same file many times only send file for that day that I have not synch it before
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525318
i'm trying with this script with no luck:

function sendfile {
   rsync 
}

touch ~/tmp/newsync
if [ -f ~/tmp/lastsync ]; then
   NEWERFLAG="-newer ~/tmp/lastsync"
else
   NEWERFLAG=""
fi

find ~/user/data/ ${NEWERFLAG} ! -newer ~/tmp/newsync | while read file; do
      sendfile ${file}
done
touch -r ~/tmp/newsync ~/tmp/lastsync

Open in new window


don't know what i'm do which is wrong
0
 
LVL 62

Expert Comment

by:gheist
ID: 40525393
rsync will just transfer differences and report via return code if anything was transferred and if anything failed (like file being written)

There is no need to wrap it like that, you just create time windows where script does not transfer new files areound this script is being ran.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525478
well if the file removed and process by DBA ETL in destination  then I don't want to send that file again ... that is why I wrap it like that to avoid duplicate file transfer
0
 
LVL 62

Expert Comment

by:gheist
ID: 40526271
rsync will not transfer a single bit twice.
it is your miss you did not say ETL in the middle as now your question is fully answered but your problem not solved.
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many old projects have bad code, but the budget doesn't exist to rewrite the codebase. You can update this code to be safer by introducing contemporary input validation, sanitation, and safer database queries.
This article discusses how to implement server side field validation and display customized error messages to the client.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses

800 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question