Solved

How to fip file with php and not to send same file multiple times

Posted on 2014-12-21
20
99 Views
Last Modified: 2014-12-31
Hello,

I have destination like:

/daily/files/ffff/

under that there is folder name for each day

2014-12-16
2014-12-15

and inside that there is files like

Main_20141216_100100_01.tar
Main_20141216_100456_01.tar
etc

so what my goal is:
- to grab files form this in SFTP and FTP to different server
- not to send same file 2 times
- navigate the directory automatically as that directory will be created for everyday new directory.
- destination path is the same as location of the files only it should create that directory for everyday and push the files with above controls.


i'm open with with any script like that work for me PHP, shell or python.

regards
0
Comment
Question by:hi4ppl
  • 10
  • 10
20 Comments
 
LVL 61

Expert Comment

by:gheist
ID: 40512103
You can use rsync where you mention sftp.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512556
Hi,

any example would be great... but when you resync I suppose it require the transferred file to be present their, in my case those file would be moved and processed form destination.
0
 
LVL 61

Expert Comment

by:gheist
ID: 40512596
Given extensive information you provided about hos it should work you sure expect script of 10000 rows that does all you need... In javascript at least.

Can you confirm that rsync works at all for transfer? SCP?
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512743
I think I would like to go with different approach as this will create things more complicated:

1- I want to get data from SFTP location automatically and I want the script to identify based on sys date the folder as for each day there will be new folder 2014-12-22 would be folder name so the script should go and grab file from that folder

2- create directory locally for each day like 2014-12-22 and put the file here

3- transfer them to another FTP location and create dynamically the folder name 2014-12-22

then I will put this in crontab to run everyday
0
 
LVL 61

Expert Comment

by:gheist
ID: 40512750
Can you elaborate on my previous question?
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40512761
Hi, no SCP is not available in all, as I only have control over my own server not otheres as they are as it's I can't modify or install anything on it as it's third party that is the reason I reduce my request on a script to do that... as for as javascript is concern no I only need php, shell or python

regards
0
 
LVL 61

Expert Comment

by:gheist
ID: 40512904
What server you have? i.e Suse/redhat/debian etc?
rsync usually is 1st party software from your Linux vendor.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40514432
hi, it's centos
0
 
LVL 61

Expert Comment

by:gheist
ID: 40514483
Can you confirm or deny working of INCLUDED rsync(rsync) and scp (openssh-clients)
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40514729
hi, yes when I type rsync/scp command it give the help details in both destination and target servers so I guess it means they are available, but one of them is centos the other is ubuntu.

thanks for help

regards
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 61

Expert Comment

by:gheist
ID: 40515685
1) use TODAY=`date +MM-DD-YYYY`

2) rsync remote:/wherever/today /here/today

Fairly easy and you can gradually transfer new and new files every five minutes if you wish. no data tandferred twice ever.

Probably you want to read "man date" to wrap up yesterday before today starts.
And make sure to examine all parameters in rsync's "-a" option - you dont need most of them, and read through rsync return codes to make sure you dont break something.

Sure there are PHP and python libraries for rsync, but I bet for your current needs a shell script with careful return checking will suffice.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40516091
Hi,

thanks... can you give me example with my current detail I have given?

regards
0
 
LVL 61

Accepted Solution

by:
gheist earned 500 total points
ID: 40516134
TODAY=`date +%Y-%M-%d`
FILENAMES=`date +%Y%M%D`
YESTERDAY=`date -d 'yesterday' +%Y-%M-%d`

rsync remote:/path/${TODAY}/Main_${FILENAMES}_01.tar /path/${TODAY}

exit 0
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525207
Hi Thanks for the help it was great but can I have one more control over this that not to copy the same file over and over again... by that what I mean is in the destination where I transfer the file they remove the file so I have to keep the log of what I have send and next time sending the files it have to check if the files have been send or not
0
 
LVL 61

Expert Comment

by:gheist
ID: 40525222
I have hard time guessing you talk about rsync or scp or sftp now.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525237
The above works but I need to have control check before rsync the file, means I should not send same file two times. as where I synch this they load the data and remove the copy so I have to make sure not to send same file many times only send file for that day that I have not synch it before
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525318
i'm trying with this script with no luck:

function sendfile {
   rsync 
}

touch ~/tmp/newsync
if [ -f ~/tmp/lastsync ]; then
   NEWERFLAG="-newer ~/tmp/lastsync"
else
   NEWERFLAG=""
fi

find ~/user/data/ ${NEWERFLAG} ! -newer ~/tmp/newsync | while read file; do
      sendfile ${file}
done
touch -r ~/tmp/newsync ~/tmp/lastsync

Open in new window


don't know what i'm do which is wrong
0
 
LVL 61

Expert Comment

by:gheist
ID: 40525393
rsync will just transfer differences and report via return code if anything was transferred and if anything failed (like file being written)

There is no need to wrap it like that, you just create time windows where script does not transfer new files areound this script is being ran.
0
 
LVL 1

Author Comment

by:hi4ppl
ID: 40525478
well if the file removed and process by DBA ETL in destination  then I don't want to send that file again ... that is why I wrap it like that to avoid duplicate file transfer
0
 
LVL 61

Expert Comment

by:gheist
ID: 40526271
rsync will not transfer a single bit twice.
it is your miss you did not say ETL in the middle as now your question is fully answered but your problem not solved.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

A set of related code is known to be a Module, it helps us to organize our code logically which is much easier for us to understand and use it. Module is an object with arbitrarily named attributes which can be used in binding and referencing. …
Strings in Python are the set of characters that, once defined, cannot be changed by any other method like replace. Even if we use the replace method it still does not modify the original string that we use, but just copies the string and then modif…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now