Solved

curl in unix

Posted on 2013-05-24
6
485 Views
Last Modified: 2013-06-03
I have pasted partial code here, but you may notice down below "Curl" is being used to download the file. I am just wondering if I can use "curl" to download latest file by date. Please help

#!/bin/ksh -xve
date

HOME_DIR=$tradehome/bmc_jobs/pace_adjustments
CUR_DATE=`date '+%Y_%m_%d'`
LOG_FILE=$HOME_DIR/logs/pace_adjustments.log
#DOWNLOAD_FILE_NAME=icecleared_gas_$CUR_DATE.dat
DOWNLOAD_FILE_NAME=icecleared_gas_2013_03_14.dat
USER_NAME=Caxton_settles
PASSWORD=icedata
HTTP_URL=https://downloads.theice.com/Settlement_Reports_CSV/Gas/$DOWNLOAD_FILE_NAME
DATA_FILE_DIR=$HOME_DIR/data
TRIES=0
MAIL_LIST=smaddirala@caxton.com
CTL_FILE=$HOME_DIR/ctl/ice_prices.ctl
SQL_LOADER_LOG_FILE=$HOME_DIR/sql_loader_log/ice_prices.log
DATA_FILE=$DATA_FILE_DIR/icecleared_gas.dat
badfname=$HOME_DIR/sql_loader_log/ice_prices.bad
dscfname=$HOME_DIR/sql_loader_log/ice_prices.dsc
logfname=$HOME_DIR/sql_loader_log/ice_prices.log


echo "Down Load File = $DOWNLOAD_FILE_NAME"
echo "URL = $HTTP_URL"


cd $HOME_DIR

export USER_NAME PASSWORD HTTP_URL

#
# Get the file from ICE
if (test -e "$DATA_FILE_DIR/$DOWNLOAD_FILE_NAME" )
then
   rm $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME
fi
sleep 30


until
   [ -f "$DATA_FILE_DIR/$DOWNLOAD_FILE_NAME" ]
do
   sleep 10
   date
   echo 'Attempt to download $$DOWNLOAD_FILE_NAME from ICE' `date`
        curl -k -u $USER_NAME:$PASSWORD $HTTP_URL > $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME

        CNT=`grep -i "<title>404 Not Found</title>" $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME | wc -l`

        echo "Not Found Count = $CNT"

        if [ $CNT -gt 0 ]
        then
                echo "Delete the bad file and try again"
                rm $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME
                sleep 10

        fi

        TRIES=`expr $TRIES + 1`

        echo "Num of attemps = $TRIES"

        if [ $TRIES -gt 3 ]
        then
            echo "$DOWNLOAD_FILE_NAME download failed after $TRIES attemps. Can't book pace adjustments for Chuck Ames. Pls look into it .." | mail ${FAIL_MAIL_LIST}
            break
        fi

done

Open in new window

0
Comment
Question by:d27m11y
  • 3
  • 2
6 Comments
 

Author Comment

by:d27m11y
ID: 39194699
Does this work

curl -z "Jun 30 2011" http://yoursite.com/file.html

Open in new window

0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 500 total points
ID: 39196039
curl?  rsync is a MUCH better tool for the job. It not only can restart,but can be told to compress, sync only changed files, use wild cards.    All built-in command lines.  rsync is on every O/S that supports curl, even windows.  (yes, you can pass logon credentials as part of the command also.
0
 
LVL 22

Expert Comment

by:blu
ID: 39196562
Rsync is great unless you don't have access to the server except via http. Is that the case here?

I am not exactly sure what you are asking. Could you elaborate? Are you asking to have curl figure out the latest file in a directory? If so, can you give a typical directory listing and what you want to happen?
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 

Author Comment

by:d27m11y
ID: 39207962
let us say I have a directory

export/home/opdu/ods/data/csv

In this directory, I have files listed as the following

missing_trades_0523.csv
missing_trades_0524.csv
missing_trades_0525.csv

How do I get the latest file based on the name of the file.

I would like to pass latest .csv file as parameter to my shell. How do I do that, please suggest.

Quick response is appreciated.
0
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
ID: 39208340
You only have one version of any particular file name as UNIX doesn't have revisioning.  Seems to me the best and easiest solution is to modify the program that generates these files so that after it creates the new missing_trades_$NNNN.csv file to
cp -f missing_trades_$NNNN.csv missing_trades_current.csv  

Then you just use whatever tool you want to always copy missing_trades_current.csv

Think outside of the box and make life easy for yourself and redo the logic to get rid of the problem altogether.

Otherwise you'd have to write a script that does ls -1 $directory/missing_trades* | sort -g -r | head - 1

(Not all flavors of unix use -g -r to get a numeric sort (-g) and reverse sort (-r)
0
 

Author Closing Comment

by:d27m11y
ID: 39216188
Useful !
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Recently, an awarded photographer, Selina De Maeyer (http://www.selinademaeyer.com/), completed a photo shoot of a beautiful event (http://www.sintjacobantwerpen.be/verslag-en-fotoreportage-van-de-sacramentsprocessie-door-antwerpen#thumbnails) in An…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

759 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now