Solved

curl in unix

Posted on 2013-05-24
6
502 Views
Last Modified: 2013-06-03
I have pasted partial code here, but you may notice down below "Curl" is being used to download the file. I am just wondering if I can use "curl" to download latest file by date. Please help

#!/bin/ksh -xve
date

HOME_DIR=$tradehome/bmc_jobs/pace_adjustments
CUR_DATE=`date '+%Y_%m_%d'`
LOG_FILE=$HOME_DIR/logs/pace_adjustments.log
#DOWNLOAD_FILE_NAME=icecleared_gas_$CUR_DATE.dat
DOWNLOAD_FILE_NAME=icecleared_gas_2013_03_14.dat
USER_NAME=Caxton_settles
PASSWORD=icedata
HTTP_URL=https://downloads.theice.com/Settlement_Reports_CSV/Gas/$DOWNLOAD_FILE_NAME
DATA_FILE_DIR=$HOME_DIR/data
TRIES=0
MAIL_LIST=smaddirala@caxton.com
CTL_FILE=$HOME_DIR/ctl/ice_prices.ctl
SQL_LOADER_LOG_FILE=$HOME_DIR/sql_loader_log/ice_prices.log
DATA_FILE=$DATA_FILE_DIR/icecleared_gas.dat
badfname=$HOME_DIR/sql_loader_log/ice_prices.bad
dscfname=$HOME_DIR/sql_loader_log/ice_prices.dsc
logfname=$HOME_DIR/sql_loader_log/ice_prices.log


echo "Down Load File = $DOWNLOAD_FILE_NAME"
echo "URL = $HTTP_URL"


cd $HOME_DIR

export USER_NAME PASSWORD HTTP_URL

#
# Get the file from ICE
if (test -e "$DATA_FILE_DIR/$DOWNLOAD_FILE_NAME" )
then
   rm $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME
fi
sleep 30


until
   [ -f "$DATA_FILE_DIR/$DOWNLOAD_FILE_NAME" ]
do
   sleep 10
   date
   echo 'Attempt to download $$DOWNLOAD_FILE_NAME from ICE' `date`
        curl -k -u $USER_NAME:$PASSWORD $HTTP_URL > $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME

        CNT=`grep -i "<title>404 Not Found</title>" $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME | wc -l`

        echo "Not Found Count = $CNT"

        if [ $CNT -gt 0 ]
        then
                echo "Delete the bad file and try again"
                rm $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME
                sleep 10

        fi

        TRIES=`expr $TRIES + 1`

        echo "Num of attemps = $TRIES"

        if [ $TRIES -gt 3 ]
        then
            echo "$DOWNLOAD_FILE_NAME download failed after $TRIES attemps. Can't book pace adjustments for Chuck Ames. Pls look into it .." | mail ${FAIL_MAIL_LIST}
            break
        fi

done

Open in new window

0
Comment
Question by:d27m11y
  • 3
  • 2
6 Comments
 

Author Comment

by:d27m11y
ID: 39194699
Does this work

curl -z "Jun 30 2011" http://yoursite.com/file.html

Open in new window

0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 500 total points
ID: 39196039
curl?  rsync is a MUCH better tool for the job. It not only can restart,but can be told to compress, sync only changed files, use wild cards.    All built-in command lines.  rsync is on every O/S that supports curl, even windows.  (yes, you can pass logon credentials as part of the command also.
0
 
LVL 22

Expert Comment

by:blu
ID: 39196562
Rsync is great unless you don't have access to the server except via http. Is that the case here?

I am not exactly sure what you are asking. Could you elaborate? Are you asking to have curl figure out the latest file in a directory? If so, can you give a typical directory listing and what you want to happen?
0
Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

 

Author Comment

by:d27m11y
ID: 39207962
let us say I have a directory

export/home/opdu/ods/data/csv

In this directory, I have files listed as the following

missing_trades_0523.csv
missing_trades_0524.csv
missing_trades_0525.csv

How do I get the latest file based on the name of the file.

I would like to pass latest .csv file as parameter to my shell. How do I do that, please suggest.

Quick response is appreciated.
0
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
ID: 39208340
You only have one version of any particular file name as UNIX doesn't have revisioning.  Seems to me the best and easiest solution is to modify the program that generates these files so that after it creates the new missing_trades_$NNNN.csv file to
cp -f missing_trades_$NNNN.csv missing_trades_current.csv  

Then you just use whatever tool you want to always copy missing_trades_current.csv

Think outside of the box and make life easy for yourself and redo the logic to get rid of the problem altogether.

Otherwise you'd have to write a script that does ls -1 $directory/missing_trades* | sort -g -r | head - 1

(Not all flavors of unix use -g -r to get a numeric sort (-g) and reverse sort (-r)
0
 

Author Closing Comment

by:d27m11y
ID: 39216188
Useful !
0

Featured Post

Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now