Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

curl in unix

Posted on 2013-05-24
6
Medium Priority
?
532 Views
Last Modified: 2013-06-03
I have pasted partial code here, but you may notice down below "Curl" is being used to download the file. I am just wondering if I can use "curl" to download latest file by date. Please help

#!/bin/ksh -xve
date

HOME_DIR=$tradehome/bmc_jobs/pace_adjustments
CUR_DATE=`date '+%Y_%m_%d'`
LOG_FILE=$HOME_DIR/logs/pace_adjustments.log
#DOWNLOAD_FILE_NAME=icecleared_gas_$CUR_DATE.dat
DOWNLOAD_FILE_NAME=icecleared_gas_2013_03_14.dat
USER_NAME=Caxton_settles
PASSWORD=icedata
HTTP_URL=https://downloads.theice.com/Settlement_Reports_CSV/Gas/$DOWNLOAD_FILE_NAME
DATA_FILE_DIR=$HOME_DIR/data
TRIES=0
MAIL_LIST=smaddirala@caxton.com
CTL_FILE=$HOME_DIR/ctl/ice_prices.ctl
SQL_LOADER_LOG_FILE=$HOME_DIR/sql_loader_log/ice_prices.log
DATA_FILE=$DATA_FILE_DIR/icecleared_gas.dat
badfname=$HOME_DIR/sql_loader_log/ice_prices.bad
dscfname=$HOME_DIR/sql_loader_log/ice_prices.dsc
logfname=$HOME_DIR/sql_loader_log/ice_prices.log


echo "Down Load File = $DOWNLOAD_FILE_NAME"
echo "URL = $HTTP_URL"


cd $HOME_DIR

export USER_NAME PASSWORD HTTP_URL

#
# Get the file from ICE
if (test -e "$DATA_FILE_DIR/$DOWNLOAD_FILE_NAME" )
then
   rm $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME
fi
sleep 30


until
   [ -f "$DATA_FILE_DIR/$DOWNLOAD_FILE_NAME" ]
do
   sleep 10
   date
   echo 'Attempt to download $$DOWNLOAD_FILE_NAME from ICE' `date`
        curl -k -u $USER_NAME:$PASSWORD $HTTP_URL > $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME

        CNT=`grep -i "<title>404 Not Found</title>" $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME | wc -l`

        echo "Not Found Count = $CNT"

        if [ $CNT -gt 0 ]
        then
                echo "Delete the bad file and try again"
                rm $DATA_FILE_DIR/$DOWNLOAD_FILE_NAME
                sleep 10

        fi

        TRIES=`expr $TRIES + 1`

        echo "Num of attemps = $TRIES"

        if [ $TRIES -gt 3 ]
        then
            echo "$DOWNLOAD_FILE_NAME download failed after $TRIES attemps. Can't book pace adjustments for Chuck Ames. Pls look into it .." | mail ${FAIL_MAIL_LIST}
            break
        fi

done

Open in new window

0
Comment
Question by:d27m11y
  • 3
  • 2
6 Comments
 

Author Comment

by:d27m11y
ID: 39194699
Does this work

curl -z "Jun 30 2011" http://yoursite.com/file.html

Open in new window

0
 
LVL 47

Assisted Solution

by:David
David earned 1000 total points
ID: 39196039
curl?  rsync is a MUCH better tool for the job. It not only can restart,but can be told to compress, sync only changed files, use wild cards.    All built-in command lines.  rsync is on every O/S that supports curl, even windows.  (yes, you can pass logon credentials as part of the command also.
0
 
LVL 22

Expert Comment

by:Brian Utterback
ID: 39196562
Rsync is great unless you don't have access to the server except via http. Is that the case here?

I am not exactly sure what you are asking. Could you elaborate? Are you asking to have curl figure out the latest file in a directory? If so, can you give a typical directory listing and what you want to happen?
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Comment

by:d27m11y
ID: 39207962
let us say I have a directory

export/home/opdu/ods/data/csv

In this directory, I have files listed as the following

missing_trades_0523.csv
missing_trades_0524.csv
missing_trades_0525.csv

How do I get the latest file based on the name of the file.

I would like to pass latest .csv file as parameter to my shell. How do I do that, please suggest.

Quick response is appreciated.
0
 
LVL 47

Accepted Solution

by:
David earned 1000 total points
ID: 39208340
You only have one version of any particular file name as UNIX doesn't have revisioning.  Seems to me the best and easiest solution is to modify the program that generates these files so that after it creates the new missing_trades_$NNNN.csv file to
cp -f missing_trades_$NNNN.csv missing_trades_current.csv  

Then you just use whatever tool you want to always copy missing_trades_current.csv

Think outside of the box and make life easy for yourself and redo the logic to get rid of the problem altogether.

Otherwise you'd have to write a script that does ls -1 $directory/missing_trades* | sort -g -r | head - 1

(Not all flavors of unix use -g -r to get a numeric sort (-g) and reverse sort (-r)
0
 

Author Closing Comment

by:d27m11y
ID: 39216188
Useful !
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machine…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses

581 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question