Link to home
Start Free TrialLog in
Avatar of Jason Yu
Jason YuFlag for United States of America

asked on

how to use curl to read file names from a column in excel file

Dear experts:

I got a task to script by using "curl" command to download hundreds of image files from a file server.

The command format is as following:

curl -o 77711-79137-p01p.png http://s7d3.scene7.com/is/image/companyname/77711-79137-p01p

where
— "http://s7d3.scene7.com/is/image/companyname/“ is common prefix
— 77711-79137-p01p is the image name
— “.png” is the postfix

The image names are from one column of an excel file ended with .xlsx, please see attachment. How could I program for this task?

Thanks.
file-name-input.xlsx
Avatar of Dave Baldwin
Dave Baldwin
Flag of United States of America image

'xlsx' files are compressed and probably not readable by cURL.  If you export the file names to a text file, you could use a batch file or script to read the names and fetch the files.
Avatar of Jason Yu

ASKER

HI, Dave:

Thanks for your quick respond. I have made a script like below, but got "curl: (3) Illegal characters found in URL" error,  could you please kindly take a look?


#!/bin/bash
    FILE="./shortname.txt"
    OUT=$(awk '{ print $1 }' $FILE)
        echo "*** File Name Are Here****"
        for file_name in $OUT
        do
          curl -o $file_name.png http://s7d3.scene7.com/is/image/AveryDennison/$file_name
        done

Open in new window

How about wget?

wget -i text_file_with_urls
how about this script:

#!/bin/bash
 2     FILE="./shortname.txt"
 3     OUT=$(awk '{ print $1 }' $FILE)
 4     echo "*** File Name Are Here****"
 5     for file_name in $OUT
 6     do
 7
 8     # echo "$file_name"
 9     wget -i http://s7d3.scene7.com/is/image/AveryDennison/$file_name
10     #  curl -o echo "$file_name".png http://s7d3.scene7.com/is/image/AveryDennison/$file_name
11     done
It looks like it doesn't support:

Warning: wildcards not supported in HTTP.
The name is too long, 245 chars total.
Trying to shorten...
New name is ▒%97▒▒}▒%1D▒b%1F'a%0C%078▒%82z▒z▒+r%95I▒%82%9DI9>▒▒▒▒mJP▒w8S%82%8A▒▒+%11YI▒.▒▒▒%07%0D%04%90B%99▒F▒%19R▒%86%9F5%1D▒▒▒^▒0%90%1Ec▒▒▒!%05▒%1D%80>*%04%01%80▒%8A`▒N<3▒%02▒c▒}F%0F▒▒|U_▒▒▒%06$▒▒N~▒WʵYz%88▒s8%17>▒_\%88▒T▒▒%D▒▒%14n%96i%1C%18▒▒.
The name is too long, 245 chars total.
Trying to shorten...
New name is ▒%97▒▒}▒%1D▒b%1F'a%0C%078▒%82z▒z▒+r%95I▒%82%9DI9>▒▒▒▒mJP▒w8S%82%8A▒▒+%11YI▒.▒▒▒%07%0D%04%90B%99▒F▒%19R▒%86%9F5%1D▒▒▒^▒0%90%1Ec▒▒▒!%05▒%1D%80>*%04%01%80▒%8A`▒N<3▒%02▒c▒}F%0F▒▒|U_▒▒▒%06$▒▒N~▒WʵYz%88▒s8%17>▒_\%88▒T▒▒%D▒▒%14n%96i%1C%18▒▒.
--2016-06-09 17:30:51--  http://s7d3.scene7.com/is/image/AveryDennison/%C3%97%C1%CB%7D%A4%1D%D3b%1F'a%0C%078%EB%82z%EEz%FD+r%95I%D5%82%9DI9%3E%B6%F2%FD%AC%E3mJP%A1w8S%82%8A%D1%E2+%11YI%BC.%AC%B3%EE%07%0D%04%90B%99%DDF%D8%19R%E0%86%9F5%1D%B6%C8%C0%5E%E70%90%1Ec%B9%DF%CC!%05%C4%1D%80%3E*%04%01%80%EE%BE%8A%60%E0N%3C3%B2%02%A5c%C0%7DF%0F%EA%F8%7CU_%AA%A3%D8%06$%A9%FEN~%D5W%CA%B5Yz%88%FDs8%17%3E%B6_%5C%88%A6T%D0%C1%25D%D1%C3%14n%96i%1C%18%C8%D8%D2%5C%E7%13%80
Reusing existing connection to s7d3.scene7.com:80.
HTTP request sent, awaiting response... 400 Bad Request
2016-06-09 17:30:51 ERROR 400: Bad Request.
How can i use the file name inside "curl" command grepped from the txt file?

curl -o $file_name.png http://s7d3.scene7.com/is/image/AveryDennison/$file_name
ASKER CERTIFIED SOLUTION
Avatar of Jason Yu
Jason Yu
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I figured it out myself