Linux Shell Scripting: Execute curl in loop

This does what I want however except it echos the commands.  Instead of using echo I want to execute the commands.
for f in *; \
do echo "curl -kX http://example.com/";
done

Open in new window

LVL 16
hankknightAsked:
Who is Participating?
 
simon3270Commented:
it's quotes again - you have single quotes round the $u and $w parameters for the first curl command.

I've split it up into separate lines to make it easier to read, and remove the unwanted semicolons:
u="https://zuhqtr5.couchappy.com/site/"
w="www/"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -i $f`
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
  echo For ${u}${w}/${f} getting rev $r with mime type $mime
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
done

Open in new window


I've used the ${u} construct to get the value of $u - the curly braces don't change the value, but they do ensure delimit the variable name.  If, for example, you wanted the value of $u immediately followed by "www", and you wrote
    $uwww
you would get the value of the uwww variable (which probably isn't set).  You should instead write
    ${u}www
0
 
simon3270Commented:
Just take out the echo and the double quotes.
for f in *
do curl -kX http://example.com/
done

Open in new window

0
 
simon3270Commented:
I've also taken out the semicolons at the ends of lines - they are not needed, and force you to have trailing backslashes in some cases.  In shell, ";" is a command separator (if you have multiple commands on one line), not a terminator.

You can still use them - for example, your loop can be written
for f in *; do curl -kX http://example.com/; done

Open in new window

0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
hankknightAuthor Commented:
Thanks but it still does not work when I integrate it with the rest of my code:
u="https://zuhqtr5.couchappy.com/site/"; \
w="www/";
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *; \
do mime=`file -i $f`;r=`curl -sk '$u""$w' | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`;curl -kX PUT $u""$w/$f""?rev=$r --data-binary @$f -H \"Content-Type: $mime\"; \
done

Open in new window

{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
0
 
hankknightAuthor Commented:
That still errors out:
 u="https://zuhqtr5.couchappy.com/site/"
$ w="www/"
$ curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
{"error":"conflict","reason":"Document update conflict."}
$ for f in *
> do 
>   mime=`file -i $f`
>   r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
>   curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
> done
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'

Open in new window

The URLs are real so you should be able to test it yourself it you want to.  Thanks.
0
 
simon3270Commented:
I'm not getting mush useful data back (I added that debug "echo" statement so we could see what was being done).

In the "r=`curl..." line,
     curl -sk ${u}${w} | sed -e 's/[{}]/''/g'
returns
    "error":"not_found","reason":"deleted"
so the "awk" statement after that doesn't find any "_rev" text, so doesn't print anything out.  That means that the "r" variable is empty, which messes up the next line.  Do you need to specify the file name in this curl request, so that you get that file's rev number?

The other problem with the curl PUT line is the -H - the double-quotes after that are escaped, so the shell doesn't use them to wrap the subsequent words as the single value for "-H".  The separate parts of what you want as the "-H" parameter are then passed to curl as files to be "PUT", which is why it says it can't find files called "hello.txt:" (note the trailing colon), "text" and "charset=us-ascii".  As far as I can tell, you want those quotes to wrap the parts of the "-H" parameter, and not to appear in the value itself.  To do that, just remove the backslash characters.

There's also a problem with the "mime" variable - it still contains the filename. To avoid this, add the "-b" option to the "file" command.

I've added a file called aa.txt (containing the single line "text file") to the URL you gave, and if I add "aa.txt" to the end of the line, I get that text back, but I can't work out a valid rev format.  What would that be?
0
 
hankknightAuthor Commented:
Turns out that the quotation marks in the revision were causing one problem and the filename included with the mime type caused another problem.

Thanks, simon3270, for all your help with this.  It now works perfectly!

u="https://zuhqtr5.couchappy.com/site/"
w="www"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -ib $f`; echo $mime;
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}' | tr -d "'"'"'`
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H "Content-Type:  $mime"
done

Open in new window

0
 
simon3270Commented:
My pleasure, and thanks for the points!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.