Solved

Linux Shell Scripting: Execute curl in loop

Posted on 2013-11-08
8
4,307 Views
Last Modified: 2013-11-08
This does what I want however except it echos the commands.  Instead of using echo I want to execute the commands.
for f in *; \
do echo "curl -kX http://example.com/";
done

Open in new window

0
Comment
Question by:hankknight
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 3
8 Comments
 
LVL 19

Expert Comment

by:simon3270
ID: 39633498
Just take out the echo and the double quotes.
for f in *
do curl -kX http://example.com/
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
ID: 39633501
I've also taken out the semicolons at the ends of lines - they are not needed, and force you to have trailing backslashes in some cases.  In shell, ";" is a command separator (if you have multiple commands on one line), not a terminator.

You can still use them - for example, your loop can be written
for f in *; do curl -kX http://example.com/; done

Open in new window

0
 
LVL 16

Author Comment

by:hankknight
ID: 39633531
Thanks but it still does not work when I integrate it with the rest of my code:
u="https://zuhqtr5.couchappy.com/site/"; \
w="www/";
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *; \
do mime=`file -i $f`;r=`curl -sk '$u""$w' | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`;curl -kX PUT $u""$w/$f""?rev=$r --data-binary @$f -H \"Content-Type: $mime\"; \
done

Open in new window

{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
0
Don't Cry: How Liquid Web is Ensuring Security

WannaCry is just the start. Read how Liquid Web is protecting itself and its customers against new threats.

 
LVL 19

Accepted Solution

by:
simon3270 earned 500 total points
ID: 39633619
it's quotes again - you have single quotes round the $u and $w parameters for the first curl command.

I've split it up into separate lines to make it easier to read, and remove the unwanted semicolons:
u="https://zuhqtr5.couchappy.com/site/"
w="www/"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -i $f`
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
  echo For ${u}${w}/${f} getting rev $r with mime type $mime
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
done

Open in new window


I've used the ${u} construct to get the value of $u - the curly braces don't change the value, but they do ensure delimit the variable name.  If, for example, you wanted the value of $u immediately followed by "www", and you wrote
    $uwww
you would get the value of the uwww variable (which probably isn't set).  You should instead write
    ${u}www
0
 
LVL 16

Author Comment

by:hankknight
ID: 39633801
That still errors out:
 u="https://zuhqtr5.couchappy.com/site/"
$ w="www/"
$ curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
{"error":"conflict","reason":"Document update conflict."}
$ for f in *
> do 
>   mime=`file -i $f`
>   r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
>   curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
> done
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'

Open in new window

The URLs are real so you should be able to test it yourself it you want to.  Thanks.
0
 
LVL 19

Expert Comment

by:simon3270
ID: 39633933
I'm not getting mush useful data back (I added that debug "echo" statement so we could see what was being done).

In the "r=`curl..." line,
     curl -sk ${u}${w} | sed -e 's/[{}]/''/g'
returns
    "error":"not_found","reason":"deleted"
so the "awk" statement after that doesn't find any "_rev" text, so doesn't print anything out.  That means that the "r" variable is empty, which messes up the next line.  Do you need to specify the file name in this curl request, so that you get that file's rev number?

The other problem with the curl PUT line is the -H - the double-quotes after that are escaped, so the shell doesn't use them to wrap the subsequent words as the single value for "-H".  The separate parts of what you want as the "-H" parameter are then passed to curl as files to be "PUT", which is why it says it can't find files called "hello.txt:" (note the trailing colon), "text" and "charset=us-ascii".  As far as I can tell, you want those quotes to wrap the parts of the "-H" parameter, and not to appear in the value itself.  To do that, just remove the backslash characters.

There's also a problem with the "mime" variable - it still contains the filename. To avoid this, add the "-b" option to the "file" command.

I've added a file called aa.txt (containing the single line "text file") to the URL you gave, and if I add "aa.txt" to the end of the line, I get that text back, but I can't work out a valid rev format.  What would that be?
0
 
LVL 16

Author Comment

by:hankknight
ID: 39634011
Turns out that the quotation marks in the revision were causing one problem and the filename included with the mime type caused another problem.

Thanks, simon3270, for all your help with this.  It now works perfectly!

u="https://zuhqtr5.couchappy.com/site/"
w="www"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -ib $f`; echo $mime;
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}' | tr -d "'"'"'`
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H "Content-Type:  $mime"
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
ID: 39634112
My pleasure, and thanks for the points!
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The purpose of this article is to demonstrate how we can use conditional statements using Python.
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question