Solved

Linux Shell Scripting: Execute curl in loop

Posted on 2013-11-08
8
3,724 Views
Last Modified: 2013-11-08
This does what I want however except it echos the commands.  Instead of using echo I want to execute the commands.
for f in *; \
do echo "curl -kX http://example.com/";
done

Open in new window

0
Comment
Question by:hankknight
  • 5
  • 3
8 Comments
 
LVL 19

Expert Comment

by:simon3270
ID: 39633498
Just take out the echo and the double quotes.
for f in *
do curl -kX http://example.com/
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
ID: 39633501
I've also taken out the semicolons at the ends of lines - they are not needed, and force you to have trailing backslashes in some cases.  In shell, ";" is a command separator (if you have multiple commands on one line), not a terminator.

You can still use them - for example, your loop can be written
for f in *; do curl -kX http://example.com/; done

Open in new window

0
 
LVL 16

Author Comment

by:hankknight
ID: 39633531
Thanks but it still does not work when I integrate it with the rest of my code:
u="https://zuhqtr5.couchappy.com/site/"; \
w="www/";
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *; \
do mime=`file -i $f`;r=`curl -sk '$u""$w' | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`;curl -kX PUT $u""$w/$f""?rev=$r --data-binary @$f -H \"Content-Type: $mime\"; \
done

Open in new window

{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
0
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 
LVL 19

Accepted Solution

by:
simon3270 earned 500 total points
ID: 39633619
it's quotes again - you have single quotes round the $u and $w parameters for the first curl command.

I've split it up into separate lines to make it easier to read, and remove the unwanted semicolons:
u="https://zuhqtr5.couchappy.com/site/"
w="www/"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -i $f`
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
  echo For ${u}${w}/${f} getting rev $r with mime type $mime
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
done

Open in new window


I've used the ${u} construct to get the value of $u - the curly braces don't change the value, but they do ensure delimit the variable name.  If, for example, you wanted the value of $u immediately followed by "www", and you wrote
    $uwww
you would get the value of the uwww variable (which probably isn't set).  You should instead write
    ${u}www
0
 
LVL 16

Author Comment

by:hankknight
ID: 39633801
That still errors out:
 u="https://zuhqtr5.couchappy.com/site/"
$ w="www/"
$ curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
{"error":"conflict","reason":"Document update conflict."}
$ for f in *
> do 
>   mime=`file -i $f`
>   r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
>   curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
> done
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'

Open in new window

The URLs are real so you should be able to test it yourself it you want to.  Thanks.
0
 
LVL 19

Expert Comment

by:simon3270
ID: 39633933
I'm not getting mush useful data back (I added that debug "echo" statement so we could see what was being done).

In the "r=`curl..." line,
     curl -sk ${u}${w} | sed -e 's/[{}]/''/g'
returns
    "error":"not_found","reason":"deleted"
so the "awk" statement after that doesn't find any "_rev" text, so doesn't print anything out.  That means that the "r" variable is empty, which messes up the next line.  Do you need to specify the file name in this curl request, so that you get that file's rev number?

The other problem with the curl PUT line is the -H - the double-quotes after that are escaped, so the shell doesn't use them to wrap the subsequent words as the single value for "-H".  The separate parts of what you want as the "-H" parameter are then passed to curl as files to be "PUT", which is why it says it can't find files called "hello.txt:" (note the trailing colon), "text" and "charset=us-ascii".  As far as I can tell, you want those quotes to wrap the parts of the "-H" parameter, and not to appear in the value itself.  To do that, just remove the backslash characters.

There's also a problem with the "mime" variable - it still contains the filename. To avoid this, add the "-b" option to the "file" command.

I've added a file called aa.txt (containing the single line "text file") to the URL you gave, and if I add "aa.txt" to the end of the line, I get that text back, but I can't work out a valid rev format.  What would that be?
0
 
LVL 16

Author Comment

by:hankknight
ID: 39634011
Turns out that the quotation marks in the revision were causing one problem and the filename included with the mime type caused another problem.

Thanks, simon3270, for all your help with this.  It now works perfectly!

u="https://zuhqtr5.couchappy.com/site/"
w="www"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -ib $f`; echo $mime;
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}' | tr -d "'"'"'`
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H "Content-Type:  $mime"
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
ID: 39634112
My pleasure, and thanks for the points!
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

816 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now