Solved

Linux Shell Scripting: Execute curl in loop

Posted on 2013-11-08
8
3,531 Views
Last Modified: 2013-11-08
This does what I want however except it echos the commands.  Instead of using echo I want to execute the commands.
for f in *; \
do echo "curl -kX http://example.com/";
done

Open in new window

0
Comment
Question by:hankknight
  • 5
  • 3
8 Comments
 
LVL 19

Expert Comment

by:simon3270
Comment Utility
Just take out the echo and the double quotes.
for f in *
do curl -kX http://example.com/
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
Comment Utility
I've also taken out the semicolons at the ends of lines - they are not needed, and force you to have trailing backslashes in some cases.  In shell, ";" is a command separator (if you have multiple commands on one line), not a terminator.

You can still use them - for example, your loop can be written
for f in *; do curl -kX http://example.com/; done

Open in new window

0
 
LVL 16

Author Comment

by:hankknight
Comment Utility
Thanks but it still does not work when I integrate it with the rest of my code:
u="https://zuhqtr5.couchappy.com/site/"; \
w="www/";
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *; \
do mime=`file -i $f`;r=`curl -sk '$u""$w' | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`;curl -kX PUT $u""$w/$f""?rev=$r --data-binary @$f -H \"Content-Type: $mime\"; \
done

Open in new window

{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
0
 
LVL 19

Accepted Solution

by:
simon3270 earned 500 total points
Comment Utility
it's quotes again - you have single quotes round the $u and $w parameters for the first curl command.

I've split it up into separate lines to make it easier to read, and remove the unwanted semicolons:
u="https://zuhqtr5.couchappy.com/site/"
w="www/"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -i $f`
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
  echo For ${u}${w}/${f} getting rev $r with mime type $mime
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
done

Open in new window


I've used the ${u} construct to get the value of $u - the curly braces don't change the value, but they do ensure delimit the variable name.  If, for example, you wanted the value of $u immediately followed by "www", and you wrote
    $uwww
you would get the value of the uwww variable (which probably isn't set).  You should instead write
    ${u}www
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 16

Author Comment

by:hankknight
Comment Utility
That still errors out:
 u="https://zuhqtr5.couchappy.com/site/"
$ w="www/"
$ curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
{"error":"conflict","reason":"Document update conflict."}
$ for f in *
> do 
>   mime=`file -i $f`
>   r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
>   curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
> done
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'

Open in new window

The URLs are real so you should be able to test it yourself it you want to.  Thanks.
0
 
LVL 19

Expert Comment

by:simon3270
Comment Utility
I'm not getting mush useful data back (I added that debug "echo" statement so we could see what was being done).

In the "r=`curl..." line,
     curl -sk ${u}${w} | sed -e 's/[{}]/''/g'
returns
    "error":"not_found","reason":"deleted"
so the "awk" statement after that doesn't find any "_rev" text, so doesn't print anything out.  That means that the "r" variable is empty, which messes up the next line.  Do you need to specify the file name in this curl request, so that you get that file's rev number?

The other problem with the curl PUT line is the -H - the double-quotes after that are escaped, so the shell doesn't use them to wrap the subsequent words as the single value for "-H".  The separate parts of what you want as the "-H" parameter are then passed to curl as files to be "PUT", which is why it says it can't find files called "hello.txt:" (note the trailing colon), "text" and "charset=us-ascii".  As far as I can tell, you want those quotes to wrap the parts of the "-H" parameter, and not to appear in the value itself.  To do that, just remove the backslash characters.

There's also a problem with the "mime" variable - it still contains the filename. To avoid this, add the "-b" option to the "file" command.

I've added a file called aa.txt (containing the single line "text file") to the URL you gave, and if I add "aa.txt" to the end of the line, I get that text back, but I can't work out a valid rev format.  What would that be?
0
 
LVL 16

Author Comment

by:hankknight
Comment Utility
Turns out that the quotation marks in the revision were causing one problem and the filename included with the mime type caused another problem.

Thanks, simon3270, for all your help with this.  It now works perfectly!

u="https://zuhqtr5.couchappy.com/site/"
w="www"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -ib $f`; echo $mime;
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}' | tr -d "'"'"'`
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H "Content-Type:  $mime"
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
Comment Utility
My pleasure, and thanks for the points!
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now