Solved

Linux Shell Scripting: Execute curl in loop

Posted on 2013-11-08
8
4,108 Views
Last Modified: 2013-11-08
This does what I want however except it echos the commands.  Instead of using echo I want to execute the commands.
for f in *; \
do echo "curl -kX http://example.com/";
done

Open in new window

0
Comment
Question by:hankknight
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 3
8 Comments
 
LVL 19

Expert Comment

by:simon3270
ID: 39633498
Just take out the echo and the double quotes.
for f in *
do curl -kX http://example.com/
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
ID: 39633501
I've also taken out the semicolons at the ends of lines - they are not needed, and force you to have trailing backslashes in some cases.  In shell, ";" is a command separator (if you have multiple commands on one line), not a terminator.

You can still use them - for example, your loop can be written
for f in *; do curl -kX http://example.com/; done

Open in new window

0
 
LVL 16

Author Comment

by:hankknight
ID: 39633531
Thanks but it still does not work when I integrate it with the rest of my code:
u="https://zuhqtr5.couchappy.com/site/"; \
w="www/";
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *; \
do mime=`file -i $f`;r=`curl -sk '$u""$w' | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`;curl -kX PUT $u""$w/$f""?rev=$r --data-binary @$f -H \"Content-Type: $mime\"; \
done

Open in new window

{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
0
Back Up Your Microsoft Windows Server®

Back up all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 19

Accepted Solution

by:
simon3270 earned 500 total points
ID: 39633619
it's quotes again - you have single quotes round the $u and $w parameters for the first curl command.

I've split it up into separate lines to make it easier to read, and remove the unwanted semicolons:
u="https://zuhqtr5.couchappy.com/site/"
w="www/"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -i $f`
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
  echo For ${u}${w}/${f} getting rev $r with mime type $mime
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
done

Open in new window


I've used the ${u} construct to get the value of $u - the curly braces don't change the value, but they do ensure delimit the variable name.  If, for example, you wanted the value of $u immediately followed by "www", and you wrote
    $uwww
you would get the value of the uwww variable (which probably isn't set).  You should instead write
    ${u}www
0
 
LVL 16

Author Comment

by:hankknight
ID: 39633801
That still errors out:
 u="https://zuhqtr5.couchappy.com/site/"
$ w="www/"
$ curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
{"error":"conflict","reason":"Document update conflict."}
$ for f in *
> do 
>   mime=`file -i $f`
>   r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}'`
>   curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H \"Content-Type: $mime\"
> done
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'hello.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'
{"error":"bad_request","reason":"Invalid rev format"}
{"error":"bad_request","reason":"Invalid rev format"}
curl: (6) Couldn't resolve host 'zzz.txt:'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'charset=us-ascii"'

Open in new window

The URLs are real so you should be able to test it yourself it you want to.  Thanks.
0
 
LVL 19

Expert Comment

by:simon3270
ID: 39633933
I'm not getting mush useful data back (I added that debug "echo" statement so we could see what was being done).

In the "r=`curl..." line,
     curl -sk ${u}${w} | sed -e 's/[{}]/''/g'
returns
    "error":"not_found","reason":"deleted"
so the "awk" statement after that doesn't find any "_rev" text, so doesn't print anything out.  That means that the "r" variable is empty, which messes up the next line.  Do you need to specify the file name in this curl request, so that you get that file's rev number?

The other problem with the curl PUT line is the -H - the double-quotes after that are escaped, so the shell doesn't use them to wrap the subsequent words as the single value for "-H".  The separate parts of what you want as the "-H" parameter are then passed to curl as files to be "PUT", which is why it says it can't find files called "hello.txt:" (note the trailing colon), "text" and "charset=us-ascii".  As far as I can tell, you want those quotes to wrap the parts of the "-H" parameter, and not to appear in the value itself.  To do that, just remove the backslash characters.

There's also a problem with the "mime" variable - it still contains the filename. To avoid this, add the "-b" option to the "file" command.

I've added a file called aa.txt (containing the single line "text file") to the URL you gave, and if I add "aa.txt" to the end of the line, I get that text back, but I can't work out a valid rev format.  What would that be?
0
 
LVL 16

Author Comment

by:hankknight
ID: 39634011
Turns out that the quotation marks in the revision were causing one problem and the filename included with the mime type caused another problem.

Thanks, simon3270, for all your help with this.  It now works perfectly!

u="https://zuhqtr5.couchappy.com/site/"
w="www"
curl -kX POST $u -H "Content-type:application/json" -d '{"_id": "'$w'"}'
for f in *
do 
  mime=`file -ib $f`; echo $mime;
  r=`curl -sk ${u}${w} | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^_rev/ {print $2}' | tr -d "'"'"'`
  curl -kX PUT ${u}${w}/${f}?rev=$r --data-binary @$f -H "Content-Type:  $mime"
done

Open in new window

0
 
LVL 19

Expert Comment

by:simon3270
ID: 39634112
My pleasure, and thanks for the points!
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
RHEL installation problems on HP Proliant DL 360 G6 server 20 148
Certificate Request CentOS/Apache 1 57
Linux kernel panic ext3-fs error 14 75
Krita v3 Linux Mint/Ubuntu 16.04 9 74
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question