[Webinar] Streamline your web hosting managementRegister Today

x
?
Solved

Execute multiple curl cmds with sleep and send output to file

Posted on 2016-08-16
10
Medium Priority
?
207 Views
Last Modified: 2016-08-16
First off, I am, in no way a Unix/Linux expert, so please bear with me. Also, I sure hope I've selected the proper topics for this question...

I need to run a few thousand curl cmds but I'm having difficulties figuring out how to go about doing this.

1. The curl is an XPOST (e.g. JSON)
For example:
curl -XPOST -H 'name1: value1' -H 'name2: value2' -H "Content-type: application/json" -d '{ "name3": [{"blah1": "blah1","aaa1": "blah2"}]}' 'http://domain.com'
curl -XPOST -H 'name1: value1' -H 'name2: value2' -H "Content-type: application/json" -d '{ "name3": [{"blah2": "blah2","aaa2": "blah2"}]}' 'http://domain.com'
curl -XPOST -H 'name1: value1' -H 'name2: value2' -H "Content-type: application/json" -d '{ "name3": [{"blah3": "blah3","aaa3": "blah2"}]}' 'http://domain.com'

The thing is I need to run this in a script so that there is a delay between each one. Also, once 1 curl is run, I get back a result which I'd like to save.

I thought I could just load all of the curl cmds in a text file and run something like:

while read line; do $line; sleep 5; done < input_curl.txt >> output_curl.txt

The resulting curl response would be something like:
{"name2":"value2","resultname1":resultvalue1,"resultvalue2":"resultname2","resultname3": etc etc...

My thought was that I'd be able to run each curl, line by line, then have it sleep for 5 seconds, and then run the next one. And, the curl result would be writen to output_curl.txt. Of course, that doesn't work.

Any help would be greatly appreciated.

Thanks,
Larry
0
Comment
Question by:biglarrrr
  • 5
  • 5
10 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758284
where the curl lines will be taken from?
0
 

Author Comment

by:biglarrrr
ID: 41758293
Hi omarfarid,

As I wrote, there are thousands of them, so I thought it would be best to get all of the curls into a file, 1 line for each, and execute that file from the cmd line. I'm not saying that's the correct way going about doing this though.

Thanks,
Larry
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758312
the while loop should work

while read line
do
sh $line
sleep 5
done < input_curl.txt >> output_curl.txt

but why you want to send all output to one file?
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 

Author Comment

by:biglarrrr
ID: 41758330
Not sure about the "sh". The input file is a text file. The reason why I need the curl response writen to a file is because I need to make sure there are no errors being returned. And, since I'm not running these curl cmds one at a time, I'd never see the results unless it's written to a file. That's why I used >> so that each curl response would get appended.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758340
what is your question for sh ?
0
 

Author Comment

by:biglarrrr
ID: 41758348
When I run that with the sh I get:

-bash-4.1$ while read line; do sh $line; sleep 5; done < input_curl.txt >> output_curl.txt
/usr/bin/curl: /usr/bin/curl: cannot execute binary file
0
 
LVL 40

Accepted Solution

by:
omarfarid earned 2000 total points
ID: 41758386
sorry, you may remove sh

the while loop should do what you want, but if you want to capture errors the you need to redirect stderr as well:

while read line
do
$line
sleep 5
done < input_curl.txt >> output_curl.txt 2>&1
0
 

Author Comment

by:biglarrrr
ID: 41758473
That did it! Thank you so much omarfarid!
0
 

Author Closing Comment

by:biglarrrr
ID: 41758474
Thank you!
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758851
Welcome :)
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Fine Tune your automatic Updates for Ubuntu / Debian
I have written articles previously comparing SARDU and YUMI.  I also included a couple of lines about Easy2boot (easy2boot.com).  I have now been using, and enjoying easy2boot as my sole multiboot utility for some years and realize that it deserves …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Course of the Month7 days, 23 hours left to enroll

607 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question