Solved

Execute multiple curl cmds with sleep and send output to file

Posted on 2016-08-16
10
142 Views
Last Modified: 2016-08-16
First off, I am, in no way a Unix/Linux expert, so please bear with me. Also, I sure hope I've selected the proper topics for this question...

I need to run a few thousand curl cmds but I'm having difficulties figuring out how to go about doing this.

1. The curl is an XPOST (e.g. JSON)
For example:
curl -XPOST -H 'name1: value1' -H 'name2: value2' -H "Content-type: application/json" -d '{ "name3": [{"blah1": "blah1","aaa1": "blah2"}]}' 'http://domain.com'
curl -XPOST -H 'name1: value1' -H 'name2: value2' -H "Content-type: application/json" -d '{ "name3": [{"blah2": "blah2","aaa2": "blah2"}]}' 'http://domain.com'
curl -XPOST -H 'name1: value1' -H 'name2: value2' -H "Content-type: application/json" -d '{ "name3": [{"blah3": "blah3","aaa3": "blah2"}]}' 'http://domain.com'

The thing is I need to run this in a script so that there is a delay between each one. Also, once 1 curl is run, I get back a result which I'd like to save.

I thought I could just load all of the curl cmds in a text file and run something like:

while read line; do $line; sleep 5; done < input_curl.txt >> output_curl.txt

The resulting curl response would be something like:
{"name2":"value2","resultname1":resultvalue1,"resultvalue2":"resultname2","resultname3": etc etc...

My thought was that I'd be able to run each curl, line by line, then have it sleep for 5 seconds, and then run the next one. And, the curl result would be writen to output_curl.txt. Of course, that doesn't work.

Any help would be greatly appreciated.

Thanks,
Larry
0
Comment
Question by:biglarrrr
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 5
10 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758284
where the curl lines will be taken from?
0
 

Author Comment

by:biglarrrr
ID: 41758293
Hi omarfarid,

As I wrote, there are thousands of them, so I thought it would be best to get all of the curls into a file, 1 line for each, and execute that file from the cmd line. I'm not saying that's the correct way going about doing this though.

Thanks,
Larry
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758312
the while loop should work

while read line
do
sh $line
sleep 5
done < input_curl.txt >> output_curl.txt

but why you want to send all output to one file?
0
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

 

Author Comment

by:biglarrrr
ID: 41758330
Not sure about the "sh". The input file is a text file. The reason why I need the curl response writen to a file is because I need to make sure there are no errors being returned. And, since I'm not running these curl cmds one at a time, I'd never see the results unless it's written to a file. That's why I used >> so that each curl response would get appended.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758340
what is your question for sh ?
0
 

Author Comment

by:biglarrrr
ID: 41758348
When I run that with the sh I get:

-bash-4.1$ while read line; do sh $line; sleep 5; done < input_curl.txt >> output_curl.txt
/usr/bin/curl: /usr/bin/curl: cannot execute binary file
0
 
LVL 40

Accepted Solution

by:
omarfarid earned 500 total points
ID: 41758386
sorry, you may remove sh

the while loop should do what you want, but if you want to capture errors the you need to redirect stderr as well:

while read line
do
$line
sleep 5
done < input_curl.txt >> output_curl.txt 2>&1
0
 

Author Comment

by:biglarrrr
ID: 41758473
That did it! Thank you so much omarfarid!
0
 

Author Closing Comment

by:biglarrrr
ID: 41758474
Thank you!
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 41758851
Welcome :)
0

Featured Post

Enroll in July's Course of the Month

July's Course of the Month is now available! Enroll to learn HTML5 and prepare for certification. It's free for Premium Members, Team Accounts, and Qualified Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Have you tried to learn about Unicode, UTF-8, and multibyte text encoding and all the articles are just too "academic" or too technical? This article aims to make the whole topic easy for just about anyone to understand.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

627 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question