Solved

Multiple HTTP::Requests

Posted on 2001-07-26
16
136 Views
Last Modified: 2013-12-25
Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
0
Comment
Question by:shodgkiss
16 Comments
 

Author Comment

by:shodgkiss
ID: 6321811
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
LVL 6

Expert Comment

by:holli
ID: 6324601
listening
0
 
LVL 84

Accepted Solution

by:
ozo earned 100 total points
ID: 6324776
Why not fork a separate process for each request?
0
 

Author Comment

by:shodgkiss
ID: 6326235
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 

Author Comment

by:shodgkiss
ID: 6326244
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
LVL 3

Expert Comment

by:interiot
ID: 6327210
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 

Author Comment

by:shodgkiss
ID: 6327368
How do you fork a seperate process for each request?
0
 

Author Comment

by:shodgkiss
ID: 6329580
Could you give me some code?
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 
LVL 84

Expert Comment

by:ozo
ID: 6330973
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 

Author Comment

by:shodgkiss
ID: 6331127
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
LVL 84

Expert Comment

by:ozo
ID: 6332500
WHat do you want to do with the results of the processing?
0
 

Author Comment

by:shodgkiss
ID: 6333395
put a string in a hash so it can be accessed in the main program to output to screen.
0
 

Author Comment

by:shodgkiss
ID: 6333406
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 

Author Comment

by:shodgkiss
ID: 6337538
but when trying to access it its not there. So i need a different way of communicating.
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 6699408
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
LVL 1

Expert Comment

by:Computer101
ID: 6904122
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0

Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Batch, VBS, and scripts in general are incredibly useful for repetitive tasks.  Some tasks can take a while to complete and it can be annoying to check back only to discover that your script finished 5 minutes ago.  Some scripts may complete nearly …
This article will show, step by step, how to integrate R code into a R Sweave document
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now