Multiple HTTP::Requests

Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
shodgkissAsked:
Who is Participating?
 
ozoConnect With a Mentor Commented:
Why not fork a separate process for each request?
0
 
shodgkissAuthor Commented:
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
holliCommented:
listening
0
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

 
shodgkissAuthor Commented:
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
shodgkissAuthor Commented:
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
interiotCommented:
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 
shodgkissAuthor Commented:
How do you fork a seperate process for each request?
0
 
shodgkissAuthor Commented:
Could you give me some code?
0
 
ozoCommented:
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 
shodgkissAuthor Commented:
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
ozoCommented:
WHat do you want to do with the results of the processing?
0
 
shodgkissAuthor Commented:
put a string in a hash so it can be accessed in the main program to output to screen.
0
 
shodgkissAuthor Commented:
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 
shodgkissAuthor Commented:
but when trying to access it its not there. So i need a different way of communicating.
0
 
MoondancerCommented:
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
Computer101Commented:
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.