Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 144
  • Last Modified:

Multiple HTTP::Requests

Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
0
shodgkiss
Asked:
shodgkiss
1 Solution
 
shodgkissAuthor Commented:
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
holliCommented:
listening
0
 
ozoCommented:
Why not fork a separate process for each request?
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
shodgkissAuthor Commented:
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
shodgkissAuthor Commented:
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
interiotCommented:
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 
shodgkissAuthor Commented:
How do you fork a seperate process for each request?
0
 
shodgkissAuthor Commented:
Could you give me some code?
0
 
ozoCommented:
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 
shodgkissAuthor Commented:
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
ozoCommented:
WHat do you want to do with the results of the processing?
0
 
shodgkissAuthor Commented:
put a string in a hash so it can be accessed in the main program to output to screen.
0
 
shodgkissAuthor Commented:
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 
shodgkissAuthor Commented:
but when trying to access it its not there. So i need a different way of communicating.
0
 
MoondancerCommented:
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
Computer101Commented:
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now