shodgkiss
asked on
Multiple HTTP::Requests
Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
listening
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
ozo, Sounds a good idea, but how do you do that???
Thanks
sH
Thanks
sH
ASKER
ozo, Sounds a good idea, but how do you do that???
Thanks
sH
Thanks
sH
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do. It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.
http://www.inf.ethz.ch/~langhein/ParallelUA/
It's an extra module that you have to install, but it can greatly speed up your program.
http://www.inf.ethz.ch/~langhein/ParallelUA/
It's an extra module that you have to install, but it can greatly speed up your program.
ASKER
How do you fork a seperate process for each request?
ASKER
Could you give me some code?
foreach my $req (@requests){
unless($pid=fork){
my $response = $ua->request($req);
#...process response
exit;
}
}
unless($pid=fork){
my $response = $ua->request($req);
#...process response
exit;
}
}
ASKER
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
WHat do you want to do with the results of the processing?
ASKER
put a string in a hash so it can be accessed in the main program to output to screen.
ASKER
eg:
my %resp = ();
foreach my $req (@requests){
unless($pid=fork){
my $response = $ua->request($req);
#...process response
# try to save in variable
$resp{$id} = $response->content;
exit;
}
}
# access it
$resp{$id}# .........
my %resp = ();
foreach my $req (@requests){
unless($pid=fork){
my $response = $ua->request($req);
#...process response
# try to save in variable
$resp{$id} = $response->content;
exit;
}
}
# access it
$resp{$id}# .........
ASKER
but when trying to access it its not there. So i need a different way of communicating.
Please update/finalize this question.
Thanks,
Moondancer
Community Support Moderator @ Experts Exchange
Thanks,
Moondancer
Community Support Moderator @ Experts Exchange
Points reduced for split. Comment from expert accepted as answer. interiot, look for your question in this topic area.
Computer101
E-E Moderator
Computer101
E-E Moderator
ASKER
foreach my $req (@requests)
{
my $response = $ua->request($req);
#...process response
}