Solved

Multiple HTTP::Requests

Posted on 2001-07-26
16
141 Views
Last Modified: 2013-12-25
Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
0
Comment
Question by:shodgkiss
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
16 Comments
 

Author Comment

by:shodgkiss
ID: 6321811
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
LVL 6

Expert Comment

by:holli
ID: 6324601
listening
0
 
LVL 84

Accepted Solution

by:
ozo earned 100 total points
ID: 6324776
Why not fork a separate process for each request?
0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 

Author Comment

by:shodgkiss
ID: 6326235
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 

Author Comment

by:shodgkiss
ID: 6326244
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
LVL 3

Expert Comment

by:interiot
ID: 6327210
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 

Author Comment

by:shodgkiss
ID: 6327368
How do you fork a seperate process for each request?
0
 

Author Comment

by:shodgkiss
ID: 6329580
Could you give me some code?
0
 
LVL 84

Expert Comment

by:ozo
ID: 6330973
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 

Author Comment

by:shodgkiss
ID: 6331127
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
LVL 84

Expert Comment

by:ozo
ID: 6332500
WHat do you want to do with the results of the processing?
0
 

Author Comment

by:shodgkiss
ID: 6333395
put a string in a hash so it can be accessed in the main program to output to screen.
0
 

Author Comment

by:shodgkiss
ID: 6333406
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 

Author Comment

by:shodgkiss
ID: 6337538
but when trying to access it its not there. So i need a different way of communicating.
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 6699408
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
LVL 1

Expert Comment

by:Computer101
ID: 6904122
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
This article will show, step by step, how to integrate R code into a R Sweave document
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

628 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question