Solved

Multiple HTTP::Requests

Posted on 2001-07-26
16
137 Views
Last Modified: 2013-12-25
Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
0
Comment
Question by:shodgkiss
16 Comments
 

Author Comment

by:shodgkiss
ID: 6321811
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
LVL 6

Expert Comment

by:holli
ID: 6324601
listening
0
 
LVL 84

Accepted Solution

by:
ozo earned 100 total points
ID: 6324776
Why not fork a separate process for each request?
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 

Author Comment

by:shodgkiss
ID: 6326235
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 

Author Comment

by:shodgkiss
ID: 6326244
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
LVL 3

Expert Comment

by:interiot
ID: 6327210
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 

Author Comment

by:shodgkiss
ID: 6327368
How do you fork a seperate process for each request?
0
 

Author Comment

by:shodgkiss
ID: 6329580
Could you give me some code?
0
 
LVL 84

Expert Comment

by:ozo
ID: 6330973
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 

Author Comment

by:shodgkiss
ID: 6331127
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
LVL 84

Expert Comment

by:ozo
ID: 6332500
WHat do you want to do with the results of the processing?
0
 

Author Comment

by:shodgkiss
ID: 6333395
put a string in a hash so it can be accessed in the main program to output to screen.
0
 

Author Comment

by:shodgkiss
ID: 6333406
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 

Author Comment

by:shodgkiss
ID: 6337538
but when trying to access it its not there. So i need a different way of communicating.
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 6699408
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
LVL 1

Expert Comment

by:Computer101
ID: 6904122
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
exchange 2007, scripts 15 79
Windows Script Host failed (Access is denied.) error 6 146
React or Angular? 6 72
Bulk Reorder File Names 4 67
The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some…
If you get a (Blue Screen of Death), your system writes a small file called a minidump. Your first step is to make certain your computer is setup to record memory dumps. Right click My Computer, choose properties. Click on the advanced tab, an…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question