Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Multiple HTTP::Requests

Posted on 2001-07-26
16
Medium Priority
?
143 Views
Last Modified: 2013-12-25
Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
0
Comment
Question by:shodgkiss
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
16 Comments
 

Author Comment

by:shodgkiss
ID: 6321811
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
LVL 6

Expert Comment

by:holli
ID: 6324601
listening
0
 
LVL 84

Accepted Solution

by:
ozo earned 400 total points
ID: 6324776
Why not fork a separate process for each request?
0
Interactive Way of Training for the AWS CSA Exam

An interactive way of learning that will help you visualize core concepts so that you can be more effective when taking your AWS certification exam.  Built for students by a student to help them understand the concepts that they are being taught.

 

Author Comment

by:shodgkiss
ID: 6326235
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 

Author Comment

by:shodgkiss
ID: 6326244
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
LVL 3

Expert Comment

by:interiot
ID: 6327210
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 

Author Comment

by:shodgkiss
ID: 6327368
How do you fork a seperate process for each request?
0
 

Author Comment

by:shodgkiss
ID: 6329580
Could you give me some code?
0
 
LVL 84

Expert Comment

by:ozo
ID: 6330973
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 

Author Comment

by:shodgkiss
ID: 6331127
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
LVL 84

Expert Comment

by:ozo
ID: 6332500
WHat do you want to do with the results of the processing?
0
 

Author Comment

by:shodgkiss
ID: 6333395
put a string in a hash so it can be accessed in the main program to output to screen.
0
 

Author Comment

by:shodgkiss
ID: 6333406
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 

Author Comment

by:shodgkiss
ID: 6337538
but when trying to access it its not there. So i need a different way of communicating.
0
 
LVL 1

Expert Comment

by:Moondancer
ID: 6699408
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
LVL 1

Expert Comment

by:Computer101
ID: 6904122
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The Windows functions GetTickCount and timeGetTime retrieve the number of milliseconds since the system was started. However, the value is stored in a DWORD, which means that it wraps around to zero every 49.7 days. This article shows how to solve t…
Originally, this post was published on Monitis Blog, you can check it here . In business circles, we sometimes hear that today is the “age of the customer.” And so it is. Thanks to the enormous advances over the past few years in consumer techno…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

704 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question