Solved

Multiple HTTP::Requests

Posted on 2001-07-26
16
135 Views
Last Modified: 2013-12-25
Hi,
I am working on a project and am trying to optimize the code to speed things up because the scripts takes a while to finish. The script goes in a foreach loop, requests some data and then processes the content.
However, when requesting to certain sites it takes a while to come back and i need a way to send out all the requests first and then process them as they come back.
Thanks in advance.
sH
0
Comment
Question by:shodgkiss
16 Comments
 

Author Comment

by:shodgkiss
Comment Utility
Heres the part that needs changing::

foreach my $req (@requests)
{
 my $response = $ua->request($req);
 #...process response
}
0
 
LVL 6

Expert Comment

by:holli
Comment Utility
listening
0
 
LVL 84

Accepted Solution

by:
ozo earned 100 total points
Comment Utility
Why not fork a separate process for each request?
0
 

Author Comment

by:shodgkiss
Comment Utility
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 

Author Comment

by:shodgkiss
Comment Utility
ozo, Sounds a good idea, but how do you do that???

Thanks
sH
0
 
LVL 3

Expert Comment

by:interiot
Comment Utility
Use ParallelUA, it allows you to request multiple URLs at a time, and fetches a specified number at once, like modern browsers do.  It uses many of the same interfaces as LWP (accepts standard Request objects), so it's easy to use.

     http://www.inf.ethz.ch/~langhein/ParallelUA/

It's an extra module that you have to install, but it can greatly speed up your program.
0
 

Author Comment

by:shodgkiss
Comment Utility
How do you fork a seperate process for each request?
0
 

Author Comment

by:shodgkiss
Comment Utility
Could you give me some code?
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 84

Expert Comment

by:ozo
Comment Utility
foreach my $req (@requests){
    unless($pid=fork){
        my $response = $ua->request($req);
        #...process response
        exit;
    }
}
0
 

Author Comment

by:shodgkiss
Comment Utility
How do i communicate between the forks and the actual script, i need to do this to get the results of the processing.
0
 
LVL 84

Expert Comment

by:ozo
Comment Utility
WHat do you want to do with the results of the processing?
0
 

Author Comment

by:shodgkiss
Comment Utility
put a string in a hash so it can be accessed in the main program to output to screen.
0
 

Author Comment

by:shodgkiss
Comment Utility
eg:

my %resp = ();
foreach my $req (@requests){
   unless($pid=fork){
       my $response = $ua->request($req);
       #...process response
       # try to save in variable
       $resp{$id} = $response->content;
       exit;
   }
}
# access it
$resp{$id}# .........
0
 

Author Comment

by:shodgkiss
Comment Utility
but when trying to access it its not there. So i need a different way of communicating.
0
 
LVL 1

Expert Comment

by:Moondancer
Comment Utility
Please update/finalize this question.

Thanks,

Moondancer
Community Support Moderator @ Experts Exchange
0
 
LVL 1

Expert Comment

by:Computer101
Comment Utility
Points reduced for split.  Comment from expert accepted as answer.  interiot, look for your question in this topic area.

Computer101
E-E Moderator
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

This tutorial will discuss the log-in process using WhizBase. In this article I assume you already know HTML. I will write the code using WhizBase Server Pages, so you need to know some basics in WBSP (you might look at some of my other articles abo…
In this tutorial I will show you how to provide a dynamic RTF document on your website generated with data from your database. For this tutorial you will need Microsoft Word or WordPad, WhizBase and Microsoft Access. In this tutorial I will show …
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now