Options for Parallel HTTP operations

Tintin
Tintin used Ask the Experts™
on
I needed to write a script that would take a whole bunch of URL's and check if the HTTP status code for each of them.  As I have a lot of URL's, I wanted to parallelise the operation, so naturally enough I thought I'd use the LWP::Parallel::UserAgent module.  Unfortunately, this module doesn't work with newer versions of libwww and looks like it is a dead module.

That got me on the path at looking at other modules.  Candidates were

IO::Lambda
AnyEvent::HTTP
HTTP::Async
HTTP::Client::Parallel

Out of the above, only IO::Lambda seemed to be the most robust, although it is quite complex.

My needs are fairly simple as I'm just doing a simple HTTP HEAD request, but one important aspect is to be able to limit the number of simultaneous connections.

So, I'm asking if anyone has experience in using any of the above modules or any suggestions for other modules/code I could use?

Note that this needs to run with ActivePerl on Linux and Windows.


Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Top Expert 2009

Commented:
You could just fork as many clients as you want.  Or possibly create multiple threads, but then you have to worry about the modules being thread safe.  
Top Expert 2007

Author

Commented:
I did think about forking/threading, but I thought it might be easier with a module that handles non-blocking IO.
Top Expert 2009

Commented:
If there is a module (and it works and is documented), that would probably be easier.  I'm not familiar with any of the modules you mentioned.  Using one, you could probably be more efficient than forking, but forking is easy to develop, and you can use existing LWP modules to handle the http protocol.
Top Expert 2007
Commented:
I ended up using HTTP::Async, which is quite a nice module, although it looks like it is no longer developed and has a small bug (which I fixed), where timeouts don't work.

Also useful would have been Parallel::ForkManager, which has an example of parallel HTTP operations (didn't see this module until today).

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial