Solved

Speed Up Execution of Perl LWP Module

Posted on 2004-04-19
8
529 Views
Last Modified: 2012-08-13
I used to have a site on a shared server running Sun Solaris 5.8 and Perl 5.6.1. We have now moved to a dedicated Linux server running Red Hat Enterprise Linux V3, Cpanel/Web Host Manager (latest), and  Perl 5.8.1. I have a perl script that collects images from a site on the web and downloads them to our server. Typcially processing between 100 to 1000 at a time. I've noticed a slight performance decrease since moving to the new RHEL server. When processing 30 urls it takes about 10 seconds longer then before. Although this difference is minor, the lag really shows up when processing thousands of images at a time. What are potential causes of the noticeable descrease in my Perl scripts performance when utilizing the LWP module?

What steps can I take to improve this.
0
Comment
Question by:davef8
  • 3
  • 2
8 Comments
 
LVL 20

Expert Comment

by:jmcg
ID: 10864686
Other people have reported on Perl 5.8 being considerably slower than Perl 5.6. You might try installing the latest version of the Perl 5.6 stream in parallel with your already-installed 5.8 and do a speed comparison.

(The evidence I've seen so far points to a heavier UTF-8 support in 5.8 vs 5.6 as one of the culprits in the slowdown. I don't know of an easy way to improve the performance while keeping the benefits of the newer version.)

The LWP module has a number of specialized features that may enable you to improve the performance (at the expense of simplicity). Using these, you may be able to reduce the number of times data is copied from memory to memory or disk to disk. Look at the options for the 'get' and 'request' methods and consider whether you can take control of the data earler in the download process, either in chunks as it comes in or by (a :content_cb) or by specifying the destination file in advance (with a :content_file).
0
 

Author Comment

by:davef8
ID: 10864718
Do you have any links to informaiton on this: "The LWP module has a number of specialized features that may enable you to improve the performance (at the expense of simplicity)."

I'll try using 5.6 as well.
0
 
LVL 20

Expert Comment

by:jmcg
ID: 10865191
Sorry, it's in the LWP::UserAgent doc:

http://search.cpan.org/~gaas/libwww-perl-5.79/lib/LWP/UserAgent.pm

Under the heading REQUEST METHODS, see the descriptions for methods

$ua->get( $url )

and

$ua->request( $request )
0
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 20

Expert Comment

by:jmcg
ID: 10865202
You might also want to check out whether using LWP::Parallel::UserAgent will help by overlapping the requests. If the requests are distributed across multiple servers, you may be able to decrease your overall run time by running the requests in parallel.

http://search.cpan.org/author/MARCLANG/ParallelUserAgent-2.57/lib/LWP/Parallel/UserAgent.pm

0
 

Author Comment

by:davef8
ID: 11086515
It turns out this issue had nothing to do with perl itself. The lag was a result of network traffic.
0
 
LVL 5

Accepted Solution

by:
Netminder earned 0 total points
ID: 11317531
PAQed, with points refunded (500)

Netminder
EE Admin
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
On Microsoft Windows, if  when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question