module for http range retrieval

I have been using LWP::Simple and WWW::Mechanize to retrieve files from http servers.

There are some files that are very large and I only need a section of them and would like to use the range retrieval capability of most http 1.1 servers.

Is there a perl module that currently supports this, and in what way?

thanks
drunnelsAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

flob9Commented:

Try "max_size" :
use LWP::UserAgent;
use HTTP::Response;
 
my $browser = LWP::UserAgent->new( );
$browser->max_size(500);
$url = 'http://www.google.com/';
my $response = $browser->get($url);
 
print $response->content( );

Open in new window

0
drunnelsAuthor Commented:
Thanks, but I'm not trying to limit the size from the beginning of the file, but rather I want to be able to specify a starting point. For instance, a may have a 500 meg file and I want to be able to ask for the download to start 400 meg into it and continue to the end of the file.
0
Build an E-Commerce Site with Angular 5

Learn how to build an E-Commerce site with Angular 5, a JavaScript framework used by developers to build web, desktop, and mobile applications.

Fairlight2cxCommented:
Actually, HTTP::Range may not be exactly what you need, since it's related to segmenting.  BUT...  The docs show that it uses the Range and Content-Range headers of the HTTP protocol.  Those are detailed at ftp://ftp.rfc-editor.org/in-notes/rfc2616.txt in the spec.  (See section 14.)

You should be able to use the header() or push_header() methods of HTTP::Request to add the appropriate information to your request and achieve your goal using full-on LWP, however.
0
flob9Commented:

$req = HTTP::Request->new(GET => "http://cdimage.debian.org/debian-cd/5.0.2/i386/iso-cd/debian-502-i386-netinst.iso"); 
$req->header(Range => "bytes=0-99"); 
$res = LWP::UserAgent->new->request($req); 
print $res->as_string; 
 
 
=> response :
 
HTTP/1.1 206 Partial Content
Connection: close
Date: Wed, 15 Jul 2009 17:22:13 GMT
Accept-Ranges: bytes
Age: 3481
ETag: "d871c8-9608000-46d7ab1025380"
Server: Apache/2.2.9 (Unix)
Content-Length: 100
Content-Range: bytes 0-99/157319168
Content-Type: application/octet-stream
Last-Modified: Mon, 29 Jun 2009 11:07:10 GMT
Client-Date: Wed, 15 Jul 2009 17:22:13 GMT
Client-Peer: 130.239.18.138:80
Client-Response-Num: 1

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
drunnelsAuthor Commented:
Thanks. This was exactly what I needed. The only thing I'd add to your answer is that to get just the page content one would add:
$content = $res->{'_content'}

0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Perl

From novice to tech pro — start learning today.