module for http range retrieval

I have been using LWP::Simple and WWW::Mechanize to retrieve files from http servers.

There are some files that are very large and I only need a section of them and would like to use the range retrieval capability of most http 1.1 servers.

Is there a perl module that currently supports this, and in what way?

thanks
drunnelsAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
flob9Connect With a Mentor Commented:

$req = HTTP::Request->new(GET => "http://cdimage.debian.org/debian-cd/5.0.2/i386/iso-cd/debian-502-i386-netinst.iso"); 
$req->header(Range => "bytes=0-99"); 
$res = LWP::UserAgent->new->request($req); 
print $res->as_string; 
 
 
=> response :
 
HTTP/1.1 206 Partial Content
Connection: close
Date: Wed, 15 Jul 2009 17:22:13 GMT
Accept-Ranges: bytes
Age: 3481
ETag: "d871c8-9608000-46d7ab1025380"
Server: Apache/2.2.9 (Unix)
Content-Length: 100
Content-Range: bytes 0-99/157319168
Content-Type: application/octet-stream
Last-Modified: Mon, 29 Jun 2009 11:07:10 GMT
Client-Date: Wed, 15 Jul 2009 17:22:13 GMT
Client-Peer: 130.239.18.138:80
Client-Response-Num: 1

Open in new window

0
 
flob9Commented:

Try "max_size" :
use LWP::UserAgent;
use HTTP::Response;
 
my $browser = LWP::UserAgent->new( );
$browser->max_size(500);
$url = 'http://www.google.com/';
my $response = $browser->get($url);
 
print $response->content( );

Open in new window

0
 
drunnelsAuthor Commented:
Thanks, but I'm not trying to limit the size from the beginning of the file, but rather I want to be able to specify a starting point. For instance, a may have a 500 meg file and I want to be able to ask for the download to start 400 meg into it and continue to the end of the file.
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
Fairlight2cxCommented:
Actually, HTTP::Range may not be exactly what you need, since it's related to segmenting.  BUT...  The docs show that it uses the Range and Content-Range headers of the HTTP protocol.  Those are detailed at ftp://ftp.rfc-editor.org/in-notes/rfc2616.txt in the spec.  (See section 14.)

You should be able to use the header() or push_header() methods of HTTP::Request to add the appropriate information to your request and achieve your goal using full-on LWP, however.
0
 
drunnelsAuthor Commented:
Thanks. This was exactly what I needed. The only thing I'd add to your answer is that to get just the page content one would add:
$content = $res->{'_content'}

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.