• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 352
  • Last Modified:

download files from web

Hi Experts,

I want to write a small script to download some files from a site.  Is there any good example on how to do this?  I need to give some options when downloading( The CGI accepts some params)
thank you.
0
ambuli
Asked:
ambuli
  • 4
  • 3
2 Solutions
 
ambuliAuthor Commented:
Hi thank you.  But, when I try using some of these code I am getting the the following error.
I tried a sample from cpan as well, and I am gettiung the same error.

Can't locate object method "get" via package "LWP::UserAgent" at download.pl line 9.
#!/usr/bin/perl -w

use LWP::UserAgent;
 
 my $ua = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 
 my $response = $ua->get('http://search.cpan.org/');
 
 if ($response->is_success) {
     print $response->decoded_content;  # or whatever
 }
 else {
     die $response->status_line;
 }

Open in new window

0
 
ZarabozoCommented:
Hello,

I hope this helps. If for some reason LWP::UserAgent is not working for you, you should check what the problem is, probably updating the module to its latest version will solve the problem.

However, this is an example code that you should be able to use to get a web page without using LWP::UserAgent:

use warnings;
use strict;
use IO::Socket;

my $host = 'search.cpan.org'; # Don't add slash at the end here
my $uri = '/'; # Put here you complete uri starting by slash always
my $port = 80;

my $s = new IO::Socket::INET (
	Proto => 'tcp',
	PeerAddr => $host,
	PeerPort => $port,
);

$s->write("GET $uri HTTP 1.0\r\n");
$s->write("HOST: $host\r\n");
$s->write("\r\n");

my $response = join '', <$s>;
$response =~ s/\r\n/\n/g;
my $header = substr($response, 0, index($response, "\n\n"));
$response = substr($response, length($header) + 2);

if ($header =~ /^.*?200\s+OK/i) {
	print "Request successful, this is the content:\n\n";
	print "$response\n";
} else {
	print "Response failed, this is the header and content:\n\n";
	print "$header\n\n$response\n";
}

Open in new window


If this solution works, please don't forget to select my asnwers as accepted solution. Thanks.

Francisco
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
ZarabozoCommented:
If you want to try a failed response with the same code I gave you in my previous answer, try this (I'm changing the $uri variable):

use warnings;
use strict;
use IO::Socket;

my $host = 'search.cpan.org'; # Don't add slash at the end here
my $uri = '/notExistentPageTest'; # Put here you complete uri starting by slash always
my $port = 80;

my $s = new IO::Socket::INET (
	Proto => 'tcp',
	PeerAddr => $host,
	PeerPort => $port,
);

$s->write("GET $uri HTTP 1.0\r\n");
$s->write("HOST: $host\r\n");
$s->write("\r\n");

my $response = join '', <$s>;
$response =~ s/\r\n/\n/g;
my $header = substr($response, 0, index($response, "\n\n"));
$response = substr($response, length($header) + 2);

if ($header =~ /^.*?200\s+OK/i) {
	print "Request successful, this is the content:\n\n";
	print "$response\n";
} else {
	print "Response failed, this is the header and content:\n\n";
	print "$header\n\n$response\n";
}

Open in new window


Hope this helps.

Francisco
0
 
ZarabozoCommented:
Also, if you want to send parameters (e.g. to actually get a search result, try modifying $uri as I do it in this example:

use warnings;
use strict;
use IO::Socket;

my $host = 'search.cpan.org'; # Don't add slash at the end here
my $uri = '/search?query=any+module&mode=all'; # Put here you complete uri starting by slash always
my $port = 80;

my $s = new IO::Socket::INET (
	Proto => 'tcp',
	PeerAddr => $host,
	PeerPort => $port,
);

$s->write("GET $uri HTTP 1.0\r\n");
$s->write("HOST: $host\r\n");
$s->write("\r\n");

my $response = join '', <$s>;
$response =~ s/\r\n/\n/g;
my $header = substr($response, 0, index($response, "\n\n"));
$response = substr($response, length($header) + 2);

if ($header =~ /^.*?200\s+OK/i) {
	print "Request successful, this is the content:\n\n";
	print "$response\n";
} else {
	print "Response failed, this is the header and content:\n\n";
	print "$header\n\n$response\n";
}

Open in new window


Francisco
0
 
ambuliAuthor Commented:
Thank you very much.
When I try it, I am not getting any response.  To give you some more context.  I have a python script that kind of works.  But, I am trying to implement/integrate that in my perl script as part of another program.

I am trying out your examples now, but if you can get anything from the following python script please let me know.

Thank you Francisco.

mport urllib
import zipfile
import cStringIO
import datetime
import os
import sys


DOWNLOAD_URL = "http://1ykf/cgi-bin/log.pl?%s"
DOWNLOAD_LOG_TYPES = ('devinfo', 'eventlog', 'javaprocesses')
TEXT_FILES = ['devinfo', 'eventlog', 'javaprocesses']
BUGDISP_FILE = 'bdispbytes'


def usage():
    print __doc__


def download_logs(ids):
    print "IDs to download:" , ids
    params = [('mode', 'bulkids'),
              ('format', 'Bulk_ZIP'),
             ]
    for eachFileType in DOWNLOAD_LOG_TYPES:
        params.append(('attachment', eachFileType))

    params.append(['id_list', ",".join(ids)])
    url = DOWNLOAD_URL % urllib.urlencode(params)

    try:
        webFile = urllib.urlopen(url)
    except IOError as ioe:
        print ioe
        sys.exit('Exiting... Failed to download logs')

    print "Downloading from: " + url
    buf = cStringIO.StringIO(webFile.read())
    webFile.close()

    return buf

if __name__ == '__main__':
    
    if(len(sys.argv)<2):
        usage()
        sys.exit(1)
            
    ids = sys.argv[1:]

Open in new window

0
 
ZarabozoCommented:
Hello Ambuli,

I don't see anything in your Python script doing anything special that Perl is not doing. There might be a problem with your Perl installation or permissions to open sockets.

Assuming you are running the script on Linux:

1) Are you running an reasonable recent version of Perl? (e.g. Perl 5.8 or higher - you can see the version by running the command perl -v)

2) Is the UID running the script the same that you use for your Python script when it works?

Besides those questions, are you behind a firewall or proxy? Do you have any kind of antivirus/firewall application that can be blocking Perl's connectivity?

HTH
0
 
ambuliAuthor Commented:
Thank you all. I am running it on windows.  I will go through a examples and try.  

C:\perl -v

This is perl, v5.8.5 built for MSWin32-x86-multi-thread

Copyright 1987-2004, Larry Wall
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now