Link to home
Start Free TrialLog in
Avatar of ambuli
ambuliFlag for United States of America

asked on

download files from web

Hi Experts,

I want to write a small script to download some files from a site.  Is there any good example on how to do this?  I need to give some options when downloading( The CGI accepts some params)
thank you.
Avatar of jimsweb
jimsweb
Flag of India image

Avatar of ambuli

ASKER

Hi thank you.  But, when I try using some of these code I am getting the the following error.
I tried a sample from cpan as well, and I am gettiung the same error.

Can't locate object method "get" via package "LWP::UserAgent" at download.pl line 9.
#!/usr/bin/perl -w

use LWP::UserAgent;
 
 my $ua = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 
 my $response = $ua->get('http://search.cpan.org/');
 
 if ($response->is_success) {
     print $response->decoded_content;  # or whatever
 }
 else {
     die $response->status_line;
 }

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of Zarabozo
Zarabozo

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Zarabozo
Zarabozo

If you want to try a failed response with the same code I gave you in my previous answer, try this (I'm changing the $uri variable):

use warnings;
use strict;
use IO::Socket;

my $host = 'search.cpan.org'; # Don't add slash at the end here
my $uri = '/notExistentPageTest'; # Put here you complete uri starting by slash always
my $port = 80;

my $s = new IO::Socket::INET (
	Proto => 'tcp',
	PeerAddr => $host,
	PeerPort => $port,
);

$s->write("GET $uri HTTP 1.0\r\n");
$s->write("HOST: $host\r\n");
$s->write("\r\n");

my $response = join '', <$s>;
$response =~ s/\r\n/\n/g;
my $header = substr($response, 0, index($response, "\n\n"));
$response = substr($response, length($header) + 2);

if ($header =~ /^.*?200\s+OK/i) {
	print "Request successful, this is the content:\n\n";
	print "$response\n";
} else {
	print "Response failed, this is the header and content:\n\n";
	print "$header\n\n$response\n";
}

Open in new window


Hope this helps.

Francisco
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of ambuli

ASKER

Thank you very much.
When I try it, I am not getting any response.  To give you some more context.  I have a python script that kind of works.  But, I am trying to implement/integrate that in my perl script as part of another program.

I am trying out your examples now, but if you can get anything from the following python script please let me know.

Thank you Francisco.

mport urllib
import zipfile
import cStringIO
import datetime
import os
import sys


DOWNLOAD_URL = "http://1ykf/cgi-bin/log.pl?%s"
DOWNLOAD_LOG_TYPES = ('devinfo', 'eventlog', 'javaprocesses')
TEXT_FILES = ['devinfo', 'eventlog', 'javaprocesses']
BUGDISP_FILE = 'bdispbytes'


def usage():
    print __doc__


def download_logs(ids):
    print "IDs to download:" , ids
    params = [('mode', 'bulkids'),
              ('format', 'Bulk_ZIP'),
             ]
    for eachFileType in DOWNLOAD_LOG_TYPES:
        params.append(('attachment', eachFileType))

    params.append(['id_list', ",".join(ids)])
    url = DOWNLOAD_URL % urllib.urlencode(params)

    try:
        webFile = urllib.urlopen(url)
    except IOError as ioe:
        print ioe
        sys.exit('Exiting... Failed to download logs')

    print "Downloading from: " + url
    buf = cStringIO.StringIO(webFile.read())
    webFile.close()

    return buf

if __name__ == '__main__':
    
    if(len(sys.argv)<2):
        usage()
        sys.exit(1)
            
    ids = sys.argv[1:]

Open in new window

Hello Ambuli,

I don't see anything in your Python script doing anything special that Perl is not doing. There might be a problem with your Perl installation or permissions to open sockets.

Assuming you are running the script on Linux:

1) Are you running an reasonable recent version of Perl? (e.g. Perl 5.8 or higher - you can see the version by running the command perl -v)

2) Is the UID running the script the same that you use for your Python script when it works?

Besides those questions, are you behind a firewall or proxy? Do you have any kind of antivirus/firewall application that can be blocking Perl's connectivity?

HTH
Avatar of ambuli

ASKER

Thank you all. I am running it on windows.  I will go through a examples and try.  

C:\perl -v

This is perl, v5.8.5 built for MSWin32-x86-multi-thread

Copyright 1987-2004, Larry Wall