We help IT Professionals succeed at work.

Check out our new AWS podcast with Certified Expert, Phil Phillips! Listen to "How to Execute a Seamless AWS Migration" on EE or on your favorite podcast platform. Listen Now

x

download files from web

ambuli
ambuli asked
on
Medium Priority
364 Views
Last Modified: 2012-05-11
Hi Experts,

I want to write a small script to download some files from a site.  Is there any good example on how to do this?  I need to give some options when downloading( The CGI accepts some params)
thank you.
Comment
Watch Question

Author

Commented:
Hi thank you.  But, when I try using some of these code I am getting the the following error.
I tried a sample from cpan as well, and I am gettiung the same error.

Can't locate object method "get" via package "LWP::UserAgent" at download.pl line 9.
#!/usr/bin/perl -w

use LWP::UserAgent;
 
 my $ua = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 
 my $response = $ua->get('http://search.cpan.org/');
 
 if ($response->is_success) {
     print $response->decoded_content;  # or whatever
 }
 else {
     die $response->status_line;
 }

Open in new window

Commented:
Unlock this solution with a free trial preview.
(No credit card required)
Get Preview

Commented:
If you want to try a failed response with the same code I gave you in my previous answer, try this (I'm changing the $uri variable):

use warnings;
use strict;
use IO::Socket;

my $host = 'search.cpan.org'; # Don't add slash at the end here
my $uri = '/notExistentPageTest'; # Put here you complete uri starting by slash always
my $port = 80;

my $s = new IO::Socket::INET (
	Proto => 'tcp',
	PeerAddr => $host,
	PeerPort => $port,
);

$s->write("GET $uri HTTP 1.0\r\n");
$s->write("HOST: $host\r\n");
$s->write("\r\n");

my $response = join '', <$s>;
$response =~ s/\r\n/\n/g;
my $header = substr($response, 0, index($response, "\n\n"));
$response = substr($response, length($header) + 2);

if ($header =~ /^.*?200\s+OK/i) {
	print "Request successful, this is the content:\n\n";
	print "$response\n";
} else {
	print "Response failed, this is the header and content:\n\n";
	print "$header\n\n$response\n";
}

Open in new window


Hope this helps.

Francisco
Commented:
Unlock this solution with a free trial preview.
(No credit card required)
Get Preview

Author

Commented:
Thank you very much.
When I try it, I am not getting any response.  To give you some more context.  I have a python script that kind of works.  But, I am trying to implement/integrate that in my perl script as part of another program.

I am trying out your examples now, but if you can get anything from the following python script please let me know.

Thank you Francisco.

mport urllib
import zipfile
import cStringIO
import datetime
import os
import sys


DOWNLOAD_URL = "http://1ykf/cgi-bin/log.pl?%s"
DOWNLOAD_LOG_TYPES = ('devinfo', 'eventlog', 'javaprocesses')
TEXT_FILES = ['devinfo', 'eventlog', 'javaprocesses']
BUGDISP_FILE = 'bdispbytes'


def usage():
    print __doc__


def download_logs(ids):
    print "IDs to download:" , ids
    params = [('mode', 'bulkids'),
              ('format', 'Bulk_ZIP'),
             ]
    for eachFileType in DOWNLOAD_LOG_TYPES:
        params.append(('attachment', eachFileType))

    params.append(['id_list', ",".join(ids)])
    url = DOWNLOAD_URL % urllib.urlencode(params)

    try:
        webFile = urllib.urlopen(url)
    except IOError as ioe:
        print ioe
        sys.exit('Exiting... Failed to download logs')

    print "Downloading from: " + url
    buf = cStringIO.StringIO(webFile.read())
    webFile.close()

    return buf

if __name__ == '__main__':
    
    if(len(sys.argv)<2):
        usage()
        sys.exit(1)
            
    ids = sys.argv[1:]

Open in new window

Commented:
Hello Ambuli,

I don't see anything in your Python script doing anything special that Perl is not doing. There might be a problem with your Perl installation or permissions to open sockets.

Assuming you are running the script on Linux:

1) Are you running an reasonable recent version of Perl? (e.g. Perl 5.8 or higher - you can see the version by running the command perl -v)

2) Is the UID running the script the same that you use for your Python script when it works?

Besides those questions, are you behind a firewall or proxy? Do you have any kind of antivirus/firewall application that can be blocking Perl's connectivity?

HTH

Author

Commented:
Thank you all. I am running it on windows.  I will go through a examples and try.  

C:\perl -v

This is perl, v5.8.5 built for MSWin32-x86-multi-thread

Copyright 1987-2004, Larry Wall
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a free trial preview!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.