Solved

Perl Script to Download Files

Posted on 2010-11-18
4
702 Views
Last Modified: 2012-06-21
I need a perl script to download this file and save it to a directory on my computer.


ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2010111800/gfs.t00z.pgrbf00.grib2

the file also exists as an http:

http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2010111800/gfs.t00z.pgrbf00.grib2

I am behind a corporate proxy that uses an automatic configuration script which complicates the file download somewhat.

Any solutions?

0
Comment
Question by:kyle972
  • 2
  • 2
4 Comments
 
LVL 10

Expert Comment

by:jeromee
ID: 34169264
There might be a better way but this worked for me.
Please change the proxy settings of course
And I used a smaller file to test the download:

use strict;
use LWP::UserAgent;

my $browser = LWP::UserAgent->new( );
$browser->proxy('http', 'http://172.17.1.18:8080');	# proxy settings here!
my $res = $browser->get('http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2010111800/gfs.t00z.pgrb2f78.idx');
print $res->content if $res->is_success;

Open in new window

0
 

Author Comment

by:kyle972
ID: 34169575
Thanks for the reply.

However, the file becomes unusable to me when reading in and then outputting it. I need to save the file like you would if you were downloading it from a browser.  I can't lose the special formatting in this file.

I just need to save it. No reading or writing to another file.

Any other ideas?
0
 
LVL 10

Accepted Solution

by:
jeromee earned 500 total points
ID: 34169714
Since it's binary file, you will need to save it as such.
See the new version of the code.
BTW, I saved it w/ my browser and compare it with the file I downloaded using Perl and both are identical.

Good luck!
use strict;
use LWP::UserAgent;

my $browser = LWP::UserAgent->new( );
$browser->proxy('http', 'http://172.17.1.18:8080');	# proxy settings here!
my $res = $browser->get('http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2010111800/gfs.t00z.pgrbf00.grib2');
if( $res->is_success ) {
   open(BINFILE, ">/tmp/binfile") || die;
   binmode BINFILE;
   print BINFILE $res->content;
   close BINFILE;
}

Open in new window

0
 

Author Closing Comment

by:kyle972
ID: 34175168
Works perfect.  Thanks
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Sending email via Perl on Windows 3 136
PERL get the value for query 4 145
git fat pull happens nothing in centos 6.3 python 2.6.6 33 207
combine multiple lines 2 42
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
A company’s greatest vulnerability is their email. CEO fraud, ransomware and spear phishing attacks are the no1 threat to a company’s security. Cybercrime is responsible for the largest loss of money to companies today with losses projected to r…

930 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now