Feeding URLs from a file to a UserAgent script/Checking if a URL is live

I need a script that will take take urls in a file and check to see if the URL is live or not. I came up with this:


#!/usr/local/bin/perl
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");


open (INPUT, '<url.txt');
open (OUTPUT, '>url.html');

 while (<INPUT>) {
       chomp;

my $req = HTTP::Request->new(POST => '$_');
my $res = $ua->request($req);

  # Check the outcome of the response
  if ($res->is_success) {
      print $res->content;
  }
  else {
      print $res->status_line, "\n";
  }

 }
 
 print OUTPUT;

 close (INPUT);
 
However when I run it I get a "400 URL must be absolute". When I plug the URL in directly into my $req = HTTP::Request->new(POST => 'www....'); it works fine, so its not my connection.
hadronsAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
ozoConnect With a Mentor Commented:
my $req = HTTP::Request->new(POST => $_);
0
 
hadronsAuthor Commented:
Excellent ... I have just additional questions:

1) is there a function that delays the next URL request so I don't hammer their servers too hard (not that they don't deserve it with all the dead URLs they sent, but still I want to be a citizen)

2) the output isn't writing to the file handlers (I can capture it by redirecting on the command line, but I prefer the file handlers.)
0
 
ozoCommented:
1) sleep
2) print OUTPUT $res->status_line, "\n";
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.