Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 284
  • Last Modified:

Feeding URLs from a file to a UserAgent script/Checking if a URL is live

I need a script that will take take urls in a file and check to see if the URL is live or not. I came up with this:


#!/usr/local/bin/perl
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");


open (INPUT, '<url.txt');
open (OUTPUT, '>url.html');

 while (<INPUT>) {
       chomp;

my $req = HTTP::Request->new(POST => '$_');
my $res = $ua->request($req);

  # Check the outcome of the response
  if ($res->is_success) {
      print $res->content;
  }
  else {
      print $res->status_line, "\n";
  }

 }
 
 print OUTPUT;

 close (INPUT);
 
However when I run it I get a "400 URL must be absolute". When I plug the URL in directly into my $req = HTTP::Request->new(POST => 'www....'); it works fine, so its not my connection.
0
hadrons
Asked:
hadrons
  • 2
1 Solution
 
ozoCommented:
my $req = HTTP::Request->new(POST => $_);
0
 
hadronsAuthor Commented:
Excellent ... I have just additional questions:

1) is there a function that delays the next URL request so I don't hammer their servers too hard (not that they don't deserve it with all the dead URLs they sent, but still I want to be a citizen)

2) the output isn't writing to the file handlers (I can capture it by redirecting on the command line, but I prefer the file handlers.)
0
 
ozoCommented:
1) sleep
2) print OUTPUT $res->status_line, "\n";
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now