Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Feeding URLs from a file to a UserAgent script/Checking if a URL is live

Posted on 2013-11-14
3
Medium Priority
?
280 Views
Last Modified: 2013-11-14
I need a script that will take take urls in a file and check to see if the URL is live or not. I came up with this:


#!/usr/local/bin/perl
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");


open (INPUT, '<url.txt');
open (OUTPUT, '>url.html');

 while (<INPUT>) {
       chomp;

my $req = HTTP::Request->new(POST => '$_');
my $res = $ua->request($req);

  # Check the outcome of the response
  if ($res->is_success) {
      print $res->content;
  }
  else {
      print $res->status_line, "\n";
  }

 }
 
 print OUTPUT;

 close (INPUT);
 
However when I run it I get a "400 URL must be absolute". When I plug the URL in directly into my $req = HTTP::Request->new(POST => 'www....'); it works fine, so its not my connection.
0
Comment
Question by:hadrons
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 2000 total points
ID: 39649469
my $req = HTTP::Request->new(POST => $_);
0
 

Author Closing Comment

by:hadrons
ID: 39649544
Excellent ... I have just additional questions:

1) is there a function that delays the next URL request so I don't hammer their servers too hard (not that they don't deserve it with all the dead URLs they sent, but still I want to be a citizen)

2) the output isn't writing to the file handlers (I can capture it by redirecting on the command line, but I prefer the file handlers.)
0
 
LVL 84

Expert Comment

by:ozo
ID: 39649833
1) sleep
2) print OUTPUT $res->status_line, "\n";
0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question