Solved

Feeding URLs from a file to a UserAgent script/Checking if a URL is live

Posted on 2013-11-14
3
278 Views
Last Modified: 2013-11-14
I need a script that will take take urls in a file and check to see if the URL is live or not. I came up with this:


#!/usr/local/bin/perl
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");


open (INPUT, '<url.txt');
open (OUTPUT, '>url.html');

 while (<INPUT>) {
       chomp;

my $req = HTTP::Request->new(POST => '$_');
my $res = $ua->request($req);

  # Check the outcome of the response
  if ($res->is_success) {
      print $res->content;
  }
  else {
      print $res->status_line, "\n";
  }

 }
 
 print OUTPUT;

 close (INPUT);
 
However when I run it I get a "400 URL must be absolute". When I plug the URL in directly into my $req = HTTP::Request->new(POST => 'www....'); it works fine, so its not my connection.
0
Comment
Question by:hadrons
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 39649469
my $req = HTTP::Request->new(POST => $_);
0
 

Author Closing Comment

by:hadrons
ID: 39649544
Excellent ... I have just additional questions:

1) is there a function that delays the next URL request so I don't hammer their servers too hard (not that they don't deserve it with all the dead URLs they sent, but still I want to be a citizen)

2) the output isn't writing to the file handlers (I can capture it by redirecting on the command line, but I prefer the file handlers.)
0
 
LVL 84

Expert Comment

by:ozo
ID: 39649833
1) sleep
2) print OUTPUT $res->status_line, "\n";
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
combine multiple lines 2 82
Matching a random pattern with one common character 2 100
Re-position the objects 7 132
Problem to macro 5 24
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question