Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Get URL content in batch mode.

Posted on 1997-12-14
1
Medium Priority
?
1,229 Views
Last Modified: 2013-12-25
I need to be able to get the content of URLs (typically Web pages) in batch mode. That means, I need to get Web pages content automatically through a command line and then be able to anlyze it. I also need to use such tool in order to get content of HTML FORM. I used a webget.pl Perl script but  
I cannot get result setting a time out value in order to fire a program if the URL is not available.

I would run this batch on my NT4.0 server.

Thanks in advance.
0
Comment
Question by:joel011197
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 

Accepted Solution

by:
akis earned 100 total points
ID: 1831647
You may want to look at LIBWWW Perl modules(ver 5.10 is the latest version, from CPAN).  These allow you to have a Perl script actually visit a web site/page, sent GET or POST requests and receive content. You can then parse that page content (received as text of course) and do your stuff.

Example:
Assuming that array @URL has all URLs that you want to visit:

##*******************************
use LWP::UserAgent;
use HTTP::Request;
my $url
my $ua = new LWP::UserAgent;
foreach $url(@URL) {
  my $request = new HTTP::Request 'GET', $url;
  my $response = $ua->request($request);
  if ($response->is_success) {
      print $response->content(); # to print URL's text
  } else {
      print "Failed:".$response->as_string()."\n";
  }
}

0

Featured Post

Not sure which OpenStack Certification to get?

So you’ve realized you might want to get certified in OpenStack, but you’re not sure what the benefits might be or even which one you should take. You know there are several certification courses you can choose from, but how do you know which one is right for you?

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this tutorial I will show you how to make a simple HTML bar chart with the usage of WhizBase, If you want more information about WhizBase please read my previous articles at http://www.experts-exchange.com/ARTH_5123186.html (http://www.experts-ex…
This article will show, step by step, how to integrate R code into a R Sweave document
Learn the basics of if, else, and elif statements in Python 2.7. Use "if" statements to test a specified condition.: The structure of an if statement is as follows: (CODE) Use "else" statements to allow the execution of an alternative, if the …
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question