Solved

Get URL content in batch mode.

Posted on 1997-12-14
1
1,213 Views
Last Modified: 2013-12-25
I need to be able to get the content of URLs (typically Web pages) in batch mode. That means, I need to get Web pages content automatically through a command line and then be able to anlyze it. I also need to use such tool in order to get content of HTML FORM. I used a webget.pl Perl script but  
I cannot get result setting a time out value in order to fire a program if the URL is not available.

I would run this batch on my NT4.0 server.

Thanks in advance.
0
Comment
Question by:joel011197
1 Comment
 

Accepted Solution

by:
akis earned 50 total points
ID: 1831647
You may want to look at LIBWWW Perl modules(ver 5.10 is the latest version, from CPAN).  These allow you to have a Perl script actually visit a web site/page, sent GET or POST requests and receive content. You can then parse that page content (received as text of course) and do your stuff.

Example:
Assuming that array @URL has all URLs that you want to visit:

##*******************************
use LWP::UserAgent;
use HTTP::Request;
my $url
my $ua = new LWP::UserAgent;
foreach $url(@URL) {
  my $request = new HTTP::Request 'GET', $url;
  my $response = $ua->request($request);
  if ($response->is_success) {
      print $response->content(); # to print URL's text
  } else {
      print "Failed:".$response->as_string()."\n";
  }
}

0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

In this tutorial I will focus on how to use WhizBase as a tool for sending ICQ messages to ICQ. Here I will use a new technology in WhizBase, published in WhizBase 5.1 version. In this tutorial I will use 3 files, pager.wbsp for the processing, e…
In this tutorial I will show you how to make a simple HTML bar chart with the usage of WhizBase, If you want more information about WhizBase please read my previous articles at http://www.experts-exchange.com/ARTH_5123186.html (http://www.experts-ex…
Learn the basics of if, else, and elif statements in Python 2.7. Use "if" statements to test a specified condition.: The structure of an if statement is as follows: (CODE) Use "else" statements to allow the execution of an alternative, if the …
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…

832 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question