Solved

Get URL content in batch mode.

Posted on 1997-12-14
1
1,208 Views
Last Modified: 2013-12-25
I need to be able to get the content of URLs (typically Web pages) in batch mode. That means, I need to get Web pages content automatically through a command line and then be able to anlyze it. I also need to use such tool in order to get content of HTML FORM. I used a webget.pl Perl script but  
I cannot get result setting a time out value in order to fire a program if the URL is not available.

I would run this batch on my NT4.0 server.

Thanks in advance.
0
Comment
Question by:joel011197
1 Comment
 

Accepted Solution

by:
akis earned 50 total points
ID: 1831647
You may want to look at LIBWWW Perl modules(ver 5.10 is the latest version, from CPAN).  These allow you to have a Perl script actually visit a web site/page, sent GET or POST requests and receive content. You can then parse that page content (received as text of course) and do your stuff.

Example:
Assuming that array @URL has all URLs that you want to visit:

##*******************************
use LWP::UserAgent;
use HTTP::Request;
my $url
my $ua = new LWP::UserAgent;
foreach $url(@URL) {
  my $request = new HTTP::Request 'GET', $url;
  my $response = $ua->request($request);
  if ($response->is_success) {
      print $response->content(); # to print URL's text
  } else {
      print "Failed:".$response->as_string()."\n";
  }
}

0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some…
This tutorial will discuss the log-in process using WhizBase. In this article I assume you already know HTML. I will write the code using WhizBase Server Pages, so you need to know some basics in WBSP (you might look at some of my other articles abo…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

759 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now