Solved

Get URL content in batch mode.

Posted on 1997-12-14
1
1,222 Views
Last Modified: 2013-12-25
I need to be able to get the content of URLs (typically Web pages) in batch mode. That means, I need to get Web pages content automatically through a command line and then be able to anlyze it. I also need to use such tool in order to get content of HTML FORM. I used a webget.pl Perl script but  
I cannot get result setting a time out value in order to fire a program if the URL is not available.

I would run this batch on my NT4.0 server.

Thanks in advance.
0
Comment
Question by:joel011197
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 

Accepted Solution

by:
akis earned 50 total points
ID: 1831647
You may want to look at LIBWWW Perl modules(ver 5.10 is the latest version, from CPAN).  These allow you to have a Perl script actually visit a web site/page, sent GET or POST requests and receive content. You can then parse that page content (received as text of course) and do your stuff.

Example:
Assuming that array @URL has all URLs that you want to visit:

##*******************************
use LWP::UserAgent;
use HTTP::Request;
my $url
my $ua = new LWP::UserAgent;
foreach $url(@URL) {
  my $request = new HTTP::Request 'GET', $url;
  my $response = $ua->request($request);
  if ($response->is_success) {
      print $response->content(); # to print URL's text
  } else {
      print "Failed:".$response->as_string()."\n";
  }
}

0

Featured Post

Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction:   Welcome to my first article ever. To begin with, the reason I write this article.  I participated in a question on Experts Exchange about the start command in Windows and there were some discussion about the usage. The discussio…
If you get a (Blue Screen of Death), your system writes a small file called a minidump. Your first step is to make certain your computer is setup to record memory dumps. Right click My Computer, choose properties. Click on the advanced tab, an…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question