Getting HTML Source from Remote URL!

I'm trying to create a small search engine submission script.

How do I get the source of a remote URL? (and) How do I access this URL with Perl?

I want to access..say..www.altavista.com/submit.cgi?url=mypage.com, then search the returned HTML code for "Successful submission".

Thanks!

Magecast
magecastAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
colindConnect With a Mentor Commented:
This is stolen almost directly from "Perl 5 by Example".  It uses the Socket module, but that's it.

sub http_get{
($_) = $address;           # Usage: httpget URL
($site, $url) = /^http:\/\/([^\/]*)(\/.*)/;
$_ = $site; ($desthost, $port) = /([^:]*)(.*)/;
$port = 80 unless $port;        # Default http port is 80
use Socket;
#chop($thishost = `hostname`);
$proto = (getprotobyname('tcp'))[2];
$port = (getservbyname($port, 'tcp'))[2] unless $port =~ /^\d+$/;
$thisaddr = (gethostbyname($thishost))[4];
$thataddr = (gethostbyname($desthost))[4] ||
        return "Unknown host '$desthost'";
        $this = pack('S n a4 x8', AF_INET, 0, $thisaddr);
        $that = pack('S n a4 x8', AF_INET, $port, $thataddr);
        # Create the connection to a remote server.
        socket(S, PF_INET, SOCK_STREAM, $proto) || print "socket: $!";
        bind(S, $this) || print "bind: $!";
        connect(S, $that) || print "connect: $!";
        select(S); $| = 1; select(stdout);
        print S "GET $url\r\n"; # Send request to get the document.
        while(<S>) { print output;
        print;}   # Read back the result.

}

0
 
b2piCommented:
Simplest case (Start with

perldoc lwpcook

if this is overly simple....)

#!/usr/bin/perl -w
use strict;

use LWP::Simple;

my($doc) = get('http://www.altavista.com/submit.cgi?url=mypage.com');
$/ = undef;
if ($doc =~ m/successful submission/i) {
    print "Hey, that worked!\n";
} else {
   print "Uhoh, need to get more complex\n";
}


0
 
b2piCommented:
By the way, you now have 4 questions locked and waiting for your response.  It is polite to either accept a given answer, ask for further clarification, or reject the answer...
0
 
magecastAuthor Commented:
Ok, I tried installing libwww-perl-5.41 but I need URI and a bunch of other stuff (where can I get this?) and it was just way too complicated.

Is there a way to do this without bringing in all these other modules?

Or is there any easy way I can just get a non-dependant module that does this job?

Thanks!
Matt

Thanks for letting me know about the locked questions b2..didn't know I had any.  I'll grade em now.
0
 
b2piCommented:
There is a way to do this without bringing in all those other modules, but you could also do it in assembly language, and not worry about any of the wheels that other people have invented.

If you're going to do any www work, you really want libwww and cgi::* installed.  

1.) Are you using activestate or the standard distribution?
2.) Do you have either visual c++ or borland c++
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.