[Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 295
  • Last Modified:

Reading page on other site

Hello,

Is it possible to start a script on an other site by clicking a URL and save the result to a file on my site??
0
mmcw
Asked:
mmcw
  • 22
  • 14
  • 3
1 Solution
 
maneshrCommented:
here is how i would try to solve the problem.

lets say there are 2 servers (A & B).

The script is on server B and the results are to be stored on server A.

using PERL i can write a script on server A that will remotely invoke a URL on server B.

this URL would then execute a script on server B, do some processing and store the results in a file on server B & send the final results to the calling script on server A.

The calling script on server A can then store the results on server A in a file and/or display it in the browser!!

if this is what you want i can write both the script (for server A and B) for you.
0
 
mmcwAuthor Commented:
On the server A I can not write/change or edit a script or do something.

I only can start a form with will produce a page with a lot of data. The data I want to use is currency data. This is on this page!!

I want to read this page what has been produced on the server A and write  and save it to my server B. The I can extract the data from this file using my server where I can use a perl script to do so!

Can you help me??
0
 
maneshrCommented:
ok. now get me get this clear.

your server, server B, will call some URL on server A. You then want to store the results on server B for later use.

if that is right, i have a script for you.

also if that is right can you provide me the URL for server B, so that i can test the script before i post the final answer here??
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
ozoCommented:
perl -MLWP::Simple -e 'getprint "http://experts-exchange.com/Q_10337436.html"'
0
 
mmcwAuthor Commented:
ServerB:
mmcw.hypermart.net
0
 
mmcwAuthor Commented:
Question to OZO:

Where do I place the comment you gave???
Could you give me some more details??

0
 
ozoCommented:
you would type it at your command prompt.
to get more details, type
perldoc LWP::Simple
0
 
mmcwAuthor Commented:
Question to OZO:

I want to add it to a perl script.
A have no tellnet access!!
0
 
ozoCommented:
use LWP::Simple;
$rc = getstore('http://experts-exchange.com/Q_10337436.html','myfile');
print status_message($rc), "\n";
0
 
mmcwAuthor Commented:
Thank you OZO!
But what if the module LWP::simple is not installed and it is not possible to install it.
0
 
mmcwAuthor Commented:
maneshr:

You told me you had an answer?
But you had to test it first.

Do you still have an answer??

Please lett me hear!
0
 
maneshrCommented:
yes i do,

however since you already have one with you, i dont think my solution would add any more to it.

0
 
mmcwAuthor Commented:
Could you tell me if your solution also uses LWP::Simple.

How to check if this module is installed on my server. On the servers I use I can not install modules.

On one of my servers I use I think I do not have installed this module!!
0
 
maneshrCommented:
no it does not use LWP::Simple. my solution uses URI and HTTP modules.

this is how i check if a module is installed on my server.

type perldoc LWP from the command prompt.
if you get the documentation, then the module is installed.

0
 
mmcwAuthor Commented:
I want to use it into the script!!

and could you pass me your solution for the question?
0
 
maneshrCommented:
here is your script...

here i am just fetching the homepage of yahoo.com and storing the output in yahoo.html in my server's tmp directory.

The same script, with slight changes, can also submit results to a search engine and get results or get daily news headlines or get whatever else you might want.

===yahoo.pl
#!/usr/local/bin/perl

require LWP;
require URI::URL;

use strict;
use CGI;

my($hdr,$server_response);
my($statement_URL)="http://www.yahoo.com";

$server_response=&browse($statement_URL,$hdr);  ##  Fire the URL

print "Content-type: text/html\n\n";

open(TMP,">/tmp/yahoo.html") || die $!;
print TMP $server_response;
close(TMP);
print $statement_URL," obtained!!<P>\n";
 
sub browse(){
  my($statement_URL,$hdr)=@_;
  my($content_type,$method);

  $content_type="application/x-www-form-urlencoded";
  $method="GET";

  my($headers)= new HTTP::Headers
    'Content-Type'   =>  $content_type,
    'MIME-Version'   =>  '1.0',
    'Date'           =>  HTTP::Date::time2str(time),
    'Accept'         =>  'text/html';

  my($ua)= new LWP::UserAgent;

  $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_
AGENT

  my($url)= new URI::URL($statement_URL);
  my($request)= new HTTP::Request($method, $url, $headers);

  my($response)= $ua->request($request);

  my($reply);

  if ($response->is_success){
    $reply=$response->content;
  }else{
    $reply=$response->error_as_HTML();
  }
  return $reply;
}
0
 
maneshrCommented:
were you able to try out this script??
how did it work??

let me know.
0
 
mmcwAuthor Commented:
I will try it!!

Another question:
Could zou take another look at my question: Read one value in file

Question: Sometimes the number looks like this: 0.4125

The script workes fine.

But somethimes the number looks like this: 1,926.22

The script will not work.

I have checked the script:

It will work till:
$html_text =~ /$$base_currency\s+=\s+(\d+\.?\d+)/;

The the sesult will be "";

Can zou fix it to make it work for both format of numbers?
0
 
mmcwAuthor Commented:
Question to OZO and Maneshr:

The script works but I need now to modules to get it to work!!

The module LWP is the same OZO used in his example!
I do not have this module installed on my computer!!

Is there an other way???

I need it urgently!!

greetings Michel
0
 
maneshrCommented:
you can go to www.cpan.org (more specifically http://www.cpan.org/authors/id/GAAS/libwww-perl-5.48.tar.gz)
download and install the module.

Libwww-perl is a collection of Perl modules which provides a simple
and consistent application programming interface (API) to the World-Wide Web.  The main focus of the library is to provide classes and functions that allow you to write WWW clients, thus libwww-perl said to be a WWW client library. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

PREREQUISITES

In order to install and use this package you will need Perl version
5.004 or better.  Some modules within this package depend on other
packages that are distributed separately from Perl.  It is recommend that you have the following packages installed before you install
libwww-perl:

  URI
  MIME-Base64
  HTML-Parser
  libnet
  Digest::MD5

These packages should be available on CPAN.

If you want to access sites using the https protocol, then you need to
install the Crypt::SSLeay or the IO::Socket::SSL module.  The
README.SSL file will tell you more about how libwww-perl supports SSL.


INSTALLATION

You install libwww-perl, as you would install any perl module library,
by running these commands:

   perl Makefile.PL
   make
   make test
   make install

Hope that helps.
0
 
mmcwAuthor Commented:
I do not have tellnet access or something to install a module myself!!
0
 
maneshrCommented:
"I do not have tellnet access or something to install a module myself!!" :(

unfortunately then the above script might not be of a lot of help to you.

i have a simpler, socket based version of the script that i gave you above. this does not use the libraries and should work with your server.

==============simple_yahoo.pl
#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";

$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/
;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print S "$request\n";
while (<S>) {
  print $_;
}
close (S);
0
 
mmcwAuthor Commented:
I tried the script but I got an 500 error: Premature end of script headers

How is the output file called and where is it situated??

Can you help me??
0
 
maneshrCommented:
Oh!! my mistake. you are supposed to run the script from the command prompt.


if you want to run it from the browser pl add .....

print "Content-type: text/html\n\n";

before ......

print S "$request\n";
0
 
mmcwAuthor Commented:
I tried this code to start from a browser:

#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";
            
$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print "Content-type: text/html\n\n";

print S "$request\n";
while (<S>) {
  print $_;
}
close (S);

without success!!
What is wrong??

And how is the output file called??
0
 
maneshrCommented:
."without success!! "

what  does/does not happen?? please provide details of what you tried and what output you got.

...."And how is the output file called?? "

i am not clear about what you are trying to say here. pl. clarify further.

Thanks
0
 
mmcwAuthor Commented:
I tried it on a hypermart server (bsdos and serverport=80) and I got an error!

I tried it on another server (freebsd and serverport=80) it workes fine only the output is to the screen. The output has to go to a file!!

How to write it to a file and what could be wrong on the hypermart server??
0
 
maneshrCommented:
to store the output to a file, change the foll piece of code ...

select (S);
                    $| = 1;
                    select (STDOUT);
                    $| = 1;

                    print "Content-type: text/html\n\n";

                    print S "$request\n";
                    while (<S>) {
                      print $_;
                    }
                    close (S);


to....
select (S);
$| = 1;

print S "$request\n";
$out_file="/tmp/yahoo.html";
open(TMP,">$out_file")||die $!;
select (TMP);
$| = 1;
while (<S>) {
  print TMP $_;
}
close (S);
close(TMP);

print "Content-type: text/html\n\n";
print $out_file," created<P>\n";
================================================

"...I tried it on a hypermart server (bsdos and serverport=80) and I got an error! "

i think its because most of these free hosting servers dont allow a socket connection to be opened from their servers to external sites.

i remember having seeing a somewhat similar problem on tripod too!!
0
 
mmcwAuthor Commented:
What I want is to copy the html file to my server.
For example I want to copy the page: http://www.yahoo.com 

to my server at:

/home/geenidee/cgi-bin/Shop/test.html

That was what  want and that is what you firsy script and OZO's script does using modules!

Can you help me??
0
 
mmcwAuthor Commented:
Thank you fot your quick answer!!

But I still can not get it to work on the other server. It worked here to print it to screen.

I tried your update to write it to a file: See the code I used:

#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";
            
$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;

print S "$request\n";
$out_file="/tmp/yahoo.html";
open(TMP,">$out_file")||die $!;
select (TMP);
$| = 1;
while (<S>) {
  print TMP $_;
}
close (S);
close(TMP);

print "Content-type: text/html\n\n";
print $out_file," created<P>\n";

But I get a error 500!
In the log file I get the message:
Premature end of script headers: /home/geenidee/cgi-bin/text.pl

What is wrong????
0
 
maneshrCommented:
the reason could be because the user under which the CGI  process runs does not have write permission to the /tmp directory or the /tmp directory does not exist.

to confirm that your self move ...

print "Content-type: text/html\n\n"

before....

open(TMP,">$out_file")||die $!;

& see if you can get any message on the browser.

let me know of the same.
0
 
mmcwAuthor Commented:
I only get the error 500 message.
The log file does only say what I said before!!

Printing to screen workes fine!
0
 
mmcwAuthor Commented:
When I print to screen the first ines of the page are:

HTTP/1.0 200 OK
Content-Length: 12125
Content-Type: text/html

Can that be the problem when I try to write to a file??
0
 
mmcwAuthor Commented:
I tried this code and it worked.

#!/bin/perl5.005_03

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";

$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print "Content-type: text/html\n\n";

$out_file = "/v/w/w/c/wwckaars/docs/news/data/yahoo.html";
open(TMP,">$out_file")||die $!;
print S "$request\n";
while (<S>) {
      print TMP $_;
}
close (S);
close (TMP);

print "$out_file created\n";

Can I use this code without any problems you think??

When I look into the writen file I see the wanted html file, but their are three lines above it!

HTTP/1.0 200 OK
Content-Length: 11945
Content-Type: text/html

Why are these lines writen to the file??
0
 
mmcwAuthor Commented:
Is it also possible to check if for this:

i think its because most of these free hosting servers dont allow a socket connection to be opened from their servers to external sites.

So I get no error when try to use the code on a server who will not allow you.
0
 
mmcwAuthor Commented:
Another little question:

die "socket ok\n" if $debug;

Where do I find this $debug??
and what will it do??

0
 
maneshrCommented:
$debug is a variable that is to be declared at the start of the program. (perhaps line 2 or 3).

$debug can take two values 0 or 1.

when $debug is 0 it means debugging is turned off.

when $debug is 1 debugging is on and all those special debugging statements (like die "socket ok\n" if $debug; ) will be executed.

its more a debugging method for the programmer/developer.

Hope that helps.
0
 
mmcwAuthor Commented:
Thank you a lot!!!
0
 
maneshrCommented:
most welcome :-)

hope you have the solution working on all your servers!!

Wish you all the best.

Rgds
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 22
  • 14
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now