Solved

Reading page on other site

Posted on 2000-05-02
39
229 Views
Last Modified: 2010-03-05
Hello,

Is it possible to start a script on an other site by clicking a URL and save the result to a file on my site??
0
Comment
Question by:mmcw
  • 22
  • 14
  • 3
39 Comments
 
LVL 16

Expert Comment

by:maneshr
ID: 2769685
here is how i would try to solve the problem.

lets say there are 2 servers (A & B).

The script is on server B and the results are to be stored on server A.

using PERL i can write a script on server A that will remotely invoke a URL on server B.

this URL would then execute a script on server B, do some processing and store the results in a file on server B & send the final results to the calling script on server A.

The calling script on server A can then store the results on server A in a file and/or display it in the browser!!

if this is what you want i can write both the script (for server A and B) for you.
0
 

Author Comment

by:mmcw
ID: 2770083
On the server A I can not write/change or edit a script or do something.

I only can start a form with will produce a page with a lot of data. The data I want to use is currency data. This is on this page!!

I want to read this page what has been produced on the server A and write  and save it to my server B. The I can extract the data from this file using my server where I can use a perl script to do so!

Can you help me??
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2770155
ok. now get me get this clear.

your server, server B, will call some URL on server A. You then want to store the results on server B for later use.

if that is right, i have a script for you.

also if that is right can you provide me the URL for server B, so that i can test the script before i post the final answer here??
0
 
LVL 84

Expert Comment

by:ozo
ID: 2770182
perl -MLWP::Simple -e 'getprint "http://experts-exchange.com/Q_10337436.html"'
0
 

Author Comment

by:mmcw
ID: 2770335
ServerB:
mmcw.hypermart.net
0
 

Author Comment

by:mmcw
ID: 2770345
Question to OZO:

Where do I place the comment you gave???
Could you give me some more details??

0
 
LVL 84

Expert Comment

by:ozo
ID: 2770618
you would type it at your command prompt.
to get more details, type
perldoc LWP::Simple
0
 

Author Comment

by:mmcw
ID: 2772175
Question to OZO:

I want to add it to a perl script.
A have no tellnet access!!
0
 
LVL 84

Expert Comment

by:ozo
ID: 2772534
use LWP::Simple;
$rc = getstore('http://experts-exchange.com/Q_10337436.html','myfile');
print status_message($rc), "\n";
0
 

Author Comment

by:mmcw
ID: 2772743
Thank you OZO!
But what if the module LWP::simple is not installed and it is not possible to install it.
0
 

Author Comment

by:mmcw
ID: 2776518
maneshr:

You told me you had an answer?
But you had to test it first.

Do you still have an answer??

Please lett me hear!
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2777012
yes i do,

however since you already have one with you, i dont think my solution would add any more to it.

0
 

Author Comment

by:mmcw
ID: 2777041
Could you tell me if your solution also uses LWP::Simple.

How to check if this module is installed on my server. On the servers I use I can not install modules.

On one of my servers I use I think I do not have installed this module!!
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2777123
no it does not use LWP::Simple. my solution uses URI and HTTP modules.

this is how i check if a module is installed on my server.

type perldoc LWP from the command prompt.
if you get the documentation, then the module is installed.

0
 

Author Comment

by:mmcw
ID: 2777334
I want to use it into the script!!

and could you pass me your solution for the question?
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2778806
here is your script...

here i am just fetching the homepage of yahoo.com and storing the output in yahoo.html in my server's tmp directory.

The same script, with slight changes, can also submit results to a search engine and get results or get daily news headlines or get whatever else you might want.

===yahoo.pl
#!/usr/local/bin/perl

require LWP;
require URI::URL;

use strict;
use CGI;

my($hdr,$server_response);
my($statement_URL)="http://www.yahoo.com";

$server_response=&browse($statement_URL,$hdr);  ##  Fire the URL

print "Content-type: text/html\n\n";

open(TMP,">/tmp/yahoo.html") || die $!;
print TMP $server_response;
close(TMP);
print $statement_URL," obtained!!<P>\n";
 
sub browse(){
  my($statement_URL,$hdr)=@_;
  my($content_type,$method);

  $content_type="application/x-www-form-urlencoded";
  $method="GET";

  my($headers)= new HTTP::Headers
    'Content-Type'   =>  $content_type,
    'MIME-Version'   =>  '1.0',
    'Date'           =>  HTTP::Date::time2str(time),
    'Accept'         =>  'text/html';

  my($ua)= new LWP::UserAgent;

  $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_
AGENT

  my($url)= new URI::URL($statement_URL);
  my($request)= new HTTP::Request($method, $url, $headers);

  my($response)= $ua->request($request);

  my($reply);

  if ($response->is_success){
    $reply=$response->content;
  }else{
    $reply=$response->error_as_HTML();
  }
  return $reply;
}
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2780939
were you able to try out this script??
how did it work??

let me know.
0
 

Author Comment

by:mmcw
ID: 2783414
I will try it!!

Another question:
Could zou take another look at my question: Read one value in file

Question: Sometimes the number looks like this: 0.4125

The script workes fine.

But somethimes the number looks like this: 1,926.22

The script will not work.

I have checked the script:

It will work till:
$html_text =~ /$$base_currency\s+=\s+(\d+\.?\d+)/;

The the sesult will be "";

Can zou fix it to make it work for both format of numbers?
0
 

Author Comment

by:mmcw
ID: 2793416
Question to OZO and Maneshr:

The script works but I need now to modules to get it to work!!

The module LWP is the same OZO used in his example!
I do not have this module installed on my computer!!

Is there an other way???

I need it urgently!!

greetings Michel
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 16

Expert Comment

by:maneshr
ID: 2793460
you can go to www.cpan.org (more specifically http://www.cpan.org/authors/id/GAAS/libwww-perl-5.48.tar.gz)
download and install the module.

Libwww-perl is a collection of Perl modules which provides a simple
and consistent application programming interface (API) to the World-Wide Web.  The main focus of the library is to provide classes and functions that allow you to write WWW clients, thus libwww-perl said to be a WWW client library. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

PREREQUISITES

In order to install and use this package you will need Perl version
5.004 or better.  Some modules within this package depend on other
packages that are distributed separately from Perl.  It is recommend that you have the following packages installed before you install
libwww-perl:

  URI
  MIME-Base64
  HTML-Parser
  libnet
  Digest::MD5

These packages should be available on CPAN.

If you want to access sites using the https protocol, then you need to
install the Crypt::SSLeay or the IO::Socket::SSL module.  The
README.SSL file will tell you more about how libwww-perl supports SSL.


INSTALLATION

You install libwww-perl, as you would install any perl module library,
by running these commands:

   perl Makefile.PL
   make
   make test
   make install

Hope that helps.
0
 

Author Comment

by:mmcw
ID: 2793522
I do not have tellnet access or something to install a module myself!!
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2793814
"I do not have tellnet access or something to install a module myself!!" :(

unfortunately then the above script might not be of a lot of help to you.

i have a simpler, socket based version of the script that i gave you above. this does not use the libraries and should work with your server.

==============simple_yahoo.pl
#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";

$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/
;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print S "$request\n";
while (<S>) {
  print $_;
}
close (S);
0
 

Author Comment

by:mmcw
ID: 2796622
I tried the script but I got an 500 error: Premature end of script headers

How is the output file called and where is it situated??

Can you help me??
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2797372
Oh!! my mistake. you are supposed to run the script from the command prompt.


if you want to run it from the browser pl add .....

print "Content-type: text/html\n\n";

before ......

print S "$request\n";
0
 

Author Comment

by:mmcw
ID: 2797474
I tried this code to start from a browser:

#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";
            
$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print "Content-type: text/html\n\n";

print S "$request\n";
while (<S>) {
  print $_;
}
close (S);

without success!!
What is wrong??

And how is the output file called??
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2797492
."without success!! "

what  does/does not happen?? please provide details of what you tried and what output you got.

...."And how is the output file called?? "

i am not clear about what you are trying to say here. pl. clarify further.

Thanks
0
 

Author Comment

by:mmcw
ID: 2797504
I tried it on a hypermart server (bsdos and serverport=80) and I got an error!

I tried it on another server (freebsd and serverport=80) it workes fine only the output is to the screen. The output has to go to a file!!

How to write it to a file and what could be wrong on the hypermart server??
0
 
LVL 16

Accepted Solution

by:
maneshr earned 100 total points
ID: 2797552
to store the output to a file, change the foll piece of code ...

select (S);
                    $| = 1;
                    select (STDOUT);
                    $| = 1;

                    print "Content-type: text/html\n\n";

                    print S "$request\n";
                    while (<S>) {
                      print $_;
                    }
                    close (S);


to....
select (S);
$| = 1;

print S "$request\n";
$out_file="/tmp/yahoo.html";
open(TMP,">$out_file")||die $!;
select (TMP);
$| = 1;
while (<S>) {
  print TMP $_;
}
close (S);
close(TMP);

print "Content-type: text/html\n\n";
print $out_file," created<P>\n";
================================================

"...I tried it on a hypermart server (bsdos and serverport=80) and I got an error! "

i think its because most of these free hosting servers dont allow a socket connection to be opened from their servers to external sites.

i remember having seeing a somewhat similar problem on tripod too!!
0
 

Author Comment

by:mmcw
ID: 2797553
What I want is to copy the html file to my server.
For example I want to copy the page: http://www.yahoo.com

to my server at:

/home/geenidee/cgi-bin/Shop/test.html

That was what  want and that is what you firsy script and OZO's script does using modules!

Can you help me??
0
 

Author Comment

by:mmcw
ID: 2797594
Thank you fot your quick answer!!

But I still can not get it to work on the other server. It worked here to print it to screen.

I tried your update to write it to a file: See the code I used:

#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";
            
$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;

print S "$request\n";
$out_file="/tmp/yahoo.html";
open(TMP,">$out_file")||die $!;
select (TMP);
$| = 1;
while (<S>) {
  print TMP $_;
}
close (S);
close(TMP);

print "Content-type: text/html\n\n";
print $out_file," created<P>\n";

But I get a error 500!
In the log file I get the message:
Premature end of script headers: /home/geenidee/cgi-bin/text.pl

What is wrong????
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2797886
the reason could be because the user under which the CGI  process runs does not have write permission to the /tmp directory or the /tmp directory does not exist.

to confirm that your self move ...

print "Content-type: text/html\n\n"

before....

open(TMP,">$out_file")||die $!;

& see if you can get any message on the browser.

let me know of the same.
0
 

Author Comment

by:mmcw
ID: 2799596
I only get the error 500 message.
The log file does only say what I said before!!

Printing to screen workes fine!
0
 

Author Comment

by:mmcw
ID: 2799604
When I print to screen the first ines of the page are:

HTTP/1.0 200 OK
Content-Length: 12125
Content-Type: text/html

Can that be the problem when I try to write to a file??
0
 

Author Comment

by:mmcw
ID: 2799651
I tried this code and it worked.

#!/bin/perl5.005_03

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";

$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print "Content-type: text/html\n\n";

$out_file = "/v/w/w/c/wwckaars/docs/news/data/yahoo.html";
open(TMP,">$out_file")||die $!;
print S "$request\n";
while (<S>) {
      print TMP $_;
}
close (S);
close (TMP);

print "$out_file created\n";

Can I use this code without any problems you think??

When I look into the writen file I see the wanted html file, but their are three lines above it!

HTTP/1.0 200 OK
Content-Length: 11945
Content-Type: text/html

Why are these lines writen to the file??
0
 

Author Comment

by:mmcw
ID: 2799659
Is it also possible to check if for this:

i think its because most of these free hosting servers dont allow a socket connection to be opened from their servers to external sites.

So I get no error when try to use the code on a server who will not allow you.
0
 

Author Comment

by:mmcw
ID: 2799727
Another little question:

die "socket ok\n" if $debug;

Where do I find this $debug??
and what will it do??

0
 
LVL 16

Expert Comment

by:maneshr
ID: 2800270
$debug is a variable that is to be declared at the start of the program. (perhaps line 2 or 3).

$debug can take two values 0 or 1.

when $debug is 0 it means debugging is turned off.

when $debug is 1 debugging is on and all those special debugging statements (like die "socket ok\n" if $debug; ) will be executed.

its more a debugging method for the programmer/developer.

Hope that helps.
0
 

Author Comment

by:mmcw
ID: 2803750
Thank you a lot!!!
0
 
LVL 16

Expert Comment

by:maneshr
ID: 2804485
most welcome :-)

hope you have the solution working on all your servers!!

Wish you all the best.

Rgds
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now