Solved

Perl Http Sockets problem...

Posted on 2001-07-05
21
375 Views
Last Modified: 2008-03-10
Hi, I'm trying to write a script that will emulate the filling in of a form on a web page using the POST method.

What I have below get's a response from the site with a confirmation page but the variables in the data seem to have been ignored.  

Does anyone have any idea's as to how to get this to work???

Or even if someone has some insight into HTTP that could be of use. I would be very appreciative.

#!PATH_TO_PERL
use IO::Socket;
use IO::Handle;

$proto = getprotobyname("tcp");
socket(SLASH, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
connect(SLASH, $paddr) or die "Connect error: $!";
autoflush SLASH 1;
print SLASH "POST name_of_page.asp\nReferer:name_of_referer_page.asp HTTP/1.0\r\n\r\nDATA_IN_HERE";

local $/ = "\n%%\n";

while(<SLASH>) {
  $out .= $_;
}
close SLASH;

@lines = split(/\r\n/, $out);

foreach(@lines) {
        s/\%/\\%/g;
        s/"/\"/g;

printf $_."\n";
}
printf "\n";
0
Comment
Question by:OnlyOne
  • 10
  • 7
  • 4
21 Comments
 
LVL 8

Expert Comment

by:bebonham
Comment Utility
it would be much easier to use LWP::UserAgent.


use LWP::UserAgent;
   $ua = new LWP::UserAgent;
   $ua->agent("AgentName/0.1 " . $ua->agent);

   # Create a request #formdatahash comes from the html page.
%FormDataHash={'field1'=>'value1', 'field2'=>'value2');
##comment out the above line one you have %FormDataHash being populated from the html.

my $qs='';
foreach $field(keys %FormDataHash)
{
$qs.=$field . '=' . $FormDataHash{$field} . '&';
}

$qs=substr($qs,0,length(qs)-1); # remove '&'
use URI::Escape;
$qs=uri_escape($qs);

$url='http://www.milestonemortgage.com/cgi-bin/pq.pl?' . $qs;

my $post = new HTTP::Request POST => "$qs";

   $post->content_type('application/x-www-form-urlencoded');

   # Pass request to the user agent and get a response back
   my $res = $ua->request($post);
   

   # Check the outcome of the response
   if ($res->is_success) {
       print $res->content;
   } else {
       print "Bad luck this time\n";
   }


hope that helps,

Bob
0
 
LVL 8

Expert Comment

by:bebonham
Comment Utility
when I had a problem like this, it turned out to be that the http headers I was sending weren't liked by the server I was hitting them with...

maneshr helped me get the headers straight...

this is the routine he wrote for me!


sub browse(){
    my($statement_URL,$hdr,$full)=@_;
    my($content_type,$method);

    $content_type="application/x-www-form-urlencoded";
    $method="POST";

    my($headers)= new HTTP::Headers
      'Content-Type'   =>  $content_type,
      'MIME-Version'   =>  '1.0',
      'Date'           =>  HTTP::Date::time2str(time),
      'Accept'         =>  'text/html';
   
    my($ua)= new LWP::UserAgent;
   
    $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_AGENT
   
    my($url)= new URI::URL($statement_URL);
    my($request)= new HTTP::Request($method, $url, $headers,$hdr);
   
    my($response)= $ua->request($request);

    my($reply);

     return ($response->as_string()) if $full;

    if ($response->is_success){
         $reply=$response->content;
    }else{
         $reply=$response->error_as_HTML();
    }
    return $reply;
}



Good Luck,

Bob
0
 
LVL 8

Expert Comment

by:bebonham
Comment Utility
of course your method is probably faster, but not a whole lot.

0
 
LVL 5

Expert Comment

by:marecs
Comment Utility
From RFC 1945

"A valid Content-Length is required on all HTTP/1.0 POST requests."

Why did you bother doing
use IO::Socket; because you use nothing in IO::Socket
What you probably meant was
use Socket;
which only loads the definitions from you socket.h file

If you want to use sockets I suggest you use them through the object orientated interface provided by IO::Socket

#!/usr/bin/perl -w

use IO::Socket;
$port = 80;
$host = "www.host.com";
$path = "/cgi-bin/formdebug.pl";
$data = "var1=one&var2=t%20w%20o&var3=tree";
$length = length $data;
$referer = "http://www.msn.com/";

$sock = IO::Socket::INET->new(PeerAddr => $host,
                                    PeerPort => $port,
                                    Proto    => 'tcp',
                                    Timeout  => 60) || die $!;
$sock->autoflush; # Not required for Perl 5.005

print $sock join("\015\012" =>
                    "POST $path HTTP/1.0",
                    "Host: $host",
                    "User-Agent: Mozilla 4.0",
                    "Referer: $referer",
                    "Content-Length: $length",
                    "", "");
print $sock $data;

while (<$sock>) {
    print;
}

$sock->close();
0
 

Author Comment

by:OnlyOne
Comment Utility
Thanks for the responses sofar.

I've tried out all of the above, the only one that is getting a response is the one from MARCES.

I think part of the reason that the others aren't working is that I'm working behind a proxy server and I don't know how to account for that in the other examples.
 
Even for MARCES's one the variables are still being ignored.
I've accounted for the proxy by setting the $host variable to my proxy server address which seem's to work.

The way I know that the variables are being ignored is that the response page has some javascript in it which has the variables set from the previous page.  
These are apprearing as "" which means that it has not read the variables in from the perl script (POST request).

I know that it can be done as what I'm using at the moment is calling the "curl" program and inputting the data using the "-d" option of curl.  I just want to do the whole program in perl as it will be more portable and the curl program will not have to be installed where ever I want to use this script.

So as it stands at the moment...

MARCES script get the response page from the server but variables are ignored (which is what my script is doing but in a messier way).

BEBONHAM's scripts could work I just don't know how to set them up for working with proxies.

Any other ideas guy's?
0
 
LVL 5

Expert Comment

by:marecs
Comment Utility
This worked for me, and adheres to RFC1945. http://www.faqs.org/rfcs/rfc1945.html

I will let bebonham give you the proxy implementation for his solution. His solution is probably the one you should be using.

#!/usr/bin/perl -w

use IO::Socket;
$port = 8080; # proxy port
$proxy = "proxy.you.com";
$url = "http://www.you.com/cgi-bin/formdebug.pl";
$data = "var1=one&var2=t%20w%20o&var3=tree";
$length = length $data;
$referer = "http://www.msn.com/";

$sock = IO::Socket::INET->new(PeerAddr => $proxy,
                                    PeerPort => $port,
                                    Proto    => 'tcp',
                                    Timeout  => 60) || die $!;
$sock->autoflush; # Not required for Perl 5.005

print $sock join("\015\012" =>
                    "POST $url HTTP/1.0",
                    "User-Agent: Mozilla 4.0",
                    "Referer: $referer",
                    "Content-Length: $length",
                    "", "");
print $sock $data;

while (<$sock>) {
    print;
}

$sock->close();
0
 
LVL 8

Expert Comment

by:bebonham
Comment Utility
this is a slight modification on maneshr's browse routine, which worked well for me in the past.

but I make no statement that lwp is better than sockets, simply that I know lwp a little better, and I think it is easier to do http with it.


sub browse(){
   my($statement_URL,$hdr,$full)=@_;
   my($content_type,$method);

   $content_type="application/x-www-form-urlencoded";
   $method="POST";

   my($headers)= new HTTP::Headers
     'Content-Type'   =>  $content_type,
     'MIME-Version'   =>  '1.0',
     'Date'           =>  HTTP::Date::time2str(time),
     'Accept'         =>  'text/html';
   
   my($ua)= new LWP::UserAgent;
   
   $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_AGENT
########################
###here we set the proxy####
$ua->proxy('http', 'http://proxy.sn.no:8080/');

   
   my($url)= new URI::URL($statement_URL);
   my($request)= new HTTP::Request($method, $url, $headers,$hdr);
   
   my($response)= $ua->request($request);

   my($reply);

    return ($response->as_string()) if $full;

   if ($response->is_success){
        $reply=$response->content;
   }else{
        $reply=$response->error_as_HTML();
   }
   return $reply;
}

0
 

Author Comment

by:OnlyOne
Comment Utility
Ok still no joy.
Here is exactly what I'm trying to do...

What I'm trying to do is to send sms messages automatically from a script.  

The code below is for the site that I'm trying to use...  
But once it works for this it would work for any site with just a few changes.

I'm getting the confirmation window with no variables set in it.

If you want to try it out you'll need to change the COUNTRY_CODE to whatever is applicable to you (the %2b is for the + sign url-encoded),
and you'll need to change the MOBILE_NUMBER to whatever your phone is with the preceeding 0 removed ie. 094 is 94.

It work's for international mobiles so if you have a mobile handy you can try it out.  The confirmation page will have the variables set in it if it has worked.

Hope this makes it easier for you to see what I'm having the problem with.

What is bugging me is that I can do it with "curl" and not completely in perl.

use IO::Socket;
$port = 80; # proxy port
$proxy = "my_proxy_address";
$url = "http://sms.planetinternet.de/free-sms/SMSC.asp";
$data = "lcode=%2bCOUNTRY_CODE&tel=MOBILE_NUMBER&Message=hello";
$length = length $data;
$referer = "http:sms.planetinternet.de/free-sms/content_smsverschicken_netscape.asp";

$sock = IO::Socket::INET->new(PeerAddr => $proxy,
PeerPort => $port,
Proto=> 'tcp',
Timeout=> 60) || die $!;
$sock->autoflush; # Not required for Perl 5.005

print $sock join("\015\012" =>
"POST $url HTTP/1.0",
"User-Agent: Mozilla 4.0",
"Referer: $referer",
"Content-Length: $length",
"", "");
print $sock $data;

while (<$sock>) {
print;
}

$sock->close();

I'm using MARECS code as it is getting the confirmation back.  I still haven't gotten your's BEBONHAM working. I pressume that I'm doing something wrong but I keep getting the "Bad luck this time" which means that I'm not getting any confirmation at all.

0
 

Author Comment

by:OnlyOne
Comment Utility
Seeing as there has been no response since my last comment for a few days, I will be deleting this question tomorrow!!

Thank's for the info guy's.
0
 
LVL 5

Expert Comment

by:marecs
Comment Utility
I thought I posted this two days ago, but the site was probably down.

I used the code below but was still not able to get it to work. What Free SMS providers tend to do is make it VERY hard for guys like you to use scripts to send SMS's. The same is true for these guys. They rely heavily on cookies. I can handle the cookies, but they may be setting cookies with JavaScript which I don't handle (But you can if you want to with a little bit of work)

I think your original question was answered, so you should award the points.


#!/usr/bin/perl -w

use HTTP::Request::Common qw(POST GET);
use LWP::UserAgent;
use HTTP::Cookies;

@MyAgent::ISA = qw(LWP::UserAgent); # set inheritance

$cookie_file = "Cookies";
$ua = MyAgent->new;
$cookie_jar = HTTP::Cookies->new(file => $cookie_file, autosave => 1);
$ua->agent('Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)');
$ua->cookie_jar($cookie_jar);
$ua->proxy('http', 'http://web-cache.surrey.ac.uk:8080/');

$req = GET 'http://sms.planetinternet.de/free-sms/content_nutzungsbedingungen.as
p';
$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->as_string();

$req = GET 'http://sms.planetinternet.de/free-sms/content_smsverschicken.asp';
$reply = $ua->request($req);
mep2cm@froude:~/Perl$ cat SMS
SMS.pl   SMS.pl~
mep2cm@froude:~/Perl$ cat SMS.pl
#!/usr/bin/perl -w

use HTTP::Request::Common qw(POST GET);
use LWP::UserAgent;
use HTTP::Cookies;

@MyAgent::ISA = qw(LWP::UserAgent); # set inheritance

$cookie_file = "Cookies";
$ua = MyAgent->new;
$cookie_jar = HTTP::Cookies->new(file => $cookie_file, autosave => 1);
$ua->agent('Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)');
$ua->cookie_jar($cookie_jar);
$ua->proxy('http', 'http://web-cache.surrey.ac.uk:8080/');

$req = GET 'http://sms.planetinternet.de/free-sms/content_nutzungsbedingungen.asp';
$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->as_string();

$req = GET 'http://sms.planetinternet.de/free-sms/content_smsverschicken.asp';
$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->as_string();

$message = "Hello";
$counter = 130 - length $message;
$country = "+44";
$phone = "7713206392";

$req = POST 'http://sms.planetinternet.de/free-sms/SMSC.asp',
[
    country => $country,
    lcode => $country,
    tel => $phone,
    counter => $counter,
    Message => $message,
    type => 'Gute Texte ...',
    huidigGetoondeLaag => '',
    adressessHidden => '',
    delayedHidden => '',
    delayedDayHidden => '',
    delayedMonthHidden => '',
    delayedYearHidden => '',
    delayedTimeHidden => '',
    processPDAHidden => 'true',
];

$reply = $ua->request($req);
print $reply->as_string();
$reply->is_success || die "Error";

print $reply->as_string();

sub MyAgent::redirect_ok { 1 } # redirects all requests
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:OnlyOne
Comment Utility
If you know of a program called "curl" which has a -d option for data and this works already, which is why I was hoping to do the full program in perl so there were no external dependancies for the script to work.  I don't think that the program is using cookies as all curl is doing is sending off the data to the server.

If there is no way of doing this then so be it.  But it would be nice to get a way of doing it...
0
 

Author Comment

by:OnlyOne
Comment Utility
Oh and don't worry I won't delete the question!!
0
 
LVL 5

Expert Comment

by:marecs
Comment Utility
cURL does handle cookies. It works now. What I needed to make it work was to support cookies, and to support redirects with POST requests. That is what the MyAgent is all about because you have to override the redirect_ok method in UserAgent which does not allow redirects of POST requests as default.

#!/usr/bin/perl -w

use HTTP::Request::Common qw(POST GET);
use LWP::UserAgent;
use HTTP::Cookies;

@MyAgent::ISA = qw(LWP::UserAgent); # set inheritance

$ua = MyAgent->new;
$cookie_jar = HTTP::Cookies->new;
$ua->agent('Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)');
$ua->cookie_jar($cookie_jar);
$ua->proxy('http', 'http://web-cache.you.com:8080/');

$message = "Hello";
$counter = 130 - length $message;
$country = "+44";
$phone = "7712345";

$req = POST 'http://sms.planetinternet.de/free-sms/SMSC.asp',
[
    country => $country,
    lcode => $country,
    tel => $phone,
    counter => $counter,
    Message => $message,
    type => 'Gute Texte ...',
    huidigGetoondeLaag => '',
    adressessHidden => '',
    delayedHidden => '',
    delayedDayHidden => '',
    delayedMonthHidden => '',
    delayedYearHidden => '',
    delayedTimeHidden => '',
    processPDAHidden => 'true',
];
$req->referer('http://sms.planetinternet.de/free-sms/content_smsverschicken.asp');

$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->content;

sub MyAgent::redirect_ok { 1 } # redirects all requests
0
 
LVL 5

Accepted Solution

by:
marecs earned 250 total points
Comment Utility
Did it work?
0
 

Author Comment

by:OnlyOne
Comment Utility
Sorry for no updates, I've been away for the last while.
I've not been able to get that code working yet.

There seem's to be an error in line 40.
$reply->is_success || die "Error";

I'm not sure what is wrong with it but when I comment it out I get nothing back at all.

I don't know if the script is working as I get no reply from the server or if there is something I'm doing wrong.

Any ideas?
0
 
LVL 5

Expert Comment

by:marecs
Comment Utility
Try

$reply->is_success || die $reply->error_as_HTML;

To see what error you get. It worked for me when I last tried it. I don't have a mobile to try it with right now.

0
 

Author Comment

by:OnlyOne
Comment Utility
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 

Author Comment

by:OnlyOne
Comment Utility
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 

Author Comment

by:OnlyOne
Comment Utility
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 

Author Comment

by:OnlyOne
Comment Utility
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 
LVL 5

Expert Comment

by:marecs
Comment Utility
My pleasure. If you have any further problems you know where to get help.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now