Perl Http Sockets problem...

Hi, I'm trying to write a script that will emulate the filling in of a form on a web page using the POST method.

What I have below get's a response from the site with a confirmation page but the variables in the data seem to have been ignored.  

Does anyone have any idea's as to how to get this to work???

Or even if someone has some insight into HTTP that could be of use. I would be very appreciative.

#!PATH_TO_PERL
use IO::Socket;
use IO::Handle;

$proto = getprotobyname("tcp");
socket(SLASH, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
connect(SLASH, $paddr) or die "Connect error: $!";
autoflush SLASH 1;
print SLASH "POST name_of_page.asp\nReferer:name_of_referer_page.asp HTTP/1.0\r\n\r\nDATA_IN_HERE";

local $/ = "\n%%\n";

while(<SLASH>) {
  $out .= $_;
}
close SLASH;

@lines = split(/\r\n/, $out);

foreach(@lines) {
        s/\%/\\%/g;
        s/"/\"/g;

printf $_."\n";
}
printf "\n";
OnlyOneAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
marecsConnect With a Mentor Commented:
Did it work?
0
 
bebonhamCommented:
it would be much easier to use LWP::UserAgent.


use LWP::UserAgent;
   $ua = new LWP::UserAgent;
   $ua->agent("AgentName/0.1 " . $ua->agent);

   # Create a request #formdatahash comes from the html page.
%FormDataHash={'field1'=>'value1', 'field2'=>'value2');
##comment out the above line one you have %FormDataHash being populated from the html.

my $qs='';
foreach $field(keys %FormDataHash)
{
$qs.=$field . '=' . $FormDataHash{$field} . '&';
}

$qs=substr($qs,0,length(qs)-1); # remove '&'
use URI::Escape;
$qs=uri_escape($qs);

$url='http://www.milestonemortgage.com/cgi-bin/pq.pl?' . $qs;

my $post = new HTTP::Request POST => "$qs";

   $post->content_type('application/x-www-form-urlencoded');

   # Pass request to the user agent and get a response back
   my $res = $ua->request($post);
   

   # Check the outcome of the response
   if ($res->is_success) {
       print $res->content;
   } else {
       print "Bad luck this time\n";
   }


hope that helps,

Bob
0
 
bebonhamCommented:
when I had a problem like this, it turned out to be that the http headers I was sending weren't liked by the server I was hitting them with...

maneshr helped me get the headers straight...

this is the routine he wrote for me!


sub browse(){
    my($statement_URL,$hdr,$full)=@_;
    my($content_type,$method);

    $content_type="application/x-www-form-urlencoded";
    $method="POST";

    my($headers)= new HTTP::Headers
      'Content-Type'   =>  $content_type,
      'MIME-Version'   =>  '1.0',
      'Date'           =>  HTTP::Date::time2str(time),
      'Accept'         =>  'text/html';
   
    my($ua)= new LWP::UserAgent;
   
    $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_AGENT
   
    my($url)= new URI::URL($statement_URL);
    my($request)= new HTTP::Request($method, $url, $headers,$hdr);
   
    my($response)= $ua->request($request);

    my($reply);

     return ($response->as_string()) if $full;

    if ($response->is_success){
         $reply=$response->content;
    }else{
         $reply=$response->error_as_HTML();
    }
    return $reply;
}



Good Luck,

Bob
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
bebonhamCommented:
of course your method is probably faster, but not a whole lot.

0
 
marecsCommented:
From RFC 1945

"A valid Content-Length is required on all HTTP/1.0 POST requests."

Why did you bother doing
use IO::Socket; because you use nothing in IO::Socket
What you probably meant was
use Socket;
which only loads the definitions from you socket.h file

If you want to use sockets I suggest you use them through the object orientated interface provided by IO::Socket

#!/usr/bin/perl -w

use IO::Socket;
$port = 80;
$host = "www.host.com";
$path = "/cgi-bin/formdebug.pl";
$data = "var1=one&var2=t%20w%20o&var3=tree";
$length = length $data;
$referer = "http://www.msn.com/";

$sock = IO::Socket::INET->new(PeerAddr => $host,
                                    PeerPort => $port,
                                    Proto    => 'tcp',
                                    Timeout  => 60) || die $!;
$sock->autoflush; # Not required for Perl 5.005

print $sock join("\015\012" =>
                    "POST $path HTTP/1.0",
                    "Host: $host",
                    "User-Agent: Mozilla 4.0",
                    "Referer: $referer",
                    "Content-Length: $length",
                    "", "");
print $sock $data;

while (<$sock>) {
    print;
}

$sock->close();
0
 
OnlyOneAuthor Commented:
Thanks for the responses sofar.

I've tried out all of the above, the only one that is getting a response is the one from MARCES.

I think part of the reason that the others aren't working is that I'm working behind a proxy server and I don't know how to account for that in the other examples.
 
Even for MARCES's one the variables are still being ignored.
I've accounted for the proxy by setting the $host variable to my proxy server address which seem's to work.

The way I know that the variables are being ignored is that the response page has some javascript in it which has the variables set from the previous page.  
These are apprearing as "" which means that it has not read the variables in from the perl script (POST request).

I know that it can be done as what I'm using at the moment is calling the "curl" program and inputting the data using the "-d" option of curl.  I just want to do the whole program in perl as it will be more portable and the curl program will not have to be installed where ever I want to use this script.

So as it stands at the moment...

MARCES script get the response page from the server but variables are ignored (which is what my script is doing but in a messier way).

BEBONHAM's scripts could work I just don't know how to set them up for working with proxies.

Any other ideas guy's?
0
 
marecsCommented:
This worked for me, and adheres to RFC1945. http://www.faqs.org/rfcs/rfc1945.html

I will let bebonham give you the proxy implementation for his solution. His solution is probably the one you should be using.

#!/usr/bin/perl -w

use IO::Socket;
$port = 8080; # proxy port
$proxy = "proxy.you.com";
$url = "http://www.you.com/cgi-bin/formdebug.pl";
$data = "var1=one&var2=t%20w%20o&var3=tree";
$length = length $data;
$referer = "http://www.msn.com/";

$sock = IO::Socket::INET->new(PeerAddr => $proxy,
                                    PeerPort => $port,
                                    Proto    => 'tcp',
                                    Timeout  => 60) || die $!;
$sock->autoflush; # Not required for Perl 5.005

print $sock join("\015\012" =>
                    "POST $url HTTP/1.0",
                    "User-Agent: Mozilla 4.0",
                    "Referer: $referer",
                    "Content-Length: $length",
                    "", "");
print $sock $data;

while (<$sock>) {
    print;
}

$sock->close();
0
 
bebonhamCommented:
this is a slight modification on maneshr's browse routine, which worked well for me in the past.

but I make no statement that lwp is better than sockets, simply that I know lwp a little better, and I think it is easier to do http with it.


sub browse(){
   my($statement_URL,$hdr,$full)=@_;
   my($content_type,$method);

   $content_type="application/x-www-form-urlencoded";
   $method="POST";

   my($headers)= new HTTP::Headers
     'Content-Type'   =>  $content_type,
     'MIME-Version'   =>  '1.0',
     'Date'           =>  HTTP::Date::time2str(time),
     'Accept'         =>  'text/html';
   
   my($ua)= new LWP::UserAgent;
   
   $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_AGENT
########################
###here we set the proxy####
$ua->proxy('http', 'http://proxy.sn.no:8080/');

   
   my($url)= new URI::URL($statement_URL);
   my($request)= new HTTP::Request($method, $url, $headers,$hdr);
   
   my($response)= $ua->request($request);

   my($reply);

    return ($response->as_string()) if $full;

   if ($response->is_success){
        $reply=$response->content;
   }else{
        $reply=$response->error_as_HTML();
   }
   return $reply;
}

0
 
OnlyOneAuthor Commented:
Ok still no joy.
Here is exactly what I'm trying to do...

What I'm trying to do is to send sms messages automatically from a script.  

The code below is for the site that I'm trying to use...  
But once it works for this it would work for any site with just a few changes.

I'm getting the confirmation window with no variables set in it.

If you want to try it out you'll need to change the COUNTRY_CODE to whatever is applicable to you (the %2b is for the + sign url-encoded),
and you'll need to change the MOBILE_NUMBER to whatever your phone is with the preceeding 0 removed ie. 094 is 94.

It work's for international mobiles so if you have a mobile handy you can try it out.  The confirmation page will have the variables set in it if it has worked.

Hope this makes it easier for you to see what I'm having the problem with.

What is bugging me is that I can do it with "curl" and not completely in perl.

use IO::Socket;
$port = 80; # proxy port
$proxy = "my_proxy_address";
$url = "http://sms.planetinternet.de/free-sms/SMSC.asp";
$data = "lcode=%2bCOUNTRY_CODE&tel=MOBILE_NUMBER&Message=hello";
$length = length $data;
$referer = "http:sms.planetinternet.de/free-sms/content_smsverschicken_netscape.asp";

$sock = IO::Socket::INET->new(PeerAddr => $proxy,
PeerPort => $port,
Proto=> 'tcp',
Timeout=> 60) || die $!;
$sock->autoflush; # Not required for Perl 5.005

print $sock join("\015\012" =>
"POST $url HTTP/1.0",
"User-Agent: Mozilla 4.0",
"Referer: $referer",
"Content-Length: $length",
"", "");
print $sock $data;

while (<$sock>) {
print;
}

$sock->close();

I'm using MARECS code as it is getting the confirmation back.  I still haven't gotten your's BEBONHAM working. I pressume that I'm doing something wrong but I keep getting the "Bad luck this time" which means that I'm not getting any confirmation at all.

0
 
OnlyOneAuthor Commented:
Seeing as there has been no response since my last comment for a few days, I will be deleting this question tomorrow!!

Thank's for the info guy's.
0
 
marecsCommented:
I thought I posted this two days ago, but the site was probably down.

I used the code below but was still not able to get it to work. What Free SMS providers tend to do is make it VERY hard for guys like you to use scripts to send SMS's. The same is true for these guys. They rely heavily on cookies. I can handle the cookies, but they may be setting cookies with JavaScript which I don't handle (But you can if you want to with a little bit of work)

I think your original question was answered, so you should award the points.


#!/usr/bin/perl -w

use HTTP::Request::Common qw(POST GET);
use LWP::UserAgent;
use HTTP::Cookies;

@MyAgent::ISA = qw(LWP::UserAgent); # set inheritance

$cookie_file = "Cookies";
$ua = MyAgent->new;
$cookie_jar = HTTP::Cookies->new(file => $cookie_file, autosave => 1);
$ua->agent('Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)');
$ua->cookie_jar($cookie_jar);
$ua->proxy('http', 'http://web-cache.surrey.ac.uk:8080/');

$req = GET 'http://sms.planetinternet.de/free-sms/content_nutzungsbedingungen.as
p';
$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->as_string();

$req = GET 'http://sms.planetinternet.de/free-sms/content_smsverschicken.asp';
$reply = $ua->request($req);
mep2cm@froude:~/Perl$ cat SMS
SMS.pl   SMS.pl~
mep2cm@froude:~/Perl$ cat SMS.pl
#!/usr/bin/perl -w

use HTTP::Request::Common qw(POST GET);
use LWP::UserAgent;
use HTTP::Cookies;

@MyAgent::ISA = qw(LWP::UserAgent); # set inheritance

$cookie_file = "Cookies";
$ua = MyAgent->new;
$cookie_jar = HTTP::Cookies->new(file => $cookie_file, autosave => 1);
$ua->agent('Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)');
$ua->cookie_jar($cookie_jar);
$ua->proxy('http', 'http://web-cache.surrey.ac.uk:8080/');

$req = GET 'http://sms.planetinternet.de/free-sms/content_nutzungsbedingungen.asp';
$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->as_string();

$req = GET 'http://sms.planetinternet.de/free-sms/content_smsverschicken.asp';
$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->as_string();

$message = "Hello";
$counter = 130 - length $message;
$country = "+44";
$phone = "7713206392";

$req = POST 'http://sms.planetinternet.de/free-sms/SMSC.asp',
[
    country => $country,
    lcode => $country,
    tel => $phone,
    counter => $counter,
    Message => $message,
    type => 'Gute Texte ...',
    huidigGetoondeLaag => '',
    adressessHidden => '',
    delayedHidden => '',
    delayedDayHidden => '',
    delayedMonthHidden => '',
    delayedYearHidden => '',
    delayedTimeHidden => '',
    processPDAHidden => 'true',
];

$reply = $ua->request($req);
print $reply->as_string();
$reply->is_success || die "Error";

print $reply->as_string();

sub MyAgent::redirect_ok { 1 } # redirects all requests
0
 
OnlyOneAuthor Commented:
If you know of a program called "curl" which has a -d option for data and this works already, which is why I was hoping to do the full program in perl so there were no external dependancies for the script to work.  I don't think that the program is using cookies as all curl is doing is sending off the data to the server.

If there is no way of doing this then so be it.  But it would be nice to get a way of doing it...
0
 
OnlyOneAuthor Commented:
Oh and don't worry I won't delete the question!!
0
 
marecsCommented:
cURL does handle cookies. It works now. What I needed to make it work was to support cookies, and to support redirects with POST requests. That is what the MyAgent is all about because you have to override the redirect_ok method in UserAgent which does not allow redirects of POST requests as default.

#!/usr/bin/perl -w

use HTTP::Request::Common qw(POST GET);
use LWP::UserAgent;
use HTTP::Cookies;

@MyAgent::ISA = qw(LWP::UserAgent); # set inheritance

$ua = MyAgent->new;
$cookie_jar = HTTP::Cookies->new;
$ua->agent('Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)');
$ua->cookie_jar($cookie_jar);
$ua->proxy('http', 'http://web-cache.you.com:8080/');

$message = "Hello";
$counter = 130 - length $message;
$country = "+44";
$phone = "7712345";

$req = POST 'http://sms.planetinternet.de/free-sms/SMSC.asp',
[
    country => $country,
    lcode => $country,
    tel => $phone,
    counter => $counter,
    Message => $message,
    type => 'Gute Texte ...',
    huidigGetoondeLaag => '',
    adressessHidden => '',
    delayedHidden => '',
    delayedDayHidden => '',
    delayedMonthHidden => '',
    delayedYearHidden => '',
    delayedTimeHidden => '',
    processPDAHidden => 'true',
];
$req->referer('http://sms.planetinternet.de/free-sms/content_smsverschicken.asp');

$reply = $ua->request($req);
$reply->is_success || die "Error";
print $reply->content;

sub MyAgent::redirect_ok { 1 } # redirects all requests
0
 
OnlyOneAuthor Commented:
Sorry for no updates, I've been away for the last while.
I've not been able to get that code working yet.

There seem's to be an error in line 40.
$reply->is_success || die "Error";

I'm not sure what is wrong with it but when I comment it out I get nothing back at all.

I don't know if the script is working as I get no reply from the server or if there is something I'm doing wrong.

Any ideas?
0
 
marecsCommented:
Try

$reply->is_success || die $reply->error_as_HTML;

To see what error you get. It worked for me when I last tried it. I don't have a mobile to try it with right now.

0
 
OnlyOneAuthor Commented:
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 
OnlyOneAuthor Commented:
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 
OnlyOneAuthor Commented:
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 
OnlyOneAuthor Commented:
That's working now!!
All I needed to see was the response code from the proxy server and I could figure out what was happening.
I've tried it out and it is working, so if you have a mobile then your in for some fun.

I'm upping the points to 250 and they're going to you
MARECS.  Well done, and good work.

Thank's a million,
Bye!!
0
 
marecsCommented:
My pleasure. If you have any further problems you know where to get help.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.