Solved

perl script to view https page that is password protected

Posted on 2007-11-14
16
800 Views
Last Modified: 2010-05-19
I have this script from adam314 to view a https page but the page requires a userid and password, which i have.  How can I change the script to insert userid and password?


use Net::SSLeay qw(get_https);

($page) = get_https('www.bacus.pt', 443, '/');
die "No page\n" unless $page;

open(OUT,">page.txt") or die "output: $!\n";
print OUT $page;
close(OUT);
0
Comment
Question by:mcgilljd
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 5
  • 4
16 Comments
 
LVL 39

Expert Comment

by:Adam314
ID: 20281310
How does it ask for a username/password when you go to the page with a browser?
0
 

Author Comment

by:mcgilljd
ID: 20282354
yes when I go to page with browser it asks for user name, password
0
 
LVL 39

Expert Comment

by:Adam314
ID: 20282476
Do you get a web page asking for user/pass?  Or a window pop-up asking for user/pass?
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 48

Accepted Solution

by:
Tintin earned 300 total points
ID: 20282613
Assuming it uses basic authentication:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => 'https://www.bacus.pt/');
$req->authorization_basic('username', 'password');

open FILE, ">page.txt" or die "Can not open page.txt $!\n";
print FILE $ua->request($req)->as_string;
close FILE;
0
 

Author Comment

by:mcgilljd
ID: 20283521
it is a popup window
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20283611
That means it uses basic authentication.
0
 

Author Comment

by:mcgilljd
ID: 20283735
Tintin, I tried to use your code.
It doesn't give an error, but it doesn't print to page.txt either

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";
$req->authorization_basic('myuserID', 'mypassword');

open FILE, ">C:\Desktop\prices.txt" or die "Can not open page.txt $!\n";
print FILE $ua->request($req)->as_string;
close FILE;


0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 300 total points
ID: 20284103
Try

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";
$req->authorization_basic('myuserID', 'mypassword');
$res = $ua->request($req);

die $res->status_line unless $res->is_success;

open FILE, ">C:/Desktop/prices.txt" or die "Can not open prices.txt $!\n";
print FILE $res->decoded_content;
close FILE;
0
 
LVL 39

Assisted Solution

by:Adam314
Adam314 earned 200 total points
ID: 20284390
A small change to Tintin's code:

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
$ua = LWP::UserAgent->new;
$us->credentials( $netloc, $realm, $uname, $pass );   #NOTE This line is new
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";
$res = $ua->request($req);

die $res->status_line unless $res->is_success;

open FILE, ">C:/Desktop/prices.txt" or die "Can not open prices.txt $!\n";
print FILE $res->decoded_content;
close FILE;

## Another option is to override the get_basic_credentials function.  For simple stuff, this isn't needed.
0
 

Author Comment

by:mcgilljd
ID: 20284391
C:\>Perl  C:\PerlScripts\login3.pl
501 Protocol scheme 'https' is not supported (Crypt::SSLeay not installed) at C:
\PerlScripts\interactivelogin3.pl line 11.

I will install Crypt::SSLeay and try again later

0
 

Author Comment

by:mcgilljd
ID: 20284459
Ok i installed crypt::SSLeay and added line
use Crypt::SSLeay ;

Now I get
401 Unauthorized at line 12

But if I use browser to go to page it works.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20284524
Possibly the site you are trying to connect to requires cookies or looks at the user agent string.

Try adding

$ua->agent("Mozilla/4.0");

and see if it makes a difference.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 20284556
You shouldn't need to "use" Crypt::SSLeay - it will be used by the other modules if needed.

It sounds as if you've provided incorrect username and/or password (and/or netloc and/or realm if using the ->credentials function)

Are you sure you have these correct?  It is likely case sensitive.
0
 

Author Comment

by:mcgilljd
ID: 20284557
added

$ua->agent("Mozilla/4.0");

no change, i will have to try later
0
 

Author Comment

by:mcgilljd
ID: 20289785
I tried using credentials method here is my script

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
my $userid ="MyID";
my $password ="P@ssword";
my $netloc ="111.222.333.444:80";
my $realm ="REALM";
my $outfile ="C:\Documents and Settings\Desktop\interactiveprices.txt";
$ua = LWP::UserAgent->new;
$ua->credentials( $netloc, $realm, $userid, $password );  
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";

$res = $ua->request($req);
die $res->status_line unless $res->is_success;

open (OUTFILE, ">$outfile") or die "Can not open prices.txt $!\n";
print OUTFILE $res->decoded_content;
close OUTFILE;


I still get 401 unauthorized
my password and username are correct but I am not sure exactly what to use for "realm"
0
 
LVL 39

Assisted Solution

by:Adam314
Adam314 earned 200 total points
ID: 20290183
If the password contains the @ symbol (or another special character), it needs to be escaped in the $password:
my $password = "P\@ssword";
Otherwise, it will be interpreted as the @ssword variable.

Is real really 'REALM'?
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question