Solved

perl script to view https page that is password protected

Posted on 2007-11-14
16
751 Views
Last Modified: 2010-05-19
I have this script from adam314 to view a https page but the page requires a userid and password, which i have.  How can I change the script to insert userid and password?


use Net::SSLeay qw(get_https);

($page) = get_https('www.bacus.pt', 443, '/');
die "No page\n" unless $page;

open(OUT,">page.txt") or die "output: $!\n";
print OUT $page;
close(OUT);
0
Comment
Question by:mcgilljd
  • 7
  • 5
  • 4
16 Comments
 
LVL 39

Expert Comment

by:Adam314
ID: 20281310
How does it ask for a username/password when you go to the page with a browser?
0
 

Author Comment

by:mcgilljd
ID: 20282354
yes when I go to page with browser it asks for user name, password
0
 
LVL 39

Expert Comment

by:Adam314
ID: 20282476
Do you get a web page asking for user/pass?  Or a window pop-up asking for user/pass?
0
 
LVL 48

Accepted Solution

by:
Tintin earned 300 total points
ID: 20282613
Assuming it uses basic authentication:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => 'https://www.bacus.pt/');
$req->authorization_basic('username', 'password');

open FILE, ">page.txt" or die "Can not open page.txt $!\n";
print FILE $ua->request($req)->as_string;
close FILE;
0
 

Author Comment

by:mcgilljd
ID: 20283521
it is a popup window
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20283611
That means it uses basic authentication.
0
 

Author Comment

by:mcgilljd
ID: 20283735
Tintin, I tried to use your code.
It doesn't give an error, but it doesn't print to page.txt either

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";
$req->authorization_basic('myuserID', 'mypassword');

open FILE, ">C:\Desktop\prices.txt" or die "Can not open page.txt $!\n";
print FILE $ua->request($req)->as_string;
close FILE;


0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 300 total points
ID: 20284103
Try

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";
$req->authorization_basic('myuserID', 'mypassword');
$res = $ua->request($req);

die $res->status_line unless $res->is_success;

open FILE, ">C:/Desktop/prices.txt" or die "Can not open prices.txt $!\n";
print FILE $res->decoded_content;
close FILE;
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 39

Assisted Solution

by:Adam314
Adam314 earned 200 total points
ID: 20284390
A small change to Tintin's code:

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
$ua = LWP::UserAgent->new;
$us->credentials( $netloc, $realm, $uname, $pass );   #NOTE This line is new
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";
$res = $ua->request($req);

die $res->status_line unless $res->is_success;

open FILE, ">C:/Desktop/prices.txt" or die "Can not open prices.txt $!\n";
print FILE $res->decoded_content;
close FILE;

## Another option is to override the get_basic_credentials function.  For simple stuff, this isn't needed.
0
 

Author Comment

by:mcgilljd
ID: 20284391
C:\>Perl  C:\PerlScripts\login3.pl
501 Protocol scheme 'https' is not supported (Crypt::SSLeay not installed) at C:
\PerlScripts\interactivelogin3.pl line 11.

I will install Crypt::SSLeay and try again later

0
 

Author Comment

by:mcgilljd
ID: 20284459
Ok i installed crypt::SSLeay and added line
use Crypt::SSLeay ;

Now I get
401 Unauthorized at line 12

But if I use browser to go to page it works.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 20284524
Possibly the site you are trying to connect to requires cookies or looks at the user agent string.

Try adding

$ua->agent("Mozilla/4.0");

and see if it makes a difference.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 20284556
You shouldn't need to "use" Crypt::SSLeay - it will be used by the other modules if needed.

It sounds as if you've provided incorrect username and/or password (and/or netloc and/or realm if using the ->credentials function)

Are you sure you have these correct?  It is likely case sensitive.
0
 

Author Comment

by:mcgilljd
ID: 20284557
added

$ua->agent("Mozilla/4.0");

no change, i will have to try later
0
 

Author Comment

by:mcgilljd
ID: 20289785
I tried using credentials method here is my script

use LWP::UserAgent;
my $urlhome ="https://website.com/WebUI/Servlet/Reviewer?axn=review";
my $userid ="MyID";
my $password ="P@ssword";
my $netloc ="111.222.333.444:80";
my $realm ="REALM";
my $outfile ="C:\Documents and Settings\Desktop\interactiveprices.txt";
$ua = LWP::UserAgent->new;
$ua->credentials( $netloc, $realm, $userid, $password );  
$req = HTTP::Request->new(GET => $urlhome) or die "no page\n";

$res = $ua->request($req);
die $res->status_line unless $res->is_success;

open (OUTFILE, ">$outfile") or die "Can not open prices.txt $!\n";
print OUTFILE $res->decoded_content;
close OUTFILE;


I still get 401 unauthorized
my password and username are correct but I am not sure exactly what to use for "realm"
0
 
LVL 39

Assisted Solution

by:Adam314
Adam314 earned 200 total points
ID: 20290183
If the password contains the @ symbol (or another special character), it needs to be escaped in the $password:
my $password = "P\@ssword";
Otherwise, it will be interpreted as the @ssword variable.

Is real really 'REALM'?
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

On Microsoft Windows, if  when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now