Solved

Perl - WWW::Mechanize with a proxy server

Posted on 2003-11-18
1
1,136 Views
Last Modified: 2012-06-27
Help!

I am using Perl WWW::Mechanize to access some websites.  The goal is to access certain pages on a corporate network and scrape information.  

I am running into a problem in that I can't get past the proxy server.  I have setup the following

my $agent = WWW::Mechanize->new();
$agent->proxy(http => "http://xxx.proxy.xxx.com:8001/");

but when I run

my $response = $agent -> get('http://www.tmng.com/HomePage/homepage.htm/');

I get a 403 error.

Now from IE, to setup the browser to run on the network, under Tools/Internet Options, undr the connection tab, in the LAN settings, you have to click on the Use Auto Configuration script option and the location is "http://xxx.proxy.xxx.com:8001".

How do I get Perl to recognize the auto config setting on the proxy server?  

Thanks
0
Comment
Question by:mattstarbuck
1 Comment
 
LVL 20

Accepted Solution

by:
jmcg earned 400 total points
ID: 9774717
The URL

http://xxx.proxy.xxx.com:8001/

may refer only to the server holding the autoconfiguration file and not itself be a proxy server. You may need to retrieve that URL and examine its content (it will look a lot like JavaScript) to determine what to set as your proxy values. I'm not aware of a way to have an LWP script do this for itself.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Suggested Solutions

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now