Solved

Perl - WWW::Mechanize with a proxy server

Posted on 2003-11-18
1
1,145 Views
Last Modified: 2012-06-27
Help!

I am using Perl WWW::Mechanize to access some websites.  The goal is to access certain pages on a corporate network and scrape information.  

I am running into a problem in that I can't get past the proxy server.  I have setup the following

my $agent = WWW::Mechanize->new();
$agent->proxy(http => "http://xxx.proxy.xxx.com:8001/");

but when I run

my $response = $agent -> get('http://www.tmng.com/HomePage/homepage.htm/');

I get a 403 error.

Now from IE, to setup the browser to run on the network, under Tools/Internet Options, undr the connection tab, in the LAN settings, you have to click on the Use Auto Configuration script option and the location is "http://xxx.proxy.xxx.com:8001".

How do I get Perl to recognize the auto config setting on the proxy server?  

Thanks
0
Comment
Question by:mattstarbuck
1 Comment
 
LVL 20

Accepted Solution

by:
jmcg earned 400 total points
ID: 9774717
The URL

http://xxx.proxy.xxx.com:8001/

may refer only to the server holding the autoconfiguration file and not itself be a proxy server. You may need to retrieve that URL and examine its content (it will look a lot like JavaScript) to determine what to set as your proxy values. I'm not aware of a way to have an LWP script do this for itself.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Along with being a a promotional video for my three-day Annielytics Dashboard Seminor, this Micro Tutorial is an intro to Google Analytics API data.

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now