Start Free Trial

asked on

unravel mystery response "401 Unauthorized xxx "

I'm getting a mysterious "401 Unauthorized xxx" response from a server that does not require authentication. I only get the response when I use a script to get and store xml (see script at bottom). I have no problem if I manually access the site from a browser. However -- as I said -- in the context of my script I get nothing but 401 errors. The same script works perfectly with other sites that provide xml, whether they reside within my company's intranet or outside it, e.g., xml from the NY Times.

Can anybody give me some clues as to how to try and address this? The folks who manage the site I'm trying to get to haven't been able to figure anything out yet. I have tried adding $ua->credentials ... but that hasn't help either (and it shouldn't be necessary, since authentication is not required).

Here's the test script:

#!/usr/local/bin/perl

use LWP::UserAgent;
use HTTP::Request::Common;

my $ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(GET =>'http://stage.mycompany.com:8312/ppisearch/application?forward=resultsxml&name=product&days=10');
$res = $ua->request($req, "c:/Apps/SDI/PPIXML/"."PPIXMLtst.xml");
if ($res->is_success) {
print "All Done!";
}
else {
print $res->status_line, "\n";
}

for the above script, here's what I find:

this url doesn't work: http://stage.mycompany.com/ppisearch/application?forward=resultsxml&name=lipitor&days=10
these urls all work fine:

http://nycweb04.mycompany.com/ics-wpd/exec/icswppro.dll?QY=product%3D%27ator%27+and+input_date%3E%3D@date-28&TN=PBD&DL=0&RL=0&NP=0&XM=1&RF=xmlBrief&MR=500&AC=QBE_QUERY

http://www.nytimes.com/services/xml/rss/nyt/Business.xml

http://nycinfoctr2.amer.mycompany.com/workshop/ppilipitorsmall.xml

Just a shot in the dark, but maybe the web server denies the user agent for some reason.

Try:

$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051111 Firefox/1.5");

before you send the request.

ASKER

Yeah...that's what I thought too. No luck. Nevertheless I tried your exact suggestion:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051111 Firefox/1.5");
$req = HTTP::Request->new(GET => 'http://ppistg.myCompany.com/ppisearch/application?forward=resultsxml&name=product&days=8');
$req->header('Accept' => 'text/html');
# send request
$res = $ua->request($req);
# check the outcome
if ($res->is_success) {
print $res->decoded_content;
}
else {
print "Error: " . $res->status_line . "\n";
}

And received back:

Error: 401 Unauthorized xxx

To which, I respond: Aaaaaaaaaaargh!

(Thanks for your suggestion though...)

Perhaps the site requires a session cookie or something? Or a referer? Try capturing the request your browser sends (via ethereal/tcpdump/windump/whathaveyou) and add all the exact same headers to your $req. Then remove them one by one until it doesn't work.

ASKER

Sorry...can you be more explicit about how I can capture what my browser sends? Meanwhile...I'll try to do what you suggest.

ASKER CERTIFIED SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

Not sure how much of this is relevant, but see what I get below. I see there's the "401 Unauthorized xxx" error, then it looks like a session cookie is set and I get through. Don't see how I can mimic a session cookie (if that's the issue). Please comment.

Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: ppistg.myCompany.com
Connection: Keep-Alive
Cookie: count=1; SignOnDefault=; PSLoginValues=EMERST|ENG|us_ns_npr; ICWEB_COOKIETEST=enabled

A±µCÈƒ
< < %%‹Ø ÐÓƒKü E (U’@ 8Îá
‡äš¬ƒ P
` IP5f8liPùçiÀ A±µC†¤ = = %%‹Ø ÐÓƒKü E /U“@ 8ÍÙ
‡äš¬ƒ P
` IP5f8liPûbs\ HTTP/1.1 401 Unauthorized xxx
Set-Cookie: ARPT=UQZVKOS10.135.201.203CKOOI; path=/
Date: Fri, 30 Dec 2005 22:14:25 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with CR136410
WWW-Authenticate: NTLM
Transfer-Encoding: Chunked

A±µCœ¤ > > %%‹Ø ÐÓƒKü E 0U”@ 8Î×
‡äš¬ƒ P
` IQ<f8liPûbì¹ 0000

A±µC«¤ 6 6 ¬ %%‹Ø E (Zã €Á¬ƒ
‡äš
` Pf8li IQDPþðw A±µC´« 8 8 ¬ %%‹Ø E *Zä €¿¬ƒ
‡äš
` Pf8li IQDPþð y GET /ppisearch/application?forward=resultsxml&name=lipitor&days=8 HTTP/1.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: ppistg.myCompany.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAABAAAAB7IIogQABAA2AAAADgAOACgAAAAFASgKAAAAD0FNUk1PUFdMUEJWTFkyQU1FUg==
Cookie: count=1; SignOnDefault=; PSLoginValues=EMERST|ENG|us_ns_npr; ICWEB_COOKIETEST=enabled; ARPT=UQZVKOS10.135.201.203CKOOI

A±µC•É < < %%‹Ø ÐÓƒKü E (U•@ 8ÎÞ
‡äš¬ƒ P
` IQDf8nkPûbe4 A±µCÕ ” ” %%‹Ø ÐÓƒKü E †U–@ 8Í
‡äš¬ƒ P
` IQDf8nkPûbîÚ HTTP/1.1 401 Unauthorized xxx
Date: Fri, 30 Dec 2005 22:14:25 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with CR136410
WWW-Authenticate: NTLM TlRMTVNTUAACAAAAAAAAAAAAAAABAgAA745uYqRHO7c=
Set-Cookie: JSESSIONID=D1xBfgPQrrZu3wJ2kyK1UVUSsZH44iFaqxB0z7pVBP0BGAwlHIoj!765801388; path=/
Transfer-Encoding: Chunked

A±µCï > > %%‹Ø ÐÓƒKü E 0U—@ 8ÎÔ
‡äš¬ƒ P
` IR¢f8nkPûbéQ 0000

A±µC 6 6 ¬ %%‹Ø E (Zæ €Á¬ƒ
‡äš
` Pf8nk IRªPÿÿw A±µC„ N N ¬ %%‹Ø E @Zç €¿t¬ƒ
‡äš
` Pf8nk IRªPÿÿ GET /ppisearch/application?forward=resultsxml&name=lipitor&days=8 HTTP/1.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: ppistg.myCompany.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAHgAAAAYABgAkAAAAAgACABIAAAADAAMAFAAAAAcABwAXAAAAAAAAACoAAAABQIAAgUBKAoAAAAPQQBNAEUAUgBFAG0AZQByAHMAVABBAE0AUgBNAE8AUABXAEwAUABCAFYATABZADIAFYB+gO411i6kfPqBARvbV0czhj6neOzmzYtS4rkffikCgTCkaGVzjkyKumgpuuG5
Cookie: A±µC– û û ¬ %%‹Ø E íZè €ÀÆ¬ƒ
‡äš
` Pf8pƒ IRªPÿÿ< count=1; SignOnDefault=; PSLoginValues=EMERST|ENG|us_ns_npr; ICWEB_COOKIETEST=enabled; ARPT=UQZVKOS10.135.201.203CKOOI; JSESSIONID=D1xBfgPQrrZu3wJ2kyK1UVUSsZH44iFaqxB0z7pVBP0BGAwlHIoj!765801388

A±µC2 < < %%‹Ø ÐÓƒKü E (U˜@ 8ÎÛ
‡äš¬ƒ P
` IRªf8pƒPûba¶ A±µCàR 6 6 ¬ %%‹Ø E (Zê €¸q¬ƒ ”¨c‘
_ P×€yÚÑt]HPÿú' A±µC+¬
< < %%‹Ø ÐÓƒKü E (U™@ 8ÎÚ
‡äš¬ƒ P
` IRªf8qHPûb`ñ E±µCÕ> %%‹Ø ÐÓƒKü E Uš@ 8Î
‡äš¬ƒ P
` IRªf8qHPûbðJ HTTP/1.1 200 OK
Date: Fri, 30 Dec 2005 22:14:25 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with CR136410
Content-Type: text/xml; charset=iso-8859-1
Transfer-Encoding: Chunked

Looks like authorization IS required. And your browser is authenticating itself.

Try reading "perldoc LWP::Authen::Ntlm".

As for accepting cookies, try:

$ua->cookie_jar({});

To use an in-memory temporary cookie jar. See "perldoc HTTP::Cookies" for more.

ASKER

Tim: Thanks for your help and especially for pointing me to the windump tool. As it turns out, the folks who manage the data I need to pull have said they'll remove authentication from the pages I target. Once they do that, I think I'll be fine.

Thanks again for your guidance!