unravel mystery response "401 Unauthorized xxx "

I'm getting a mysterious "401 Unauthorized xxx" response from a server that does not require authentication. I only get the response when I use a script to get and store xml (see script at bottom). I have no problem if I manually access the site from a browser. However -- as I said -- in the context of my script I get nothing but 401 errors. The same script works perfectly with other sites that provide xml, whether they reside within my company's intranet or outside it, e.g., xml from the NY Times.

Can anybody give me some clues as to how to try and address this? The folks who manage the site I'm trying to get to haven't been able to figure anything out yet. I have tried adding $ua->credentials ...    but that hasn't help either (and it shouldn't be necessary, since authentication is not required).

Here's the test script:

#!/usr/local/bin/perl

use LWP::UserAgent;
use HTTP::Request::Common;

my $ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(GET =>'http://stage.mycompany.com:8312/ppisearch/application?forward=resultsxml&name=product&days=10');
$res = $ua->request($req, "c:/Apps/SDI/PPIXML/"."PPIXMLtst.xml");
if ($res->is_success) {
   print "All Done!";
}
else {
   print $res->status_line, "\n";
}

for the above script, here's what I find:


this url doesn't work:  http://stage.mycompany.com/ppisearch/application?forward=resultsxml&name=lipitor&days=10
these urls all work fine:

http://nycweb04.mycompany.com/ics-wpd/exec/icswppro.dll?QY=product%3D%27ator%27+and+input_date%3E%3D@date-28&TN=PBD&DL=0&RL=0&NP=0&XM=1&RF=xmlBrief&MR=500&AC=QBE_QUERY

http://www.nytimes.com/services/xml/rss/nyt/Business.xml

http://nycinfoctr2.amer.mycompany.com/workshop/ppilipitorsmall.xml


GessWurkerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Tim_UtschigCommented:
Just a shot in the dark, but maybe the web server denies the user agent for some reason.

Try:

$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051111 Firefox/1.5");

before you send the request.
GessWurkerAuthor Commented:
Yeah...that's what I thought too. No luck. Nevertheless I tried your exact suggestion:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051111 Firefox/1.5");
$req = HTTP::Request->new(GET => 'http://ppistg.myCompany.com/ppisearch/application?forward=resultsxml&name=product&days=8');
$req->header('Accept' => 'text/html');
# send request
$res = $ua->request($req);
# check the outcome
if ($res->is_success) {
   print $res->decoded_content;
}
else {
   print "Error: " . $res->status_line . "\n";
}

And received back:

Error: 401 Unauthorized xxx

To which, I respond: Aaaaaaaaaaargh!

(Thanks for your suggestion though...)
Tim_UtschigCommented:
Perhaps the site requires a session cookie or something?  Or a referer?  Try capturing the request your browser sends (via ethereal/tcpdump/windump/whathaveyou) and add all the exact same headers to your $req.  Then remove them one by one until it doesn't work.
Exploring ASP.NET Core: Fundamentals

Learn to build web apps and services, IoT apps, and mobile backends by covering the fundamentals of ASP.NET Core and  exploring the core foundations for app libraries.

GessWurkerAuthor Commented:
Sorry...can you be more explicit about how I can capture what my browser sends? Meanwhile...I'll try to do what you suggest.
Tim_UtschigCommented:
If Unix:

   tcpdump -pni eth0 -s0 -w browser-request.pcap tcp port 80

If Winders (get the right interface index for passing to the -i option using "windump -D"):

   windump -pni 1 -s0 -w browser-request.pcap tcp port 80

windump is available here: http://www.winpcap.org/windump/

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
GessWurkerAuthor Commented:
Not sure how much of this is relevant, but see what I get below. I see there's the "401 Unauthorized xxx" error, then it looks like a session cookie is set and I get through. Don't see how I can mimic a session cookie (if that's the issue). Please comment.

Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: ppistg.myCompany.com
Connection: Keep-Alive
Cookie: count=1; SignOnDefault=; PSLoginValues=EMERST|ENG|us_ns_npr; ICWEB_COOKIETEST=enabled

A±µCȃ
 <   <    %%‹Ø ÐÓƒKü E  (U’@ 8Îá
‡äš¬ƒ  P
` IP5f8liPùçiÀ        A±µC†¤  =  =   %%‹Ø ÐÓƒKü E /U“@ 8ÍÙ
‡äš¬ƒ  P
` IP5f8liPûbs\  HTTP/1.1 401 Unauthorized xxx
Set-Cookie: ARPT=UQZVKOS10.135.201.203CKOOI; path=/
Date: Fri, 30 Dec 2005 22:14:25 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with CR136410
WWW-Authenticate: NTLM
Transfer-Encoding: Chunked

A±µCœ¤  >   >    %%‹Ø ÐÓƒKü E  0U”@ 8Î×
‡äš¬ƒ  P
` IQ<f8liPûbì¹  0000

A±µC«¤  6   6     ¬ %%‹Ø E  (Zã  €Á¬ƒ
‡äš
` Pf8li IQDPþðw  A±µC´«  8  8     ¬ %%‹Ø E *Zä  €¿¬ƒ
‡äš
` Pf8li IQDPþð y  GET /ppisearch/application?forward=resultsxml&name=lipitor&days=8 HTTP/1.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: ppistg.myCompany.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAABAAAAB7IIogQABAA2AAAADgAOACgAAAAFASgKAAAAD0FNUk1PUFdMUEJWTFkyQU1FUg==
Cookie: count=1; SignOnDefault=; PSLoginValues=EMERST|ENG|us_ns_npr; ICWEB_COOKIETEST=enabled; ARPT=UQZVKOS10.135.201.203CKOOI

A±µC•É  <   <    %%‹Ø ÐÓƒKü E  (U•@ 8ÎÞ
‡äš¬ƒ  P
` IQDf8nkPûbe4        A±µCÕ  ”  ”   %%‹Ø ÐÓƒKü E †U–@ 8Í
‡äš¬ƒ  P
` IQDf8nkPûbîÚ  HTTP/1.1 401 Unauthorized xxx
Date: Fri, 30 Dec 2005 22:14:25 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with CR136410
WWW-Authenticate: NTLM TlRMTVNTUAACAAAAAAAAAAAAAAABAgAA745uYqRHO7c=
Set-Cookie: JSESSIONID=D1xBfgPQrrZu3wJ2kyK1UVUSsZH44iFaqxB0z7pVBP0BGAwlHIoj!765801388; path=/
Transfer-Encoding: Chunked

A±µCï  >   >    %%‹Ø ÐÓƒKü E  0U—@ 8ÎÔ
‡äš¬ƒ  P
` IR¢f8nkPûbéQ  0000

A±µC        6   6     ¬ %%‹Ø E  (Zæ  €Á¬ƒ
‡äš
` Pf8nk IRªPÿÿw  A±µC„  N  N     ¬ %%‹Ø E @Zç  €¿t¬ƒ
‡äš
` Pf8nk IRªPÿÿ   GET /ppisearch/application?forward=resultsxml&name=lipitor&days=8 HTTP/1.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: ppistg.myCompany.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAHgAAAAYABgAkAAAAAgACABIAAAADAAMAFAAAAAcABwAXAAAAAAAAACoAAAABQIAAgUBKAoAAAAPQQBNAEUAUgBFAG0AZQByAHMAVABBAE0AUgBNAE8AUABXAEwAUABCAFYATABZADIAFYB+gO411i6kfPqBARvbV0czhj6neOzmzYtS4rkffikCgTCkaGVzjkyKumgpuuG5
Cookie: A±µC–  û   û     ¬ %%‹Ø E  íZè  €ÀƬƒ
‡äš
` Pf8pƒ IRªPÿÿ<  count=1; SignOnDefault=; PSLoginValues=EMERST|ENG|us_ns_npr; ICWEB_COOKIETEST=enabled; ARPT=UQZVKOS10.135.201.203CKOOI; JSESSIONID=D1xBfgPQrrZu3wJ2kyK1UVUSsZH44iFaqxB0z7pVBP0BGAwlHIoj!765801388

A±µC2  <   <    %%‹Ø ÐÓƒKü E  (U˜@ 8ÎÛ
‡äš¬ƒ  P
` IRªf8pƒPûba¶        A±µCàR  6   6     ¬ %%‹Ø E  (Zê  €¸q¬ƒ ”¨c‘
_ P×€yÚÑt]HPÿú'  A±µC+¬
 <   <    %%‹Ø ÐÓƒKü E  (U™@ 8ÎÚ
‡äš¬ƒ  P
` IRªf8qHPûb`ñ        E±µCÕ>      %%‹Ø ÐÓƒKü E  Uš@ 8Î
‡äš¬ƒ  P
` IRªf8qHPûbðJ  HTTP/1.1 200 OK
Date: Fri, 30 Dec 2005 22:14:25 GMT
Server: WebLogic XMLX Module 8.1 SP1 Fri Jun 20 23:06:40 PDT 2003 271009 with CR136410
Content-Type: text/xml; charset=iso-8859-1
Transfer-Encoding: Chunked
Tim_UtschigCommented:
Looks like authorization IS required.  And your browser is authenticating itself.

Try reading "perldoc LWP::Authen::Ntlm".

As for accepting cookies, try:

$ua->cookie_jar({});

To use an in-memory temporary cookie jar.  See "perldoc HTTP::Cookies" for more.
GessWurkerAuthor Commented:
Tim: Thanks for your help and especially for pointing me to the windump tool. As it turns out, the folks who manage the data I need to pull have said they'll remove authentication from the pages I target. Once they do that, I think I'll be fine.

Thanks again for your guidance!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Perl

From novice to tech pro — start learning today.