?
Solved

LWP::UserAgent   cookie problem

Posted on 2003-02-19
6
Medium Priority
?
644 Views
Last Modified: 2012-06-21
i get a error where it says does not seem to contain cookies


i am trying to do a search on 50 results instead of the 15 msn does it on. when i change the resuts to 50 it saves it as a cookie and not in the url line. So how would i send a request to get the source code for resluts of 1-50 not just 1-15

here is the cookie that it saves and the co=50 is the number of results



msnsrchnosp=0&co=50&RS=1search.msn.com/085208755229619988313043364829546562*


#!/usr/local/bin/perl -w

use LWP::UserAgent;




@keyword=( "map", "maps", "usa+maps", "county+maps", "us+regional+maps", "us+regional+map", "us+map", "us+maps", "state+maps", "map+of+the+us", "zip+code+maps", "city+maps", "census+tracts", "custom+maps", "town+map", "zip+code+map", "radius+map", "radius+maps");


#market map part
print  "*******  MARKET MAPS!!!!!!!   *******  \n \n";
open FILE, ">>key.txt";
print FILE "*******  msn!!!!!!!   *******  \n \n";
print FILE "*******  MARKET MAPS!!!!!!!   *******  \n \n";
close FILE;

#1-50
foreach $element (@keyword)
{
  $count=0;

  print $element;
  print "*******  MARKET MAPS!!!!!!!   *******  \n \n";
  $cookie_jar = HTTP::Cookies->new(file => 'tsearch.txt');
  $user_agent=new LWP::UserAgent;
  $request= new HTTP::Request('GET',

"http://search.msn.com/results.asp?q=$element&origq=market%2Bmaps&RS=CHECKED&FORM=SMCRT&v=1&cfg=SMCINITIAL&nosp=0&thr=");
 
 
  $cookie_jar->add_cookie_header($request);
   
  $user_agent->agent('Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826');
  $request->header('Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/msword,

application/vnd.ms-powerpoint, */*');
  $request->header('Accept-Language' => 'en;q=1.0');
  $cookie_jar->extract_cookies($response);
 
  $user_agent->cookie_jar($cookie_jar);
 
 $textd=$cookie_jar->as_string();
print $textd;
 
  $response=$user_agent->request($request);                    
  $content=$response->{_content};  
 

 while($content=~ /<span class="clsResultURL"><i>(.*?)<\/i>/sg)
  {
    $count=$count+1;
    if($1=~ /market.*?maps.*?\.com/s)
    {

      open FILE, ">>key.txt";
      print FILE "keyword: $element \n";
      print FILE "rank: $count \n";
      close FILE;

      last;
    }  
  }


 
 
}
0
Comment
Question by:kerrston
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
6 Comments
 

Author Comment

by:kerrston
ID: 8121121
test
0
 
LVL 20

Accepted Solution

by:
jmcg earned 0 total points
ID: 8121831
We saw your question all right, it just wasn't apparent to any of us what an answer might be.


The error message saying "$file does not seem to contain cookies" happens when you specify a file for your cookie-jar that was not created by the "save" method.

One thing you might try is to add the following near the beginning of your script:

$cookie_jar = HTTP::Cookies->new();
$cookie_jar = HTTP::Cookies->save('tsearch.txt');

There are in fact quite a few things that should arguably be moved out of your main foreach loop. You don't need to create new user_agents, cookie-jars, or open/close FILE on every value. I'm also suspicious that you're using extract_cookies($response) before you've processed the request. And you should not have to call add_cookie_header or extract_cookies at all if you've associated the cookie-jar with the user agent.

0
 
LVL 20

Expert Comment

by:jmcg
ID: 8121839
Oops, cut'n'paste'n'get-distracted....

$cookie_jar = HTTP::Cookies->new();
$cookie_jar->save('tsearch.txt');
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:kerrston
ID: 8122024
i did the project another way already bit longer progarm to run but works all the same
0
 
LVL 20

Expert Comment

by:jmcg
ID: 8122354
Well if you solved it yourself, there's no need to waste your points. Drop a request in the Community Support forum and request that my answer be unaccepted and your points refunded. I think they can do that for you.
0
 

Expert Comment

by:SpideyMod
ID: 8139195
Actually, a question once answered cannot be unanswered.  I can and will refund the points for the question.  I will also change the grade from a C to an A so it isn't a bad mark on the recipient.

Future PAQ purchasers:
This question remains unanswered so you may or may not obtain any value from the responses.  Since it cost you nothing to view, all should be well.

SpideyMod
Community Support Moderator @Experts Exchange
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans
Suggested Courses

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question