Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

LWP::UserAgent   cookie problem

Posted on 2003-02-19
6
Medium Priority
?
655 Views
Last Modified: 2012-06-21
i get a error where it says does not seem to contain cookies


i am trying to do a search on 50 results instead of the 15 msn does it on. when i change the resuts to 50 it saves it as a cookie and not in the url line. So how would i send a request to get the source code for resluts of 1-50 not just 1-15

here is the cookie that it saves and the co=50 is the number of results



msnsrchnosp=0&co=50&RS=1search.msn.com/085208755229619988313043364829546562*


#!/usr/local/bin/perl -w

use LWP::UserAgent;




@keyword=( "map", "maps", "usa+maps", "county+maps", "us+regional+maps", "us+regional+map", "us+map", "us+maps", "state+maps", "map+of+the+us", "zip+code+maps", "city+maps", "census+tracts", "custom+maps", "town+map", "zip+code+map", "radius+map", "radius+maps");


#market map part
print  "*******  MARKET MAPS!!!!!!!   *******  \n \n";
open FILE, ">>key.txt";
print FILE "*******  msn!!!!!!!   *******  \n \n";
print FILE "*******  MARKET MAPS!!!!!!!   *******  \n \n";
close FILE;

#1-50
foreach $element (@keyword)
{
  $count=0;

  print $element;
  print "*******  MARKET MAPS!!!!!!!   *******  \n \n";
  $cookie_jar = HTTP::Cookies->new(file => 'tsearch.txt');
  $user_agent=new LWP::UserAgent;
  $request= new HTTP::Request('GET',

"http://search.msn.com/results.asp?q=$element&origq=market%2Bmaps&RS=CHECKED&FORM=SMCRT&v=1&cfg=SMCINITIAL&nosp=0&thr=");
 
 
  $cookie_jar->add_cookie_header($request);
   
  $user_agent->agent('Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020826');
  $request->header('Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/msword,

application/vnd.ms-powerpoint, */*');
  $request->header('Accept-Language' => 'en;q=1.0');
  $cookie_jar->extract_cookies($response);
 
  $user_agent->cookie_jar($cookie_jar);
 
 $textd=$cookie_jar->as_string();
print $textd;
 
  $response=$user_agent->request($request);                    
  $content=$response->{_content};  
 

 while($content=~ /<span class="clsResultURL"><i>(.*?)<\/i>/sg)
  {
    $count=$count+1;
    if($1=~ /market.*?maps.*?\.com/s)
    {

      open FILE, ">>key.txt";
      print FILE "keyword: $element \n";
      print FILE "rank: $count \n";
      close FILE;

      last;
    }  
  }


 
 
}
0
Comment
Question by:kerrston
  • 3
  • 2
6 Comments
 

Author Comment

by:kerrston
ID: 8121121
test
0
 
LVL 20

Accepted Solution

by:
jmcg earned 0 total points
ID: 8121831
We saw your question all right, it just wasn't apparent to any of us what an answer might be.


The error message saying "$file does not seem to contain cookies" happens when you specify a file for your cookie-jar that was not created by the "save" method.

One thing you might try is to add the following near the beginning of your script:

$cookie_jar = HTTP::Cookies->new();
$cookie_jar = HTTP::Cookies->save('tsearch.txt');

There are in fact quite a few things that should arguably be moved out of your main foreach loop. You don't need to create new user_agents, cookie-jars, or open/close FILE on every value. I'm also suspicious that you're using extract_cookies($response) before you've processed the request. And you should not have to call add_cookie_header or extract_cookies at all if you've associated the cookie-jar with the user agent.

0
 
LVL 20

Expert Comment

by:jmcg
ID: 8121839
Oops, cut'n'paste'n'get-distracted....

$cookie_jar = HTTP::Cookies->new();
$cookie_jar->save('tsearch.txt');
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 

Author Comment

by:kerrston
ID: 8122024
i did the project another way already bit longer progarm to run but works all the same
0
 
LVL 20

Expert Comment

by:jmcg
ID: 8122354
Well if you solved it yourself, there's no need to waste your points. Drop a request in the Community Support forum and request that my answer be unaccepted and your points refunded. I think they can do that for you.
0
 

Expert Comment

by:SpideyMod
ID: 8139195
Actually, a question once answered cannot be unanswered.  I can and will refund the points for the question.  I will also change the grade from a C to an A so it isn't a bad mark on the recipient.

Future PAQ purchasers:
This question remains unanswered so you may or may not obtain any value from the responses.  Since it cost you nothing to view, all should be well.

SpideyMod
Community Support Moderator @Experts Exchange
0

Featured Post

Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans
Suggested Courses

572 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question