Solved

Getting cookies with Indy BIG PROB

Posted on 2006-07-12
28
726 Views
Last Modified: 2012-08-14
I`m not sure where`s the prob but it seems I get all the time wrong cookies. I mean not all the cookies! I~ve been spending 2 days on this and don`t know what`s happening.

Here`s a simple example:

  HTML := IdHTTP1.Get('http://feedmelinks.com/');

  Cookies := TStringList.Create;

  Cookies.Clear;
  Count := IdCookieManager.CookieCollection.count;
  for i := 1 to count do
    Cookies.Add(IdCookieManager.CookieCollection.Items[i-1].CookieText);

  ShowMessage(Cookies.text);

  Cookies.Free;

It seems I don`t get all the cookies from ANY site ??!
I want at least for that to see what I`m doing wrong, or what I am not doing. PLS!
0
Comment
Question by:crystyan
  • 18
  • 9
28 Comments
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17094383
do you have HandleRedirects set to true?
also, it should be
for i := 0 to count -1 do...
0
 

Author Comment

by:crystyan
ID: 17094528
it`s Items[i-1] so it`s ok. and I`ve set the redirect to true.
did u see what I mean ?
if u would use a sniffer u would see what I mean.
I`m staying at this for a lot of time :((
0
 

Author Comment

by:crystyan
ID: 17095089
what I get is ONLY the end of cookies, not all of them :((
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 28

Expert Comment

by:2266180
ID: 17095251
I made a fast sniff and this is what I found to be loaded and setting cookies:
/favicon.ico (I don't think this is mandatory since not all browsers get this icon, but it's worth keeping an eye on) (I got a 404m but still with a set-cookie for PHPSESSID)

also, the main site only spits one cookie: a PHPSESSID

what other cookies are you seeing?
0
 

Author Comment

by:crystyan
ID: 17095285
I see only the PHPSESSID and path.
0
 
LVL 28

Expert Comment

by:2266180
ID: 17095297
path is not a cookie. it is the path to which the cookie aplies.
and the code you are running doesn't get the PHPSESSID?
0
 

Author Comment

by:crystyan
ID: 17095304
0
 
LVL 28

Expert Comment

by:2266180
ID: 17095346
that is what I have told you in the other question about "resources that set cookies". that is just a particular naming I guess. but such resources do not need to be gifs. I've seen jpg's used for the same purpose. even imahes that were needed (like a check code in a registration form, or a header or background image). these will be loaded by browser or other content displaying softeware but will not be accessed by crawlers (webmasters don't exactly care about the crawlers, they tend to want information from real users, so this is a good way to get it)
0
 

Author Comment

by:crystyan
ID: 17095378
try this:

  GetHTML('http://feedmelinks.com/');
  GetCookies;

  SetCookies;
  HTML := PostHTML('http://feedmelinks.com/login', 'userId=' +
    User + '&password=' + Pass + '&op=login');

  Result := pos('<div class="welcome">Welcome, <b>' +
    User + '</b>!<br />', HTML) > 0;

  SetCookies;
  HTML := GetHTML('http://feedmelinks.com/add');
  SaveHTML(html);

U`ll see that the html is going to a page that says I`m not logged. Though I set the PHPSESSID cookie. I guess I need the rest of them cuz there is the id too in a cookie :(((
0
 
LVL 28

Expert Comment

by:2266180
ID: 17095397
I was about to shutdown, so I'll check it tomorrow
0
 

Author Comment

by:crystyan
ID: 17095403
ok, please
thanks a lot !
I hope u can find where those cookies are set :|
0
 

Author Comment

by:crystyan
ID: 17095521
0
 
LVL 28

Expert Comment

by:2266180
ID: 17097663
can you post a username and password. I am not exactly trilled that I always have to create an account just to test something
0
 

Author Comment

by:crystyan
ID: 17100000
user: crystyignat
pass: mypassword
0
 
LVL 28

Expert Comment

by:2266180
ID: 17104279
I just got home (I've been out with my girlfriend :D) I't kind of late, so I'll look into it tomorrow morning. sorry for the delay
0
 

Author Comment

by:crystyan
ID: 17104400
np :)
THANKS for ur help !!! I really appreciate it :-)
0
 

Author Comment

by:crystyan
ID: 17113834
hi ciuly,

did u have some free time ? any good news :-) ?
don`t wanna push u ... I hope u can find the prob there ...

thanks
Cristian
0
 
LVL 28

Expert Comment

by:2266180
ID: 17117223
hi,

sorry for not saying anything. I left country saturday morning. I tried to get a fix until then, but couldn.t so I thought I'd research it on my spare time here (I'm on bsssines here).

I managed to find te problem though nothing I tried seems to solve it. it could be a bug in indy, but I am not sure yet.
the way it goes is like this:
- after you do the post command, you will receive an aswer that sets 2 cookies" user name and password thing. now, what I managed so far si to have the username cookie included. this is very ackward as both username and password cookies are being processed the same, but the last one does not get included. this I still have to see why.

I will post again when I have mor einfo. if I happen not to resond for more than 3 days, then please do post a "reminder" message :) (It happens to me that I forget things)

thanks
0
 

Author Comment

by:crystyan
ID: 17117261
hi,

thanks a lot so far ! I`m not sure ... it could be indy bug cuz I`ve seen on more pages :(((((((((((((((
Ufff I need to solve this :(

thanks!
0
 

Author Comment

by:crystyan
ID: 17127023
hi there,

would u have time these days to find out more pls ?
other component that works would be great too. but I doubt there`s any other FREE component that does what Indy does ...

thanks ciuly,
0
 

Author Comment

by:crystyan
ID: 17141099
hi ciuly,

did u forget me :-s ?

thanks
0
 
LVL 28

Expert Comment

by:2266180
ID: 17143795
hi,

didn't forget, just that as I said I am out of the country and only have internet access at this office in which I have to work so my free time windows are very limited. but I am working on it when I get an open window and have an idea to test. I hope to get to the bottom of this issue this week.

keep buzzing me if I don't reply in a 3 day period :)

thanks
0
 

Author Comment

by:crystyan
ID: 17171707
hi ciuly,

anything new ? I should move on with the project I`m working on :((( grr ...

thanks!
0
 

Author Comment

by:crystyan
ID: 17172082
check out this component. though I`m not sure how to get the cookies with it http://www.torry.net/vcl/internet/http/advhttp.zip
0
 

Author Comment

by:crystyan
ID: 17172094
or on this link http://www.torry.net/pages.php?id=215 lots of components related.
0
 

Author Comment

by:crystyan
ID: 17172257
ciuly could u take a look on this plss http://www.torry.net/authorsmore.php?id=5735 ?
it has TONS of options there !!! I`ve installed it but I can`t get it how to use the post method ... I mean I`m not sure where to put the parameters yet ...
0
 
LVL 28

Accepted Solution

by:
2266180 earned 500 total points
ID: 17174994
many sorry's. I am still not in my country and that is why I worked so slowly on EE in general (lost about 50000 points to russel and lost 1st place from the top15 yearly :( )

anyway, I am almost stupid :)) so I was trying a lot of things here but probably got the reading o
f the ethereal wrong, because the cookies were set ok even before I left the country. It just that I had an extra get that messed up the logged in verification (see commented code)
so, the code is a little messy, I won't clean it up as it's not a big deal to clean it up. jut delete the unnecessary comments, and do some fixes for the hardcoded stuff and you should get it working. it might work with the original code I changed as well provided that you don't use that extra get I did on the /portal. I won't check this now, but you can check it if you want.

let me know if there are any isues with this code.

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, IdCookieManager, IdBaseComponent, IdComponent, IdTCPConnection,
  IdTCPClient, IdHTTP;

type
  TForm1 = class(TForm)
    IdHTTP1: TIdHTTP;
    IdCookieManager1: TIdCookieManager;
    procedure FormCreate(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
var
 Params: TStringList;
 HTML, loginurl, myuser: String;
 count,i:integer;
 _cookies, cookies:tstringlist;
 ll:boolean;
 name,value:String;

 procedure setcookies;
 var j:integer; s:string;
 begin
   count:=cookies.count;
   s:='';
   for j:=1 to count do
   begin
     IdCookieManager1.AddCookie(cookies[j-1],IdHTTP1.url.Host);
     s:=s+'; '+cookies[j-1];
   end;
   if s<>'' then
   begin
     delete(s,1,2);
     s:=s+';';
     IdHTTP1.Request.CustomHeaders.Values['Cookie']:=s;
     IdHTTP1.Request.RawHeaders.Values['Cookie']:=s;
     //('Cookie'+IdHTTP1.Request.RawHeaders.NameValueSeparator+s);
   end;{}
 end;

 procedure extractcookie(cookie:string; var name,value:string);
 var i,k:integer;
 begin
   i:=pos('=',cookie);
   k:=pos(';',cookie);
   if k=0 then k:=length(cookie);
   if i>0 then
   begin
     name:=copy(cookie,1,i-1);
     value:=copy(cookie,i+1,k-i-1);
   end      else
   begin
     name:='';
     value:='';
   end;
 end;

 procedure savecookies;
 var j:integer;
 begin
   count:=IdCookieManager1.CookieCollection.count;
   for j:=1 to count do
   begin
     extractcookie(IdCookieManager1.CookieCollection.Items[j-1].CookieText,name,value);
     cookies.Values[name]:=value;
   end;
//   IdCookieManager1.CookieCollection.Clear;
 end;

 procedure saveit(name:string);
 begin
   with tfilestream.create(name,fmcreate) do
   try
     write(pansichar(html)^,length(html));
   finally
     free;
   end;
 end;

begin
  ll:=false;
  loginurl:='http://feedmelinks.com/login';
  Params := TStringList.Create;
  try
    cookies:=tstringlist.Create;
//    cookies.Duplicates:=dupIgnore;
//    cookies.Sorted:=true;

    idhttp1.Host:='feedmelinks.com';
    html:=idhttp1.Get('http://feedmelinks.com/');// first get; get first cookie(s)
    savecookies;

    setcookies;
    html:=idhttp1.Get(loginUrl);// next get; this is clean: used for retrieving the viewstate
    savecookies;

    myuser:='crystyignat';
    Params.Values['userId'] := myuser;
    Params.Values['password'] := 'mypassword';
    Params.Values['op'] := 'login';

    IdHTTP1.HandleRedirects:=false;// now this made the buzz, because the cookies were not set when following the redirect
    try
      setcookies;
      HTML := IdHTTP1.Post(loginurl, Params);// now do the log in

      _Cookies := TStringList.Create;
      IdHTTP1.Response.RawHeaders.Extract('Set-cookie', _Cookies);
      for i := 0 to _Cookies.Count - 1 do
      begin
//        IdCookieManager1.AddCookie(_Cookies[i], IdHTTP1.URL.Host);
        extractcookie(_Cookies[i],name,value);
        cookies.Values[name]:=value;
      end;
      _cookies.free;
//      savecookies;

      if pos('<div class="welcome">Welcome, <b>'+myuser+'</b>',html)>0 then
      begin
        setCookies;
        html:=idhttp1.Get('http://feedmelinks.com/'); // software redirect
        savecookies;

        saveit('hhh.html');

//        setCookies;
//        html:=idhttp1.Get('http://feedmelinks.com/portal'); // another software redirect
        //savecookies;

        ll:=pos('<a class="tn" href="logout">log out',html)>0;
      end;
    except on e: EIdHTTPProtocolException do
    begin
      if e.ReplyErrorCode<>302 then
         raise e;
        // now this is the redirect
        count:=IdCookieManager1.CookieCollection.count;// get the next cookie (this will be the userid)
        for i:=1 to count do
          cookies.Add(IdCookieManager1.CookieCollection.Items[i-1].CookieText);

        setcookies;
        html:=idhttp1.Get(IdHTTP1.Response.Location);// follow redirect
      end;
    end;

    cookies.free;
  except on e: EIdHTTPProtocolException do
    begin
      showmessage(idHTTP1.response.ResponseText);
    end;
  end;
  Params.Free;
  showmessage('logged in? : '+booltostr(ll,true));
end;

end.

(regarding those components .. they don't worth the hassle. indy is good: it's just the user (me at this point) that is stupid :D )
0
 

Author Comment

by:crystyan
ID: 17177277
AWESOME !!!! so I should take a look on in the header response and look after the "set-cookie" I guess that`s why I wasn`t "seeing" all the cookies.

THANKS A LOT!

P.S. Am obs k esti din Romania :D Nice Work !!!
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
TEMBEDDEDWB how can i change its user agent ? 8 84
Multi-layered image in FireMonkey 9 68
Wincontrol not (correctly) drawn 15 49
enhance the following code 3 42
The uses clause is one of those things that just tends to grow and grow. Most of the time this is in the main form, as it's from this form that all others are called. If you have a big application (including many forms), the uses clause in the in…
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
Finding and deleting duplicate (picture) files can be a time consuming task. My wife and I, our three kids and their families all share one dilemma: Managing our pictures. Between desktops, laptops, phones, tablets, and cameras; over the last decade…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question