Solved

Getting cookies with Indy BIG PROB

Posted on 2006-07-12
28
722 Views
Last Modified: 2012-08-14
I`m not sure where`s the prob but it seems I get all the time wrong cookies. I mean not all the cookies! I~ve been spending 2 days on this and don`t know what`s happening.

Here`s a simple example:

  HTML := IdHTTP1.Get('http://feedmelinks.com/');

  Cookies := TStringList.Create;

  Cookies.Clear;
  Count := IdCookieManager.CookieCollection.count;
  for i := 1 to count do
    Cookies.Add(IdCookieManager.CookieCollection.Items[i-1].CookieText);

  ShowMessage(Cookies.text);

  Cookies.Free;

It seems I don`t get all the cookies from ANY site ??!
I want at least for that to see what I`m doing wrong, or what I am not doing. PLS!
0
Comment
Question by:crystyan
  • 18
  • 9
28 Comments
 
LVL 17

Expert Comment

by:TheRealLoki
Comment Utility
do you have HandleRedirects set to true?
also, it should be
for i := 0 to count -1 do...
0
 

Author Comment

by:crystyan
Comment Utility
it`s Items[i-1] so it`s ok. and I`ve set the redirect to true.
did u see what I mean ?
if u would use a sniffer u would see what I mean.
I`m staying at this for a lot of time :((
0
 

Author Comment

by:crystyan
Comment Utility
what I get is ONLY the end of cookies, not all of them :((
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
I made a fast sniff and this is what I found to be loaded and setting cookies:
/favicon.ico (I don't think this is mandatory since not all browsers get this icon, but it's worth keeping an eye on) (I got a 404m but still with a set-cookie for PHPSESSID)

also, the main site only spits one cookie: a PHPSESSID

what other cookies are you seeing?
0
 

Author Comment

by:crystyan
Comment Utility
I see only the PHPSESSID and path.
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
path is not a cookie. it is the path to which the cookie aplies.
and the code you are running doesn't get the PHPSESSID?
0
 

Author Comment

by:crystyan
Comment Utility
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
that is what I have told you in the other question about "resources that set cookies". that is just a particular naming I guess. but such resources do not need to be gifs. I've seen jpg's used for the same purpose. even imahes that were needed (like a check code in a registration form, or a header or background image). these will be loaded by browser or other content displaying softeware but will not be accessed by crawlers (webmasters don't exactly care about the crawlers, they tend to want information from real users, so this is a good way to get it)
0
 

Author Comment

by:crystyan
Comment Utility
try this:

  GetHTML('http://feedmelinks.com/');
  GetCookies;

  SetCookies;
  HTML := PostHTML('http://feedmelinks.com/login', 'userId=' +
    User + '&password=' + Pass + '&op=login');

  Result := pos('<div class="welcome">Welcome, <b>' +
    User + '</b>!<br />', HTML) > 0;

  SetCookies;
  HTML := GetHTML('http://feedmelinks.com/add');
  SaveHTML(html);

U`ll see that the html is going to a page that says I`m not logged. Though I set the PHPSESSID cookie. I guess I need the rest of them cuz there is the id too in a cookie :(((
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
I was about to shutdown, so I'll check it tomorrow
0
 

Author Comment

by:crystyan
Comment Utility
ok, please
thanks a lot !
I hope u can find where those cookies are set :|
0
 

Author Comment

by:crystyan
Comment Utility
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
can you post a username and password. I am not exactly trilled that I always have to create an account just to test something
0
 

Author Comment

by:crystyan
Comment Utility
user: crystyignat
pass: mypassword
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 28

Expert Comment

by:ciuly
Comment Utility
I just got home (I've been out with my girlfriend :D) I't kind of late, so I'll look into it tomorrow morning. sorry for the delay
0
 

Author Comment

by:crystyan
Comment Utility
np :)
THANKS for ur help !!! I really appreciate it :-)
0
 

Author Comment

by:crystyan
Comment Utility
hi ciuly,

did u have some free time ? any good news :-) ?
don`t wanna push u ... I hope u can find the prob there ...

thanks
Cristian
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
hi,

sorry for not saying anything. I left country saturday morning. I tried to get a fix until then, but couldn.t so I thought I'd research it on my spare time here (I'm on bsssines here).

I managed to find te problem though nothing I tried seems to solve it. it could be a bug in indy, but I am not sure yet.
the way it goes is like this:
- after you do the post command, you will receive an aswer that sets 2 cookies" user name and password thing. now, what I managed so far si to have the username cookie included. this is very ackward as both username and password cookies are being processed the same, but the last one does not get included. this I still have to see why.

I will post again when I have mor einfo. if I happen not to resond for more than 3 days, then please do post a "reminder" message :) (It happens to me that I forget things)

thanks
0
 

Author Comment

by:crystyan
Comment Utility
hi,

thanks a lot so far ! I`m not sure ... it could be indy bug cuz I`ve seen on more pages :(((((((((((((((
Ufff I need to solve this :(

thanks!
0
 

Author Comment

by:crystyan
Comment Utility
hi there,

would u have time these days to find out more pls ?
other component that works would be great too. but I doubt there`s any other FREE component that does what Indy does ...

thanks ciuly,
0
 

Author Comment

by:crystyan
Comment Utility
hi ciuly,

did u forget me :-s ?

thanks
0
 
LVL 28

Expert Comment

by:ciuly
Comment Utility
hi,

didn't forget, just that as I said I am out of the country and only have internet access at this office in which I have to work so my free time windows are very limited. but I am working on it when I get an open window and have an idea to test. I hope to get to the bottom of this issue this week.

keep buzzing me if I don't reply in a 3 day period :)

thanks
0
 

Author Comment

by:crystyan
Comment Utility
hi ciuly,

anything new ? I should move on with the project I`m working on :((( grr ...

thanks!
0
 

Author Comment

by:crystyan
Comment Utility
check out this component. though I`m not sure how to get the cookies with it http://www.torry.net/vcl/internet/http/advhttp.zip
0
 

Author Comment

by:crystyan
Comment Utility
or on this link http://www.torry.net/pages.php?id=215 lots of components related.
0
 

Author Comment

by:crystyan
Comment Utility
ciuly could u take a look on this plss http://www.torry.net/authorsmore.php?id=5735 ?
it has TONS of options there !!! I`ve installed it but I can`t get it how to use the post method ... I mean I`m not sure where to put the parameters yet ...
0
 
LVL 28

Accepted Solution

by:
ciuly earned 500 total points
Comment Utility
many sorry's. I am still not in my country and that is why I worked so slowly on EE in general (lost about 50000 points to russel and lost 1st place from the top15 yearly :( )

anyway, I am almost stupid :)) so I was trying a lot of things here but probably got the reading o
f the ethereal wrong, because the cookies were set ok even before I left the country. It just that I had an extra get that messed up the logged in verification (see commented code)
so, the code is a little messy, I won't clean it up as it's not a big deal to clean it up. jut delete the unnecessary comments, and do some fixes for the hardcoded stuff and you should get it working. it might work with the original code I changed as well provided that you don't use that extra get I did on the /portal. I won't check this now, but you can check it if you want.

let me know if there are any isues with this code.

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, IdCookieManager, IdBaseComponent, IdComponent, IdTCPConnection,
  IdTCPClient, IdHTTP;

type
  TForm1 = class(TForm)
    IdHTTP1: TIdHTTP;
    IdCookieManager1: TIdCookieManager;
    procedure FormCreate(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
var
 Params: TStringList;
 HTML, loginurl, myuser: String;
 count,i:integer;
 _cookies, cookies:tstringlist;
 ll:boolean;
 name,value:String;

 procedure setcookies;
 var j:integer; s:string;
 begin
   count:=cookies.count;
   s:='';
   for j:=1 to count do
   begin
     IdCookieManager1.AddCookie(cookies[j-1],IdHTTP1.url.Host);
     s:=s+'; '+cookies[j-1];
   end;
   if s<>'' then
   begin
     delete(s,1,2);
     s:=s+';';
     IdHTTP1.Request.CustomHeaders.Values['Cookie']:=s;
     IdHTTP1.Request.RawHeaders.Values['Cookie']:=s;
     //('Cookie'+IdHTTP1.Request.RawHeaders.NameValueSeparator+s);
   end;{}
 end;

 procedure extractcookie(cookie:string; var name,value:string);
 var i,k:integer;
 begin
   i:=pos('=',cookie);
   k:=pos(';',cookie);
   if k=0 then k:=length(cookie);
   if i>0 then
   begin
     name:=copy(cookie,1,i-1);
     value:=copy(cookie,i+1,k-i-1);
   end      else
   begin
     name:='';
     value:='';
   end;
 end;

 procedure savecookies;
 var j:integer;
 begin
   count:=IdCookieManager1.CookieCollection.count;
   for j:=1 to count do
   begin
     extractcookie(IdCookieManager1.CookieCollection.Items[j-1].CookieText,name,value);
     cookies.Values[name]:=value;
   end;
//   IdCookieManager1.CookieCollection.Clear;
 end;

 procedure saveit(name:string);
 begin
   with tfilestream.create(name,fmcreate) do
   try
     write(pansichar(html)^,length(html));
   finally
     free;
   end;
 end;

begin
  ll:=false;
  loginurl:='http://feedmelinks.com/login';
  Params := TStringList.Create;
  try
    cookies:=tstringlist.Create;
//    cookies.Duplicates:=dupIgnore;
//    cookies.Sorted:=true;

    idhttp1.Host:='feedmelinks.com';
    html:=idhttp1.Get('http://feedmelinks.com/');// first get; get first cookie(s)
    savecookies;

    setcookies;
    html:=idhttp1.Get(loginUrl);// next get; this is clean: used for retrieving the viewstate
    savecookies;

    myuser:='crystyignat';
    Params.Values['userId'] := myuser;
    Params.Values['password'] := 'mypassword';
    Params.Values['op'] := 'login';

    IdHTTP1.HandleRedirects:=false;// now this made the buzz, because the cookies were not set when following the redirect
    try
      setcookies;
      HTML := IdHTTP1.Post(loginurl, Params);// now do the log in

      _Cookies := TStringList.Create;
      IdHTTP1.Response.RawHeaders.Extract('Set-cookie', _Cookies);
      for i := 0 to _Cookies.Count - 1 do
      begin
//        IdCookieManager1.AddCookie(_Cookies[i], IdHTTP1.URL.Host);
        extractcookie(_Cookies[i],name,value);
        cookies.Values[name]:=value;
      end;
      _cookies.free;
//      savecookies;

      if pos('<div class="welcome">Welcome, <b>'+myuser+'</b>',html)>0 then
      begin
        setCookies;
        html:=idhttp1.Get('http://feedmelinks.com/'); // software redirect
        savecookies;

        saveit('hhh.html');

//        setCookies;
//        html:=idhttp1.Get('http://feedmelinks.com/portal'); // another software redirect
        //savecookies;

        ll:=pos('<a class="tn" href="logout">log out',html)>0;
      end;
    except on e: EIdHTTPProtocolException do
    begin
      if e.ReplyErrorCode<>302 then
         raise e;
        // now this is the redirect
        count:=IdCookieManager1.CookieCollection.count;// get the next cookie (this will be the userid)
        for i:=1 to count do
          cookies.Add(IdCookieManager1.CookieCollection.Items[i-1].CookieText);

        setcookies;
        html:=idhttp1.Get(IdHTTP1.Response.Location);// follow redirect
      end;
    end;

    cookies.free;
  except on e: EIdHTTPProtocolException do
    begin
      showmessage(idHTTP1.response.ResponseText);
    end;
  end;
  Params.Free;
  showmessage('logged in? : '+booltostr(ll,true));
end;

end.

(regarding those components .. they don't worth the hassle. indy is good: it's just the user (me at this point) that is stupid :D )
0
 

Author Comment

by:crystyan
Comment Utility
AWESOME !!!! so I should take a look on in the header response and look after the "set-cookie" I guess that`s why I wasn`t "seeing" all the cookies.

THANKS A LOT!

P.S. Am obs k esti din Romania :D Nice Work !!!
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Suggested Solutions

The uses clause is one of those things that just tends to grow and grow. Most of the time this is in the main form, as it's from this form that all others are called. If you have a big application (including many forms), the uses clause in the in…
In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now