Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Using cookies with HttpWebRequest

Posted on 2004-03-20
12
Medium Priority
?
1,291 Views
Last Modified: 2007-12-19
I'm working on building a program that will log into a web site using cookies and extract data from the page. I'm using HttpWebRequest. I know there is a CookieContainer property but I haven't quite figured out how to get this to work yet. Ideally I'd like to be able to load the cookie data from a file on my computer and access the page using HttpWebRequest. Does anybody have some code samples I could try?
0
Comment
Question by:alc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 6
12 Comments
 
LVL 12

Expert Comment

by:esteban_felipe
ID: 10643010
Hi alc,

You will have to follow these steps

1.- Make an HttpWebRequest and set yourRequest.CookieContainer = new CookieContainer.
2.- Point the request the login page and post login/password info.
3.- Set your httpwebresponse.cookies to request.CookieContainer.GetCookies(request.RequestUri)
4.- Store the cookies somewhere for later use
5.- For future requests add your stored cookies in the cookieContainer object of your request.



Esteban Felipe
www.estebanf.com
0
 

Author Comment

by:alc
ID: 10643055
Esteban,

In step 2 you say to post the login/password info. I'm not quite sure how to do this. When I review the cookies on my computer I can see my user name and the encrypted password. Where does this information go in the Cookie? The Cookie object has a Value property but I'm not sure how this is used. Can you post some sample code that does this?
0
 
LVL 12

Expert Comment

by:esteban_felipe
ID: 10643131
To post data you will need to:

string strPost="login=yourlogin&password=yourpassword";
HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(thetargeturlloginpage);
objRequest.CookieContainer = new CookieContainer();
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-urlencoded";
StreamWriter writer = new StreamWriter(objRequest.GetRequestStream());
writer.write(strPost);
writer.close();
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
objResponse.Cookies = objRequest.CookieContainer.GetCookies(objRequest.RequestUri);
foreach(Cookie cookie in objResponse.Cookies)
{
//Save to cookies. I would binary serialize to disk, but any method should work
}


the actual string to post will depend of your target page. You are supposed to send all data that would be send by a html form. So, check out the html source code and find what values it's expecting. For example if you find that the source code have something like:
Login = <input type="text" id="username">
password = <input type="password" id="userpassword">
remember me <input type="checkbox" id="rememberme">

you will have to post something like ="username=YOURUSERNAME&userpassword=YOURPASSWORd&rememberme=true"

In later request you would do


HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(thetargetpageurlthatrequireloggeduser);
objRequest.CookieContainer = new CookieContainer();
objRequest.CookieContainer.Add(YOURSAVEDCOOKIES); //if there's more than one, add them all
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
//Work with objResponse from here.

Esteban Felipe
www.estebanf.com
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:alc
ID: 10644213
Esteban,

That helped, but I'm still not getting the response I expected. I can see that I'm now getting cookies in the response. However, the response stream is empty. I'm not sure why this is. Any ideas?
0
 
LVL 12

Expert Comment

by:esteban_felipe
ID: 10644600
Can i see your code please?
0
 

Author Comment

by:alc
ID: 10645401
try
{

      HttpWebRequest oReq =
            (HttpWebRequest)WebRequest.Create("http://www.investors.com/login.asp?/register/default.asp?location=/member/checkup/checkup.asp?t=ge&ss=yes&survey=falsewinname=registrationend:;");
      oReq.CookieContainer = new CookieContainer();
      string strPost = "htmUserName=<username>&htmPassword=<password>";                  oReq.Method = "POST";
      oReq.ContentLength = strPost.Length;
      oReq.ContentType = "application/x-www-form-urlencoded";
      StreamWriter writer = new StreamWriter(oReq.GetRequestStream());
      writer.Write(strPost);
      writer.Close();

      HttpWebResponse oResp = (HttpWebResponse)oReq.GetResponse();
      oResp.Cookies = oReq.CookieContainer.GetCookies(oReq.RequestUri);

      // Print the properties of each cookie.
      foreach (Cookie cook in oResp.Cookies)
      {
            Debug.WriteLine("Cookie:");
            Debug.WriteLine(cook.Name + " = " + cook.Value);
            Debug.WriteLine("Domain: " + cook.Domain);
            Debug.WriteLine("Path: " + cook.Path);
            Debug.WriteLine("Port: " + cook.Port);
            Debug.WriteLine("Secure: " + cook.Secure);
   
            Debug.WriteLine("When issued: " + cook.TimeStamp);
            Debug.WriteLine("Expires: " + cook.Expires + " (expired? " + cook.Expired + ")");
            Debug.WriteLine("Don't save: " + cook.Discard);    
            Debug.WriteLine("Comment: " + cook.Comment);
            Debug.WriteLine("Uri for comments: " + cook.CommentUri);
            Debug.WriteLine("Version: RFC " + (cook.Version == 1 ? "2109" : "2965"));

            // Show the string representation of the cookie.
            Debug.WriteLine ("String: " + cook.ToString());
      }

      Stream oStream = oResp.GetResponseStream();
      Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
      StreamReader oStreamRead = new StreamReader( oStream, encode );

      WriteStream(oStreamRead);

      oResp.Close();
      oStreamRead.Close();
      oStream.Close();
}
catch(WebException e)
{
      Debug.WriteLine("Web Exception"+
            "\n\nException Message :" + e.Message);
      if(e.Status == WebExceptionStatus.ProtocolError)
      {
            Debug.WriteLine("Status Code : " + ((HttpWebResponse)e.Response).StatusCode);
            Debug.WriteLine("Status Description : " + ((HttpWebResponse)e.Response).StatusDescription);
      }
}

private void WriteStream(StreamReader oStreamRead)
{
      string strLine;

      while ((strLine = oStreamRead.ReadLine()) != null)
      {
            Debug.WriteLine(strLine);
      }

}
0
 
LVL 12

Expert Comment

by:esteban_felipe
ID: 10646212
Questions:
1.- Do you get any cookie printed?
2.- What response are you getting and what response are you expecting to get?
0
 

Author Comment

by:alc
ID: 10646432
1 - Yes, I get one cookie in the response.
2 - There is nothing in the Response Stream. The first ReadLine() is null.
0
 
LVL 12

Expert Comment

by:esteban_felipe
ID: 10646454
uhmm... weird... Tell me something, ¿what's the normal behavior of that page? Is theres any chance that after you login you get a response.redirect?.
0
 

Author Comment

by:alc
ID: 10646512
Yes, I think it is doing a response redirect. The URL changes once I login and another IE window is loaded with the page I want.
0
 
LVL 12

Accepted Solution

by:
esteban_felipe earned 2000 total points
ID: 10646525
Yeah, I saw it (I opened an account there for testing). Now I'm trying to figure out how to "get aware" of the redirect.

But this may be getting off topic. You have the cookie now, you should be able to access a "logged" page now. Did you tried that?
0
 

Author Comment

by:alc
ID: 10646530
Yes, I tried accessing the page directly once I had the cookies but for some reason it didn't have any HTML. It only had a bunch of Javascript.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Extention Methods in C# 3.0 by Ivo Stoykov C# 3.0 offers extension methods. They allow extending existing classes without changing the class's source code or relying on inheritance. These are static methods invoked as instance method. This…
This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…
Is your data getting by on basic protection measures? In today’s climate of debilitating malware and ransomware—like WannaCry—that may not be enough. You need to establish more than basics, like a recovery plan that protects both data and endpoints.…

610 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question