alc
asked on
Using cookies with HttpWebRequest
I'm working on building a program that will log into a web site using cookies and extract data from the page. I'm using HttpWebRequest. I know there is a CookieContainer property but I haven't quite figured out how to get this to work yet. Ideally I'd like to be able to load the cookie data from a file on my computer and access the page using HttpWebRequest. Does anybody have some code samples I could try?
ASKER
Esteban,
In step 2 you say to post the login/password info. I'm not quite sure how to do this. When I review the cookies on my computer I can see my user name and the encrypted password. Where does this information go in the Cookie? The Cookie object has a Value property but I'm not sure how this is used. Can you post some sample code that does this?
In step 2 you say to post the login/password info. I'm not quite sure how to do this. When I review the cookies on my computer I can see my user name and the encrypted password. Where does this information go in the Cookie? The Cookie object has a Value property but I'm not sure how this is used. Can you post some sample code that does this?
To post data you will need to:
string strPost="login=yourlogin&p assword=yo urpassword ";
HttpWebRequest objRequest = (HttpWebRequest)WebRequest .Create(th etargeturl loginpage) ;
objRequest.CookieContainer = new CookieContainer();
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-ur lencoded";
StreamWriter writer = new StreamWriter(objRequest.Ge tRequestSt ream());
writer.write(strPost);
writer.close();
HttpWebResponse objResponse = (HttpWebResponse)objReques t.GetRespo nse();
objResponse.Cookies = objRequest.CookieContainer .GetCookie s(objReque st.Request Uri);
foreach(Cookie cookie in objResponse.Cookies)
{
//Save to cookies. I would binary serialize to disk, but any method should work
}
the actual string to post will depend of your target page. You are supposed to send all data that would be send by a html form. So, check out the html source code and find what values it's expecting. For example if you find that the source code have something like:
Login = <input type="text" id="username">
password = <input type="password" id="userpassword">
remember me <input type="checkbox" id="rememberme">
you will have to post something like ="username=YOURUSERNAME&us erpassword =YOURPASSW ORd&rememb erme=true"
In later request you would do
HttpWebRequest objRequest = (HttpWebRequest)WebRequest .Create(th etargetpag eurlthatre quirelogge duser);
objRequest.CookieContainer = new CookieContainer();
objRequest.CookieContainer .Add(YOURS AVEDCOOKIE S); //if there's more than one, add them all
HttpWebResponse objResponse = (HttpWebResponse)objReques t.GetRespo nse();
//Work with objResponse from here.
Esteban Felipe
www.estebanf.com
string strPost="login=yourlogin&p
HttpWebRequest objRequest = (HttpWebRequest)WebRequest
objRequest.CookieContainer
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-ur
StreamWriter writer = new StreamWriter(objRequest.Ge
writer.write(strPost);
writer.close();
HttpWebResponse objResponse = (HttpWebResponse)objReques
objResponse.Cookies = objRequest.CookieContainer
foreach(Cookie cookie in objResponse.Cookies)
{
//Save to cookies. I would binary serialize to disk, but any method should work
}
the actual string to post will depend of your target page. You are supposed to send all data that would be send by a html form. So, check out the html source code and find what values it's expecting. For example if you find that the source code have something like:
Login = <input type="text" id="username">
password = <input type="password" id="userpassword">
remember me <input type="checkbox" id="rememberme">
you will have to post something like ="username=YOURUSERNAME&us
In later request you would do
HttpWebRequest objRequest = (HttpWebRequest)WebRequest
objRequest.CookieContainer
objRequest.CookieContainer
HttpWebResponse objResponse = (HttpWebResponse)objReques
//Work with objResponse from here.
Esteban Felipe
www.estebanf.com
ASKER
Esteban,
That helped, but I'm still not getting the response I expected. I can see that I'm now getting cookies in the response. However, the response stream is empty. I'm not sure why this is. Any ideas?
That helped, but I'm still not getting the response I expected. I can see that I'm now getting cookies in the response. However, the response stream is empty. I'm not sure why this is. Any ideas?
Can i see your code please?
ASKER
try
{
HttpWebRequest oReq =
(HttpWebRequest)WebRequest .Create("http://www.investors.com/login.asp?/register/default.asp?location=/member/checkup/checkup.asp?t=ge&ss=yes&survey=falsewinname=registrationend: ;");
oReq.CookieContainer = new CookieContainer();
string strPost = "htmUserName=<username>&ht mPassword= <password> "; oReq.Method = "POST";
oReq.ContentLength = strPost.Length;
oReq.ContentType = "application/x-www-form-ur lencoded";
StreamWriter writer = new StreamWriter(oReq.GetReque stStream() );
writer.Write(strPost);
writer.Close();
HttpWebResponse oResp = (HttpWebResponse)oReq.GetR esponse();
oResp.Cookies = oReq.CookieContainer.GetCo okies(oReq .RequestUr i);
// Print the properties of each cookie.
foreach (Cookie cook in oResp.Cookies)
{
Debug.WriteLine("Cookie:") ;
Debug.WriteLine(cook.Name + " = " + cook.Value);
Debug.WriteLine("Domain: " + cook.Domain);
Debug.WriteLine("Path: " + cook.Path);
Debug.WriteLine("Port: " + cook.Port);
Debug.WriteLine("Secure: " + cook.Secure);
Debug.WriteLine("When issued: " + cook.TimeStamp);
Debug.WriteLine("Expires: " + cook.Expires + " (expired? " + cook.Expired + ")");
Debug.WriteLine("Don't save: " + cook.Discard);
Debug.WriteLine("Comment: " + cook.Comment);
Debug.WriteLine("Uri for comments: " + cook.CommentUri);
Debug.WriteLine("Version: RFC " + (cook.Version == 1 ? "2109" : "2965"));
// Show the string representation of the cookie.
Debug.WriteLine ("String: " + cook.ToString());
}
Stream oStream = oResp.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEn coding("ut f-8");
StreamReader oStreamRead = new StreamReader( oStream, encode );
WriteStream(oStreamRead);
oResp.Close();
oStreamRead.Close();
oStream.Close();
}
catch(WebException e)
{
Debug.WriteLine("Web Exception"+
"\n\nException Message :" + e.Message);
if(e.Status == WebExceptionStatus.Protoco lError)
{
Debug.WriteLine("Status Code : " + ((HttpWebResponse)e.Respon se).Status Code);
Debug.WriteLine("Status Description : " + ((HttpWebResponse)e.Respon se).Status Descriptio n);
}
}
private void WriteStream(StreamReader oStreamRead)
{
string strLine;
while ((strLine = oStreamRead.ReadLine()) != null)
{
Debug.WriteLine(strLine);
}
}
{
HttpWebRequest oReq =
(HttpWebRequest)WebRequest
oReq.CookieContainer = new CookieContainer();
string strPost = "htmUserName=<username>&ht
oReq.ContentLength = strPost.Length;
oReq.ContentType = "application/x-www-form-ur
StreamWriter writer = new StreamWriter(oReq.GetReque
writer.Write(strPost);
writer.Close();
HttpWebResponse oResp = (HttpWebResponse)oReq.GetR
oResp.Cookies = oReq.CookieContainer.GetCo
// Print the properties of each cookie.
foreach (Cookie cook in oResp.Cookies)
{
Debug.WriteLine("Cookie:")
Debug.WriteLine(cook.Name + " = " + cook.Value);
Debug.WriteLine("Domain: " + cook.Domain);
Debug.WriteLine("Path: " + cook.Path);
Debug.WriteLine("Port: " + cook.Port);
Debug.WriteLine("Secure: " + cook.Secure);
Debug.WriteLine("When issued: " + cook.TimeStamp);
Debug.WriteLine("Expires: " + cook.Expires + " (expired? " + cook.Expired + ")");
Debug.WriteLine("Don't save: " + cook.Discard);
Debug.WriteLine("Comment: " + cook.Comment);
Debug.WriteLine("Uri for comments: " + cook.CommentUri);
Debug.WriteLine("Version: RFC " + (cook.Version == 1 ? "2109" : "2965"));
// Show the string representation of the cookie.
Debug.WriteLine ("String: " + cook.ToString());
}
Stream oStream = oResp.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEn
StreamReader oStreamRead = new StreamReader( oStream, encode );
WriteStream(oStreamRead);
oResp.Close();
oStreamRead.Close();
oStream.Close();
}
catch(WebException e)
{
Debug.WriteLine("Web Exception"+
"\n\nException Message :" + e.Message);
if(e.Status == WebExceptionStatus.Protoco
{
Debug.WriteLine("Status Code : " + ((HttpWebResponse)e.Respon
Debug.WriteLine("Status Description : " + ((HttpWebResponse)e.Respon
}
}
private void WriteStream(StreamReader oStreamRead)
{
string strLine;
while ((strLine = oStreamRead.ReadLine()) != null)
{
Debug.WriteLine(strLine);
}
}
Questions:
1.- Do you get any cookie printed?
2.- What response are you getting and what response are you expecting to get?
1.- Do you get any cookie printed?
2.- What response are you getting and what response are you expecting to get?
ASKER
1 - Yes, I get one cookie in the response.
2 - There is nothing in the Response Stream. The first ReadLine() is null.
2 - There is nothing in the Response Stream. The first ReadLine() is null.
uhmm... weird... Tell me something, ¿what's the normal behavior of that page? Is theres any chance that after you login you get a response.redirect?.
ASKER
Yes, I think it is doing a response redirect. The URL changes once I login and another IE window is loaded with the page I want.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Yes, I tried accessing the page directly once I had the cookies but for some reason it didn't have any HTML. It only had a bunch of Javascript.
You will have to follow these steps
1.- Make an HttpWebRequest and set yourRequest.CookieContaine
2.- Point the request the login page and post login/password info.
3.- Set your httpwebresponse.cookies to request.CookieContainer.Ge
4.- Store the cookies somewhere for later use
5.- For future requests add your stored cookies in the cookieContainer object of your request.
Esteban Felipe
www.estebanf.com