We help IT Professionals succeed at work.

Retaining cookies using net.HttpWebRequest

mrl72
mrl72 asked
on
Medium Priority
460 Views
Last Modified: 2012-05-06
I am trying to write an asp.net application that scrapes some information from a website. The application requires you to be currently logged into the website. A link on the website, when clicked opens up my application which then gets the referring URL and scrapes the information from it. I am using HttpWebRequest to open the referring page and scrape the information using HttpWebResponse and GetResponseStream. The problem is once a call to HttpWebRequest has been made no cookies are brought over into the request and thus the referring page redirects to a login page. Reading on MS website it says cookies are automatically disabled, so how to you keep the cookies from the referring page and pass them through to the HttpWebRequest?
Comment
Watch Question

David RobitailleAnalyst Programmer

Commented:
Please check my comment at the end of this question: the trick is to use CookieContainer.
http://www.experts-exchange.com/Programming/Languages/.NET/ASP.NET/Q_24045173.html

Author

Commented:
I don't understand how that would help. I don't need to pass any username/password as I've already logged in on the previous web page and I don't want to log in again, just use the cookie from the requesting URL. If that makes sense..
Analyst Programmer
Commented:
It just show how to "link" a cookie into 2 request object. Has i said, the trick is to use the CookieContainer.
 reading futere i just noticed thst it<s not that simple. it<s because your page.request use a cookie colection and HttpWebRequest use a cookie colection.
here a link that might help:
http://blogs.microsoft.co.il/blogs/shair/archive/2008/02/26/cookie-collection-and-cookiecontainer.aspx
 and a E-E links
http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_20926170.html 

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts

Author

Commented:
Ok this is what I've done so far. When I run the script I see two cookies: cookie_l10n=en-us and Bus cookie_intl=deleted. Whatever they are I don't know. I don't see any of the other cookies from the browser sessions I have open.

dim sUrl as string = "http://sitetopullcookiesfrom"
Dim CC As New CookieContainer()
Dim Req As HttpWebRequest 
Dim Res as HttpWebResponse
Req =  CType(WebRequest.Create(sUrl), HttpWebRequest)
 
req.Proxy = Nothing
Req.UseDefaultCredentials = True
 
'YOU MUST ASSIGN A COOKIE CONTAINER FOR THE REQUEST TO PULL THE COOKIES
 
Req.CookieContainer = CC
Res = DirectCast(Req.GetResponse(), HttpWebResponse)
 
'DUMP THE COOKIES
 
If Res.Cookies IsNot Nothing AndAlso Res.Cookies.Count <> 0 Then
    For Each c As Cookie In Res.Cookies
        response.write(vbTab & c.ToString())
    Next
Else
    response.write("No Cookies")
End If

Open in new window

David RobitailleAnalyst Programmer

Commented:
I`m afraid I cannot help you further on that, you will have to wait for another expert to check that question.

Author

Commented:
Although no complete solution provided the expert gave enough direction for me to find an answer to my problem.
David RobitailleAnalyst Programmer

Commented:
I`m glad you figured out how to make it work!
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.