• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 430
  • Last Modified:

Retaining cookies using net.HttpWebRequest

I am trying to write an asp.net application that scrapes some information from a website. The application requires you to be currently logged into the website. A link on the website, when clicked opens up my application which then gets the referring URL and scrapes the information from it. I am using HttpWebRequest to open the referring page and scrape the information using HttpWebResponse and GetResponseStream. The problem is once a call to HttpWebRequest has been made no cookies are brought over into the request and thus the referring page redirects to a login page. Reading on MS website it says cookies are automatically disabled, so how to you keep the cookies from the referring page and pass them through to the HttpWebRequest?
0
mrl72
Asked:
mrl72
  • 4
  • 3
1 Solution
 
David RobitailleAnalyst ProgrammerCommented:
Please check my comment at the end of this question: the trick is to use CookieContainer.
http://www.experts-exchange.com/Programming/Languages/.NET/ASP.NET/Q_24045173.html
0
 
mrl72Author Commented:
I don't understand how that would help. I don't need to pass any username/password as I've already logged in on the previous web page and I don't want to log in again, just use the cookie from the requesting URL. If that makes sense..
0
 
David RobitailleAnalyst ProgrammerCommented:
It just show how to "link" a cookie into 2 request object. Has i said, the trick is to use the CookieContainer.
 reading futere i just noticed thst it<s not that simple. it<s because your page.request use a cookie colection and HttpWebRequest use a cookie colection.
here a link that might help:
http://blogs.microsoft.co.il/blogs/shair/archive/2008/02/26/cookie-collection-and-cookiecontainer.aspx
 and a E-E links
http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_20926170.html 
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 
mrl72Author Commented:
Ok this is what I've done so far. When I run the script I see two cookies: cookie_l10n=en-us and Bus cookie_intl=deleted. Whatever they are I don't know. I don't see any of the other cookies from the browser sessions I have open.

dim sUrl as string = "http://sitetopullcookiesfrom"
Dim CC As New CookieContainer()
Dim Req As HttpWebRequest 
Dim Res as HttpWebResponse
Req =  CType(WebRequest.Create(sUrl), HttpWebRequest)
 
req.Proxy = Nothing
Req.UseDefaultCredentials = True
 
'YOU MUST ASSIGN A COOKIE CONTAINER FOR THE REQUEST TO PULL THE COOKIES
 
Req.CookieContainer = CC
Res = DirectCast(Req.GetResponse(), HttpWebResponse)
 
'DUMP THE COOKIES
 
If Res.Cookies IsNot Nothing AndAlso Res.Cookies.Count <> 0 Then
    For Each c As Cookie In Res.Cookies
        response.write(vbTab & c.ToString())
    Next
Else
    response.write("No Cookies")
End If

Open in new window

0
 
David RobitailleAnalyst ProgrammerCommented:
I`m afraid I cannot help you further on that, you will have to wait for another expert to check that question.
0
 
mrl72Author Commented:
Although no complete solution provided the expert gave enough direction for me to find an answer to my problem.
0
 
David RobitailleAnalyst ProgrammerCommented:
I`m glad you figured out how to make it work!
0

Featured Post

The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now