mrl72
asked on
Retaining cookies using net.HttpWebRequest
I am trying to write an asp.net application that scrapes some information from a website. The application requires you to be currently logged into the website. A link on the website, when clicked opens up my application which then gets the referring URL and scrapes the information from it. I am using HttpWebRequest to open the referring page and scrape the information using HttpWebResponse and GetResponseStream. The problem is once a call to HttpWebRequest has been made no cookies are brought over into the request and thus the referring page redirects to a login page. Reading on MS website it says cookies are automatically disabled, so how to you keep the cookies from the referring page and pass them through to the HttpWebRequest?
ASKER
I don't understand how that would help. I don't need to pass any username/password as I've already logged in on the previous web page and I don't want to log in again, just use the cookie from the requesting URL. If that makes sense..
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ok this is what I've done so far. When I run the script I see two cookies: cookie_l10n=en-us and Bus cookie_intl=deleted. Whatever they are I don't know. I don't see any of the other cookies from the browser sessions I have open.
dim sUrl as string = "http://sitetopullcookiesfrom"
Dim CC As New CookieContainer()
Dim Req As HttpWebRequest
Dim Res as HttpWebResponse
Req = CType(WebRequest.Create(sUrl), HttpWebRequest)
req.Proxy = Nothing
Req.UseDefaultCredentials = True
'YOU MUST ASSIGN A COOKIE CONTAINER FOR THE REQUEST TO PULL THE COOKIES
Req.CookieContainer = CC
Res = DirectCast(Req.GetResponse(), HttpWebResponse)
'DUMP THE COOKIES
If Res.Cookies IsNot Nothing AndAlso Res.Cookies.Count <> 0 Then
For Each c As Cookie In Res.Cookies
response.write(vbTab & c.ToString())
Next
Else
response.write("No Cookies")
End If
I`m afraid I cannot help you further on that, you will have to wait for another expert to check that question.
ASKER
Although no complete solution provided the expert gave enough direction for me to find an answer to my problem.
I`m glad you figured out how to make it work!
https://www.experts-exchange.com/questions/24045173/Prgrammatically-log-a-user-into-a-secure-site-ASP-NET.html