zorba111
asked on
How do I access the cookies in chrome programmatically from a seperate process (.NET)
I'm writing a web-bot in C#. One of my websites of interest has a login and uses cookies.
To save progamming I want to be able to login to the website using Chrome, and navigate to where the data is that I want to harvest.
At this point my bot will take over. First my bot will read the cookie values out of Chrome, and then it will use them in subsequent HTTP requests from my web-bot, so that it can stay authenticated while harvesting the data we need.
Is this possible? If so how?
I think I've read that Chrome exposes a JavaScript API for handling cookies, but how to access this from the C# code of another process.
Otherwise I will have to programmatically do all the initial login and navigation, using C#'s cookie classes to handle all the cookie transactions from the get-go (which I'd really prefer to do in Chrome as its just a one-time set up task).
To save progamming I want to be able to login to the website using Chrome, and navigate to where the data is that I want to harvest.
At this point my bot will take over. First my bot will read the cookie values out of Chrome, and then it will use them in subsequent HTTP requests from my web-bot, so that it can stay authenticated while harvesting the data we need.
Is this possible? If so how?
I think I've read that Chrome exposes a JavaScript API for handling cookies, but how to access this from the C# code of another process.
Otherwise I will have to programmatically do all the initial login and navigation, using C#'s cookie classes to handle all the cookie transactions from the get-go (which I'd really prefer to do in Chrome as its just a one-time set up task).
David Johnson, CD
retrieving the cookie from another application is problematic at best and there may be several cookies that you need i.e. session cookie, status cookie, and other web server defined cookies. they also may not be stored in an individual as sent by the web server but the in a browser specific format that the browser knows how to return to the server when requested.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi David, thanks for your answer. It was *all* the cookies used by this site that I needed, not just one. Yes, I agree, it seems problematic at best. Thanks for your experience.
I wasn't too sure what exactly you meant in the second sentence. Individual what? response? Some typos here that mask the meaning for me, sorry. If you could restate this I'd appreciate it, thanks.
I wasn't too sure what exactly you meant in the second sentence. Individual what? response? Some typos here that mask the meaning for me, sorry. If you could restate this I'd appreciate it, thanks.
ASKER
Hi Steve, thanks for your answer!
We are actually a paying subscriber to this website, and that is why I am able to login!
"Modern browsers are secure environments. Or at least, they try to be. The APIs made available in Chrome are meant to be used from within Chrome. That means creating a Chrome extension or Chrome app to run within the Chrome sandbox. .."
Yes, this answers my question. Can I access these APIs from another process? The answer is therefore NO (unless I suppose I could find the function calls exposed by the DLL and I made a low-level call into them, but all this would be undocumented, and yes it would still be in the context of my crawler process).
"Finally, as mentioned by David Johnson (#40910053), the browser stores the cookies SOMEWHERE. From the the info I've seen on the web, Chrome stores them in a SQLite database file. That's not a huge obstacle, though."
Ok, well I guess I could find them that way. Or I could scrape them in from the Chrome settings (theres a window for managing cookies which lists them by URL) using windows programming.
In the end without a direct API exposed to other processes to access cookies, it was just easier to code up the login and initial navigation to the data pages (into my web-bot).
We are actually a paying subscriber to this website, and that is why I am able to login!
"Modern browsers are secure environments. Or at least, they try to be. The APIs made available in Chrome are meant to be used from within Chrome. That means creating a Chrome extension or Chrome app to run within the Chrome sandbox. .."
Yes, this answers my question. Can I access these APIs from another process? The answer is therefore NO (unless I suppose I could find the function calls exposed by the DLL and I made a low-level call into them, but all this would be undocumented, and yes it would still be in the context of my crawler process).
"Finally, as mentioned by David Johnson (#40910053), the browser stores the cookies SOMEWHERE. From the the info I've seen on the web, Chrome stores them in a SQLite database file. That's not a huge obstacle, though."
Ok, well I guess I could find them that way. Or I could scrape them in from the Chrome settings (theres a window for managing cookies which lists them by URL) using windows programming.
In the end without a direct API exposed to other processes to access cookies, it was just easier to code up the login and initial navigation to the data pages (into my web-bot).