I'm writing a personal news aggregator that pulls press releases from some of my favorite news websites, but I've come across a problem: some websites require you to log in before accessing their content.
BusinessWire.com, for example, will tell me the URL of a story via their RSS feed, but when I go to the address I will only see the story if I'm logged in. Otherwise, I see some generic login page.
I have a function GetHTML(url) that grabs the HTML from any url, but if I try to do that on a businesswire page, I just get the HTML from the login page. Is there any way I can have my program "log in" so that when it retrieves the HTML it actually gets page containing the article I'm looking for and not the login page? I'm assuming the "logged in" status is done using cookies... can the VB internet functions have their own cookies?
Here's a sample URL from businesswire that will only show you the story if you're logged in: