I'm building a little web crawler, and for each web page, I extract all links to navigate further.
Today I'm downloading html source by the WebClient class and then searches for <a> tags.
I have realized that I don't get all links that I want this way.
If I navigate to a web page with FireFox and open FireBug, I can find the HTML source that I'm looking for, but is there any component out there that does this in .net?
It's also important that my console app can run in multi threading mode.