We help IT Professionals succeed at work.
Get Started

Get rendered HTML source in .net c#

3,538 Views
Last Modified: 2012-09-06
Hello,

I'm building a little web crawler, and for each web page, I extract all links to navigate further.

Today I'm downloading html source by the WebClient class and then searches for <a> tags.

I have realized that I don't get all links that I want this way.
Some pages render html by javascript and ajax after page is loaded.

If I navigate to a web page with FireFox and open FireBug, I can find the HTML source that I'm looking for, but is there any component out there that does this in .net?

It's also important that my console app can run in multi threading mode.


Thanks
Comment
Watch Question
Network Operations Manager
CERTIFIED EXPERT
Commented:
This problem has been solved!
Unlock 1 Answer and 5 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE