Get rendered HTML source in .net c#


I'm building a little web crawler, and for each web page, I extract all links to navigate further.

Today I'm downloading html source by the WebClient class and then searches for <a> tags.

I have realized that I don't get all links that I want this way.
Some pages render html by javascript and ajax after page is loaded.

If I navigate to a web page with FireFox and open FireBug, I can find the HTML source that I'm looking for, but is there any component out there that does this in .net?

It's also important that my console app can run in multi threading mode.

Who is Participating?
Ron MalmsteadInformation Services ManagerCommented:
You might consider switching to an invisible WebBrowser Object in an invisible Win Form project rather than WebClient in a console.

That will allow you to access the DOM document after it is fully rendered.

HtmlElementCollection Collection = WebBrowser1.Document.GetElementsByTagName("a");
Try this mehtod

        public static String RenderHtml<T>(this Control control) where T : Control
            StringWriter controlString = new StringWriter();
            control.RenderControl(new HtmlTextWriter(controlString));
            return controlString.ToString();
jimmieanderssonAuthor Commented:
Thanks for your reply.
Not sure if I used it correctly, But I tried this:

                var control = new LiteralControl(webClientData.Source);
                var html = control.RenderHtml<LiteralControl>();

Open in new window

the html return didn't change anything. exactly the same as input
jimmieanderssonAuthor Commented:
It works. But unfortunately not as fast as I was hoping
Ron MalmsteadInformation Services ManagerCommented:
That's probably because it waits until the page is fully rendered... all little bits and peices.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.