downloading a webpage

RGuillermo
RGuillermo used Ask the Experts™
on
Hello Experts
I am trying to download a webpage
and then programmatically clck on any button

AS an example I am using https://www.msn.com/en-us

My visual c# code is as follows

        private void button2_Click_1(object sender, EventArgs e)
        {
            string url = textBox_url.Text;
            WebClient webClient = new WebClient();
            webClient.DownloadFileCompleted += new AsyncCompletedEventHandler(Completed);
            webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(ProgressChanged);
            webClient.DownloadFileAsync(new Uri(url), @"c:\tmp\aaaaaaa___myfile.txt");
        }

the problem is once I get the source code
I do not get the information of the lower part of the web page.
Like the small squares with additional news not as important as the main body

Why am I not getting the full code of the web page?
Are these js scripts that should be downloaded separately?
REgards,
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
This is because contents below the page are getting downloaded when web page is scrolled down.
So on initial page load you will not be getting those contents.

As a solution, what you can do is through JavaScript scroll the page till bottom and then fire your code.
Dr. KlahnPrincipal Software Engineer
Commented:
You are getting the source for the web page skeleton, and that only.  Where the page calls APIs on other domains, or scripts, that additional source necessary render the entire page properly may not be pulled in.

Pulling in complete page source ecame more difficult over the last few years due to ad blockers.  The ad blockers figure out how to deal with a site pulling an ad from ad supplier B using method gamma, the site shoots back with a new method iota, the ad blocker figures out how to deal with that, and "source" may cascade down ten or twenty levels.
RGuillermoProject Manager

Author

Commented:
Thank you so much Experts!!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial