[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

Reading source of a web page created dynamically

Posted on 2016-10-03
6
Medium Priority
?
53 Views
Last Modified: 2016-10-08
I would like to be able to read the contents of a table on a web page.  
I can use the WebCient.OpenRead to read the source HTML code.  Using FireFox I can view the source and they seem to be identical.  So far so good.
Also using Firefox I can inspect an element - the table I want to read.  The ID of the table doesn't exist in the source code, in fact no tables exist in the source code of the website.  Looking at the source I think there is a frame which is being created dynamically and this frame then has the contents copied and pasted into a div.

In the source from the WebClient.OpenRead I get:   <div id="xyz_abc_div"></div>
In the browser (inspect element) I see a load more HTML with the table I want.

Question.  How can I get the contents of this dynamically filled div ?
0
Comment
Question by:AndyAinscow
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 41826879
It is more likely that the data is being loaded thru AJAX / javascript after the page is loaded.  It would not show in the view source in that case.  That is a common method these days.  The only way to get that content is to run the javascript that makes the request.
0
 
LVL 45

Author Comment

by:AndyAinscow
ID: 41827005
Hmmm, thanks.

I think it is javascript (numbers of .js scripts being loaded).  I also see a function called LoadFrame with a frame and then (snippet)

                  var iframe = document.getElementById('xyz_frame');
                  iframe.src = 'path goes here';
                  iframe.onload = function(){
                        xyz_copyFrameToDiv(iframe, 'xyz_div')


>>The only way to get that content is to run the javascript that makes the request.
I understand the words but the result exists in the browser, can't I get at that.
0
 
LVL 84

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 2000 total points
ID: 41827028
The browsers run javascript and your function does not.  WebCient.OpenRead basically just reads a single file from a 'resource'.  It does not do any processing.  https://msdn.microsoft.com/en-us/library/ms144209(v=vs.110).aspx    The javascript, CSS, and images are often in other files which are not loaded with the original request.  In addition, even if you download the javascript files, you must have a javascript interpreter and the HTML DOM to make any sense out of them.  Javascript is a programming language, just reading the source file doesn't create the results.
0
Will your db performance match your db growth?

In Percona’s white paper “Performance at Scale: Keeping Your Database on Its Toes,” we take a high-level approach to what you need to think about when planning for database scalability.

 
LVL 45

Accepted Solution

by:
AndyAinscow earned 0 total points
ID: 41827632
Found what I wanted.  I can use the DocumentText property of a WebBrowserControl to get the currently used HTML of the page after the browser has run the js functions.
0
 
LVL 45

Author Closing Comment

by:AndyAinscow
ID: 41834916
Thanks for the info on how it is working in the background, it made me think of an alternative to get what I wanted.
0
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 41835186
You're welcome, glad to help.
0

Featured Post

Enroll in October's Free Course of the Month

Do you work with and analyze data? Enroll in October's Course of the Month for 7+ hours of SQL training, allowing you to quickly and efficiently store or retrieve data. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Australian government abolished Visa 457 earlier this April and this article describes how this decision might affect Australian IT scene and IT experts.
CTAs encourage people to do something specific to show interest in your company, product or service. Keep reading to learn why CTAs should always be thought of as extremely important, albeit small, sections of websites.
The viewer will learn how to dynamically set the form action using jQuery.
The is a quite short video tutorial. In this video, I'm going to show you how to create self-host WordPress blog with free hosting service.

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question