• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 127
  • Last Modified:

In vb.net, why would my webclient.DownloadString instruction produce a string that is missing several Links

I'm writing an application to read a website that includes a link to a spreadsheet that I need to download on a daily basis. I want to automate this.

Dim strWebPageSource As String
strWebPageSource = New System.Net.WebClient().DownloadString("websiteURL")

String is returned successfully, and parts of the web page are included. However, some of the parts are not included.

When I actually look at the web page manually and do a "VIEW SOURCE", I can see the link (<a href="...">xxx</a>) to the spreadsheet just fine. But the entire tag is missing from my string. There are other selective tags missing, also.

Is this because of some Javascript somewhere? And is there anything I can do to capture these tags?
  • 4
  • 2
1 Solution
käµfm³d 👽Commented:
Is this because of some Javascript somewhere?
Most likely.

And is there anything I can do to capture these tags?
Probably not with WebClient. You may want to switch to a a WebBrowser control, which is a scaled down version of IE. The WB control has the ability to run Javascript. However, since you are basically working with an instance of IE, you may be limited by any browser-specific coding the site owner has done.
Usually VIEW SOURCE will be a good check what your code should also pick up. The source will only have the original javascript code. not the evaluated/interpreted javascript commands.
To double check, use Wireshark and your code, and see what's captured. Also check if this is the same as what is captured with your browser (but that should be the same as your VIEW SOURCE).
If the raw data is really different, a thing you cannot control could be some server side code (for instance, if browser type/version = x, then return this page. Or if cookie this, then return this page). But since this is server code, you can only guess what is causing the different returns (you could try to fake your response the same way as your browser, set all the headers the same).
TexanDonnaPAuthor Commented:
What do you mean, "Set all the headers the same?"
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Setting the headers the same is ONLY useful IF:

You capture data from your code: returns OTHER code then the captured data from your browser.

In this case, to check if the server is indeed sending other data depending on headers, look at the full headers sent by the browser and imitate as many as possible in your code:
Since your line is one code to insert the webclient, you have to adjust it to enable to add parameters (make more lines, use more variables)

myWebClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded")
myWebClient.Headers.Add("User-Agent", "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17")

etc etc

Then check if your code now finally gets different data.
TexanDonnaPAuthor Commented:
Seems that neither answer here was pertinent. Found that the "source" I screen-scraped on a previous page had converted part of my site's HREF info from "&" to "&amp;". I'm a web newbie, and I didn't realize this conversion was done, and it caused my WebClient to return a partial webpage, along with an Error Message. I had missed the Error Message.
Once I took the URL string, and replaced the "&amp;" with "&", the webpage string was returned correctly using the WebClient.
Thanks so much to KAUFMED and KIMPUTER  for helping.
TexanDonnaPAuthor Commented:
Sorry my response took so long. I had a week of vacation!
TexanDonnaPAuthor Commented:
This problem is resolved.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

  • 4
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now