Learn how to a build a cloud-first strategyRegister Now


In vb.net, why would my webclient.DownloadString instruction produce a string that is missing several Links

Posted on 2014-08-14
Medium Priority
Last Modified: 2016-06-21
I'm writing an application to read a website that includes a link to a spreadsheet that I need to download on a daily basis. I want to automate this.

Dim strWebPageSource As String
strWebPageSource = New System.Net.WebClient().DownloadString("websiteURL")

String is returned successfully, and parts of the web page are included. However, some of the parts are not included.

When I actually look at the web page manually and do a "VIEW SOURCE", I can see the link (<a href="...">xxx</a>) to the spreadsheet just fine. But the entire tag is missing from my string. There are other selective tags missing, also.

Is this because of some Javascript somewhere? And is there anything I can do to capture these tags?
Question by:TexanDonnaP
  • 4
  • 2
LVL 75

Expert Comment

by:käµfm³d 👽
ID: 40260766
Is this because of some Javascript somewhere?
Most likely.

And is there anything I can do to capture these tags?
Probably not with WebClient. You may want to switch to a a WebBrowser control, which is a scaled down version of IE. The WB control has the ability to run Javascript. However, since you are basically working with an instance of IE, you may be limited by any browser-specific coding the site owner has done.
LVL 37

Expert Comment

ID: 40260789
Usually VIEW SOURCE will be a good check what your code should also pick up. The source will only have the original javascript code. not the evaluated/interpreted javascript commands.
To double check, use Wireshark and your code, and see what's captured. Also check if this is the same as what is captured with your browser (but that should be the same as your VIEW SOURCE).
If the raw data is really different, a thing you cannot control could be some server side code (for instance, if browser type/version = x, then return this page. Or if cookie this, then return this page). But since this is server code, you can only guess what is causing the different returns (you could try to fake your response the same way as your browser, set all the headers the same).

Author Comment

ID: 40261124
What do you mean, "Set all the headers the same?"
Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

LVL 37

Expert Comment

ID: 40262687
Setting the headers the same is ONLY useful IF:

You capture data from your code: returns OTHER code then the captured data from your browser.

In this case, to check if the server is indeed sending other data depending on headers, look at the full headers sent by the browser and imitate as many as possible in your code:
Since your line is one code to insert the webclient, you have to adjust it to enable to add parameters (make more lines, use more variables)

myWebClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded")
myWebClient.Headers.Add("User-Agent", "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17")

etc etc

Then check if your code now finally gets different data.

Accepted Solution

TexanDonnaP earned 0 total points
ID: 40286798
Seems that neither answer here was pertinent. Found that the "source" I screen-scraped on a previous page had converted part of my site's HREF info from "&" to "&amp;". I'm a web newbie, and I didn't realize this conversion was done, and it caused my WebClient to return a partial webpage, along with an Error Message. I had missed the Error Message.
Once I took the URL string, and replaced the "&amp;" with "&", the webpage string was returned correctly using the WebClient.
Thanks so much to KAUFMED and KIMPUTER  for helping.

Author Comment

ID: 40286802
Sorry my response took so long. I had a week of vacation!

Author Comment

ID: 40286803
This problem is resolved.

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Thoughout my experience working on eCommerce web applications I have seen applications succumbing to increased user demand and throughput. With increased loads the response times started to spike, which leads to user frustration and lost sales. I ha…
Simulator games are perfect for generating sample realistic data streams, especially for learning data analysis. It is even useful for demoing offerings such as Azure stream analytics, PowerBI etc.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.
Suggested Courses
Course of the Month21 days, 6 hours left to enroll

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question