VB .Net - web browser control. Automate clicking on and manipulating popup windows after downloading .exe file

I believe that what you need to do with the System.Windows.Forms.WebBrowser control is to handle the Navigating event, set e.Cancel = True, and handle the file downloading yourself, instead of letting the WebBrowser control do that.  You should be able to get the URL from the e.Url.ToString() value.

Thank you both very much for the responses.   I will try theLearnedOne's  approach 1st as it seems to be the simpler of the two.    

*** TheLearnedOne****

- I'm a beginner to VB .Net so please forgive me for asking what may seem obvious.   Please see my questions and comments below.

- Could you please elaborate more on what you meant by handling the navigating event?    Do I need to create a new subroutine using Handles webbrowser3.navigating? (see code snippet below)

- Please see the approach taken below and in code snippet let me know if I'm going in the right direction.

- I'm not familiar with how to download a file without the webbrowser.   Do you have a code example you could share?   I tried using the "DownloadFile " method of the webClient class.   Not sure if this is the correct approach (see code snippet)

Instead of the approach you suggested, e.Url.ToString(), I got the url already by looping through an element collection of <a> tags.   url = curElement2.GetAttribute("href")

In my original approach I was going to navigate to the url of the .exe file - WebBrowser3.Navigate(url).    Should I bypass this altogether or should I navigate to the url and then cancel on the navigate event of the webbrowser?   (see if what I did in code snippet is headed in the right direction)

Private nav As Boolean
nav = False
 
...
...
...
 
' Once this program navigates to the page needed, retrieve the url (to the .exe file) to download the file.   Instead of navigating to the file through the webbrowser, get it another way such as below for example:
 
****Notice the "nav" flag I set to true here that is used in the Webbrowser3_Navigating subroutine (See at bottom of code snippet)***
 
nav = True
WebBrowser3.Navigate(url)
 
' this doesn't seem to work
Dim webc As New System.Net.WebClient()
webc.DownloadFile(url, "test")
 
 
'HERE IS A SUBROUTING TO HANDLE WEBBROWSER3.NAVIGATING EVENT
Private Sub Webbrowser3_Navigating(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserNavigatingEventArgs) Handles WebBrowser3.Navigating
        If nav = True Then
            e.Cancel = True
            docComplete = True
        End If
End Sub

Select allOpen in new window

1) You already have a Navigating event handler, so you don't need another one.

2) The System.Net.WebClient is a simplified wrapper for the System.Net.HttpWebRequest, which you can use to download files with the DownloadFile.

Thanks.

If I understand your correctly, I don't need to create the event handler that I included in the code snippet is that right?

If I already have a navigating event handler, where or in what way do I handle the Navigating event, you mentioned, "set e.Cancel = True"   Can you give an example?   Where do I do that?

Also, it also sounds like you are saying that I can use system.net.webclient to download the file. (as you said, it's just a simplified wrapper).   Of all the available methods, "DownloadFile" seems to be the most relevant but is it the right method to use?   Looking at the parameters the function accepts, it doesn't seem like this function will allow me save the file, give it a name, and save to a particular directory on a server or my own machine.

' this doesn't seem to work
Dim webc As New System.Net.WebClient()
webc.DownloadFile(url, "test")

Here's what I find in object browser for this function:

Public Sub DownloadFile(ByVal address As String, ByVal fileName As String)
     Member of System.Net.WebClient
Summary:
Downloads the resource with the specified URI to a local file.

Parameters:
address: The URI from which to download data.
fileName: The name of the local file that is to receive the data.

Exceptions:
System.Net.WebException: The URI formed by combining System.Net.WebClient.BaseAddress and address is invalid.  -or- filename is null or System.String.Empty.  -or- The file does not exist.  -or- An error occurred while downloading data.
System.NotSupportedException: The method has been called simultaneously on multiple threads.

It sounds like you have a little more complicated setup than you showed.  If you have the right address, which you get from e.Url.ToString(), for the download file, then you cancel the navigation, and download the file with the WebClient.  I don't have any examples for what you are trying to do, so I hope that we can come up with this together.

"System.Net.WebException: The URI formed by combining System.Net.WebClient.BaseAddress and address is invalid."
What arguments are you passing for DownloadFile?

"System.NotSupportedException: The method has been called simultaneously on multiple threads."
This is a little tougher to discern, since you might have implied multiple threads, instead of explicit threads.

If I type e.Url.ToString().   Intellisense, does not pick up anything for url.  it's not an option to choose.

For downloadFile I pass the following:
Dim webc As New System.Net.WebClient()
webc.DownloadFile(url, "test")

The url string looks something like this:
http://XXXXXX.com/user/u_report_giver.asp?Report=f_iv200907.exe

I assign a value to the string as follows in the code below:

       For Each curElement3 As HtmlElement In theElementCollection4
                    Dim controlText2 As String = curElement3.GetAttribute("InnerText").ToString
                    Debug.Print(controlText2)
                    If controlText2 = "XXXXInventory" Then
                        url = curElement3.GetAttribute("href")

You mentioned instead of of using the webbrowser control, I should be able to download myself.    I thought perhaps this approach is to eliminate the 2 securities warnings (in the attachment) because they could be related specifically to the webbrowser.

The other important step in the process I want to be sure came across is to unzip a file to a specific directory.   When running through the flow manually on the browser, clicking pass the 2 browser security warnings, then unzipping the file (clicking on the button to unzip and selecting location), there is no option to choose a file name to save as.    Clicking on unzip creates a file named IV200907.txt and saves to the location specified.   I need the ability to change the name of this file.

Does your application work with a single web page, or should it be dynamic to support multiple web pages?

The application starts at a particular url and navigates through multiple web pages to get to a destination web page.    When it gets to this page, it finds a specific link and clicks the link to download.    

Unless I answered your question, could you elaborate on what you meant by it being dynamic to support multiple web pages?    

Thanks.

So that we're on the same page, let me clarify...
 
The automation script I wrote uses webbrowser to navigates through multiple web pages.   When it gets to the destination page, this page has multiple hyperlinks to reports such as below:

Feb 2009 Inventory
March 2009 Inventory
April 2009 Inventory
May 2009 Inventory
June 2009 Inventory
July 2009 Inventory
August 2009 Inventory

Say I click on the link for "March 2009 Inventory" link the url is:
http://xxxxxxx.com/user/u_report_giver.asp?Report=f_iv200903.exe

All of the other hyperlinks will be essentially the same except the name of the exe file will differ based on the date of the report selected.    

For July 2009 Inventory" link the url will be:
http://xxxxxxx.com/user/u_report_giver.asp?Report=f_iv200907.exe

Does this make more sense now?

Do you have to navigate through all the pages, or can you go directly to the download page, with the appropriate arguments in the URL?

I have to navigate through all the pages.

I have to log in to the website first, at the next page I choose from a list of states all hyperlinks.  
- AZ (hyperlink)
- CA (hyperlink)
- NV (hyperlink)
....
.....
....

I'll choose, AZ for example.....

After clicking on the AZ hyperlink and reaching the next page, I choose from a variety of report links based on date as described above.    
e.g. July 2009 Inventory (hyperlink)

After I download the file, the webbrowser needs to go back to the previous web page and choose the next state, and the same report link to download the report again for the relevant month.

OK, sometimes you can bypass all the startup stuff, by passing a URL with the proper query string values.

I don't see any reason for parsing through the HTML to get the URL for the download.  If you navigate the pages, and check the URLs in the DocumentComplete event handler, and then perform the necessary operations to get to the final download page.  Then, you can parse through the HTML, get the links for all the downloads, and download all the files at one time, without having to navigate between pages.

In my case, the page with all "States" names, each a hyperlink.
The url for this page is
http://xxxxx.com/user/u_user.asp

When I scroll over the hyperlink for each state you can see the following at the bottom of the web page.
e.g. AZ (hyperlink) .... javascript:frmSubmit('AZ', 1,1);
e.g. CA (hyperlink).... javascript:frmSubmit('CA', 14,1);

After clicking any hyperlink, the next page url is always the same.  
http://XXXXXX.com/user/u_Report.asp

I don't see any name value pairs, etc. passed in the url.

I am trying to point out different options, but I clearly don't completely understand what you are working with (on the outside looking in).  

I have done a lot of work with the WebBrowser control, HttpWebRequest, WebClient, Microsoft.mshtml, HTML parsers, etc., to automate web pages, but each situation is unique.  

I am trying to determine your exact requirements, and give you good directions to take.

I just created a document and added it as an attachment (visuals say so much more).   I appreciate all your time and effort on this!    Please let me know if this makes things more clear and any other questions to help clarify the requirements.
Workflow.docx

awesome, that helps a lot.    thank you!       Now I just need to figure out how to programatically set those values in the form and submit.     Also, need to download the file address the windows popups, winzip extractor, etc.

I feel already I should give points but we have been addressing some other questions other than the original (which is good!) and still have yet to address the original question.     I have more questions and don't want you to feel cheated of points or having to spend too much time on one question.    Do you want me to open up new questions concerning this project or keep it all together?  

function frmSubmit(astrStateCode, aintDivisionID, aintStateLevelFlag)
{
	document.frmRetrieveFiles.action = "u_Report.asp";
	document.frmRetrieveFiles.hdnStateCode.value = astrStateCode;
	document.frmRetrieveFiles.hdnDivisionId.value = aintDivisionID;
	document.frmRetrieveFiles.hdnStateLevelFlag.value = aintStateLevelFlag;
	document.frmRetrieveFiles.submit();
	}
 
<input type="hidden" name="hdnStateCode" id="hdnStateCode">
<input type="hidden" name="hdnDivisionId" id="hdnDivisionId">
<input type="hidden" name="hdnSupplierCD" id="hdnSupplierCD">	
<input type="hidden" name="hdnStateLevelFlag" id="hdnStateLevelFlag">
<input type="hidden" name="hdnUserLevel" id="hdnUserLevel" value="SU">	
 
 
<form name="frmRetrieveFiles" id="frmRetrieveFiles" method="post"><!--table border="0" cellspacing="0" cellpadding="4" width="100%" align="center" border="0">
 
<a href="javascript:frmSubmit('AZ', 1, 1);">Arizona</A></font></b></td></tr><tr><td>&nbsp;&nbsp;&nbsp;<b><font face="Verdana,Arial,Helvetica" size="1">
<A href="javascript:frmSubmit('CA',14,1);">California</A></font></b></td></tr><tr><td>&nbsp;&nbsp;&nbsp;<b><font face="Verdana,Arial,Helvetica" size="1">
<A href="javascript:frmSubmit('CO',17,1);">Colorado</A></font></b></td></tr><tr><td>&nbsp;&nbsp;&nbsp;<b><font face="Verdana,Arial,Helvetica" size="1">
<A href="javascript:frmSubmit('FL', 15, 1);">Florida</A></font></b></td></tr><tr><td>&nbsp;&nbsp;&nbsp;<b><font face="Verdana,Arial,Helvetica" size="1">
<A href="javascript:frmSubmit('HI', 7, 1);">Hawaii</A></font></b></td></tr><tr><td>&nbsp;&nbsp;&nbsp;<b><font face="Verdana,Arial,Helvetica" size="1">
...
...
...

Select allOpen in new window

Instead of finding a way to programaticaly click the browser back button, to go from
a. www.xxxxxx.com/user/u_report.asp
back to
b. www.xxxxxx.com/user/u_user.asp

Seems I can just use webbrowser1.navigate(www.xxxxx.com/user/u_user.asp)

I could set the values programatically (doing the same as what javascript does):
    webbrowser1.Document.Body.All("hdnStateCode").SetAttribute("Value", "CA")
    webbrowser1.Document.Body.All("hdnDivisionID").SetAttribute("Value", "14")
    webbrowser1.Document.Body.All("hdnStatelevelflag").SetAttribute("Value", "1")

You mentioned being able to simulate the u_user.asp post but is there any benefit to being able to simulate this over just navigating back to the u_user.asp and programatically clicking on the hyperlink

Why not just let javascript do the job of populating those values?

I think the direction you were headed was to find a way to to download all the files at once right?    It would be great, but do you think that is still possible?

I created a new thread specifically requesting help on downloading the file and the best approach.

https://www.experts-exchange.com/questions/24672888/VB-Net-trouble-automating-download-of-a-file-using-urldownloadtofile.html

Will leave this one open and can award points for further help to download all files at once if possible and to get past user window popups for browser security and winzip extract to unzip file.

Please Dismiss this earlier question to you -  ""You mentioned being able to simulate the u_user.asp post but is there any benefit to being able to simulate this over just navigating back to the u_user.asp and programatically clicking on the hyperlink    Why not just let javascript do the job of populating those values?"

The information you provided helped me to simplify my code and an error I was getting earlier.  Instead of retrieving an HTMLElement collection and looping through to find what I needed ... I am just assigning the input values for the form directly myself using setattribute() and those values for each state, that I found on the page.  

Again, if you have any thoughts on how to a way to approach downloading all the files at once, please let me know.   thanks and I hope you have a great weekend!

I am going to close this and award some points for the help provided so far.   I have create a new question as a continuation.

https://www.experts-exchange.com/index.jsp?qid=24675255

Thank you both for your you suggestions and help on this issue.  I really appreciate your time.   Your comments and feedback have definitely helped me to certain degree.   This is the first time I've developed in VB .Net and I haven't done any development for years so I'm struggling!    I opened a new question related and continuined from this question.   If you have time to assist please see  - https://www.experts-exchange.com/index.jsp?qid=24675255