Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 277
  • Last Modified:

Problem with downloading a file from a web in Visual Basic 2008

Hello,
I am writting a program in visual basic 2008 where by clicking a button it will download ebill from a website and save it to my local drive. Here is the manual process downloading ebill:
1. login with userid/passwd
2. go to the ebill section
3. press Download button on their website
4. save the file
When I click Download button manually I saw a name 'downloadEbill.do' at the bottom of the page. The actual file name is ebill_month.zip and when I put this file name I get 404 error. But when I use 'downloadEbill.do' as a filename I do not get any error and did get that file in my local drive.  So the problem I am having is:
1. I can't open downloadEbill.do file to see whats inside. How to open it?
2. The main file is ebill_month.zip which I want.  How do I have to program it to get this file?

Could anybody please help me? I would really appreciate it. If you need more information please let me know.
Thanks so much in advance.
0
guavamay
Asked:
guavamay
  • 5
  • 5
  • 2
  • +1
4 Solutions
 
oobaylyCommented:
If you examine the code on the ebill site, you'll probably find that the Download button posts some data to downloadebill.do, this server side script then generates the bill, writes it to the response along with a few headers describing what type of file it is along with the desired filename. ebill_month.zip won't actually exist on the webserver.

Can you show us the code you're currently using to download the file in your app?
0
 
guavamayAuthor Commented:
Oh, I see, that make sense now. But then how do I incorporate that into my application?
 Here is the code  as per your request.......
------------------------------------
Imports System.Net
Imports System.IO
Public Class Form1
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim webAddr As String = "https://secure.mywebsitename.com/bill/"
        'Dim dfileName As String = "eBill_apr.zip"
        Dim dfileName As String = "downloadebill.do"
        Dim myWebResource As String = Nothing

        'create a new webclient instance
        Dim myWebClient As New WebClient()
        'concatenate the web address with the web resource filename.
        myWebResource = webAddr + dfileName
        Console.WriteLine("Downloading File ""{0}"" from ""{1}"" ....." + ControlChars.Cr + ControlChars.Cr, dfileName, myWebResource)
        'adding userid and passwd
        myWebClient.Credentials = New NetworkCredential("myuserid", "mypasswd", "https://secure.mywebsitename.com/bill/")

        'the downloadfile() method downloads the web resource file and saves it into the current folder
        myWebClient.DownloadFile(myWebResource, dfileName)

        Console.WriteLine("Successfully downloaded file ""{0}"" from ""{1}""", dfileName, myWebResource)

        MsgBox("after consol with sucess...")
               
    End Sub
End Class

Thanks so much.

0
 
oobaylyCommented:
Ok, a couple of things:
Unless you login to the site using a dialog like the attached file (The browser displays it when it received 401 Response with a WWW-Authenticate header), you cannot use the Credentials property.
You'll need to create a CookieContainer, which is attached to every web request you make so that the web server can keep track of your session.
You'll have to make more than one request, one to authenticate and one to retrieve the file.

Have a look at this code, it posts login credentials to a server, the create another request to retrieve the file (setting the same CookieCollection). You'll have to inspect the forms to determine what fields have to be posted.
'' Cookie container so that we can keep track of the session
''       * Use it for any further requests made
''       
 
Dim cookies As New CookieContainer()
 
' Create the post data
Dim username As String = ""
Dim password As String = ""
Dim postData As String = String.Format("username={0}&password={1}", username, password)
 
'' Create the request
Dim req As HttpWebRequest = DirectCast(HttpWebRequest.Create("http://www.domain.com/login.php"), HttpWebRequest)
req.Method = "POST"
req.CookieContainer = cookies
 
'' Add the post data
req.ContentLength = postData.Length
Dim writer As New StreamWriter(req.GetRequestStream())
writer.Write(postData)
writer.Flush()
writer.Close()
 
'' Send the request, making sure to close the stream
Dim resp As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
resp.Close()

Open in new window

401-auth.PNG
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
guavamayAuthor Commented:
Hello,
Is that mean in the middle of download processing, my credential userid/passwd getting lost, hence the website couldn't find the file with filename ebill_apr.zip?

Also when you said "You'll have to inspect the forms to determine what fields have to be posted", there is no form that I have to fill in. In manual process, once I login, I go to ebill and they have a button called 'Download' which brings the filename and it asks to save it or open it or cancel it. Would it be possible for you to explain a bit like what kind of information as an example I should be looking for and also where will I add them - at the post data section?

Sorry I am sort of confused (I am still very new to vb).
Thanks so much for your help in advance.



0
 
oobaylyCommented:
Well, what does the login form look like? Is it like the dialog I posted a screenshot of, or is it a standard html web form?

By inspecting a form I mean looking at the html page source and seeing what the names of the elements are. My example made the assumption that the there was a textbox called "username", and a password element called "password".
Also, just because there's no form elements visible (ie. Selectbox, Textbox) it doesn't mean that there's no form. It's quite possible that the button submits a form populated with an array of hidden fields.
Of course, it's quite possible that all you need is a standard GET HttpWebRequest, but you will need to duplicate all the querystring fields.

When automating a function that does something like this it's always necessary to have a very good idea of how the target web page works.
0
 
guavamayAuthor Commented:
Hi oobayly,
Thanks for the example code. I have been trying using your example for past few days to download the file but unfortunately no success yet!  I have learned a lot though........
At one point I did get the response (I think!) but I don't see the file. After I added more stuff in postdata now I am getting following error message:
"The underlying connection was closed: An unexpected error occurred on a receive."
I have couple of questions:
1. Can I ask you what this error means and what causes it?
2.Should I send my download filename as one of the parameter in postdata?
3.If I would like to save the file in my local drive to a specific directory - how do I send that information?
Thanks so much in advance.
0
 
jake072Commented:
guavamay,

I would recommend that you use a normal web browser to manually download the file, and have wireshark open so you can see all the data travelling between the browser and the server.

Then you will know what parameters you need...  Otherwise, it's pretty much a crap shoot.

Jake
0
 
HairbrushCommented:
HTTP fiddler (http://www.fiddler2.com/fiddler2/) is a free tool that would also be useful in this situation, for inspecting the raw HTTP transactions between the browser and server.
0
 
oobaylyCommented:
Been a hectic few days at work, so I haven't even been able to check this thread.

I agree with jake072. Wireshark's a fantastic tool for learning how any network protocols work, though may take a bit of getting used to. As for HTTP fiddler, I've never heard of it, but it certainly sounds like it's worth a look.

As for your questions:
1). We'd need to see the code you're using. It normally points towards a network error.
2). jake072 & Hairbrush gave their suggestions for analysing the post data required. Another option would be to use the Firebug (https://addons.mozilla.org/firefox/addon/1843) or WebDeveloper (https://addons.mozilla.org/en-US/firefox/addon/60) plugins for Firefox. Don't even think about using IE to work out the html on a 3rd party website unless you already read, speak & dream html.
3). Saving the downloaded file is simply a matter of copying the data returned in the HttpWebRequest's ResponseStream into a FileStream. You could use something like the following code.

Finally, another option would be to use a customised WebClient. You can't use the standard WebClient as it doesn't support cookies. This way you can use the UploadValues method which allows you to send a NameValue collection of parameters, and retrieves a byte array containing the data returned. This byte array can then be written to a FileStream.

http://couldbedone.blogspot.com/2007/08/webclient-handling-cookies.html





Private Sub CopyStream(ByVal src As Stream, ByVal trg As Stream)
    Dim buff As Byte() = New Byte(4095) {}
    While True
        Dim read As Integer = src.Read(buff, 0, buff.Length)
        If read = 0 Then Exit While
        
        trg.Write(buff, 0, read)
    End While
    trg.Flush()
End Sub

Open in new window

0
 
guavamayAuthor Commented:
Hi Jake and Hairbrush: Thank You both so much for the valuable information which I didn't know those tools exists. I  need to spend some time to learn how to use them.

Hi oobayly: Thank you so much for answering my questions. I managed to solve the problem with the error messg which I was getting before. But now I am getting something different...
Here is my code....
        Dim cookies As New CookieContainer()
        ' Create the post data
         Dim postData As String = String.Format("username={0}&password={1}&src={2}&filename={3}&application={4}", username, password, src,filename, application)
        '' Create the request
        Dim req As HttpWebRequest = DirectCast(HttpWebRequest.Create("my uri"), HttpWebRequest)
        req.Method = "POST"
        req.CookieContainer = cookies
         '' Add the post data
        req.ContentLength = postData.Length
        Dim writer As New StreamWriter(req.GetRequestStream())
        'use try for exception
        Try
            writer.Write(postData)
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
        writer.Flush()
        writer.Close()
       
        '' Send the request
        Dim resp As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
        MsgBox(req.HaveResponse)
        MsgBox(resp.StatusDescription)
        MsgBox(resp.Headers.ToString)
        Dim sr As New StreamReader(resp.GetResponseStream())
        Dim sTR As String = sr.ReadToEnd()
        MsgBox(sTR.ToString)
        resp.Close()
        MsgBox("at the end")
    End Sub

I know I am getting the response back but the MsgBox(sTR.ToString) displays the following msg:
"Your current session has expired. Please re-enter your login/passwd to access"
Is anything wrong with my cookie statement? Or am I missing something?
Thanks again in advance.






0
 
jake072Commented:
guavamay,

Undoubtedly there is something wrong with the cookie stream...  Again, I would suggest using Wireshark or even a FireFox plugin as oobayly suggests...  You need to duplicate the exact cookie stream, but note that obviously you can't hardcode certain things, like session id, etc...

It's a steep learning curve...  I remember it took me like 15 hours of solid non-stop work to do an ebay sniper program...

Good luck,

Jake
0
 
oobaylyCommented:
Try setting the POST request's ContentType. I'd forgotten to include it in the code I posted. Also, if the src, filename & application values contain characters like ampersands and spaces, you may need to url-encode them. To use the System.Web.HttpUtility, you may have to a add a reference to System.Web
req.ContentType = "application/x-www-form-urlencoded"
 
'' URL Encode all the post data values
Dim postData As String = String.Format("username={0}&password={1}&src={2}&filename={3}&application={4}", _
  System.Web.HttpUtility.UrlEncode(username), _
  System.Web.HttpUtility.UrlEncode(password), _
  System.Web.HttpUtility.UrlEncode(src), _
  System.Web.HttpUtility.UrlEncode(filename), _
  System.Web.HttpUtility.UrlEncode(application))

Open in new window

0
 
guavamayAuthor Commented:
Thanks oobayly.
I added the ContentType and the URL Encode all my post data values, unfortunately, it still didn't work. I am still getting the same error mesg --- "Your current session has expired. Please re-enter your login/passwd to access".  
Am I missing anything else?
I am not sure why I am still getting this error?
I will spend more time to study the html code as well as the two tools that jake072 and Hairbrush mentioned.
Thanks so much for your help.
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

  • 5
  • 5
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now