Solved

Problem with downloading a file from a web in Visual Basic 2008

Posted on 2009-05-04
14
254 Views
Last Modified: 2012-06-27
Hello,
I am writting a program in visual basic 2008 where by clicking a button it will download ebill from a website and save it to my local drive. Here is the manual process downloading ebill:
1. login with userid/passwd
2. go to the ebill section
3. press Download button on their website
4. save the file
When I click Download button manually I saw a name 'downloadEbill.do' at the bottom of the page. The actual file name is ebill_month.zip and when I put this file name I get 404 error. But when I use 'downloadEbill.do' as a filename I do not get any error and did get that file in my local drive.  So the problem I am having is:
1. I can't open downloadEbill.do file to see whats inside. How to open it?
2. The main file is ebill_month.zip which I want.  How do I have to program it to get this file?

Could anybody please help me? I would really appreciate it. If you need more information please let me know.
Thanks so much in advance.
0
Comment
Question by:guavamay
  • 5
  • 5
  • 2
  • +1
14 Comments
 
LVL 15

Expert Comment

by:oobayly
ID: 24296358
If you examine the code on the ebill site, you'll probably find that the Download button posts some data to downloadebill.do, this server side script then generates the bill, writes it to the response along with a few headers describing what type of file it is along with the desired filename. ebill_month.zip won't actually exist on the webserver.

Can you show us the code you're currently using to download the file in your app?
0
 

Author Comment

by:guavamay
ID: 24297315
Oh, I see, that make sense now. But then how do I incorporate that into my application?
 Here is the code  as per your request.......
------------------------------------
Imports System.Net
Imports System.IO
Public Class Form1
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim webAddr As String = "https://secure.mywebsitename.com/bill/"
        'Dim dfileName As String = "eBill_apr.zip"
        Dim dfileName As String = "downloadebill.do"
        Dim myWebResource As String = Nothing

        'create a new webclient instance
        Dim myWebClient As New WebClient()
        'concatenate the web address with the web resource filename.
        myWebResource = webAddr + dfileName
        Console.WriteLine("Downloading File ""{0}"" from ""{1}"" ....." + ControlChars.Cr + ControlChars.Cr, dfileName, myWebResource)
        'adding userid and passwd
        myWebClient.Credentials = New NetworkCredential("myuserid", "mypasswd", "https://secure.mywebsitename.com/bill/")

        'the downloadfile() method downloads the web resource file and saves it into the current folder
        myWebClient.DownloadFile(myWebResource, dfileName)

        Console.WriteLine("Successfully downloaded file ""{0}"" from ""{1}""", dfileName, myWebResource)

        MsgBox("after consol with sucess...")
               
    End Sub
End Class

Thanks so much.

0
 
LVL 15

Expert Comment

by:oobayly
ID: 24297956
Ok, a couple of things:
Unless you login to the site using a dialog like the attached file (The browser displays it when it received 401 Response with a WWW-Authenticate header), you cannot use the Credentials property.
You'll need to create a CookieContainer, which is attached to every web request you make so that the web server can keep track of your session.
You'll have to make more than one request, one to authenticate and one to retrieve the file.

Have a look at this code, it posts login credentials to a server, the create another request to retrieve the file (setting the same CookieCollection). You'll have to inspect the forms to determine what fields have to be posted.
'' Cookie container so that we can keep track of the session

''       * Use it for any further requests made

''       
 

Dim cookies As New CookieContainer()
 

' Create the post data

Dim username As String = ""

Dim password As String = ""

Dim postData As String = String.Format("username={0}&password={1}", username, password)
 

'' Create the request

Dim req As HttpWebRequest = DirectCast(HttpWebRequest.Create("http://www.domain.com/login.php"), HttpWebRequest)

req.Method = "POST"

req.CookieContainer = cookies
 

'' Add the post data

req.ContentLength = postData.Length

Dim writer As New StreamWriter(req.GetRequestStream())

writer.Write(postData)

writer.Flush()

writer.Close()
 

'' Send the request, making sure to close the stream

Dim resp As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)

resp.Close()

Open in new window

401-auth.PNG
0
 

Author Comment

by:guavamay
ID: 24298253
Hello,
Is that mean in the middle of download processing, my credential userid/passwd getting lost, hence the website couldn't find the file with filename ebill_apr.zip?

Also when you said "You'll have to inspect the forms to determine what fields have to be posted", there is no form that I have to fill in. In manual process, once I login, I go to ebill and they have a button called 'Download' which brings the filename and it asks to save it or open it or cancel it. Would it be possible for you to explain a bit like what kind of information as an example I should be looking for and also where will I add them - at the post data section?

Sorry I am sort of confused (I am still very new to vb).
Thanks so much for your help in advance.



0
 
LVL 15

Expert Comment

by:oobayly
ID: 24298513
Well, what does the login form look like? Is it like the dialog I posted a screenshot of, or is it a standard html web form?

By inspecting a form I mean looking at the html page source and seeing what the names of the elements are. My example made the assumption that the there was a textbox called "username", and a password element called "password".
Also, just because there's no form elements visible (ie. Selectbox, Textbox) it doesn't mean that there's no form. It's quite possible that the button submits a form populated with an array of hidden fields.
Of course, it's quite possible that all you need is a standard GET HttpWebRequest, but you will need to duplicate all the querystring fields.

When automating a function that does something like this it's always necessary to have a very good idea of how the target web page works.
0
 

Author Comment

by:guavamay
ID: 24351429
Hi oobayly,
Thanks for the example code. I have been trying using your example for past few days to download the file but unfortunately no success yet!  I have learned a lot though........
At one point I did get the response (I think!) but I don't see the file. After I added more stuff in postdata now I am getting following error message:
"The underlying connection was closed: An unexpected error occurred on a receive."
I have couple of questions:
1. Can I ask you what this error means and what causes it?
2.Should I send my download filename as one of the parameter in postdata?
3.If I would like to save the file in my local drive to a specific directory - how do I send that information?
Thanks so much in advance.
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 14

Assisted Solution

by:jake072
jake072 earned 50 total points
ID: 24364035
guavamay,

I would recommend that you use a normal web browser to manually download the file, and have wireshark open so you can see all the data travelling between the browser and the server.

Then you will know what parameters you need...  Otherwise, it's pretty much a crap shoot.

Jake
0
 
LVL 12

Assisted Solution

by:Hairbrush
Hairbrush earned 50 total points
ID: 24364600
HTTP fiddler (http://www.fiddler2.com/fiddler2/) is a free tool that would also be useful in this situation, for inspecting the raw HTTP transactions between the browser and server.
0
 
LVL 15

Assisted Solution

by:oobayly
oobayly earned 400 total points
ID: 24365321
Been a hectic few days at work, so I haven't even been able to check this thread.

I agree with jake072. Wireshark's a fantastic tool for learning how any network protocols work, though may take a bit of getting used to. As for HTTP fiddler, I've never heard of it, but it certainly sounds like it's worth a look.

As for your questions:
1). We'd need to see the code you're using. It normally points towards a network error.
2). jake072 & Hairbrush gave their suggestions for analysing the post data required. Another option would be to use the Firebug (https://addons.mozilla.org/firefox/addon/1843) or WebDeveloper (https://addons.mozilla.org/en-US/firefox/addon/60) plugins for Firefox. Don't even think about using IE to work out the html on a 3rd party website unless you already read, speak & dream html.
3). Saving the downloaded file is simply a matter of copying the data returned in the HttpWebRequest's ResponseStream into a FileStream. You could use something like the following code.

Finally, another option would be to use a customised WebClient. You can't use the standard WebClient as it doesn't support cookies. This way you can use the UploadValues method which allows you to send a NameValue collection of parameters, and retrieves a byte array containing the data returned. This byte array can then be written to a FileStream.

http://couldbedone.blogspot.com/2007/08/webclient-handling-cookies.html





Private Sub CopyStream(ByVal src As Stream, ByVal trg As Stream)

    Dim buff As Byte() = New Byte(4095) {}

    While True

        Dim read As Integer = src.Read(buff, 0, buff.Length)

        If read = 0 Then Exit While

        

        trg.Write(buff, 0, read)

    End While

    trg.Flush()

End Sub

Open in new window

0
 

Author Comment

by:guavamay
ID: 24387448
Hi Jake and Hairbrush: Thank You both so much for the valuable information which I didn't know those tools exists. I  need to spend some time to learn how to use them.

Hi oobayly: Thank you so much for answering my questions. I managed to solve the problem with the error messg which I was getting before. But now I am getting something different...
Here is my code....
        Dim cookies As New CookieContainer()
        ' Create the post data
         Dim postData As String = String.Format("username={0}&password={1}&src={2}&filename={3}&application={4}", username, password, src,filename, application)
        '' Create the request
        Dim req As HttpWebRequest = DirectCast(HttpWebRequest.Create("my uri"), HttpWebRequest)
        req.Method = "POST"
        req.CookieContainer = cookies
         '' Add the post data
        req.ContentLength = postData.Length
        Dim writer As New StreamWriter(req.GetRequestStream())
        'use try for exception
        Try
            writer.Write(postData)
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
        writer.Flush()
        writer.Close()
       
        '' Send the request
        Dim resp As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
        MsgBox(req.HaveResponse)
        MsgBox(resp.StatusDescription)
        MsgBox(resp.Headers.ToString)
        Dim sr As New StreamReader(resp.GetResponseStream())
        Dim sTR As String = sr.ReadToEnd()
        MsgBox(sTR.ToString)
        resp.Close()
        MsgBox("at the end")
    End Sub

I know I am getting the response back but the MsgBox(sTR.ToString) displays the following msg:
"Your current session has expired. Please re-enter your login/passwd to access"
Is anything wrong with my cookie statement? Or am I missing something?
Thanks again in advance.






0
 
LVL 14

Expert Comment

by:jake072
ID: 24389220
guavamay,

Undoubtedly there is something wrong with the cookie stream...  Again, I would suggest using Wireshark or even a FireFox plugin as oobayly suggests...  You need to duplicate the exact cookie stream, but note that obviously you can't hardcode certain things, like session id, etc...

It's a steep learning curve...  I remember it took me like 15 hours of solid non-stop work to do an ebay sniper program...

Good luck,

Jake
0
 
LVL 15

Expert Comment

by:oobayly
ID: 24394186
Try setting the POST request's ContentType. I'd forgotten to include it in the code I posted. Also, if the src, filename & application values contain characters like ampersands and spaces, you may need to url-encode them. To use the System.Web.HttpUtility, you may have to a add a reference to System.Web
req.ContentType = "application/x-www-form-urlencoded"
 

'' URL Encode all the post data values

Dim postData As String = String.Format("username={0}&password={1}&src={2}&filename={3}&application={4}", _

  System.Web.HttpUtility.UrlEncode(username), _

  System.Web.HttpUtility.UrlEncode(password), _

  System.Web.HttpUtility.UrlEncode(src), _

  System.Web.HttpUtility.UrlEncode(filename), _

  System.Web.HttpUtility.UrlEncode(application))

Open in new window

0
 

Accepted Solution

by:
guavamay earned 0 total points
ID: 24473220
Thanks oobayly.
I added the ContentType and the URL Encode all my post data values, unfortunately, it still didn't work. I am still getting the same error mesg --- "Your current session has expired. Please re-enter your login/passwd to access".  
Am I missing anything else?
I am not sure why I am still getting this error?
I will spend more time to study the html code as well as the two tools that jake072 and Hairbrush mentioned.
Thanks so much for your help.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Introduction As chip makers focus on adding processor cores over increasing clock speed, developers need to utilize the features of modern CPUs.  One of the ways we can do this is by implementing parallel algorithms in our software.   One recent…
What my article will show is if you ever had to do processing to a listbox without being able to just select all the items in it. My software Visual Studio 2008 crystal report v11 My issue was I wanted to add crystal report to a form and show…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now