Posting multipart/form-data as if the content was an uploaded file

Posted on 2004-10-15
Medium Priority
Last Modified: 2010-05-18
I am trying to post data to a web site as if the data was an uploaded file, here is my code:

Dim boundary as String = "---xxx"
Dim postData as String
postData += boundary  & Chr(10)
postData += "Content-Disposition: form-data; name=""upfile""; filename=""c:\xyz.xyz""" & Chr(10)
postData += "Content-Type: text/plain" & Chr(10)
postData += Chr(10)
postData += "file line 1" & Chr(10)
postData += "file line 2" & Chr(10)
postData += "file line 3" & Chr(10)
postData += boundary & Chr(10)
postData += "Content-Disposition: form-data; name=""submit""" & Chr(10)
postData += Chr(10)
ostData += "Upload()" & Chr(10)
postData += boundary & "--"

Dim req As Net.HttpWebRequest = System.Net.WebRequest.Create(url)
req.Method = "POST"
req.Headers.Add("Cookie", cookie)
req.ContentType = ""multipart/form-data; boundary=" & boundary
Dim postBuff() As Byte = System.Text.Encoding.GetEncoding(1252).GetBytes(postData)
req.ContentLength = postBuff.Length
Dim reqStream As System.IO.Stream = req.GetRequestStream()
reqStream.Write(postBuff, 0, postBuff.Length)
Dim res As System.Net.HttpWebResponse = req.GetResponse()
Dim rdr As New System.Io.StreamReader(res.GetResponseStream(), System.Text.Encoding.ASCII)
Dim retVal as string = rdr.ReadToEnd()

However, this does not work!! Please help me, I am stuck.

Thanks, more points can easily be made available for a swift answer, I might have to post another question to do this as I can't see how I increase points right now
Question by:duncanlatimer
  • 5
  • 4

Expert Comment

ID: 12329389
There could be various things wrong; it's rather hard to help since you're not telling _how_ it doesn't work. At least encoding issues are possible culprits; that 1252 is somewhat suspicious (though probably harmless), as is ASCII for the file contents (quite likely less harmless). For some quick tips, try reading the following blog entry: <http://blogs.ronaco.com/Blogs/rwilson/articles/174.aspx>. It is in C#, but it could help you to squash some issues.

Let's get back to this after you've posted some details on _how_ it doesn't work.

Author Comment

ID: 12332997
Sorry for not being able to be more specific here, the web site accepts the "file" with no error message and returns the same response as I would get if I had posted a "real file", however it does not process the information in the file, so I have to suppose that in some way the file is not processible. The format of the "file" content posted is correct, so I assume the way I post it isn't. I suppose my question is does my code emulate what happens when a file is posted and are there any other tricks I should try, I will look at the encoding, thanks.

As it is not my web site I am posting to I have absolutely no idea what is happening on the server side and obviously these people are expecting a "real file" so are unlikely to be very helpful if I ask them.

Author Comment

ID: 12333585
I have tried this with CRLF and well as just LF, still didn't work, also I use the very same post code to logon to the site, so this is known to work fine, I have changed the encoding to UTF8 and this works fine for login, but this file upload stuff still doesn't work. Basically what my problem seems to be (but I may be wrong) is that the data I send is not the same as that which would be sent has I actually uploaded a file, if that makes sense?

I checked out that article and we are basically doing the same thing, with the exception that he seems to use ASCII encoding for his final boundary and UTF8 for the preceeding ones, I can't see this making any difference, but if it could let me know and bag yourself some points (assuming that it fixes may problem)

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Expert Comment

ID: 12333677
Another difference might be the fact that your content length is calculated differently. But please wait a moment; I've been toying around with the HTTP upload tonight. I'll post some code soon once I've tested it properly (during the next few hours).

Expert Comment

ID: 12333880
Without analysing all your code (I believe the central issue is the mixture with strings, encodings et al), I delved into the issue and wrote some code myself. My blog entry at <http://www.heikniemi.net/hc/archives/000150.html> contains a better description and the original source in C#. I understand you want the answer in VB.net, so I quickly translated it. Please understand that I'm not a VB programmer; I did test the code below (it both works and compiles), but it's not likely to be a good example of VB. Also, it lacks all the comments. Please refer to the C# version for some more details.

I tested the code with some rudimentary PHP server side scripts. Please try this and see if it works for you. If it doesn't, it's likely that the other end of your transmission is doing some sort of hidden validation. If that's the case, the situation needs more thinking and working; perhaps it's a hidden form field they're expecting? Anyway, the code below should upload simple files and byte arrays nicely.

Imports System
Imports System.IO
Imports System.Net
Namespace JouniHeikniemi.Tools.Http

    Public Class HttpFileUpload

        Private Sub New()
        End Sub

        Public Structure UploadSpec
            Public Contents As Byte()
            Public FileName As String
            Public FieldName As String

            Public Sub New(ByVal contents As Byte(), ByVal fileName As String, ByVal fieldName As String)
                Me.Contents = contents
                Me.FileName = fileName
                Me.FieldName = fieldName
            End Sub

            Public Sub New(ByVal pathname As String, ByVal fieldName As String)
                Dim inFile As FileStream = New FileStream(pathname, FileMode.Open)
                Dim inBytes(inFile.Length) As Byte
                inFile.Read(inBytes, 0, inBytes.Length)
                CType(inFile, IDisposable).Dispose()
                Me.Contents = inBytes
                Me.FileName = Path.GetFileName(pathname)
                Me.FieldName = fieldName
            End Sub
        End Structure

        Public Shared Function UploadFile(ByVal pathname As String, ByVal url As String, ByVal fieldName As String, ByVal cookies As CookieContainer, ByVal credentials As CredentialCache) As HttpWebResponse
            Dim us(0) As UploadSpec
            us(0) = New UploadSpec(pathname, fieldName)
            Return Upload(url, cookies, credentials, us)
        End Function

        Public Shared Function UploadByteArray(ByVal data As Byte(), ByVal fileName As String, ByVal url As String, ByVal fieldName As String, ByVal cookies As CookieContainer, ByVal credentials As CredentialCache) As HttpWebResponse
            Dim us(0) As UploadSpec
            us(0) = New UploadSpec(data, fileName, fieldName)
            Return Upload(url, cookies, credentials, us)
        End Function

        Public Shared Function Upload(ByVal url As String, ByVal cookies As CookieContainer, ByVal credentials As CredentialCache, ByRef objects As UploadSpec()) As HttpWebResponse
            Dim req As HttpWebRequest = CType(WebRequest.Create(url), HttpWebRequest)
            If Not (cookies Is Nothing) Then
                req.CookieContainer = cookies
            End If
            If Not (credentials Is Nothing) Then
                req.Credentials = credentials
            End If
            Dim boundary As String = Guid.NewGuid.ToString.Replace("-", "")
            req.ContentType = "multipart/form-data; boundary=" + boundary
            req.Method = "POST"
            Dim postData As MemoryStream = New MemoryStream
            Dim newLine As String = "" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & ""
            Dim sw As StreamWriter = New StreamWriter(postData)
            For Each us As UploadSpec In objects
                sw.Write("--" + boundary + newLine)
                sw.Write("Content-Disposition: form-data; name=""{0}""; filename=""{1}""{2}", us.FieldName, us.FileName, newLine)
                sw.Write("Content-Type: application/octet-stream" + newLine + newLine)
                postData.Write(us.Contents, 0, us.Contents.Length)
            sw.Write("--{0}--{1}", boundary, newLine)
            req.ContentLength = postData.Length
            Dim s As Stream = req.GetRequestStream
            CType(s, IDisposable).Dispose()
            Return CType(req.GetResponse, HttpWebResponse)
        End Function
    End Class
End Namespace

Expert Comment

ID: 12336843
You might need to coordinate with the owner of the URL and get their specification for the "file".

If their specification states that you should POST to their URL, then simply POST the file to them without any artificial formatting. Just POST the content as-is and use the properties of HttpWebRequest to set any HTTP header settings.

"Real file"? A receiver of a web POSTed file sees only a stream. It even has to fake receiving the filename using maybe a hidden form field for the filename or some property in the request. So maybe, you need to review their actual form and find the fields that the URL requires.

A successful POST should indicate a response code of 200. Otherwise, there's a problem.

Have fun.

Author Comment

ID: 12337431
der_jth, I have looked at your code and I am struggling to see the difference between what you are doing and what I do, I'm pretty sure this isn't an encoding issue, the length of my post buffer in bytes is exactly the same as the length of the string in chars and the buffer contains the chars in the string. I have tried changing the encoding to UTF8 and ASCII, neither worked.

etmendz, The response code I get is 200 and to the best of my knowledge the file format is specified correctly, however one problem with the site I am using is that uploading junk would also give me a 200.

This is the post data captured using HTTPLook from a genuine file upload:
POST /batch/edit_batch.php HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer: http://www.ibetx.com/batch/edit_batch.php
Accept-Language: en-gb
Content-Type: multipart/form-data; boundary=---------------------------7d41e252c04d2
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: www.ibetx.com
Content-Length: 389
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: PHPSESSID=071a688b7aa19120dfaa732b8b6c2b0e

Content-Disposition: form-data; name="upfile"; filename="C:\Documents and Settings\Duncan\Desktop\sample.txt"
Content-Type: text/plain

U,95057,716682,35744013,L,2.00,1.00,Ali Deo
Content-Disposition: form-data; name="submit"


And it is that post data that I am trying to emulate here, there is of course no actual file in my program, it is the content of a ficticious file if you like, my reference to a real file was simply trying to differenciate between what a browser would do when you uploaded a file that did exist locally and what my program did, does that clarify things (or make matters worse)?

Author Comment

ID: 12337448
I dunno if this helps explain the problem here but here is the actual code used to post currently:

Private Function httpPost(ByRef url As String, ByRef postData As String, ByRef cookie As String, ByRef contentType As String)

            Dim req As Net.HttpWebRequest = Net.WebRequest.Create(url)
            req.Method = "POST"
            req.Headers.Add("Cookie", cookie)
            req.ContentType = contentType
            Dim postBuff() As Byte = Text.Encoding.UTF8.GetBytes(postData)
            req.ContentLength = postBuff.Length
            Dim reqStream As IO.Stream = req.GetRequestStream()
            reqStream.Write(postBuff, 0, postBuff.Length)
            Dim res As System.Net.HttpWebResponse = req.GetResponse()
            Dim rdr As New IO.StreamReader(res.GetResponseStream(), Text.Encoding.UTF8)
            httpPost = rdr.ReadToEnd()
        Catch e As Net.WebException
            Console.WriteLine("Woops, web exception thrown")
            httpPost = ""
        End Try

    End Function

Accepted Solution

der_jth earned 1500 total points
ID: 12337762
Now that you did post the actual file contents (text only), it certainly doesn't look like an encoding issue. Anyway, if posting it with my code doesn't work, then the receiving end is doing some validation you're failing. It can be something very trivial, ranging from PHPSESSID being wrong (outdated) to some form field issues as both I and etmendz have referred to earlier. Your posting code itself looks pretty fine, but the construction of postData is at least equally important. But on a quick look, it seems fine too.

Try to capture an actual post made with your browser and search for the differences?

Author Comment

ID: 12357473
Hi all, the answer was that the bondary in the data should be the boundary specified in the header preceded by two dashes. This was evident in both of der-jth's examples, but I didn't notice until I went through the POSTs byte by byte as suggested by der_jth, so he gets the points, perhaps a good tip for people wishing to do this kind of stuff would be don't use dashes in your boundary, it will then be more obvious what is going on, I did wonder why the code examples I looked at we adding the dashes to the begining of the boundary, and with hind sight................ but I know for the future

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: jpaulino
XML Literals are a great way to handle XML files and the community doesn’t use it as much as it should.  An XML Literal is like a String (http://msdn.microsoft.com/en-us/library/system.string.aspx) Literal, only instead of starting and ending with w…
Introduction When many people think of the WebBrowser (http://msdn.microsoft.com/en-us/library/2te2y1x6%28v=VS.85%29.aspx) control, they immediately think of a control which allows the viewing and navigation of web pages. While this is true, it's a…
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
With just a little bit of  SQL and VBA, many doors open to cool things like synchronize a list box to display data relevant to other information on a form.  If you have never written code or looked at an SQL statement before, no problem! ...  give i…
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question