Posting multipart/form-data as if the content was an uploaded file

There could be various things wrong; it's rather hard to help since you're not telling _how_ it doesn't work. At least encoding issues are possible culprits; that 1252 is somewhat suspicious (though probably harmless), as is ASCII for the file contents (quite likely less harmless). For some quick tips, try reading the following blog entry: <http://blogs.ronaco.com/Blogs/rwilson/articles/174.aspx>. It is in C#, but it could help you to squash some issues.

Let's get back to this after you've posted some details on _how_ it doesn't work.

Sorry for not being able to be more specific here, the web site accepts the "file" with no error message and returns the same response as I would get if I had posted a "real file", however it does not process the information in the file, so I have to suppose that in some way the file is not processible. The format of the "file" content posted is correct, so I assume the way I post it isn't. I suppose my question is does my code emulate what happens when a file is posted and are there any other tricks I should try, I will look at the encoding, thanks.

As it is not my web site I am posting to I have absolutely no idea what is happening on the server side and obviously these people are expecting a "real file" so are unlikely to be very helpful if I ask them.

I have tried this with CRLF and well as just LF, still didn't work, also I use the very same post code to logon to the site, so this is known to work fine, I have changed the encoding to UTF8 and this works fine for login, but this file upload stuff still doesn't work. Basically what my problem seems to be (but I may be wrong) is that the data I send is not the same as that which would be sent has I actually uploaded a file, if that makes sense?

I checked out that article and we are basically doing the same thing, with the exception that he seems to use ASCII encoding for his final boundary and UTF8 for the preceeding ones, I can't see this making any difference, but if it could let me know and bag yourself some points (assuming that it fixes may problem)

Cheers

Another difference might be the fact that your content length is calculated differently. But please wait a moment; I've been toying around with the HTTP upload tonight. I'll post some code soon once I've tested it properly (during the next few hours).

Without analysing all your code (I believe the central issue is the mixture with strings, encodings et al), I delved into the issue and wrote some code myself. My blog entry at <http://www.heikniemi.net/hc/archives/000150.html> contains a better description and the original source in C#. I understand you want the answer in VB.net, so I quickly translated it. Please understand that I'm not a VB programmer; I did test the code below (it both works and compiles), but it's not likely to be a good example of VB. Also, it lacks all the comments. Please refer to the C# version for some more details.

I tested the code with some rudimentary PHP server side scripts. Please try this and see if it works for you. If it doesn't, it's likely that the other end of your transmission is doing some sort of hidden validation. If that's the case, the situation needs more thinking and working; perhaps it's a hidden form field they're expecting? Anyway, the code below should upload simple files and byte arrays nicely.

--
Imports System
Imports System.IO
Imports System.Net
Namespace JouniHeikniemi.Tools.Http

    Public Class HttpFileUpload

        Private Sub New()
        End Sub

        Public Structure UploadSpec
            Public Contents As Byte()
            Public FileName As String
            Public FieldName As String

            Public Sub New(ByVal contents As Byte(), ByVal fileName As String, ByVal fieldName As String)
                Me.Contents = contents
                Me.FileName = fileName
                Me.FieldName = fieldName
            End Sub

            Public Sub New(ByVal pathname As String, ByVal fieldName As String)
                Dim inFile As FileStream = New FileStream(pathname, FileMode.Open)
                Dim inBytes(inFile.Length) As Byte
                inFile.Read(inBytes, 0, inBytes.Length)
                CType(inFile, IDisposable).Dispose()
                Me.Contents = inBytes
                Me.FileName = Path.GetFileName(pathname)
                Me.FieldName = fieldName
            End Sub
        End Structure

        Public Shared Function UploadFile(ByVal pathname As String, ByVal url As String, ByVal fieldName As String, ByVal cookies As CookieContainer, ByVal credentials As CredentialCache) As HttpWebResponse
            Dim us(0) As UploadSpec
            us(0) = New UploadSpec(pathname, fieldName)
            Return Upload(url, cookies, credentials, us)
        End Function

        Public Shared Function UploadByteArray(ByVal data As Byte(), ByVal fileName As String, ByVal url As String, ByVal fieldName As String, ByVal cookies As CookieContainer, ByVal credentials As CredentialCache) As HttpWebResponse
            Dim us(0) As UploadSpec
            us(0) = New UploadSpec(data, fileName, fieldName)
            Return Upload(url, cookies, credentials, us)
        End Function

        Public Shared Function Upload(ByVal url As String, ByVal cookies As CookieContainer, ByVal credentials As CredentialCache, ByRef objects As UploadSpec()) As HttpWebResponse
            Dim req As HttpWebRequest = CType(WebRequest.Create(url), HttpWebRequest)
            If Not (cookies Is Nothing) Then
                req.CookieContainer = cookies
            End If
            If Not (credentials Is Nothing) Then
                req.Credentials = credentials
            End If
            Dim boundary As String = Guid.NewGuid.ToString.Replace("-", "")
            req.ContentType = "multipart/form-data; boundary=" + boundary
            req.Method = "POST"
            Dim postData As MemoryStream = New MemoryStream
            Dim newLine As String = "" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & ""
            Dim sw As StreamWriter = New StreamWriter(postData)
            For Each us As UploadSpec In objects
                sw.Write("--" + boundary + newLine)
                sw.Write("Content-Disposition: form-data; name=""{0}""; filename=""{1}""{2}", us.FieldName, us.FileName, newLine)
                sw.Write("Content-Type: application/octet-stream" + newLine + newLine)
                sw.Flush()
                postData.Write(us.Contents, 0, us.Contents.Length)
                sw.Write(newLine)
            Next
            sw.Write("--{0}--{1}", boundary, newLine)
            sw.Flush()
            req.ContentLength = postData.Length
            Dim s As Stream = req.GetRequestStream
            postData.WriteTo(s)
            CType(s, IDisposable).Dispose()
            postData.Close()
            Return CType(req.GetResponse, HttpWebResponse)
        End Function
    End Class
End Namespace
--

You might need to coordinate with the owner of the URL and get their specification for the "file".

If their specification states that you should POST to their URL, then simply POST the file to them without any artificial formatting. Just POST the content as-is and use the properties of HttpWebRequest to set any HTTP header settings.

"Real file"? A receiver of a web POSTed file sees only a stream. It even has to fake receiving the filename using maybe a hidden form field for the filename or some property in the request. So maybe, you need to review their actual form and find the fields that the URL requires.

A successful POST should indicate a response code of 200. Otherwise, there's a problem.

Have fun.

der_jth, I have looked at your code and I am struggling to see the difference between what you are doing and what I do, I'm pretty sure this isn't an encoding issue, the length of my post buffer in bytes is exactly the same as the length of the string in chars and the buffer contains the chars in the string. I have tried changing the encoding to UTF8 and ASCII, neither worked.

etmendz, The response code I get is 200 and to the best of my knowledge the file format is specified correctly, however one problem with the site I am using is that uploading junk would also give me a 200.

This is the post data captured using HTTPLook from a genuine file upload:
POST /batch/edit_batch.php HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer: http://www.ibetx.com/batch/edit_batch.php
Accept-Language: en-gb
Content-Type: multipart/form-data; boundary=---------------------------7d41e252c04d2
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: www.ibetx.com
Content-Length: 389
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: PHPSESSID=071a688b7aa19120dfaa732b8b6c2b0e

-----------------------------7d41e252c04d2
Content-Disposition: form-data; name="upfile"; filename="C:\Documents and Settings\Duncan\Desktop\sample.txt"
Content-Type: text/plain

~~,unmatched
U,95057,716682,35744013,L,2.00,1.00,Ali Deo
-----------------------------7d41e252c04d2
Content-Disposition: form-data; name="submit"

Upload
-----------------------------7d41e252c04d2--

And it is that post data that I am trying to emulate here, there is of course no actual file in my program, it is the content of a ficticious file if you like, my reference to a real file was simply trying to differenciate between what a browser would do when you uploaded a file that did exist locally and what my program did, does that clarify things (or make matters worse)?

I dunno if this helps explain the problem here but here is the actual code used to post currently:

Private Function httpPost(ByRef url As String, ByRef postData As String, ByRef cookie As String, ByRef contentType As String)

        Try
            Dim req As Net.HttpWebRequest = Net.WebRequest.Create(url)
            req.Method = "POST"
            req.Headers.Add("Cookie", cookie)
            req.ContentType = contentType
            Dim postBuff() As Byte = Text.Encoding.UTF8.GetBytes(postData)
            req.ContentLength = postBuff.Length
            Dim reqStream As IO.Stream = req.GetRequestStream()
            reqStream.Write(postBuff, 0, postBuff.Length)
            reqStream.Flush()
            reqStream.Close()
            Dim res As System.Net.HttpWebResponse = req.GetResponse()
            Dim rdr As New IO.StreamReader(res.GetResponseStream(), Text.Encoding.UTF8)
            httpPost = rdr.ReadToEnd()
            Console.WriteLine(res.StatusCode)
            res.Close()
        Catch e As Net.WebException
            Console.WriteLine("Woops, web exception thrown")
            httpPost = ""
        End Try

    End Function

Hi all, the answer was that the bondary in the data should be the boundary specified in the header preceded by two dashes. This was evident in both of der-jth's examples, but I didn't notice until I went through the POSTs byte by byte as suggested by der_jth, so he gets the points, perhaps a good tip for people wishing to do this kind of stuff would be don't use dashes in your boundary, it will then be more obvious what is going on, I did wonder why the code examples I looked at we adding the dashes to the begining of the boundary, and with hind sight................ but I know for the future