Using MSXML to transmit files through SSL

magic keyword: SOAP (transmission protocol)

Oh.... perhaps I should have said

"I'm looking for actual code examples.... no theories, magic potions, or magic phrases please ;)"

HAHAHA j/k....

-M

MEL,

"...is there an easier/more efficient way to do this using these same components? "

1. Since we can't see your existing code, how can we tell you if there is a more efficient way to do what you want to do?

2. You might be constrained in solving your problem by assumptions and requirements you've already accepted.  The "powers that be" might have to live with their decisions or change the solution constraints.

3. I might suggest compressing the 13MB file and attach it to an secure email transmission.  The receiving email system and automatically process it.

4. Is your problem the speed or something else?

5. Since you already have some transmission working, what's your REAL problem?  (may have been answered in 4)

Perhaps I didn't make it clear enough... the speed at which it transmits is the issue... that is what I was referring to when I stated:

"Well it works, but to transmit a 13MB file it takes a little over 6 minutes
to do it... and that's locally!  I don't even want to make a guess as to how long it would take to transmit
through the Internet!!"

Unfortunately e-mail isn't an option... I had already wanted to perform something similar with SMIME but it was rejected due to the vendor we are sending the file to.

Code?

Here is the section that does the actual sending from my proof of concept project:

------------------------------------------------------
    Dim objSend    As New MSXML2.XMLHTTP30
    Dim URL        As String
   
    Dim DataBeingPushed As String
    Dim FileName        As String
    Dim FileSize        As Long
    Dim PartNumber      As Long
    Dim TotalParts      As Long
    Dim AllSent         As Boolean
    Dim CountTries      As Integer
    Dim FilePart        As String
    Dim fs              As New Scripting.FileSystemObject
    Dim objFile         As File
    Dim objStream       As TextStream
   
    Dim PacketSize      As Long
    Dim StartPos        As Long
    Dim EndPos          As Long
   
    Dim StartTime       As Date
    Dim EndTime         As Date
   
    'Open the text file to be sent
    Set objFile = fs.GetFile(Trim(txtfilename.Text))
   
    'Set Packet Size
    PacketSize = 10000
   
    'Determine file size
    FileSize = objFile.Size
    Debug.Print Now & " - Length of Data File: " & FileSize
   
    'Determine number of "packets"
    TotalParts = Int(FileSize / PacketSize)
    If FileSize Mod PacketSize > 0 Then
        TotalParts = TotalParts + 1
    End If
   
    'Filename to be sent to client
    FileName = Trim(txtcatchname.Text)
   
    Debug.Print Now & " - Data set to transmit.... starting to push"
   
    AllSent = False
    CountTries = 1
    PartNumber = 0
   
    'Target machine
    URL = Trim(txttarget.Text)
   
    StartTime = Now
   
    'Open file to be transmitted
    Set objStream = objFile.OpenAsTextStream(ForReading)
   
    Do Until AllSent
   
        'Check if it is a ReTry
        If CountTries = 1 Then
            PartNumber = PartNumber + 1
           
            'Build Part to Send
            StartPos = (PartNumber - 1) * PacketSize + 1
            EndPos = StartPos + PacketSize - 1
            If EndPos > FileSize Then
                EndPos = FileSize
            End If
           
            'Read the part from the file
            FilePart = objStream.Read((EndPos - StartPos + 1))
        End If
       
       
        objSend.open "POST", URL, False
        objSend.setRequestHeader "FileName", FileName       'Send filename to target
        objSend.setRequestHeader "PartNumber", PartNumber   'Send partnumber to target
       
        Debug.Print Now & " - Sending Filename: " & FileName & " Part Number: " & Format(PartNumber, "###,##0") & " of " & Format(TotalParts, "###,##0") & " Positions: " & Format(StartPos, "###,###,##0") & "-" & Format(EndPos, "###,###,##0") & ". Try#" & CountTries
       
        On Error Resume Next
        'Send the packet
        objSend.send (FilePart)
        On Error GoTo 0
         
        If objSend.responseText = "SUCCESS" Then
            'SUCCESS text is sent back from target machine as a response
            If PartNumber = TotalParts Then
                AllSent = True
            Else
                'Reset tries counter
                CountTries = 1
            End If
        Else
            'Anything else is a failure... retry if applicable
            If CountTries > 3 Then
                Exit Do
            Else
                CountTries = CountTries + 1
            End If
        End If
       
    Loop
   
    EndTime = Now
   
    If AllSent Then
        Debug.Print Now & " - All parts for file " & FileName & " have been successfully sent. " & Now
        Debug.Print "Total Transfer time: " & Format(DateDiff("s", StartTime, EndTime), "###,###") & " seconds."
        MsgBox "Total Transfer time: " & Format(DateDiff("s", StartTime, EndTime), "###,###") & " seconds."
    Else
        Debug.Print Now & " - There was a problem with the transmission of one or more parts to this file - " & FileName
    End If
   
    'Close the file
    objStream.Close

----------------------------------------------------------

Here is the section of ASP code that is doing the receiving:

      if trim(Request.ServerVariables("HTTP_FileName")) > "" _
      and trim(Request.ServerVariables("HTTP_PartNumber")) > "" then
            
            dim objFileStream
            dim testfile
            dim strFileName
            dim intPartNumber
            dim bData
            dim strData
            dim lngLength
            
            'Retrieve Filename and Part Number
            strFileName = Request.ServerVariables("HTTP_FileName")
            intPartNumber = Request.ServerVariables("HTTP_PartNumber")
            strFileName = "D:\TestPush\" & strFileName
            
            Set objFileStream = createobject("Scripting.FileSystemObject")
            
            If intPartNumber = 1 Then
                  'Create Text File
                  Set testfile = objFileStream.CreateTextFile(strFileName,true,false)
            Else
                  'Open Text File for Append
                  Set testfile = objFileStream.OpenTextFile(strfilename,8,false)
            End If
            
            bData = Request.BinaryRead(Request.ServerVariables("HTTP_Content_Length"))
            
            'Convert to Binary
            dim i
            for i = 1 to lenb(bData)
                  strData = strData & Chr(AscB(MidB(bData,i,1)))
            next
            
            testfile.write(strData)
            
            testfile.close
            
            Set testfile = Nothing
            Set objFileStream = Nothing
            
            Response.Write "SUCCESS"
      else
            Response.Write "FAILURE"
      end if

-------------------------------------------------------

It's down and dirty code at the present... so I'll apologize to begin with  ;)

Thanks.

-M

Man... even my comments are nasty in that code...
an addendum... the section in the catch code that says it is converting *to* binary is actually converting *FROM* binary...  I'm thinking that this might be part of my slowdown since there has to be a more efficient way of doing this conversion.  But I don't think that'll solve my entire problem.

-M

Mel,

The idea behind FTP is to kick off the transfer and forget about it.  The protocol includes transmission checks to make sure the entire file gets there.  If you want transmission assurances, sent a short message (with the destination filename and its CRC32 value) to a server program that is monitoring the destination directory.  When successfully received, the server program should send a message back to your client program, confirming transmission and allowing the deletion of the file on the client's PC.  Stop doing this yourself.

Before transmission, COMPRESS THE FILE!!!!
Note: You can also encrypt the file when you compress it.

Your performance problem is I/O bound.  You must address that constraint first.

If you have a business "process", you will probably want to look at the BizTalk architecture.  Part of the BizTalk services is SOAP.

Well BizTalk is not going to help me... and FTP will not work because, it needs to run through HTTPS.  You see, we have certain guidlines and a bunch of nasty firewalls  ;-)  One of these guidlines is that we cannot have an outside vendor come into our firewall with FTP, and from our side we are only allowed to *pull* with FTP (unless we want to go through a TON of hoops and paperwork... AND *wait* for the corporate office to prioritize the work they would need to do, etc. etc. ad nauseum.  Hence the requirement for HTTPS.. it not only allows us to run through an already opened port in the firewall, but since the rules for push/pull are already established (and OPEN!), we can get things done in a much quicker timeframe.

-M

I repeat "Have you looked at SOAP?"  It uses HTTPS.

No, I'm not all that familiar with it.. do you have any  code examples (or links that I can check out)?

There is a SOAP toolkit documentation:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/soap/htm/soap_guidelines_9soj.asp

Excellent page of SOAP/XML-Messaging links:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsrvspec/html/globalxmlwebsrvinterop.asp

SOAP toolkit page:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsrvspec/html/globalxmlwebsrvinterop.asp

==================================================
please note that all of these pages come from msdn.microsoft.com.

Sorry about the delay... year end projects and all kinda took up my time... (and still are ;-)

Those links really don't help me out that much.. I'm one of those people that need to actually step through code to see how something new works.

Any other more specific links to code?

Not sure when I'll be able to get back to this.. but I'll resolve this question one way or another soon... (I HOPE!!)

-M

Hi mel.. i think i have a possible solution for you.  I have used his technology quite a bit. and it has worked for me in many diffent situations.  This is what i would do (architecturally speaking).  Build yourself an asp page that calls your vb component.  have the asp page sit at the webserver where the ssl is located.  compress the file and then convert the compressed file to a string.  load the file with xml headers around it and use the Send method of the msxml dom object to send the file to the corresponding ASP page.  In the Asp page you will catch the file in the response object move it to whatever location you like convert the file back to its non string format.. then decompress the file inside your firewall.

I think this will work.  if you need some source code let me know.

This should handle the file size.. security and the firewall issues you currently have.

That is pretty much what I am doing now... the only difference is that I am not compressing the file. The compressed file would still have to be broken up into chunks and passed along that way... so really the only thing you are offering to me (unless I'm mistaken... which I might be! ;-) is the compression.. which really doesn't address the issue.

actually mel what i am offering to you is not passing the binary byte data... you should pass a string.  its the same way a webservice would work...  it does not pass binary data across the wire.  it serializes data by passing the data as a string.  you would convert the file to a string.. not the data in the file.. but the actual file itself can be converted to a string. then a compressed string.. then you can pass it across the web.(done it hundreds of times.)

hmmm sounds like it might be what I am looking for...

could you supply some code examples?

Since dware never did get back to me on those code samples....

This question has been pending deletion.  Questions with content cannot be deleted without the next two things in the process.

1) post your intentions to delete the question and why.  Give experts a couple of days to comment and/or add what you need in the event the solution evolves and can be awarded

2) if no activity within the next few days, ask Moderator to refund and delete, or refund and PAQ, if there is value contained within this question that may help others who search our PAQ (Previously Asked Questions) database.

We have been migrating servers and had some Email outages that may account for the lack of response here, and this message may trigger a new response so that you can get what you need here.  Please let us know in your Community Support question to refund/PAQ or delete this, if noting evolves here the next few days.

Thanks,
Moondancer - EE Moderator

I think the thread is worth seeing.  The transmission duration problem could be solved a number of ways, mostly through compression.  MELeBlanc seemed to have ignored these remarks altogether.

Is this still a problem?

If not, how was it solved?

Thank you, aikimark, for returning and expounding further.
Hopefully MELeBlanc will respond and this can be finalized.
Moondancer - EE Moderator

Compression is *not* what I am looking for.
Becuase even a large compressed file will cause the same transfer problems I am encountering now.

When you pull a file through the MSXML object the transfer happens extremely quickly, If I take that same file and push it to a machine is when it takes a lot longer.  My question from the beginning was whether or not there is a way to send the file similar to the way one receives it through the XML object.  The code that I posted above was what I came up with to push the file... and basically, using the same concept, was there a better way to do it.

The process that dware was referring to sounded as close as I was going to get to my answer.. that was the reason for me asking for some code samples.

-M

If this is a "problem" for either one of you then I'll just randomly accept something as an answer.. and I'll even be kind enough to give it an 'A'.  I really don't care about the points.... I was just trying to keep the area clean.

For non-.NET applications, turn the KEEP-ALIVE flag turned off.  This should increase packet transmissions from 5/sec to 200/sec.

Sounds like an option... where is that option set?

it is an HTTP-related setting on your server.

Hmmm... sounded good...
I ran numerous tests and found that the perfomance of the application went down when  I turned off keep-alives.  On a 5MB file it actually increased the transfer time in the neighborhood of 30 seconds.

-M

is this a VB.NET application?

Nope.. the whole thing was written in VB6.

That first link is just about the exact way I am pulling data from a couple of our customers... in that direction it flies because it is actually streaming the information down to me.

That second link looks *EXTREMELY* interesting.. and might be exactly what I was looking for!

I'll look into it in more detail tomorrow when I return to work!

Thanks!

-M

Oh.. and as for bottlenecks...
I'm thinking that the main bottleneck is a combination of two things:
1. I'm not streaming the data to the target.. I'm sending it in chunks.
2. The catch script running at the target site is doing a conversion from binary.

Hence, the reason why that second link looks real good.. no conversion.. and it appears to be streaming the data!!

While you're looking, check out your Timeout-related settings.