.NET FTP get large without landing file on the server

Posted on 2009-12-24
Last Modified: 2013-11-07
I have a process that gets a file from an FTP server, processes it and inserts data into a Database.  The one problem I have is that the data is sensitive so one of the requirements is that I cannot 'land' the file on the server anywhere, it has to be streamed from the FTP server through my process and into the Database.  Basically what I have now is the FTP Get method takes a Stream as one of it's input parameters, so I pass it a MemoryStream object and then create a StreamReader object to read from the MemoryStream object returned back.  This process works with smaller files but one of the files I have to process is 600+MB.  So when I try to create the MemoryStream object I get a System.OutOfMemory Exception.  I have posted the section of code that I'm using to stream the file.

I've seen some posts on using an UnmanagedMemoryStream object, but that didn't work either.  I know the code posted is in VB.Net, but responses in VB.Net or C# would be fine.  The FTP Client I'm using is a third party tool from Rebex, they have SFTP .Net libraries.

Any help on this would be appreciated.

Thanks in Advance.
Happy Holidays!!!
'file_size = the size of the file on the FTP server

'sf = FTP Client object

'sr = StreamReader object

Using ms As New MemoryStream(Convert.ToInt32(file_size))

 sf.GetFile("/dir/dir1/lrgFile.txt", ms)

 'Not sure if the code below is needed....

 ms.Position = 0

 sr = New StreamReader(ms)

End Using

Open in new window

Question by:chuck620
    LVL 57

    Expert Comment

    I am assuming by "landing" on the server you mean that it can be in a file.

    Must you "receive" the whole file before processing it?  Or can you process chunks of the file as you receive it?

    Author Comment

    Yes, exactly, it cannot not be in a physical file form on the server.

    I do not have receive the whole file first before processing it, I can do it in chunks.
    LVL 57

    Accepted Solution

    O.K., I've never done .NET programming, in fact no Windows programming.  My experience is OS level programming on IBM mainframes.  So this is logically how you might be able to accomplish this.

    Can you set it up so that the memory stream is like an array of a specific size? That is multiple buffers of say 1-2 MB per buffer.
    Once buffer#1 is full, buffer#2 is allocated and the ftp process starts writing to it and you start processing the records in buffer#1.  Once you have processed all of the records in buffer#1, you free it.

    Author Comment

    I'm not sure how I could do that.  Plus I'm reading each line and parsing out the data I need, if I use the method you mentioned how can I guarantee that I'll get a complete line of data when I'm processing it.

    Author Closing Comment

    After reading your comment I went back to the third party SFTP documentation and saw that there was a way to grab a partial file.  I was able to grab the file in chunks as to not get the OutOfMemoryException.  I also built in logic to determine if I had a complete line of data and how to handle it if I did not.
    LVL 57

    Expert Comment

    You can't guarantee that you will get a complete line within a single buffer, this is something that you would have to track by checking for EOR.  

    If the last bit in the buffer does not have a EOR, you would need to move that to another (much smaller) temporary buffer, free the "big buffer", and then combine the first part of the partial record with the last part of the partial record, which would be the 1st piece of the next buffer.

    You will have to track where the end of the partial record is so you know where to append the rest of if.


    Featured Post

    Why You Should Analyze Threat Actor TTPs

    After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

    Join & Write a Comment

    Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
    If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
    Need more eyes on your posted question? Go ahead and follow the quick steps in this video to learn how to Request Attention to your question. *Log into your Experts Exchange account *Find the question you want to Request Attention for *Go to the e…
    This video is in connection to the article "The case of a missing mobile phone (". It will help one to understand clearly the steps to track a lost android phone.

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now