• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 284
  • Last Modified:

.NET FTP get large without landing file on the server

I have a process that gets a file from an FTP server, processes it and inserts data into a Database.  The one problem I have is that the data is sensitive so one of the requirements is that I cannot 'land' the file on the server anywhere, it has to be streamed from the FTP server through my process and into the Database.  Basically what I have now is the FTP Get method takes a Stream as one of it's input parameters, so I pass it a MemoryStream object and then create a StreamReader object to read from the MemoryStream object returned back.  This process works with smaller files but one of the files I have to process is 600+MB.  So when I try to create the MemoryStream object I get a System.OutOfMemory Exception.  I have posted the section of code that I'm using to stream the file.

I've seen some posts on using an UnmanagedMemoryStream object, but that didn't work either.  I know the code posted is in VB.Net, but responses in VB.Net or C# would be fine.  The FTP Client I'm using is a third party tool from Rebex, they have SFTP .Net libraries.

Any help on this would be appreciated.

Thanks in Advance.
and
Happy Holidays!!!
'file_size = the size of the file on the FTP server
'sf = FTP Client object
'sr = StreamReader object

Using ms As New MemoryStream(Convert.ToInt32(file_size))
 sf.GetFile("/dir/dir1/lrgFile.txt", ms)
 'Not sure if the code below is needed....
 ms.Position = 0
 sr = New StreamReader(ms)
End Using

Open in new window

0
chuck620
Asked:
chuck620
  • 3
  • 3
1 Solution
 
giltjrCommented:
I am assuming by "landing" on the server you mean that it can be in a file.

Must you "receive" the whole file before processing it?  Or can you process chunks of the file as you receive it?
0
 
chuck620Author Commented:
Yes, exactly, it cannot not be in a physical file form on the server.

I do not have receive the whole file first before processing it, I can do it in chunks.
0
 
giltjrCommented:
O.K., I've never done .NET programming, in fact no Windows programming.  My experience is OS level programming on IBM mainframes.  So this is logically how you might be able to accomplish this.

Can you set it up so that the memory stream is like an array of a specific size? That is multiple buffers of say 1-2 MB per buffer.
Once buffer#1 is full, buffer#2 is allocated and the ftp process starts writing to it and you start processing the records in buffer#1.  Once you have processed all of the records in buffer#1, you free it.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
chuck620Author Commented:
I'm not sure how I could do that.  Plus I'm reading each line and parsing out the data I need, if I use the method you mentioned how can I guarantee that I'll get a complete line of data when I'm processing it.
0
 
chuck620Author Commented:
After reading your comment I went back to the third party SFTP documentation and saw that there was a way to grab a partial file.  I was able to grab the file in chunks as to not get the OutOfMemoryException.  I also built in logic to determine if I had a complete line of data and how to handle it if I did not.
0
 
giltjrCommented:
You can't guarantee that you will get a complete line within a single buffer, this is something that you would have to track by checking for EOR.  

If the last bit in the buffer does not have a EOR, you would need to move that to another (much smaller) temporary buffer, free the "big buffer", and then combine the first part of the partial record with the last part of the partial record, which would be the 1st piece of the next buffer.

You will have to track where the end of the partial record is so you know where to append the rest of if.

0

Featured Post

[Webinar On Demand] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now