I have a process that gets a file from an FTP server, processes it and inserts data into a Database. The one problem I have is that the data is sensitive so one of the requirements is that I cannot 'land' the file on the server anywhere, it has to be streamed from the FTP server through my process and into the Database. Basically what I have now is the FTP Get method takes a Stream as one of it's input parameters, so I pass it a MemoryStream object and then create a StreamReader object to read from the MemoryStream object returned back. This process works with smaller files but one of the files I have to process is 600+MB. So when I try to create the MemoryStream object I get a System.OutOfMemory Exception. I have posted the section of code that I'm using to stream the file.
I've seen some posts on using an UnmanagedMemoryStream object, but that didn't work either. I know the code posted is in VB.Net, but responses in VB.Net or C# would be fine. The FTP Client I'm using is a third party tool from Rebex, they have SFTP .Net libraries.
Any help on this would be appreciated.
Thanks in Advance.
'file_size = the size of the file on the FTP server
'sf = FTP Client object
'sr = StreamReader object
Using ms As New MemoryStream(Convert.ToInt32(file_size))
'Not sure if the code below is needed....
ms.Position = 0
sr = New StreamReader(ms)