troubleshooting Question

.NET FTP get large without landing file on the server

Avatar of chuck620
chuck620Flag for United States of America asked on
.NET ProgrammingC#Visual Basic.NET
6 Comments1 Solution324 ViewsLast Modified:
I have a process that gets a file from an FTP server, processes it and inserts data into a Database.  The one problem I have is that the data is sensitive so one of the requirements is that I cannot 'land' the file on the server anywhere, it has to be streamed from the FTP server through my process and into the Database.  Basically what I have now is the FTP Get method takes a Stream as one of it's input parameters, so I pass it a MemoryStream object and then create a StreamReader object to read from the MemoryStream object returned back.  This process works with smaller files but one of the files I have to process is 600+MB.  So when I try to create the MemoryStream object I get a System.OutOfMemory Exception.  I have posted the section of code that I'm using to stream the file.

I've seen some posts on using an UnmanagedMemoryStream object, but that didn't work either.  I know the code posted is in VB.Net, but responses in VB.Net or C# would be fine.  The FTP Client I'm using is a third party tool from Rebex, they have SFTP .Net libraries.

Any help on this would be appreciated.

Thanks in Advance.
Happy Holidays!!!
'file_size = the size of the file on the FTP server
'sf = FTP Client object
'sr = StreamReader object

Using ms As New MemoryStream(Convert.ToInt32(file_size))
 sf.GetFile("/dir/dir1/lrgFile.txt", ms)
 'Not sure if the code below is needed....
 ms.Position = 0
 sr = New StreamReader(ms)
End Using
Join our community to see this answer!
Unlock 1 Answer and 6 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 6 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros