We help IT Professionals succeed at work.
Get Started

Memory Error Writing Text File

azyet24
azyet24 asked
on
495 Views
Last Modified: 2012-05-10
I am exporting a large dataset to a delimited file.  Unless I run the script on a system that has a ton of memory, I get a memory error.

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

The file that is being created actually never gains in size (remains 0 KB) until the file is completly writting, which tells me that all of this work is being done in memory.  How can I do this so that while it is writing the file, the file actually gains size which reduces the memory overhead?  See code attached.
Dim Seperator As String
        Seperator = vbTab
        Dim linea As String
        Dim sw As StreamWriter
        Dim strBuilder As New System.Text.StringBuilder
        sw = File.CreateText("pathtofile.txt")
        Try            
            Dim objCmd As New OdbcCommand(Sql, objconn)
            Dim myAdapter As New OdbcDataAdapter(objCmd)
            Dim ds As New DataSet()     
            Try
                'Fill Dataset
                myAdapter.Fill(ds, "mln")
            Catch ex As Exception
                Console.WriteLine("Error filling dataset: " & ex.tostring)
            End Try
           
            For Each Dr As DataRow In ds.Tables(0).Rows
                strBuilder.Append("")

                strBuilder.Append(Dr(0) & Seperator) 
                strBuilder.Append(Dr(1) & Seperator) 
                strBuilder.Append(Dr(2) & Seperator) 
                strBuilder.Append(Dr(3) & Seperator) 
                ' Plus many more

                strBuilder.AppendLine()               
            Next
            linea = strBuilder.ToString
            sw.WriteLine(linea)
            strBuilder.Remove(0, strBuilder.Length)
        Catch ex As Exception
            Console.WriteLine(ex.ToString)    
        End Try

Open in new window

Comment
Watch Question
Top Expert 2010
Commented:
This problem has been solved!
Unlock 2 Answers and 12 Comments.
See Answers
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE