How can I speed up my could and het rid of out of memeory error...?

I'm getting an out of memory error, after line 75.
Well a user of mine is.

I think the size of the data can't be handle by a string or the Inflate function.

Can someone offer some ideas how I can solve this problem?
Friend Sub ExportTopicsDetails(ByVal pstrOutputDir As String, ByVal pstrTopics() As Export.Topics, ByVal pbooAllTopics As Boolean, _
        ByRef pstrTD() As Export.TopicDetail)
        Dim lcnn1 As New ADODB.Connection()
        Dim lsnaLists As New ADODB.Recordset()
        Dim strChunk As String = ""
        Dim Offset As Long
        Dim Totalsize As Long
        Dim ChunkSize As Long
        Dim lstrSQL As String
 
        AddDebugComment("Templates.ExportTopicsDetails") 
        Dim lintDoeventsInc As Integer
 
        lcnn1 = New ADODB.Connection()
        lcnn1.Open(gstrConnectionString)
 
        Dim lintArrInc As Integer
 
        lstrSQL = "SELECT * FROM TopicDetail"
 
        gstrLastSQL = lstrSQL 
 
        lsnaLists.Open(lstrSQL, lcnn1, ADODB.CursorTypeEnum.adOpenStatic, ADODB.LockTypeEnum.adLockReadOnly)
 
        Dim lintTArrInc As Integer
        With lsnaLists
            Do While Not .EOF
                Dim lstrThisTopicCode As String = .Fields("TopicCode").Value
                For lintArrInc = 0 To pstrTopics.GetUpperBound(0)
                    If pstrTopics(lintArrInc).TrueTopicCode = lstrThisTopicCode Then
                        Dim lstrThisTemplateID As String = .Fields("TemplateID").Value & ""
                      
                        Try
                            lstrThisTemplateID = lstrThisTemplateID.ToUpper
                        Catch
                            '
                        End Try
                   
                        Dim lstrOptions As String = .Fields("options").Value
                        AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID, lstrOptions)
 
                        strChunk = ""
                       
                        Dim lstrCurrentTopicFile As String = pstrOutputDir & lstrThisTopicCode & ".top" 
 
                        ReDim Preserve lstrTempFiles(lstrTempFiles.GetUpperBound(0) + 1) 
                        lstrTempFiles(lstrTempFiles.GetUpperBound(0)) = lstrCurrentTopicFile 
 
                        Dim lbooProceed As Boolean = True
                        If File.Exists(lstrCurrentTopicFile) = True Then
                            If Microsoft.VisualBasic.FileLen(lstrCurrentTopicFile) <> 0 Then
                                lbooProceed = False
                            End If
                        End If
 
                        If lbooProceed = True Then 
 
                            Dim OpenFile As FileStream = New FileStream(lstrCurrentTopicFile, _
                                FileMode.CreateNew, FileAccess.Write, FileShare.Write)
                                
                            Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.GetEncoding("ISO-8859-1"))
 
                            Offset = 0
                            ChunkSize = 1000
                            Totalsize = lsnaLists("Detail").ActualSize
 
                            Do While Offset < Totalsize
                                strChunk = strChunk + lsnaLists("Detail").GetChunk(ChunkSize)
                                Offset = Offset + ChunkSize
                            Loop
                            
                                           
                            Dim lintByteLength As Integer = lsnaLists("ByteLen").Value
                            If lintByteLength <> -1 And strChunk <> "" Then 'added And strChunk <> "" 
                                AddDebugComment("Inflate")
                                Dim InfDef As New InflateDeflate()
                                strChunk = InfDef.Inflate(strChunk, lintByteLength)
                            End If
                      
                            'output strChunk
                            StreamWriter.WriteLine(strChunk)
                            StreamWriter.Close()
                            OpenFile.Close() 
                        End If 
                    End If
                Next lintArrInc
 
                lintDoeventsInc += 1
                If lintDoeventsInc > 10 Then
                    System.Windows.Forms.Application.DoEvents()
                    lintDoeventsInc = 0
                End If
 
                .MoveNext()
            Loop
        End With
 
        lsnaLists.Close()
        lcnn1.Close()
 
    End Sub

Open in new window

mindwarpltdAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

SameerJagdaleCommented:
I suggest you use StringBuilder with capacity (set it to max you would expect) instead of string. Have you tried this?
0
mindwarpltdAuthor Commented:
No, can you give me some lines of code where and how I'd use that?
0
SameerJagdaleCommented:
stringbuilder has an append method to add dynamic strings..further this is mutable class. you can try the type of strChunk to stringbuilder instead of string..
check the below links to get some sample code.
http://www.vbdotnetheaven.com/UploadFile/mahesh/StringBuilder04212005073651AM/StringBuilder.aspx
http://www.vbdotnetheaven.com/UploadFile/mahesh/StringsinDotnet04212005074227AM/StringsinDotnet.aspx
 
0
Exploring ASP.NET Core: Fundamentals

Learn to build web apps and services, IoT apps, and mobile backends by covering the fundamentals of ASP.NET Core and  exploring the core foundations for app libraries.

mindwarpltdAuthor Commented:
Does this look right?

I've left my comments in this time.
 Friend Sub ExportTopicsDetails(ByVal pstrOutputDir As String, ByVal pstrTopics() As Export.Topics, ByVal pbooAllTopics As Boolean, _
        ByRef pstrTD() As Export.TopicDetail)
        Dim lcnn1 As New ADODB.Connection()
        Dim lsnaLists As New ADODB.Recordset()
        'Dim strChunk As String = ""
        Dim builder As New System.Text.StringBuilder() 'JM 12/11/2008 317
 
        Dim Offset As Long
        Dim Totalsize As Long
        Dim ChunkSize As Long
        Dim lstrSQL As String
 
        AddDebugComment("Templates.ExportTopicsDetails") 'JM 26/02/2004
        Dim lintDoeventsInc As Integer 'JM 04/08/2003
 
        lcnn1 = New ADODB.Connection()
        lcnn1.Open(gstrConnectionString)
 
        Dim lintArrInc As Integer
 
        lstrSQL = "SELECT * FROM TopicDetail"
 
        gstrLastSQL = lstrSQL 'JM 05/01/2005
 
        lsnaLists.Open(lstrSQL, lcnn1, ADODB.CursorTypeEnum.adOpenStatic, ADODB.LockTypeEnum.adLockReadOnly)
 
        Dim lintTArrInc As Integer
        With lsnaLists
            Do While Not .EOF
                Dim lstrThisTopicCode As String = .Fields("TopicCode").Value
                For lintArrInc = 0 To pstrTopics.GetUpperBound(0)
                    If pstrTopics(lintArrInc).TrueTopicCode = lstrThisTopicCode Then
                        Dim lstrThisTemplateID As String = .Fields("TemplateID").Value & ""
                        '--- 'JM 25/10/2003 ---
                        Try
                            lstrThisTemplateID = lstrThisTemplateID.ToUpper
                        Catch
                            '
                        End Try
                        'not needed 'JM 04/03/2004
                        'Dim lintByteLen As Integer = .Fields("ByteLen").Value 'JM 24/02/2004
                        '--- 'JM 25/10/2003 ---
                        'AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID)
                        'JM 24/02/2004
                        'AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID)
                        'JM 25/04/2004
                        Dim lstrOptions As String = .Fields("options").Value
                        AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID, lstrOptions)
 
                        'strChunk = ""
                        builder = Nothing 'JM 12/11/2008 317
 
                        'JM 23/10/2003 - commented out
                        'Try
                        '    System.IO.File.Delete(pstrOutputDir & lstrThisTopicCode & ".top")
                        'Catch ex As Exception
                        'End Try
 
                        Dim lstrCurrentTopicFile As String = pstrOutputDir & lstrThisTopicCode & ".top" 'JM 04/08/2003
 
                        ReDim Preserve lstrTempFiles(lstrTempFiles.GetUpperBound(0) + 1) 'JM 04/08/2003
                        lstrTempFiles(lstrTempFiles.GetUpperBound(0)) = lstrCurrentTopicFile 'JM 04/08/2003
 
                        '--- 'JM 23/10/2003 ---
                        Dim lbooProceed As Boolean = True
                        If File.Exists(lstrCurrentTopicFile) = True Then
                            If Microsoft.VisualBasic.FileLen(lstrCurrentTopicFile) <> 0 Then
                                lbooProceed = False
                            End If
                        End If
                        '--- 'JM 23/10/2003 ---
 
                        If lbooProceed = True Then 'JM 23/10/2003 - cater for linked topics
 
                            Dim OpenFile As FileStream = New FileStream(lstrCurrentTopicFile, _
                                FileMode.CreateNew, FileAccess.Write, FileShare.Write) 'JM 04/08/2003
                            'Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile)
                            'JM 16/09/2003
                            Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.GetEncoding("ISO-8859-1"))
                            ''JM 04/03/2004 - catering for unicode
                            ''Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.UTF8)
                            ''Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.BigEndianUnicode)
                            'Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.Unicode)
 
                            Offset = 0
                            ChunkSize = 1000
                            Totalsize = lsnaLists("Detail").ActualSize
 
                            Do While Offset < Totalsize
                                'strChunk = strChunk + lsnaLists("Detail").GetChunk(ChunkSize)
                                builder.Append(lsnaLists("Detail").GetChunk(ChunkSize)) 'JM 12/11/2008 317
                                Offset = Offset + ChunkSize
                            Loop
                            'TODO: ChunkSize // JM 04/06/2008 156
                            '--- 'JM 04/03/2004 ---                            
                            Dim lintByteLength As Integer = lsnaLists("ByteLen").Value
                            If lintByteLength <> -1 And builder.ToString <> "" Then 'JM 12/11/2008 317'added And strChunk <> "" 'JM 05/03/2004
                                AddDebugComment("Inflate")
                                Dim InfDef As New InflateDeflate()
                                'strChunk = InfDef.Inflate(strChunk, lintByteLength)
                                builder.Append(InfDef.Inflate(builder.ToString, lintByteLength)) 'JM 12/11/2008 317
                            End If
                            '--- 'JM 04/03/2004 ---
 
                            'output strChunk
                            StreamWriter.WriteLine(builder.ToString) 'JM 12/11/2008 317'strChunk)
                            StreamWriter.Close()
                            OpenFile.Close() 'JM 04/08/2003
                        End If 'JM 23/10/2003
                    End If
                Next lintArrInc
 
                '--- 'JM 04/08/2003 ---
                lintDoeventsInc += 1
                If lintDoeventsInc > 10 Then
                    System.Windows.Forms.Application.DoEvents()
                    lintDoeventsInc = 0
                End If
                '--- 'JM 04/08/2003 ---
                .MoveNext()
            Loop
        End With
 
        lsnaLists.Close()
        lcnn1.Close()
 
    End Sub

Open in new window

0
mindwarpltdAuthor Commented:
I've tried running the code and get a null error on this line..

builder.Append(lsnaLists("Detail").GetChunk(ChunkSize))

An unhandled exception of type 'System.NullReferenceException' occurred in MyProgram.exe

Additional information: Object reference not set to an instance of an object.
0
SameerJagdaleCommented:
you are setting it to null...
'strChunk = ""
builder = Nothing 'JM 12/11/2008 317

Do you want to add a blank character (as per the comment) or want to set it to null. if you are setting it to nothing.. you need to initialize it again.. with new operator.

HTH
0
mindwarpltdAuthor Commented:
Ok that seems to work now :0

Can you improve on this for me.
I'm thinking the tostring will take time...

If lintByteLength <> -1 And builder.ToString <> "" Then
0
mindwarpltdAuthor Commented:
Also on this line do you think its the best way to handle this...?

 builder.Append(InfDef.Inflate(builder.ToString, lintByteLength))

I'm passing in the contents then re-writing it.
0
SameerJagdaleCommented:
i can think of 2 suggestions:

If lintByteLength <> -1 And builder.Length > 0 Then 'JM 12/11/2008 317'added And strChunk <> "" 'JM 05/03/2004
'strChunk = InfDef.Inflate(strChunk, lintByteLength)
'assuming you want to scrap the old value and input the new string..if not just ignore 'these lines and your code should work.
'store the original one
Dim tempString As String = builder.ToString
'reinitialize the object
builder = New System.Text.StringBuilder()
'now append it
builder.Append(InfDef.Inflate(tempString, lintByteLength)) 'JM 12/11/2008 317
End If
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mindwarpltdAuthor Commented:
Thanks,

i don't think assigning back to a string is a good idea.

However I will bare it in mind.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
.NET Programming

From novice to tech pro — start learning today.