[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

How can I speed up my could and het rid of out of memeory error...?

Posted on 2008-11-12
10
Medium Priority
?
301 Views
Last Modified: 2013-11-07
I'm getting an out of memory error, after line 75.
Well a user of mine is.

I think the size of the data can't be handle by a string or the Inflate function.

Can someone offer some ideas how I can solve this problem?
Friend Sub ExportTopicsDetails(ByVal pstrOutputDir As String, ByVal pstrTopics() As Export.Topics, ByVal pbooAllTopics As Boolean, _
        ByRef pstrTD() As Export.TopicDetail)
        Dim lcnn1 As New ADODB.Connection()
        Dim lsnaLists As New ADODB.Recordset()
        Dim strChunk As String = ""
        Dim Offset As Long
        Dim Totalsize As Long
        Dim ChunkSize As Long
        Dim lstrSQL As String
 
        AddDebugComment("Templates.ExportTopicsDetails") 
        Dim lintDoeventsInc As Integer
 
        lcnn1 = New ADODB.Connection()
        lcnn1.Open(gstrConnectionString)
 
        Dim lintArrInc As Integer
 
        lstrSQL = "SELECT * FROM TopicDetail"
 
        gstrLastSQL = lstrSQL 
 
        lsnaLists.Open(lstrSQL, lcnn1, ADODB.CursorTypeEnum.adOpenStatic, ADODB.LockTypeEnum.adLockReadOnly)
 
        Dim lintTArrInc As Integer
        With lsnaLists
            Do While Not .EOF
                Dim lstrThisTopicCode As String = .Fields("TopicCode").Value
                For lintArrInc = 0 To pstrTopics.GetUpperBound(0)
                    If pstrTopics(lintArrInc).TrueTopicCode = lstrThisTopicCode Then
                        Dim lstrThisTemplateID As String = .Fields("TemplateID").Value & ""
                      
                        Try
                            lstrThisTemplateID = lstrThisTemplateID.ToUpper
                        Catch
                            '
                        End Try
                   
                        Dim lstrOptions As String = .Fields("options").Value
                        AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID, lstrOptions)
 
                        strChunk = ""
                       
                        Dim lstrCurrentTopicFile As String = pstrOutputDir & lstrThisTopicCode & ".top" 
 
                        ReDim Preserve lstrTempFiles(lstrTempFiles.GetUpperBound(0) + 1) 
                        lstrTempFiles(lstrTempFiles.GetUpperBound(0)) = lstrCurrentTopicFile 
 
                        Dim lbooProceed As Boolean = True
                        If File.Exists(lstrCurrentTopicFile) = True Then
                            If Microsoft.VisualBasic.FileLen(lstrCurrentTopicFile) <> 0 Then
                                lbooProceed = False
                            End If
                        End If
 
                        If lbooProceed = True Then 
 
                            Dim OpenFile As FileStream = New FileStream(lstrCurrentTopicFile, _
                                FileMode.CreateNew, FileAccess.Write, FileShare.Write)
                                
                            Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.GetEncoding("ISO-8859-1"))
 
                            Offset = 0
                            ChunkSize = 1000
                            Totalsize = lsnaLists("Detail").ActualSize
 
                            Do While Offset < Totalsize
                                strChunk = strChunk + lsnaLists("Detail").GetChunk(ChunkSize)
                                Offset = Offset + ChunkSize
                            Loop
                            
                                           
                            Dim lintByteLength As Integer = lsnaLists("ByteLen").Value
                            If lintByteLength <> -1 And strChunk <> "" Then 'added And strChunk <> "" 
                                AddDebugComment("Inflate")
                                Dim InfDef As New InflateDeflate()
                                strChunk = InfDef.Inflate(strChunk, lintByteLength)
                            End If
                      
                            'output strChunk
                            StreamWriter.WriteLine(strChunk)
                            StreamWriter.Close()
                            OpenFile.Close() 
                        End If 
                    End If
                Next lintArrInc
 
                lintDoeventsInc += 1
                If lintDoeventsInc > 10 Then
                    System.Windows.Forms.Application.DoEvents()
                    lintDoeventsInc = 0
                End If
 
                .MoveNext()
            Loop
        End With
 
        lsnaLists.Close()
        lcnn1.Close()
 
    End Sub

Open in new window

0
Comment
Question by:mindwarpltd
  • 6
  • 4
10 Comments
 
LVL 13

Expert Comment

by:SameerJagdale
ID: 22938264
I suggest you use StringBuilder with capacity (set it to max you would expect) instead of string. Have you tried this?
0
 

Author Comment

by:mindwarpltd
ID: 22938385
No, can you give me some lines of code where and how I'd use that?
0
 
LVL 13

Expert Comment

by:SameerJagdale
ID: 22939763
stringbuilder has an append method to add dynamic strings..further this is mutable class. you can try the type of strChunk to stringbuilder instead of string..
check the below links to get some sample code.
http://www.vbdotnetheaven.com/UploadFile/mahesh/StringBuilder04212005073651AM/StringBuilder.aspx
http://www.vbdotnetheaven.com/UploadFile/mahesh/StringsinDotnet04212005074227AM/StringsinDotnet.aspx
 
0
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

 

Author Comment

by:mindwarpltd
ID: 22939952
Does this look right?

I've left my comments in this time.
 Friend Sub ExportTopicsDetails(ByVal pstrOutputDir As String, ByVal pstrTopics() As Export.Topics, ByVal pbooAllTopics As Boolean, _
        ByRef pstrTD() As Export.TopicDetail)
        Dim lcnn1 As New ADODB.Connection()
        Dim lsnaLists As New ADODB.Recordset()
        'Dim strChunk As String = ""
        Dim builder As New System.Text.StringBuilder() 'JM 12/11/2008 317
 
        Dim Offset As Long
        Dim Totalsize As Long
        Dim ChunkSize As Long
        Dim lstrSQL As String
 
        AddDebugComment("Templates.ExportTopicsDetails") 'JM 26/02/2004
        Dim lintDoeventsInc As Integer 'JM 04/08/2003
 
        lcnn1 = New ADODB.Connection()
        lcnn1.Open(gstrConnectionString)
 
        Dim lintArrInc As Integer
 
        lstrSQL = "SELECT * FROM TopicDetail"
 
        gstrLastSQL = lstrSQL 'JM 05/01/2005
 
        lsnaLists.Open(lstrSQL, lcnn1, ADODB.CursorTypeEnum.adOpenStatic, ADODB.LockTypeEnum.adLockReadOnly)
 
        Dim lintTArrInc As Integer
        With lsnaLists
            Do While Not .EOF
                Dim lstrThisTopicCode As String = .Fields("TopicCode").Value
                For lintArrInc = 0 To pstrTopics.GetUpperBound(0)
                    If pstrTopics(lintArrInc).TrueTopicCode = lstrThisTopicCode Then
                        Dim lstrThisTemplateID As String = .Fields("TemplateID").Value & ""
                        '--- 'JM 25/10/2003 ---
                        Try
                            lstrThisTemplateID = lstrThisTemplateID.ToUpper
                        Catch
                            '
                        End Try
                        'not needed 'JM 04/03/2004
                        'Dim lintByteLen As Integer = .Fields("ByteLen").Value 'JM 24/02/2004
                        '--- 'JM 25/10/2003 ---
                        'AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID)
                        'JM 24/02/2004
                        'AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID)
                        'JM 25/04/2004
                        Dim lstrOptions As String = .Fields("options").Value
                        AddTopicDetailArrayItem(pstrTD, ReturnTrueTopicStr(lstrThisTopicCode), lstrThisTemplateID, lstrOptions)
 
                        'strChunk = ""
                        builder = Nothing 'JM 12/11/2008 317
 
                        'JM 23/10/2003 - commented out
                        'Try
                        '    System.IO.File.Delete(pstrOutputDir & lstrThisTopicCode & ".top")
                        'Catch ex As Exception
                        'End Try
 
                        Dim lstrCurrentTopicFile As String = pstrOutputDir & lstrThisTopicCode & ".top" 'JM 04/08/2003
 
                        ReDim Preserve lstrTempFiles(lstrTempFiles.GetUpperBound(0) + 1) 'JM 04/08/2003
                        lstrTempFiles(lstrTempFiles.GetUpperBound(0)) = lstrCurrentTopicFile 'JM 04/08/2003
 
                        '--- 'JM 23/10/2003 ---
                        Dim lbooProceed As Boolean = True
                        If File.Exists(lstrCurrentTopicFile) = True Then
                            If Microsoft.VisualBasic.FileLen(lstrCurrentTopicFile) <> 0 Then
                                lbooProceed = False
                            End If
                        End If
                        '--- 'JM 23/10/2003 ---
 
                        If lbooProceed = True Then 'JM 23/10/2003 - cater for linked topics
 
                            Dim OpenFile As FileStream = New FileStream(lstrCurrentTopicFile, _
                                FileMode.CreateNew, FileAccess.Write, FileShare.Write) 'JM 04/08/2003
                            'Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile)
                            'JM 16/09/2003
                            Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.GetEncoding("ISO-8859-1"))
                            ''JM 04/03/2004 - catering for unicode
                            ''Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.UTF8)
                            ''Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.BigEndianUnicode)
                            'Dim StreamWriter As StreamWriter = New StreamWriter(OpenFile, System.Text.Encoding.Unicode)
 
                            Offset = 0
                            ChunkSize = 1000
                            Totalsize = lsnaLists("Detail").ActualSize
 
                            Do While Offset < Totalsize
                                'strChunk = strChunk + lsnaLists("Detail").GetChunk(ChunkSize)
                                builder.Append(lsnaLists("Detail").GetChunk(ChunkSize)) 'JM 12/11/2008 317
                                Offset = Offset + ChunkSize
                            Loop
                            'TODO: ChunkSize // JM 04/06/2008 156
                            '--- 'JM 04/03/2004 ---                            
                            Dim lintByteLength As Integer = lsnaLists("ByteLen").Value
                            If lintByteLength <> -1 And builder.ToString <> "" Then 'JM 12/11/2008 317'added And strChunk <> "" 'JM 05/03/2004
                                AddDebugComment("Inflate")
                                Dim InfDef As New InflateDeflate()
                                'strChunk = InfDef.Inflate(strChunk, lintByteLength)
                                builder.Append(InfDef.Inflate(builder.ToString, lintByteLength)) 'JM 12/11/2008 317
                            End If
                            '--- 'JM 04/03/2004 ---
 
                            'output strChunk
                            StreamWriter.WriteLine(builder.ToString) 'JM 12/11/2008 317'strChunk)
                            StreamWriter.Close()
                            OpenFile.Close() 'JM 04/08/2003
                        End If 'JM 23/10/2003
                    End If
                Next lintArrInc
 
                '--- 'JM 04/08/2003 ---
                lintDoeventsInc += 1
                If lintDoeventsInc > 10 Then
                    System.Windows.Forms.Application.DoEvents()
                    lintDoeventsInc = 0
                End If
                '--- 'JM 04/08/2003 ---
                .MoveNext()
            Loop
        End With
 
        lsnaLists.Close()
        lcnn1.Close()
 
    End Sub

Open in new window

0
 

Author Comment

by:mindwarpltd
ID: 22940000
I've tried running the code and get a null error on this line..

builder.Append(lsnaLists("Detail").GetChunk(ChunkSize))

An unhandled exception of type 'System.NullReferenceException' occurred in MyProgram.exe

Additional information: Object reference not set to an instance of an object.
0
 
LVL 13

Expert Comment

by:SameerJagdale
ID: 22947354
you are setting it to null...
'strChunk = ""
builder = Nothing 'JM 12/11/2008 317

Do you want to add a blank character (as per the comment) or want to set it to null. if you are setting it to nothing.. you need to initialize it again.. with new operator.

HTH
0
 

Author Comment

by:mindwarpltd
ID: 22948329
Ok that seems to work now :0

Can you improve on this for me.
I'm thinking the tostring will take time...

If lintByteLength <> -1 And builder.ToString <> "" Then
0
 

Author Comment

by:mindwarpltd
ID: 22948340
Also on this line do you think its the best way to handle this...?

 builder.Append(InfDef.Inflate(builder.ToString, lintByteLength))

I'm passing in the contents then re-writing it.
0
 
LVL 13

Accepted Solution

by:
SameerJagdale earned 2000 total points
ID: 22949718
i can think of 2 suggestions:

If lintByteLength <> -1 And builder.Length > 0 Then 'JM 12/11/2008 317'added And strChunk <> "" 'JM 05/03/2004
'strChunk = InfDef.Inflate(strChunk, lintByteLength)
'assuming you want to scrap the old value and input the new string..if not just ignore 'these lines and your code should work.
'store the original one
Dim tempString As String = builder.ToString
'reinitialize the object
builder = New System.Text.StringBuilder()
'now append it
builder.Append(InfDef.Inflate(tempString, lintByteLength)) 'JM 12/11/2008 317
End If
0
 

Author Comment

by:mindwarpltd
ID: 22949854
Thanks,

i don't think assigning back to a string is a good idea.

However I will bare it in mind.
0

Featured Post

Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Entering time in Microsoft Access can be difficult. An input mask often bothers users more than helping them and won't catch all typing errors. This article shows how to create a textbox for 24-hour time input with full validation politely catching …
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
Screencast - Getting to Know the Pipeline

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question