Efficient method to store/load array of 2097152 numbers

Posted on 2006-06-01
Last Modified: 2012-06-27
I have a large lookup table which is stored in an array of 2,097,152 Integers.  My values that I am looking up could fit into a byte (0-255), but it appears VB6 stores "everything" in 16 bit data storage anyway.

I need an efficient manner to store/retrieve this array of numbers to/from a file.

Since VB6 is not a language I use frequently I first dumped them to a text file using FileSystemObject and WriteLine to write each number.  As you can expect this took a bit of time.  I'd like to do fewer and bigger disk accesses.  Maybe access chunks of the array at once instead of one element at a time?

Question by:dwieringa
    LVL 13

    Assisted Solution

    Hi dwieringa

    The followin code declares a byte array (instead of integers) and has two methods one for reading from a specified point in a file and a specified length and the other writes to a file at a specified point. Add a command Button on a form and paste the following code:

    Dim mFileNum As Long
    Dim A() As Byte

    Private Function WriteBytes(iData() As Byte, Optional CursorPos As Long = 1) As Boolean
    On Error GoTo Er1
    Put #mFileNum, CursorPos, iData
    WriteBytes = True
    Exit Function
        ErrIO = False
    End Function

    Private Function ReadBytes(iData() As Byte, Optional CursorPos As Long = 1) As Boolean
    On Error GoTo Er1
    Get #mFileNum, CursorPos, iData
    ReadBytes = True
    Exit Function
        ReadBytes = False
    End Function

    Private Sub Command1_Click()
    Dim tmpFile As Long
    Dim StartFrom As Long
    tmpFileNum = FreeFile

    Open "C:\ff.txt" For Binary As tmpFileNum
    mFileNum = tmpFileNum

    ' Specify the location to start reading from:
    StartFrom = 10
    ' Specify the ammount of data (bytes) to read:
    ReDim A(20)
    ' To read the whole file to array:
    ' StartFrom = 1
    ' ReDim A(LOF(mFileNum))

    ' Read the Bytes
    ReadBytes A

    Close #mFileNum
    tmpFileNum = FreeFile

    Open "C:\zz.txt" For Binary As tmpFileNum
    mFileNum = tmpFileNum

    ' Read the Bytes
    WriteBytes A

    Close #mFileNum

    End Sub

    LVL 35

    Expert Comment

    by:[ fanpages ]
    You could use ActiveX Data Objects to access the flat ASCII file as a "database" (by way of a Text ISAM), read the entire file contents using a simple SQL statement, and then manipulate the resultant recordset as a whole unit, copying to a local array...

    [ ]

    Open Text
    If a text file (for example, TestFile.txt) is located at c:\somepath, where C is the drive, and where somepath is the folder that contains TestFile.txt, as follows:

    cnn.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
           "Data Source=c:\somepath;" & _
           "Extended Properties=""text;HDR=Yes;FMT=Delimited;"";"

    specify the file name in the SQL statement as follows:

    rst.Open "Select * From TextFile.txt", cnn, , , adCmdText

    The Text ISAM permits you to handle multiple text file formats. You cannot define all characteristics of a text file through the connection string. For example, if you want to open a fixed-width file, or you want to use a delimiter other than the comma, you must specify all these settings in a Schema.INI file. For more information about about Schema.INI files, visit the Microsoft Developer Network (MSDN) Library at the following Web site:

    (Where rst is a Recordset object, and cnn is a Database object)

    See also:
    "Much ADO About Text Files"
    [ ]


    "So How Do I Use ADO to Query a Text File?"

    On Error Resume Next
    Const adOpenStatic = 3
    Const adLockOptimistic = 3
    Const adCmdText = &H0001

    Set objConnection = CreateObject("ADODB.Connection")
    Set objRecordSet = CreateObject("ADODB.Recordset")

    strPathtoTextFile = "C:\Databases\"

    objConnection.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
              "Data Source=" & strPathtoTextFile & ";" & _
              "Extended Properties=""text;HDR=YES;FMT=Delimited"""

    objRecordset.Open "SELECT * FROM PhoneList.csv", _
              objConnection, adOpenStatic, adLockOptimistic, adCmdText

    Do Until objRecordset.EOF
        Wscript.Echo "Name: " & objRecordset.Fields.Item("Name")
        Wscript.Echo "Department: " & _
        Wscript.Echo "Extension: " & objRecordset.Fields.Item("Extension")  


    LVL 10

    Accepted Solution

    It's quite simple really, opening a file for binary output prepares it for writing a string of arbitrary bytes from memory to the file, opening for binary input is the reverse. An array of bytes, integers, longs etc., is such an arbitrary string of bytes in memory.
    These two subs are abstracts from a program I wrote that load and save a 'state' file for a tanker loading system.

    Regards .. Alan

    Private Sub SaveArray(Myarray() As Integer, MyPath As String)
    Dim f As Integer
        f = FreeFile
        Open MyPath For Binary As #fo
        Put #f, , Myarray
        Close #f
    End Sub

    Private Sub LoadArray(Myarray() As Integer, MyPath As String)
    Dim f As Integer
    Dim s As Long
        f = FreeFile
        Open MyPath For Binary As #f
        s = LOF(f) / Len(f) ' Size of the file / size of an integer
        ReDim Myarray(s) ' only needed if size of array in file is not known
        Get #f, , Myarray
    End Sub
    LVL 10

    Assisted Solution

    in the SaveArray sub . .
    >    Open MyPath For Binary As #fo
    should be
    >    Open MyPath For Binary As #f
    And in the LoadArray sub . .
    >    Get #f, , Myarray
    >End Sub
    Should be
    >    Get #f, , Myarray
    >    Close #f
    >End Sub

    (comes from cutting and pasting from existing code. . .)

    .. Alan

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Do You Know the 4 Main Threat Actor Types?

    Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

    Have you ever wanted to restrict the users input in a textbox to numbers, and while doing that make sure that they can't 'cheat' by pasting in non-numeric text? Of course you can do that with code you write yourself but it's tedious and error-prone …
    If you have ever used Microsoft Word then you know that it has a good spell checker and it may have occurred to you that the ability to check spelling might be a nice piece of functionality to add to certain applications of yours. Well the code that…
    As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
    Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…

    761 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    8 Experts available now in Live!

    Get 1:1 Help Now