• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 380
  • Last Modified:

Efficient method to store/load array of 2097152 numbers

I have a large lookup table which is stored in an array of 2,097,152 Integers.  My values that I am looking up could fit into a byte (0-255), but it appears VB6 stores "everything" in 16 bit data storage anyway.

I need an efficient manner to store/retrieve this array of numbers to/from a file.

Since VB6 is not a language I use frequently I first dumped them to a text file using FileSystemObject and WriteLine to write each number.  As you can expect this took a bit of time.  I'd like to do fewer and bigger disk accesses.  Maybe access chunks of the array at once instead of one element at a time?

Suggestions?
0
dwieringa
Asked:
dwieringa
  • 2
3 Solutions
 
iHadiCommented:
Hi dwieringa

The followin code declares a byte array (instead of integers) and has two methods one for reading from a specified point in a file and a specified length and the other writes to a file at a specified point. Add a command Button on a form and paste the following code:

Dim mFileNum As Long
Dim A() As Byte

Private Function WriteBytes(iData() As Byte, Optional CursorPos As Long = 1) As Boolean
On Error GoTo Er1
Put #mFileNum, CursorPos, iData
WriteBytes = True
Exit Function
Er1:
    ErrIO = False
End Function

Private Function ReadBytes(iData() As Byte, Optional CursorPos As Long = 1) As Boolean
On Error GoTo Er1
Get #mFileNum, CursorPos, iData
ReadBytes = True
Exit Function
Er1:
    ReadBytes = False
End Function

Private Sub Command1_Click()
Dim tmpFile As Long
Dim StartFrom As Long
'******************Read********************
tmpFileNum = FreeFile

Open "C:\ff.txt" For Binary As tmpFileNum
mFileNum = tmpFileNum

' Specify the location to start reading from:
StartFrom = 10
' Specify the ammount of data (bytes) to read:
ReDim A(20)
' To read the whole file to array:
' StartFrom = 1
' ReDim A(LOF(mFileNum))

' Read the Bytes
ReadBytes A

Close #mFileNum
'******************Write********************
tmpFileNum = FreeFile

Open "C:\zz.txt" For Binary As tmpFileNum
mFileNum = tmpFileNum

' Read the Bytes
WriteBytes A

Close #mFileNum

End Sub

0
 
[ fanpages ]IT Services ConsultantCommented:
You could use ActiveX Data Objects to access the flat ASCII file as a "database" (by way of a Text ISAM), read the entire file contents using a simple SQL statement, and then manipulate the resultant recordset as a whole unit, copying to a local array...

[ http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B326548 ]

===
Open Text
If a text file (for example, TestFile.txt) is located at c:\somepath, where C is the drive, and where somepath is the folder that contains TestFile.txt, as follows:

cnn.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
       "Data Source=c:\somepath;" & _
       "Extended Properties=""text;HDR=Yes;FMT=Delimited;"";"
                        

specify the file name in the SQL statement as follows:

rst.Open "Select * From TextFile.txt", cnn, , , adCmdText
                        


The Text ISAM permits you to handle multiple text file formats. You cannot define all characteristics of a text file through the connection string. For example, if you want to open a fixed-width file, or you want to use a delimiter other than the comma, you must specify all these settings in a Schema.INI file. For more information about about Schema.INI files, visit the Microsoft Developer Network (MSDN) Library at the following Web site:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcjetschema_ini_file.asp
===

(Where rst is a Recordset object, and cnn is a Database object)


See also:
"Much ADO About Text Files"
[ http://msdn.microsoft.com/library/en-us/dnclinic/html/scripting03092004.asp ]

Specifically...

"So How Do I Use ADO to Query a Text File?"
===
<snip>

On Error Resume Next
Const adOpenStatic = 3
Const adLockOptimistic = 3
Const adCmdText = &H0001

Set objConnection = CreateObject("ADODB.Connection")
Set objRecordSet = CreateObject("ADODB.Recordset")

strPathtoTextFile = "C:\Databases\"

objConnection.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
          "Data Source=" & strPathtoTextFile & ";" & _
          "Extended Properties=""text;HDR=YES;FMT=Delimited"""

objRecordset.Open "SELECT * FROM PhoneList.csv", _
          objConnection, adOpenStatic, adLockOptimistic, adCmdText

Do Until objRecordset.EOF
    Wscript.Echo "Name: " & objRecordset.Fields.Item("Name")
    Wscript.Echo "Department: " & _
        objRecordset.Fields.Item("Department")
    Wscript.Echo "Extension: " & objRecordset.Fields.Item("Extension")  
    objRecordset.MoveNext
Loop
===


BFN,

fp.
0
 
ADSaundersCommented:
It's quite simple really, opening a file for binary output prepares it for writing a string of arbitrary bytes from memory to the file, opening for binary input is the reverse. An array of bytes, integers, longs etc., is such an arbitrary string of bytes in memory.
These two subs are abstracts from a program I wrote that load and save a 'state' file for a tanker loading system.

Regards .. Alan


Private Sub SaveArray(Myarray() As Integer, MyPath As String)
Dim f As Integer
    f = FreeFile
    Open MyPath For Binary As #fo
    Put #f, , Myarray
    Close #f
End Sub

Private Sub LoadArray(Myarray() As Integer, MyPath As String)
Dim f As Integer
Dim s As Long
    f = FreeFile
    Open MyPath For Binary As #f
    s = LOF(f) / Len(f) ' Size of the file / size of an integer
    ReDim Myarray(s) ' only needed if size of array in file is not known
    Get #f, , Myarray
End Sub
0
 
ADSaundersCommented:
Sorry,
in the SaveArray sub . .
>    Open MyPath For Binary As #fo
should be
>    Open MyPath For Binary As #f
And in the LoadArray sub . .
>    Get #f, , Myarray
>End Sub
Should be
>    Get #f, , Myarray
>    Close #f
>End Sub

(comes from cutting and pasting from existing code. . .)

.. Alan
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now