freejaso
asked on
Reading a Unix File
I am trying to read an extremly large Unix file with the following code:
Open txtFile.Text For Input As #1
Length = LOF(1)
tmpData = Input(Length, #1)
Close #1
I am getting an run-time error because the string variable tmpData is not large enough to store the data. I cannot use the normal line input functions that vb provides because at the end of each line is just a line feed and NOT a ctr/line feed like a dos file. Also, I cannot open the file for random access because each line has a different length. Do you have any ideas how to read this file?
Open txtFile.Text For Input As #1
Length = LOF(1)
tmpData = Input(Length, #1)
Close #1
I am getting an run-time error because the string variable tmpData is not large enough to store the data. I cannot use the normal line input functions that vb provides because at the end of each line is just a line feed and NOT a ctr/line feed like a dos file. Also, I cannot open the file for random access because each line has a different length. Do you have any ideas how to read this file?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
How large is extremely large, by the way?
How big is the file, the following reads in a 13mg unix file
Dim tmpData As String
Dim Length As Long
Open "C:\temp\export.new" For Input As #1
Length = LOF(1)
tmpData = Input(Length, #1)
Close #1
Dim tmpData As String
Dim Length As Long
Open "C:\temp\export.new" For Input As #1
Length = LOF(1)
tmpData = Input(Length, #1)
Close #1
ASKER
The file is much larger than 13mg, around 46mg and bigger.
>Do you have any ideas how to read this file?
Use binary file methods to read it in chunks and split it into lines using split function. Discard your end chunk unless it ends in LF (chr(10) and start the next chunk at the first byte after the last LF. I don't have any time right now to write the sample code...
Disregard the code I posted above, I don't think it will help if the file is so big it's giving you memory problems.
Use binary file methods to read it in chunks and split it into lines using split function. Discard your end chunk unless it ends in LF (chr(10) and start the next chunk at the first byte after the last LF. I don't have any time right now to write the sample code...
Disregard the code I posted above, I don't think it will help if the file is so big it's giving you memory problems.
ASKER
The file is much larger than 13mg, around 46mg and bigger.
actually, you may discover that the end-of-line character in the UNIX file is in fact an ASCII 0, and not a vbLF character.
Arthur Wood
Arthur Wood
ASKER
The file is much larger than 13mg, around 46mg and bigger.
ASKER
The the following code PaulHews mentioned works great but when I try to split it I get a run-time error.
Open txtFile.Text For Binary As #1
tmpData = Space(LOF(1))
Get #1, , tmpData
Close #1
Data() = Split(tmpData, vbLf) 'I am getting an error here!
Open txtFile.Text For Binary As #1
tmpData = Space(LOF(1))
Get #1, , tmpData
Close #1
Data() = Split(tmpData, vbLf) 'I am getting an error here!
Try this
Data = Split(tmpData, vbLf)
Data = Split(tmpData, vbLf)
What is the error?
ASKER
The error is:
Run-Time Error '7';
Out of memory
Run-Time Error '7';
Out of memory
Right, so read in chunks instead as I suggested above.
ASKER
I am not real sure how to do that. I have never worked with binary files. Could you get me on the correct path?
This is likely full of bugs because I don't have time to test it and debug it and I wrote it in a hurry, but it might give you an idea:
Private Sub Command1_Click()
Dim hFile As Integer
Dim strChunk As String
Dim vData As Variant
Const ChunkSize = 65536
Dim lngPointer As Long
Dim lngChunkPointer As Long
Dim lngFileSize As Long
Dim i As Long
hFile = FreeFile
Open "c:\temp.txt" For Binary As #hFile
lngFileSize = LOF(hFile)
lngPointer = 1
Do While Seek(hFile) < lngFileSize
If lngPointer + ChunkSize > lngFileSize Then
strChunk = Space(lngFileSize - lngPointer)
Else
strChunk = Space(ChunkSize)
End If
'Get a chunk of data from position lngPointer
Get #hFile, lngPointer, strChunk
'Find last LF in the string
lngChunkPointer = InStrRev(strChunk, vbLf)
'Reposition the pointer
lngPointer = lngPointer + lngChunkPointer
If lngChunkPointer > 0 Then
strChunk = Left$(strChunk, lngChunkPointer)
vData = Split(strChunk, vbLf)
For i = LBound(vData) To UBound(vData)
'process a line of data
Call sProcess(vData(i))
Next i
Else
strChunk = RTrim(strChunk)
Call sProcess(strChunk)
End If
Loop
End Sub
Private Sub Command1_Click()
Dim hFile As Integer
Dim strChunk As String
Dim vData As Variant
Const ChunkSize = 65536
Dim lngPointer As Long
Dim lngChunkPointer As Long
Dim lngFileSize As Long
Dim i As Long
hFile = FreeFile
Open "c:\temp.txt" For Binary As #hFile
lngFileSize = LOF(hFile)
lngPointer = 1
Do While Seek(hFile) < lngFileSize
If lngPointer + ChunkSize > lngFileSize Then
strChunk = Space(lngFileSize - lngPointer)
Else
strChunk = Space(ChunkSize)
End If
'Get a chunk of data from position lngPointer
Get #hFile, lngPointer, strChunk
'Find last LF in the string
lngChunkPointer = InStrRev(strChunk, vbLf)
'Reposition the pointer
lngPointer = lngPointer + lngChunkPointer
If lngChunkPointer > 0 Then
strChunk = Left$(strChunk, lngChunkPointer)
vData = Split(strChunk, vbLf)
For i = LBound(vData) To UBound(vData)
'process a line of data
Call sProcess(vData(i))
Next i
Else
strChunk = RTrim(strChunk)
Call sProcess(strChunk)
End If
Loop
End Sub
ASKER
Thanks for you help!
Anytime, I just hope it works out for you.
Open txtFile.txt For Input As #1
Open nFile.txt For Output As #2
Do Until EOF(1)
Line Input #1, strDat
strRepl = Replace(strDat, vbLf, vbCrLf)
Print #2, strRepl
Loop
Close #1
Close #2