Go Premium for a chance to win a PS4. Enter to Win


URGENT - Dealing with Files in VB

Posted on 2001-08-15
Medium Priority
Last Modified: 2010-05-19
I have files of size like 4 to 8 M.B. It has data seperated with a perticular delimiter. I am making an application which will read this file.. parse the data using delimiters & then save it as different files. i am using Scriptiing.filesystemobject to read the file & then parse it with the delimeter & then making different files & saving it. Thhe Code is as follows

Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)

After i get all the data of the file in arr i use split function using the delimiter & then make new files. Everything works fien if the file size is small like 1 M.B. .. but when i use file up to 4 M.B. ts.ReadAll just takes too much time which i cannot afford...Please suggest other alternative ..
thanking u in advance.
Question by:N_D
  • 4
  • 2
  • 2
  • +6

Expert Comment

ID: 6389459
Instead of using the FSO, you can use the VB native File I/O routines, or you can use some Win API functions to read/write the files.

In addition, if you display the status onscreen, instead of updating the status display for each and every record, do so only after every 10 or 100 records.

Reading the fine in chunks (instead of line-by-line) will also improve performance.

Others may have additional suggestions.

-Dennis Borg
President PAFSO
LVL 10

Expert Comment

ID: 6389548
Dim Filenr As Integer
Dim ByteArray() As Byte
Filenr = FreeFile
Open SourceFile For Binary As #Filenr
ReDim ByteArray(0 To LOF(Filenr) - 1)
Get #Filenr, , ByteArray()
Close #Filenr

You can try this, especially for large files. I got it from the web somewhere, claimed the fastest way to read
large files (I havn't test it)


Author Comment

ID: 6389562
i need to know the I/O Routines Please suggest me a site from where i can find more info
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!


Expert Comment

ID: 6389685
try the memory mapping method

Accepted Solution

DennisBorg earned 303 total points
ID: 6389859
N D:

>i need to know the I/O Routines Please suggest me a site from where i can find more info

Your VB Documentation ... look up:

   Line Input

-Dennis Borg

Expert Comment

ID: 6389861

instead of reading the whole file into memory, why don't you read it line by line, or block by block...

read n-bytes, scan for delimiter, if found, process it and read next block , so on...

it would help if you give us more details on your problem. what is the file type, and so on.

Expert Comment

ID: 6389878
Since the objective is efficience, you would not want to read line-by-line. You'd want to read block by block, unless the file was relatively small, then I'd read the whoe file in at once. However, the size of the files in this case would warrant a block-by-block reading of the files.

Expert Comment

ID: 6390027
Hi, N_D.

I had this same issue while using the FileSystemObject in another language.  You're right about how an increase in the file size can slow down the textstream's readall method.

In my own application, I could not predict what the file size would be.  So I experimented with block sizes and found 65536 to be about right for the wide range of file sizes that I tested.

<--------------- Sample Code Begins ----------------->
Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)

arr = ""
do while not ts.AtEndOfStream
  arr = arr & ts.Read( 65536 )
<--------------- Sample Code Ends ----------------->

Just in case ...
If you ever need to read in a binary file, use SoftArtisans' FileManager control instead.  I've had problems with the textstream object reading a lot of chr(0)'s.

(download the evaluation - it's uncrippled freeware.)

Bye. -e2

Expert Comment

ID: 6390403
or, you can do this:

Dim Block() as Byte, FileHandle as Long, AllData As String, FileLength as Long
FileHandle = FreeFile
ReDim Block(0 to 65535)
Open file_parse For Binary as #FileHandle
FileLength = LOF(FileHandle)
Do Until CurrentPosition >= FileLength
If (CurrentPosition + 65536) >= FileLength Then ReDim Block(0 to FileLength - CurrentPosition)
Get #FileHandle, CurrentPosition, Block
CurrentPosition = CurrentPosition + UBound(Block) + 1
AllData = AllData + StrConv(Block, vbUnicode)

Try that, I'm not sure if it will work as I coded it off the top of my head (I'm at work without access to VB).

The StrConv may or may not be needed, it depends.
LVL 14

Expert Comment

ID: 6390648
N_D.. please post the code you are presently using - as it is easier for us to show you what to do that way.

By the way.. when it somes to speed.. Binary file access will absolutely blow the doors off of ANY File System Object and/or Line Input means.. <smile>.

Expert Comment

by:Valliappan AN
ID: 6391378

Expert Comment

ID: 6392883
Hi, wsh2.

>Binary file access will absolutely blow the doors off of
>ANY File System Object and/or Line Input means

Just to let you know...

I have a sucky Novell Client for NT.  FileSystemObject's TextStream can open files an order of magnitude faster than binary file access.  The textstream almost never takes longer than 2.5 sec on any size file while binary file access (and any other file access technique) takes at least 30 sec even on the smallest files.

#Anecdotal Evidence
Even on PCs with an updated Novell Client (non-NT), FSO is still a little faster when it comes to accessing files ...

1. that the Novell server has compressed due to non-activity
2. that are stored in large directories (ie. >20,000 files)
3. where both conditions exist.

Note: I wouldn't have even considered FileSystemObject's TextStream (or SoftArtisan-FileManager's BinaryStream) if network access to these files weren't so slow.

Bye. -e2

Author Comment

ID: 6397627
DennisBorg  thnks man.. ur list of functions helped me a lot.. i am cunfused here.. whom should i give the points... plz suggest me..
thanks all for the help I appreciate it..

Expert Comment

ID: 6398546
N D:

>DennisBorg  thnks man.. ur list of functions helped me a
>lot.. i am cunfused here.. whom should i give the
>points... plz suggest me..

You're welcome. I'm glad I could help.

You would give the points to the person who best helped you reach your working solution.

If there is no clear-cut best solution (i.e. you feel that two or three people all helped you equally), then perhaps splitting the points would be the best thing to do. Customer Service can help you split the points if that is what you choose to do.

I can't really tell you who you should give the points to; that is really a decision which is up to you.

-Dennis Borg

Featured Post

Important Lessons on Recovering from Petya

In their most recent webinar, Skyport Systems explores ways to isolate and protect critical databases to keep the core of your company safe from harm.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When trying to find the cause of a problem in VBA or VB6 it's often valuable to know what procedures were executed prior to the error. You can use the Call Stack for that but it is often inadequate because it may show procedures you aren't intereste…
If you have ever used Microsoft Word then you know that it has a good spell checker and it may have occurred to you that the ability to check spelling might be a nice piece of functionality to add to certain applications of yours. Well the code that…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…
This lesson covers basic error handling code in Microsoft Excel using VBA. This is the first lesson in a 3-part series that uses code to loop through an Excel spreadsheet in VBA and then fix errors, taking advantage of error handling code. This l…
Suggested Courses

916 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question