URGENT - Dealing with Files in VB

Posted on 2001-08-15
Last Modified: 2010-05-19
I have files of size like 4 to 8 M.B. It has data seperated with a perticular delimiter. I am making an application which will read this file.. parse the data using delimiters & then save it as different files. i am using Scriptiing.filesystemobject to read the file & then parse it with the delimeter & then making different files & saving it. Thhe Code is as follows

Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)

After i get all the data of the file in arr i use split function using the delimiter & then make new files. Everything works fien if the file size is small like 1 M.B. .. but when i use file up to 4 M.B. ts.ReadAll just takes too much time which i cannot afford...Please suggest other alternative ..
thanking u in advance.
Question by:N_D
  • 4
  • 2
  • 2
  • +6

Expert Comment

ID: 6389459
Instead of using the FSO, you can use the VB native File I/O routines, or you can use some Win API functions to read/write the files.

In addition, if you display the status onscreen, instead of updating the status display for each and every record, do so only after every 10 or 100 records.

Reading the fine in chunks (instead of line-by-line) will also improve performance.

Others may have additional suggestions.

-Dennis Borg
President PAFSO
LVL 10

Expert Comment

ID: 6389548
Dim Filenr As Integer
Dim ByteArray() As Byte
Filenr = FreeFile
Open SourceFile For Binary As #Filenr
ReDim ByteArray(0 To LOF(Filenr) - 1)
Get #Filenr, , ByteArray()
Close #Filenr

You can try this, especially for large files. I got it from the web somewhere, claimed the fastest way to read
large files (I havn't test it)


Author Comment

ID: 6389562
i need to know the I/O Routines Please suggest me a site from where i can find more info
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.


Expert Comment

ID: 6389685
try the memory mapping method

Accepted Solution

DennisBorg earned 101 total points
ID: 6389859
N D:

>i need to know the I/O Routines Please suggest me a site from where i can find more info

Your VB Documentation ... look up:

   Line Input

-Dennis Borg

Expert Comment

ID: 6389861

instead of reading the whole file into memory, why don't you read it line by line, or block by block...

read n-bytes, scan for delimiter, if found, process it and read next block , so on...

it would help if you give us more details on your problem. what is the file type, and so on.

Expert Comment

ID: 6389878
Since the objective is efficience, you would not want to read line-by-line. You'd want to read block by block, unless the file was relatively small, then I'd read the whoe file in at once. However, the size of the files in this case would warrant a block-by-block reading of the files.

Expert Comment

ID: 6390027
Hi, N_D.

I had this same issue while using the FileSystemObject in another language.  You're right about how an increase in the file size can slow down the textstream's readall method.

In my own application, I could not predict what the file size would be.  So I experimented with block sizes and found 65536 to be about right for the wide range of file sizes that I tested.

<--------------- Sample Code Begins ----------------->
Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)

arr = ""
do while not ts.AtEndOfStream
  arr = arr & ts.Read( 65536 )
<--------------- Sample Code Ends ----------------->

Just in case ...
If you ever need to read in a binary file, use SoftArtisans' FileManager control instead.  I've had problems with the textstream object reading a lot of chr(0)'s.
(download the evaluation - it's uncrippled freeware.)

Bye. -e2

Expert Comment

ID: 6390403
or, you can do this:

Dim Block() as Byte, FileHandle as Long, AllData As String, FileLength as Long
FileHandle = FreeFile
ReDim Block(0 to 65535)
Open file_parse For Binary as #FileHandle
FileLength = LOF(FileHandle)
Do Until CurrentPosition >= FileLength
If (CurrentPosition + 65536) >= FileLength Then ReDim Block(0 to FileLength - CurrentPosition)
Get #FileHandle, CurrentPosition, Block
CurrentPosition = CurrentPosition + UBound(Block) + 1
AllData = AllData + StrConv(Block, vbUnicode)

Try that, I'm not sure if it will work as I coded it off the top of my head (I'm at work without access to VB).

The StrConv may or may not be needed, it depends.
LVL 14

Expert Comment

ID: 6390648
N_D.. please post the code you are presently using - as it is easier for us to show you what to do that way.

By the way.. when it somes to speed.. Binary file access will absolutely blow the doors off of ANY File System Object and/or Line Input means.. <smile>.

Expert Comment

by:Valliappan AN
ID: 6391378

Expert Comment

ID: 6392883
Hi, wsh2.

>Binary file access will absolutely blow the doors off of
>ANY File System Object and/or Line Input means

Just to let you know...

I have a sucky Novell Client for NT.  FileSystemObject's TextStream can open files an order of magnitude faster than binary file access.  The textstream almost never takes longer than 2.5 sec on any size file while binary file access (and any other file access technique) takes at least 30 sec even on the smallest files.

#Anecdotal Evidence
Even on PCs with an updated Novell Client (non-NT), FSO is still a little faster when it comes to accessing files ...

1. that the Novell server has compressed due to non-activity
2. that are stored in large directories (ie. >20,000 files)
3. where both conditions exist.

Note: I wouldn't have even considered FileSystemObject's TextStream (or SoftArtisan-FileManager's BinaryStream) if network access to these files weren't so slow.

Bye. -e2

Author Comment

ID: 6397627
DennisBorg  thnks man.. ur list of functions helped me a lot.. i am cunfused here.. whom should i give the points... plz suggest me..
thanks all for the help I appreciate it..

Expert Comment

ID: 6398546
N D:

>DennisBorg  thnks man.. ur list of functions helped me a
>lot.. i am cunfused here.. whom should i give the
>points... plz suggest me..

You're welcome. I'm glad I could help.

You would give the points to the person who best helped you reach your working solution.

If there is no clear-cut best solution (i.e. you feel that two or three people all helped you equally), then perhaps splitting the points would be the best thing to do. Customer Service can help you split the points if that is what you choose to do.

I can't really tell you who you should give the points to; that is really a decision which is up to you.

-Dennis Borg

Featured Post

ScreenConnect 6.0 Free Trial

Explore all the enhancements in one game-changing release, ScreenConnect 6.0, based on partner feedback. New features include a redesigned UI, app configurations and chat acknowledgement to improve customer engagement!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Not long ago I saw a question in the VB Script forum that I thought would not take much time. You can read that question (Question ID  ( Here (http…
Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
Show developers how to use a criteria form to limit the data that appears on an Access report. It is a common requirement that users can specify the criteria for a report at runtime. The easiest way to accomplish this is using a criteria form that a…

776 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question