URGENT - Dealing with Files in VB

I have files of size like 4 to 8 M.B. It has data seperated with a perticular delimiter. I am making an application which will read this file.. parse the data using delimiters & then save it as different files. i am using Scriptiing.filesystemobject to read the file & then parse it with the delimeter & then making different files & saving it. Thhe Code is as follows

Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)
arr=ts.ReadAll

After i get all the data of the file in arr i use split function using the delimiter & then make new files. Everything works fien if the file size is small like 1 M.B. .. but when i use file up to 4 M.B. ts.ReadAll just takes too much time which i cannot afford...Please suggest other alternative ..
thanking u in advance.
LVL 1
N_DAsked:
Who is Participating?
 
DennisBorgConnect With a Mentor Commented:
N D:

>i need to know the I/O Routines Please suggest me a site from where i can find more info

Your VB Documentation ... look up:

   Open
   Close
   FreeFile
   Input#
   Input()
   Line Input
   Get#
   Put#
   Print#
   Write#
   FileLen
   LOF()
   EOF()


-Dennis Borg
0
 
DennisBorgCommented:
Instead of using the FSO, you can use the VB native File I/O routines, or you can use some Win API functions to read/write the files.

In addition, if you display the status onscreen, instead of updating the status display for each and every record, do so only after every 10 or 100 records.

Reading the fine in chunks (instead of line-by-line) will also improve performance.


Others may have additional suggestions.

-Dennis Borg
President PAFSO
0
 
GoodJunCommented:
Dim Filenr As Integer
Dim ByteArray() As Byte
Filenr = FreeFile
Open SourceFile For Binary As #Filenr
ReDim ByteArray(0 To LOF(Filenr) - 1)
Get #Filenr, , ByteArray()
Close #Filenr

You can try this, especially for large files. I got it from the web somewhere, claimed the fastest way to read
large files (I havn't test it)

Cheers,
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
N_DAuthor Commented:
i need to know the I/O Routines Please suggest me a site from where i can find more info
0
 
selim007Commented:
try the memory mapping method
0
 
jchewCommented:
hi,

instead of reading the whole file into memory, why don't you read it line by line, or block by block...

read n-bytes, scan for delimiter, if found, process it and read next block , so on...

it would help if you give us more details on your problem. what is the file type, and so on.
0
 
DennisBorgCommented:
Since the objective is efficience, you would not want to read line-by-line. You'd want to read block by block, unless the file was relatively small, then I'd read the whoe file in at once. However, the size of the files in this case would warrant a block-by-block reading of the files.
0
 
q2eddieCommented:
Hi, N_D.

I had this same issue while using the FileSystemObject in another language.  You're right about how an increase in the file size can slow down the textstream's readall method.

In my own application, I could not predict what the file size would be.  So I experimented with block sizes and found 65536 to be about right for the wide range of file sizes that I tested.

<--------------- Sample Code Begins ----------------->
Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)

arr = ""
do while not ts.AtEndOfStream
  arr = arr & ts.Read( 65536 )
loop
<--------------- Sample Code Ends ----------------->

#Comments
Just in case ...
If you ever need to read in a binary file, use SoftArtisans' FileManager control instead.  I've had problems with the textstream object reading a lot of chr(0)'s.

http://www.softartisans.com/softartisans/safilemanager.html
(download the evaluation - it's uncrippled freeware.)

Bye. -e2
0
 
JanusFuryCommented:
or, you can do this:

Dim Block() as Byte, FileHandle as Long, AllData As String, FileLength as Long
FileHandle = FreeFile
ReDim Block(0 to 65535)
Open file_parse For Binary as #FileHandle
FileLength = LOF(FileHandle)
Do Until CurrentPosition >= FileLength
If (CurrentPosition + 65536) >= FileLength Then ReDim Block(0 to FileLength - CurrentPosition)
Get #FileHandle, CurrentPosition, Block
CurrentPosition = CurrentPosition + UBound(Block) + 1
AllData = AllData + StrConv(Block, vbUnicode)
Loop

Try that, I'm not sure if it will work as I coded it off the top of my head (I'm at work without access to VB).

The StrConv may or may not be needed, it depends.
0
 
wsh2Commented:
N_D.. please post the code you are presently using - as it is easier for us to show you what to do that way.

By the way.. when it somes to speed.. Binary file access will absolutely blow the doors off of ANY File System Object and/or Line Input means.. <smile>.
0
 
Valliappan ANSenior Tech ConsultantCommented:
<ping>
0
 
q2eddieCommented:
Hi, wsh2.

>Binary file access will absolutely blow the doors off of
>ANY File System Object and/or Line Input means

Just to let you know...

I have a sucky Novell Client for NT.  FileSystemObject's TextStream can open files an order of magnitude faster than binary file access.  The textstream almost never takes longer than 2.5 sec on any size file while binary file access (and any other file access technique) takes at least 30 sec even on the smallest files.

#Anecdotal Evidence
Even on PCs with an updated Novell Client (non-NT), FSO is still a little faster when it comes to accessing files ...

1. that the Novell server has compressed due to non-activity
2. that are stored in large directories (ie. >20,000 files)
3. where both conditions exist.

Note: I wouldn't have even considered FileSystemObject's TextStream (or SoftArtisan-FileManager's BinaryStream) if network access to these files weren't so slow.

Bye. -e2
0
 
N_DAuthor Commented:
DennisBorg  thnks man.. ur list of functions helped me a lot.. i am cunfused here.. whom should i give the points... plz suggest me..
thanks all for the help I appreciate it..
Cheers
Nehal.
0
 
DennisBorgCommented:
N D:

>DennisBorg  thnks man.. ur list of functions helped me a
>lot.. i am cunfused here.. whom should i give the
>points... plz suggest me..


You're welcome. I'm glad I could help.

You would give the points to the person who best helped you reach your working solution.

If there is no clear-cut best solution (i.e. you feel that two or three people all helped you equally), then perhaps splitting the points would be the best thing to do. Customer Service can help you split the points if that is what you choose to do.

I can't really tell you who you should give the points to; that is really a decision which is up to you.


-Dennis Borg
0
All Courses

From novice to tech pro — start learning today.