Solved

URGENT - Dealing with Files in VB

Posted on 2001-08-15
14
242 Views
Last Modified: 2010-05-19
I have files of size like 4 to 8 M.B. It has data seperated with a perticular delimiter. I am making an application which will read this file.. parse the data using delimiters & then save it as different files. i am using Scriptiing.filesystemobject to read the file & then parse it with the delimeter & then making different files & saving it. Thhe Code is as follows

Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)
arr=ts.ReadAll

After i get all the data of the file in arr i use split function using the delimiter & then make new files. Everything works fien if the file size is small like 1 M.B. .. but when i use file up to 4 M.B. ts.ReadAll just takes too much time which i cannot afford...Please suggest other alternative ..
thanking u in advance.
0
Comment
Question by:N_D
  • 4
  • 2
  • 2
  • +6
14 Comments
 
LVL 8

Expert Comment

by:DennisBorg
Comment Utility
Instead of using the FSO, you can use the VB native File I/O routines, or you can use some Win API functions to read/write the files.

In addition, if you display the status onscreen, instead of updating the status display for each and every record, do so only after every 10 or 100 records.

Reading the fine in chunks (instead of line-by-line) will also improve performance.


Others may have additional suggestions.

-Dennis Borg
President PAFSO
0
 
LVL 10

Expert Comment

by:GoodJun
Comment Utility
Dim Filenr As Integer
Dim ByteArray() As Byte
Filenr = FreeFile
Open SourceFile For Binary As #Filenr
ReDim ByteArray(0 To LOF(Filenr) - 1)
Get #Filenr, , ByteArray()
Close #Filenr

You can try this, especially for large files. I got it from the web somewhere, claimed the fastest way to read
large files (I havn't test it)

Cheers,
0
 
LVL 1

Author Comment

by:N_D
Comment Utility
i need to know the I/O Routines Please suggest me a site from where i can find more info
0
 
LVL 2

Expert Comment

by:selim007
Comment Utility
try the memory mapping method
0
 
LVL 8

Accepted Solution

by:
DennisBorg earned 101 total points
Comment Utility
N D:

>i need to know the I/O Routines Please suggest me a site from where i can find more info

Your VB Documentation ... look up:

   Open
   Close
   FreeFile
   Input#
   Input()
   Line Input
   Get#
   Put#
   Print#
   Write#
   FileLen
   LOF()
   EOF()


-Dennis Borg
0
 

Expert Comment

by:jchew
Comment Utility
hi,

instead of reading the whole file into memory, why don't you read it line by line, or block by block...

read n-bytes, scan for delimiter, if found, process it and read next block , so on...

it would help if you give us more details on your problem. what is the file type, and so on.
0
 
LVL 8

Expert Comment

by:DennisBorg
Comment Utility
Since the objective is efficience, you would not want to read line-by-line. You'd want to read block by block, unless the file was relatively small, then I'd read the whoe file in at once. However, the size of the files in this case would warrant a block-by-block reading of the files.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 7

Expert Comment

by:q2eddie
Comment Utility
Hi, N_D.

I had this same issue while using the FileSystemObject in another language.  You're right about how an increase in the file size can slow down the textstream's readall method.

In my own application, I could not predict what the file size would be.  So I experimented with block sizes and found 65536 to be about right for the wide range of file sizes that I tested.

<--------------- Sample Code Begins ----------------->
Set obj1 = New Scripting.FileSystemObject
Dim ts As TextStream
Dim arr as String
Set ts = obj1.OpenTextFile(file_parse, ForReading)

arr = ""
do while not ts.AtEndOfStream
  arr = arr & ts.Read( 65536 )
loop
<--------------- Sample Code Ends ----------------->

#Comments
Just in case ...
If you ever need to read in a binary file, use SoftArtisans' FileManager control instead.  I've had problems with the textstream object reading a lot of chr(0)'s.

http://www.softartisans.com/softartisans/safilemanager.html
(download the evaluation - it's uncrippled freeware.)

Bye. -e2
0
 
LVL 2

Expert Comment

by:JanusFury
Comment Utility
or, you can do this:

Dim Block() as Byte, FileHandle as Long, AllData As String, FileLength as Long
FileHandle = FreeFile
ReDim Block(0 to 65535)
Open file_parse For Binary as #FileHandle
FileLength = LOF(FileHandle)
Do Until CurrentPosition >= FileLength
If (CurrentPosition + 65536) >= FileLength Then ReDim Block(0 to FileLength - CurrentPosition)
Get #FileHandle, CurrentPosition, Block
CurrentPosition = CurrentPosition + UBound(Block) + 1
AllData = AllData + StrConv(Block, vbUnicode)
Loop

Try that, I'm not sure if it will work as I coded it off the top of my head (I'm at work without access to VB).

The StrConv may or may not be needed, it depends.
0
 
LVL 14

Expert Comment

by:wsh2
Comment Utility
N_D.. please post the code you are presently using - as it is easier for us to show you what to do that way.

By the way.. when it somes to speed.. Binary file access will absolutely blow the doors off of ANY File System Object and/or Line Input means.. <smile>.
0
 
LVL 9

Expert Comment

by:Valliappan AN
Comment Utility
<ping>
0
 
LVL 7

Expert Comment

by:q2eddie
Comment Utility
Hi, wsh2.

>Binary file access will absolutely blow the doors off of
>ANY File System Object and/or Line Input means

Just to let you know...

I have a sucky Novell Client for NT.  FileSystemObject's TextStream can open files an order of magnitude faster than binary file access.  The textstream almost never takes longer than 2.5 sec on any size file while binary file access (and any other file access technique) takes at least 30 sec even on the smallest files.

#Anecdotal Evidence
Even on PCs with an updated Novell Client (non-NT), FSO is still a little faster when it comes to accessing files ...

1. that the Novell server has compressed due to non-activity
2. that are stored in large directories (ie. >20,000 files)
3. where both conditions exist.

Note: I wouldn't have even considered FileSystemObject's TextStream (or SoftArtisan-FileManager's BinaryStream) if network access to these files weren't so slow.

Bye. -e2
0
 
LVL 1

Author Comment

by:N_D
Comment Utility
DennisBorg  thnks man.. ur list of functions helped me a lot.. i am cunfused here.. whom should i give the points... plz suggest me..
thanks all for the help I appreciate it..
Cheers
Nehal.
0
 
LVL 8

Expert Comment

by:DennisBorg
Comment Utility
N D:

>DennisBorg  thnks man.. ur list of functions helped me a
>lot.. i am cunfused here.. whom should i give the
>points... plz suggest me..


You're welcome. I'm glad I could help.

You would give the points to the person who best helped you reach your working solution.

If there is no clear-cut best solution (i.e. you feel that two or three people all helped you equally), then perhaps splitting the points would be the best thing to do. Customer Service can help you split the points if that is what you choose to do.

I can't really tell you who you should give the points to; that is really a decision which is up to you.


-Dennis Borg
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This article is the result of a quest to better understand Task Scheduler 2.0 and all the newer objects available in vbscript in this version over  the limited options we had scripting in Task Scheduler 1.0.  As I started my journey of knowledge I f…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
Get people started with the process of using Access VBA to control Outlook using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Microsoft Outlook. Using automation, an Access applic…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now