Solved

Q:  Custom File Programming

Posted on 1997-06-12
5
149 Views
Last Modified: 2010-05-18
Although I have quite a bit of experience programming, all of my apps thus far have been database apps.  You learn the confines and the constructs and go with it.  I am currently writing an app that calls for "manual" file handling and I would like some pointers from those who have "been there, done that"

I Need:

Fixed length "Customer Information Record"
Multiple, variable length string identifiers related to a customer

The fixed length customer record sounds easy enough, random file, record structures, etc..

It is the other side that I would like some direction...

-  Do I store the multiple strings in the same physical file?
-  If so, how?  what about updates?  when the string grows?stability?
-  Do I reference a customer # tag in my string structure?
-  Should I implement a pointer type system to point to next/prev string?  Or is the maintenance of these pointers prohibitive?
-  Should I store the strings in a seperate "detail" file?
-  Should I use files, or FileStreams?
-  If I use FileStreams, could/should I use TStringList stream methods?
-  What about performance?

Sheesh, that's a lotta questions.  You'd think I was a newbie or something.  Anyways...

TIA to everyone's(anyone's) help..
0
Comment
Question by:meech
  • 3
  • 2
5 Comments
 
LVL 4

Expert Comment

by:erajoj
ID: 1336981
What kind of performances are we talking about here?
What kind of searching do you have in mind? Substring? None?
Do you mind a separate index file?
Hugeness?

/// John


0
 

Author Comment

by:meech
ID: 1336982
On average the app will have under 100 master records, each with approx 20-30 (avg) string "paramaters" (detail records).  Master record size under 2K.  Avg detail string length -- about 200 bytes.  

Performance-wise I was saying, ideally, i don't want to have to read 3000 records to get the last one.  Indexing in some fashion would be preferred.  

Were you thinking something along the lines of storing the master file in a record, the details in a Filestreams type, and then a third "index" file which cross references record numbers to Stream offsets or something like that?  

Also, i dealing with the sizes I described above it is not inconievable that the entire database be loaded into memory at once (maybe for file compacting/restructuring/rebuilding/reindexing, etc.)
 

String searching - None.  The detail file (I think of them as parameters) will need to contain the string and an identifier.  No searching will be required.  

It is actually a more/less traditional master/client relationship.  I just would like variable text length (for file size) and Paradox memo fields scare me.  I really like the Jet engine - not an option.
0
 
LVL 4

Accepted Solution

by:
erajoj earned 100 total points
ID: 1336983
I really don't know what to answer.
You seem to have gotten the picture already.
What I had in mind is xactly what you describe.

Here's my humble approach:
The Index file just contains pointers (=32 bit Integers).
The "string"-file contains records of the form:
  size, length: Integer;
  strData     : lots of chars;
With this approach, the index file can be rebuilt from the "string"-file data if it becomes corrupt or erased.
"Size" is the binary size of the string the first time it's added to the file (can be exaggerated).
"Length" is the string length.
If you change the string to be smaller than "Size" bytes then just change the "Length".
If you change the string to be greater than "Size" bytes then change the "Length" to 0 (zero) and Append the changed string to the end of the file with new "size" and "length" and update the index file.

Maybe not the most elegant way to do it but it's quite fast and very simple to code.
Don't forget to "compact" the database if it bloats.

There is always the really simple way to use the LoadFromFile & SaveToFile methods of TStringList. But then the whole file will reside in memory. This approach is so simple that it's sickening.

/// John

0
 

Author Comment

by:meech
ID: 1336984
I get the idea, the only questions I have remaining are:

1)  For the string file, I couldn't use record structures because records need to be a fixed length.  I would have to use a TFileStream and store offsets in the index, right?

2)  I thought about using the TStringList.SaveToStream/LoadFromStream methods.  But would 1 stream accomodate multiple TStringLists?

3)  For working with Streams:  I know the "new" strings are null terminated unlike the old 1.0 strings.  Can I just use @MyString to write the string to a stream?

4)  What's your experience (if any) with TFileStreams?  Do they commit automatically?  Stability?

Thanks for the help....
0
 
LVL 4

Expert Comment

by:erajoj
ID: 1336985
I think you're trying to complicate things here.
Use Delphis native file handling.
Take a look at BlockRead & BlockWrite.

/// John

0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…
In this video I am going to show you how to back up and restore Office 365 mailboxes using CodeTwo Backup for Office 365. Learn more about the tool used in this video here: http://www.codetwo.com/backup-for-office-365/ (http://www.codetwo.com/ba…

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now