In Memory Database

Posted on 2006-05-15
Last Modified: 2010-04-05
I'm looking for a light weight database component to replace a large dataset.. currently I have an array that is now growing to 1Million+ rows.. and memory usage isn't that efficient.

When I search through it I just loop throught the array.. I never expected that it would grow this large and now I need to fix the problem of high memory usage.

The data is only relavent for the session, and is retrieved from the server upon login.

Does anybody have a suggestion for a local database that I can compile in? I have found one so far: here

Question by:rbohac
    LVL 28

    Assisted Solution

    here are a few I found:
    I haven't worked with such components so I cannot tell you which one is better.
    LVL 26

    Expert Comment

    LVL 3

    Accepted Solution

    Well. if its efficiency of datamanagement you want then you could use a MemoryDataset. but that wont take up less memory..... probably more. your array will be hard as hell to use, so I agree in changing it, but a memory dataset will have overheads. There are many vendors around that provide memory based TDatasets.
    with a million rows perhaps you want to cache it to disk? not memory, by compiling in a lite DBF database or Lite SQL database that is embedded. Also look at flashfiler from TurboPower which is now opensource.

    LVL 17

    Assisted Solution

    if you're holidng 1 million+ records, a memory dataset is not the best solution
    Storing on disk is probably the way to go. (even though it is temporary), but slower.

    You could opt for a typed file, with an index file

    // the main database file
        DBaseRec = Record
                     Deleted : Boolean;
                     Name: string[64];
       DBase : DBaseRec;
       DBaseFile: File of DBaseRec;

    The index file will hold a list of "search fields" and record numbers, so you can quickly "seek" to the record you want.
    (the idea is to only have to search the index file, then use it's "record pointer" to getthe "full" data from teh actual database file)

    If you're talking Delphi 5, then the BDE will do a (somewhat ridiculed) decent job with very little effort.
    (requires installing bde on the machine of course)

    If you have a later version of Delphi, I think the BDE is removed. (can probably still downlaod it from borland though)
    If you want free, and a DBMS that does not require an administrator, then Firebird is a good choice
    It also has an embedded install if no remote access is required
    LVL 26

    Assisted Solution

    How about Firebird embedded. This would give you total control and full SQL compliancy.

    You can store your database as a resource, save it to disk and connect, then when you are done,
    delete it.
    LVL 6

    Author Comment

    wow what a great response. I'm starting to go through and look at all of these this morning.

    LVL 1

    Assisted Solution


    Another option is Absolute Database which supports both disk and memory based structures.
    LVL 6

    Author Comment

    I actually stuck with a compressed stream type that I created, but all of the answers were very helpful. The only thing I can do that will be fair is increase the points and split them

    thank you everybody

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Top 6 Sources for Identifying Threat Actor TTPs

    Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

    A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
    In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
    Sending a Secure fax is easy with eFax Corporate ( First, Just open a new email message.  In the To field, type your recipient's fax number You can even send a secure international fax — just include t…
    Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now