Avatar of DSOM
DSOM
Flag for United States of America asked on

Storage format / engine for my database

I have a data set with ~7 million 32-character strings.  I am trying to find the most efficient storage for it.

It needs to be encrypted.
It needs fast searching.
It needs to be stand-alone.
It needs to be as compact as possible for online distribution.

I have tried Absolute Database but the resulting data file is about 800K.
I tried mOrmot SQLite3 and the data file is about 400K.
I tried a simple in-memory string list but it gobbles up about 450K of RAM.

What are my options?
DelphiStorage

Avatar of undefined
Last Comment
DSOM

8/22/2022 - Mon
Geert G

have you got any realistic numbers of what to expect ?

32x7000000 chars = 224000000 bytes (or 213 Mb)

400K / 213Mb is a factor of 1/532,
so thats a very good compression factor by the look of it

it looks like you already have what you look for

or is there an other problem ?
DSOM

ASKER
I don't know what to expect or what my options are which is why I asked.

I am hoping there might be something more suitable that I haven't found yet.

If it is indeed the best option that is fine too, I just need to know it.
Gerald Connolly

Well, i guess it depends on what you are really doing with this.

213MBytes  (NB "b" = bits, "B" = Bytes) is a tiny database and you would have to go back probably 10 years to find a memory stick it couldnt fit on!
Even on 4Mb broadband it will only take 5 minutes or so to download.

Why does it need to be compressed? It will just make it slower to access.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
DSOM

ASKER
It only needs to be compressed during distribution/installation.  It is expanded on the end-users' computer.

Ugh, I just realized what I did.  Posting when tired and not braining properly.

It doesn't look like I can edit my original question.  Should have been:

I have tried Absolute Database but the resulting data file is about 800MB.
I tried mOrmot SQLite3 and the data file is about 400MB.
I tried a simple in-memory string list but it gobbles up about 450MB of RAM.
Sinisa Vuk

How does your strings look like? Is it something like hex numbers? or just random characters.
Gerald Connolly

why is 800MB a problem?

How are you distributing this db and how often are there updates?
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
DSOM

ASKER
It is too big.  Customers still on slow connections or even dialup have problems with it.  The download installer at almost 1GB is using up almost 5TB a month in bandwidth.  That is just for the initial distribution/installation.

It is being distributed via my website.  Updates are daily but just incrementally so that isn't a big problem.
Geert G

have you tried putting the strings in a text file and zipping it ?
zip is very good at text compression
DSOM

ASKER
I tried using a compressed string list.  But it takes up too much RAM when loaded.  From my original question "I tried a simple in-memory string list but it gobbles up about 450MB of RAM."
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
ASKER CERTIFIED SOLUTION
DSOM

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Geert G

lol, i wasn't meaning for you to load as a stringlist
>> my idea was:
put all your text in a file.txt
> zip this file.txt

when you start your app or appinstaller:
> unzip the file
> load it into a database in the local environment

the local environment could be a company which hosts an enterprise database like sql server, oracle, etc ...
that way you can have 1 dictionary (at least i think it's a dictionary)
for all pc's in 1 local environment and you'd only have to do the upload once
and set the connection string on the others

>> i know it's a differrent ball game like this
DSOM

ASKER
When I do an incremental update it is a zipped text file that is imported.  However the original distribution contains the full current database.  The end-user won't have a database server aside from the one I run which they access over the Internet.  This isn't for enterprise customers, it's average home users.

The database on the customer's home computer is a stand-alone SQL database.
DSOM

ASKER
This has ended up being the best option
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.