WannaCry is just the start. Read how Liquid Web is protecting itself and its customers against new threats.
Course Of The Month
6 days, 11 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Storage format / engine for my database
Posted on 2012-12-31
I have a data set with ~7 million 32-character strings. I am trying to find the most efficient storage for it.
It needs to be encrypted.
It needs fast searching.
It needs to be stand-alone.
It needs to be as compact as possible for online distribution.
I have tried Absolute Database but the resulting data file is about 800K.
I tried mOrmot SQLite3 and the data file is about 400K.
I tried a simple in-memory string list but it gobbles up about 450K of RAM.
What are my options?
It needs to be encrypted.
It needs fast searching.
It needs to be stand-alone.
It needs to be as compact as possible for online distribution.
I have tried Absolute Database but the resulting data file is about 800K.
I tried mOrmot SQLite3 and the data file is about 400K.
I tried a simple in-memory string list but it gobbles up about 450K of RAM.
What are my options?
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
-
7
3
2- +1
13 Comments
Active 1 day ago
have you got any realistic numbers of what to expect ?
32x7000000 chars = 224000000 bytes (or 213 Mb)
400K / 213Mb is a factor of 1/532,
so thats a very good compression factor by the look of it
it looks like you already have what you look for
or is there an other problem ?
32x7000000 chars = 224000000 bytes (or 213 Mb)
400K / 213Mb is a factor of 1/532,
so thats a very good compression factor by the look of it
it looks like you already have what you look for
or is there an other problem ?
I don't know what to expect or what my options are which is why I asked.
I am hoping there might be something more suitable that I haven't found yet.
If it is indeed the best option that is fine too, I just need to know it.
I am hoping there might be something more suitable that I haven't found yet.
If it is indeed the best option that is fine too, I just need to know it.
Active 2 days ago
Well, i guess it depends on what you are really doing with this.
213MBytes (NB "b" = bits, "B" = Bytes) is a tiny database and you would have to go back probably 10 years to find a memory stick it couldnt fit on!
Even on 4Mb broadband it will only take 5 minutes or so to download.
Why does it need to be compressed? It will just make it slower to access.
213MBytes (NB "b" = bits, "B" = Bytes) is a tiny database and you would have to go back probably 10 years to find a memory stick it couldnt fit on!
Even on 4Mb broadband it will only take 5 minutes or so to download.
Why does it need to be compressed? It will just make it slower to access.
It only needs to be compressed during distribution/installation.
Ugh, I just realized what I did. Posting when tired and not braining properly.
It doesn't look like I can edit my original question. Should have been:
I have tried Absolute Database but the resulting data file is about 800MB.
I tried mOrmot SQLite3 and the data file is about 400MB.
I tried a simple in-memory string list but it gobbles up about 450MB of RAM.
Ugh, I just realized what I did. Posting when tired and not braining properly.
It doesn't look like I can edit my original question. Should have been:
I have tried Absolute Database but the resulting data file is about 800MB.
I tried mOrmot SQLite3 and the data file is about 400MB.
I tried a simple in-memory string list but it gobbles up about 450MB of RAM.
Active 1 day ago
How does your strings look like? Is it something like hex numbers? or just random characters.
Active 2 days ago
why is 800MB a problem?
How are you distributing this db and how often are there updates?
How are you distributing this db and how often are there updates?
It is too big. Customers still on slow connections or even dialup have problems with it. The download installer at almost 1GB is using up almost 5TB a month in bandwidth. That is just for the initial distribution/installation.
It is being distributed via my website. Updates are daily but just incrementally so that isn't a big problem.
It is being distributed via my website. Updates are daily but just incrementally so that isn't a big problem.
Active 1 day ago
have you tried putting the strings in a text file and zipping it ?
zip is very good at text compression
zip is very good at text compression
I tried using a compressed string list. But it takes up too much RAM when loaded. From my original question "I tried a simple in-memory string list but it gobbles up about 450MB of RAM."
I am trying SynSQLite3 right now and it looks promising. About 500MB on disk and almost zero memory footprint. That database will compress down to about 300MB for distribution which is a huge improvement over almost 1GB.
Active 1 day ago
lol, i wasn't meaning for you to load as a stringlist
>> my idea was:
put all your text in a file.txt
> zip this file.txt
when you start your app or appinstaller:
> unzip the file
> load it into a database in the local environment
the local environment could be a company which hosts an enterprise database like sql server, oracle, etc ...
that way you can have 1 dictionary (at least i think it's a dictionary)
for all pc's in 1 local environment and you'd only have to do the upload once
and set the connection string on the others
>> i know it's a differrent ball game like this
>> my idea was:
put all your text in a file.txt
> zip this file.txt
when you start your app or appinstaller:
> unzip the file
> load it into a database in the local environment
the local environment could be a company which hosts an enterprise database like sql server, oracle, etc ...
that way you can have 1 dictionary (at least i think it's a dictionary)
for all pc's in 1 local environment and you'd only have to do the upload once
and set the connection string on the others
>> i know it's a differrent ball game like this
When I do an incremental update it is a zipped text file that is imported. However the original distribution contains the full current database. The end-user won't have a database server aside from the one I run which they access over the Internet. This isn't for enterprise customers, it's average home users.
The database on the customer's home computer is a stand-alone SQL database.
The database on the customer's home computer is a stand-alone SQL database.
Featured Post
Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.
One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
Finding original email is quite difficult due to their duplicates. From this article, you will come to know why multiple duplicates of same emails appear and how to delete duplicate emails from Outlook securely and instantly while vital emails remai…
The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions. Here are my tips for handling such a transfer.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility.
Plug in the external drive you wish to encrypt:
Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting.
This w…
Suggested Courses
Course of the Month6 days, 11 hours left to enroll
Earn Certification
CompTIA IT Fundamentals
$49.00$44.10
724 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.