In memory database, shared memory, etc

Posted on 2004-09-13
Last Modified: 2010-04-17
We have an application that is real-time and must update every second.   Sadly, we are using MS Access as are database.

The issue is the front end is handling drawing all the information on the screen which it gets from the database, and we have another service that uses a socket to retrieve info from a unix system and put this info into the MS Access DB.

I'm really worried about speed and having the socket program write all the time, it failing etc.

Isn't there a way i can do shared memory between applications in some sort of DLL?   Or is there a better mechanism between sharing high speed data between two apps (one running vb 6.0 right now, soon to be C#), other application is C#.

I'm just a little paranoid about a real-time application having two programs trying to retrieve info from a  Database every second!


Question by:rkneal
LVL 11

Expert Comment

ID: 12045106
have a look at Berkley DB, which supposedly supplies fast (optionally) in RAM databases... URL :

Btw, you could always replace the Access database with a strong opensource DB (PostgreSQL might be a good choice)...

Expert Comment

ID: 12046233
One or two database accesses per second shouldn't be too much of a performance problem.  Perhaps your newly designed application could use a messaging strategy instead of a polling strategy - that is, when something changes, a signal is sent to your GUI which then updates its display.

If you're looking for a small, high-performance database I would personally recommend not to try PGSQL and take a look at something like firebird (  Postgre has great features but performance is not one of them.

Author Comment

ID: 12046268
can you send an external event to a program from another program???

Lets say i'm in C# code that is doing the interface, could my vb 6.0 program send an event to it? Or are you saying we might want to look at something like a socket program between the two
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.


Expert Comment

ID: 12046582
A socket would certainly be one way to do it.  Your C# program could listen on port XXX, and the vb program could send messages to let it know that the data has changed.  This has the added "bonus" of letting you run the processes on separate machines or across the globe if you ever needed to, but that's probably not desirable anyways for a real-time application.

Expert Comment

ID: 12047848
If you must avoid the overhead of TCP/IP based communication, then it is possible to have windows programs communicate via shared memory using the Win32 API. Look up CreateFileMapping, MapViewOfFile and OpenFileMapping. The process that creates the shared memory would use CreateFileMapping and the process wishing to use the shared memory would use OpenFileMapping.

Hope this helps.

Accepted Solution

Validor earned 250 total points
ID: 12049073
How much information is being moved from the Unix database to MSAccess?  You should only move as little information as possible, and move it mostly when other things are idle, if possible.  If the Unix source is fast enough, consider writing a web service to access the Unix source directly, and fetch only the necessary info for the task at hand.  If there is no way around the Unix -> Access shuffle that you're doing now, I would definitely suggest MSDE, Interbase, Firebird or MySQL.  Any of them would be faster than Access.  MSDE supports bulk-inserts which will handle HUGE inserts from the Unix source (sometimes 20,000 rows per second), but inserts ONLY- not updates.  MySQL is generally one of the fastest databases around (and is fairly cheap), but doesn't support all the features that the others do.  Interbase (v6.0) and Firebird are open-source and are embeddable.

There are several ways that you can share data between applications.  Sockets are one of the fastest for transmitting data and there are a host of libraries to help make the job easy.  C# comes with some swell objects in the FCL right off the bat.  I would also recommend the signalling methodology mentioned by barryfandango.  It will make updates faster and minimize traffic.  For simple notifications, Mailslots are easier than sockets, but are one-way and not guaranteed to be delivered (though I've NEVER seen them fail).

Memory-mapped files are the method mentioned by KeithWatson.  it works, but requires the data to already be there and the programs STILL have to communicate which parts are of this huge data store new and which parts are not, or they have to reload the whole data store every time they update.  It serves the same purpose as two applications using a database, which is usually better anyway.

My suggestion is to use MSDE with bulk-inserts to absorb updates from the Unix system, or a webservice to query the UNIX system directly.  If there is not much data coming in from the Unix system (even if it is frequntly), this should not be a problem.  I would still recommend another database over MSAccess, though.

Author Comment

ID: 12049117
well, it is not a lot of data, but often. It's only basically 138 rows, need to read/write from them them every second.

I just hate MS Access, it's such a dog.  We have some experience at MySQL, so will try that or that berkely DB that can do RAM database.


Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
Whether you've completed a degree in computer sciences or you're a self-taught programmer, writing your first lines of code in the real world is always a challenge. Here are some of the most common pitfalls for new programmers.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

773 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question