PSQL v9.1 - after a certain load, low performance with shared Btrieve files

Let me explain shortly what our application is currently doing.

Our software is a voice mail system. People call in on a number where they can manage their mailbox, listen to messages, changing settings.
All these settings and the call flow is stored in Btrieve BTV files stored locally on a telecomnode. This system can handle more than 60 simultaneous calls.

Remark: We only use one single pervasive client-id per Telecom node, since its our application that connects to the BTV files trough the Btrieve API.
We open the btrieve files locally (on the same machine).  Pervasive Server v8 is installed.

Our application is also using record locking and transactional updates.
Another application is also using ODBC access from a .Net Application (not applicable for the moment).

****All this is working great at the moment.****

What we would like to setup.

In a production environment we want to set up
"  2 of our applications on 2 different machines (telecom nodes) connecting to the same shared BTrieve files.
"  These BTrieve files should be located on a Cluster environment.
"  1 telecom node should handle 120 simultaneous calls.

Telecoms nodes:
"  Windows server 2003 standard edition (SP2)
"  3GB ram

Cluster environment:
"  Windows server 2003 enterprise edition (SP2)
"  3GB ram

We already set up 1 telecom node connecting to the shared BTrieve files.
"  On the server (location of the BTrieve files) we have installed Pervasive Server v9.1, according to the Clustering documentation from Pervasive.
"  On the Telecom node, we have installed the Pervasive Client (available on the Pervasive Server CD-ROM).

All this is working functionally OK, but we have performance problems when we reach more than +/- 35 simultaneous calls (on 1 telecom node).
"  Up to 35 callers --> performance ok
"  >35 callers --> performance very slow

We have read about MEFS (multi engine file sharing), which would allow us to use the same application without modification but which is substituted by other technologies like Pervasive Workgroup Engine and Pervasive Gateway, unfortunately we didnt find any documentation about how to install/configure this.  

Do you have any suggestion how to set up this platform without performance issues?
client/server or server/server ???

Thanks in advance.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bill BachPresident and Btrieve GuruCommented:
I would first recommend patching your PSQLv9.1 setup to PSQLv9.5 (Service Pack 2), and then applying the latest 9.52 update.  Both patches can be found at Pervasive's web site.  

Second, you should review your memory configuration to see if you have adequate L1 and L2 cache, and decide if you want to use System Cache or not.

The last thing is to review your use of the ClientID field and threading.  If you run a number of clients through the same connection, then they will be single-threaded at the networking layer.  This is probably the biggest detractor from performance in what you are doing, and is why you didn't see a problem on a standalone machine.  You may find better parallelism by implementing a new process for each call with its own database connection.

To answer your original question, MEFS is an implementation detail from Btrieve 6.15, which was eliminated in the PSQL7 engine due to its severe performance limitations in multi-user environments.  It was ONLY suitable for small workgroup setups.  The Pervasive Workgroup Engine uses Single-Engine File Sharing (like the server) and may be an option for you to reduce costs, depending on your application design and the Pervasive licensing agreement, which I suggest that you read in detail.  The WGE tops out at 5 concurrent users, but from your description, you may only have 2.
ArtiliumAuthor Commented:
we have enabled a 2nd workstation (installed Pervasive CLIENT° 9.52) and launched 30 calls on this one... so on the server (also pervasive 9.52 SERVER) we currently have 4 communication threads and 2 licences used of the 6... and in total 60 calls without performance loss...

if we launch 50 calls on one of the clients.... we notice that calls on THAT client have a performance loss;
the 2nd client has no performance loss.

Now our programmers told us they have 120 "threads" towards the btrieve client ... but with netstat we only see 1 unc and 1 btrieve pipe towards the server (hence 2 communication threads per workstation on the server)....

Is there something they have to code to have really multithreading connectivity with the btrieve client?
or is there a hidden registry setting for this to be enabled?

We can install a 3rd box...4th box... but the idea is that 1 box could handle the load of the 2 if one fails...
and it should be 120 simultaneous calls per box... not 30

but currently 30 seems to be the limit for each "btrieve" thread...
Bill BachPresident and Btrieve GuruCommented:
This is from a conversation with the original developer of the code itself.  It's a bit dated, but you get the idea:

1) Is the PSQL2000i NSL still single-threading requests to the server?

2) Any chance that this will change soon? PSQL8? PSQL9?
It will some time, maybe in PSQL8 but no guarantees at this point.  BTW, it is not just that the app needs to be multi-threaded; it also needs to use different Btrieve clientIDs because the MKDE itself only allows one request per clientID to be active at any one time.

3) Do the ODBC drivers have the same single-outstanding-request limitation?
There can only be one outstanding request per database connection.  If the app opens multiple database connections and make separate requests on each connection on different threads I think the requests would be handled in parallel.  

4) What about the new SP3 OLEDB components?
Ultimately, the OLEDB components all wind up going through the MIF and thus through NSL for remote requests; therefore remote requests would be single threaded through NSL.

The most telling side is the Engine Monitor, which is reporting one connection from each computer, right?  This essentially indicates that the program is single-threading Btrieve requests through the communications layer.

A network analyzer might give you some more insight into what is happening.  With it, you could see the response times between the various Btrieve calls and replies, and you'll see the ClientID buried in the data block so that you can see the various threads at play.  Look specifically for database calls that take an inordinate amount of time (anything over a millisecond or so), which could indicate waiting for disk activity on the server.  

You MIGHT be able to address this by adjusting cache, keeping file sizes small, and other such things, but you'll eventually run into the same issue as you continue to scale -- single-threading at the network layer.

There is a page in the on-line manuals called "Btrieve Clients" that you may wish to review.  I believe, however, that you must actually spawn child processes, not just child threads, for this to work the way that you want.

Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

ArtiliumAuthor Commented:
okay our developper has changed the usage of "BTRV" into "BTRVID" whereas an extra parameter "clientid" is now presented. We have successfully launched upto 32 different clientid's from one application to the server; but when we want to use them we get a btrieve error status 3 telling us the files are not "open"...
when we launch 120x 1 clientid the problem does not represent itself... but we still have a limit of max 30 calls on 1 clientid :(

another thread ( explains to me that changing clientid will not matter hence files are locked to a certain clientid... which explains why other clientid's can not open the btv we need :(
Bill BachPresident and Btrieve GuruCommented:
Files are never "locked" to a client ID, unless you specifically apply a lock on them.  Instead, each client ID is just like an independent connection from an independent computer or process.  This means that when you create the session, you must also open the database files inside EACH connection.  Each CID also gets its own record locks, and of course must remember to close the files when it is finished.

When you do all of this, the "Active Users" screen WILL show a different connection for each user, as it should.  However, the telltale sign will be the number of communications sessions, which is reported in the Pervasive Monitor under Communications Statistics.  This MUST show the proper number.  If it still shows 2, then your threads are all still sharing the same communications session, and you still won't be able to get past your limits.  

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ArtiliumAuthor Commented:
    microkernel  -> active files  2  app x 4 clientid, 240+120 voice    Active microkernel files:  24  
    Selected file's handles :   8  
    Microkernel -> Active Users  
    Active Microkernel Users:  8  
    Selected user's handles:  2159  
    microkernel -> resource  usage  
    Files:  24  
    handles:  13240/22004  
    clients:  8/13  
    worker threads (MAX  1024)  1/9  
    licences in use (MAX  6)  2/2  
    transactions  0/0  
    locks/peak  1/4  
    Microkernel ->  communications  
    Total Requests :  1392907516  
    spx requests:  0  
    tcp/ip requests:  1392907516  
    netbios requests:  0  
    connection timeouts:  0  
    connection recoveries  0  
    communication thread current/peak (MAX 16)  3/9  
    total remote sessions  current/peak  9/10  
    SPX remote sessions current/peak  0/0  
    TCP/IP remote sessions current/peak  9/10  
    Netbios remote sessions current/peak  0/0  
    port 139 and/or 445  2  
    Port 3351  8  
ArtiliumAuthor Commented:
Thnx for the tip...BTRVID is indeed a better solution than BTRV
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.