[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.

Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

6.8

BerkeleyDB: High Volume database hitting limits

Asked by wevouch in Berkeley DB, Linux, Operating Systems Miscellaneous, Databases Miscellaneous, Miscellaneous Hardware

Tags: Sleepycat, BerkeleyDB, C++ API

I am using Berkeley DB to store real time ticks for market data. To give you an idea of the amount of data...
- I get roughly 15000 ticks/sec on an average
- by the end of the day the database file grows upto 50GB.

My system configuration is...
$ uname -a
Linux gritsbox.rsi.com 2.6.18-92.1.17.el5 #1 SMP Wed Oct 22 04:19:38 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
- Quad core 1.6 Ghz
- 4GB RAM
- I have set the Berkeley db cache to 2GB and I am using the BTREE access method.

I started out using one database to store everything and it works well for 3/4th of the trading day. However, Db:puts increasingly start to take longer and towards the end of the day the delay is so high that I start loosing ticks. I figured that this could be because of BTREE trying to rebalance a heavy tree on every PUT (is this correct?)

I then decided to open 10000 (yeah!!!) databases in the environment, 1 for each symbol(ticker) that i am subscribed to. my idea behind this was that balancing a much smaller tree would be faster for Berkeley DB. However when i run the code, i get a "Db::open: Cannot allocate memory" error after opening just 300 something databases.

How do I make this work?

thanks
[+][-]12/19/08 08:48 PM, ID: 23217143Administrative Comment

Experts Exchange has a courteous staff of administrators who help members get the most out of the website by means of administrative comments like this one.

Start your 30-day free trial to view this Administrative Comment or ask the Experts your question.

 
[+][-]12/21/08 03:36 AM, ID: 23221459Administrative Comment

Experts Exchange has a courteous staff of administrators who help members get the most out of the website by means of administrative comments like this one.

Start your 30-day free trial to view this Administrative Comment or ask the Experts your question.

 
[+][-]12/21/08 04:37 AM, ID: 23221699Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]12/22/08 03:53 PM, ID: 23230614Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]12/23/08 01:19 AM, ID: 23232232Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zones: Berkeley DB, Linux, Operating Systems Miscellaneous, Databases Miscellaneous, Miscellaneous Hardware
Tags: Sleepycat, BerkeleyDB, C++ API
Sign Up Now!
Solution Provided By: Gns
Participating Experts: 1
Solution Grade: B
 
 
Loading Advertisement...
20091111-EE-VQP-89 - Hierarchy / EE_QW_2_20070628