?
Solved

What is best way to store high frequency changing and high frequency query data for web user?

Posted on 2014-02-21
4
Medium Priority
?
530 Views
Last Modified: 2014-03-21
I have stock price data that
- many users will queries quite often
- about 2000-3000 rows at times per symbol
- yet data appending pretty fast.

what would be the best way to tackle this problem.
- sql is currently has high cpu because users keep repeating requests
- sometime data change fast sometime not - but users have habit for repeat queries
- I am thinking about making data available on memory entirely but
-- Can we make sql on memory entirely and will this help my situation
-- If not using sql but create cache in memory - what would be the best way so web user can see the price rows and most up to date price rows without much cpu per calling

Many thanks,
0
Comment
Question by:JSW21
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 66

Assisted Solution

by:Jim Horn
Jim Horn earned 500 total points
ID: 39876926
How often is this data updated?  

As long as we're not talking realtime, one solution would be to create a couple of materialized views, indexed on the columns that users regularly search.  

Then in the front end, have it interpret which term(s) the users entered as search criteria, and call the appropriate view.

Then after the nightly backup/restore create a job that drops and re-executes these views.
0
 
LVL 69

Accepted Solution

by:
Scott Pletcher earned 500 total points
ID: 39877086
The single best way to address is to create the best clustered index possible on the table, one that resolves the majority of common lookups.

Unfortunately, I don't know enough detail about your table to assist with that at this point.

But you need to look at the JOIN and WHERE conditions in the most common queries.  Then determine if those columns are appropriate for the clustered index, and in what order the key column(s) should be.


If you need historical stock info, an indexed view might indeed help.  SQL can, and often will, use that view even if you don't explicitly reference it in the query.
0
 

Author Comment

by:JSW21
ID: 39902539
You refer to View as feature in the sql right?
So by creating view and indexed it properly will increase performance for repeated queries.

"Then after the nightly backup/restore create a job that drops and re-executes these views."

If I create a view why do I have to drop and recreate it again nightly? Would the View update automatically when data changes?
0
 
LVL 66

Expert Comment

by:Jim Horn
ID: 39945034
Thanks for the split.  Good luck with your project.  -Jim
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this series, we will discuss common questions received as a database Solutions Engineer at Percona. In this role, we speak with a wide array of MySQL and MongoDB users responsible for both extremely large and complex environments to smaller singl…
This post looks at MongoDB and MySQL, and covers high-level MongoDB strengths, weaknesses, features, and uses from the perspective of an SQL user.
Via a live example, show how to setup several different housekeeping processes for a SQL Server.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question