Solved

What is best way to store high frequency changing and high frequency query data for web user?

Posted on 2014-02-21
4
484 Views
Last Modified: 2014-03-21
I have stock price data that
- many users will queries quite often
- about 2000-3000 rows at times per symbol
- yet data appending pretty fast.

what would be the best way to tackle this problem.
- sql is currently has high cpu because users keep repeating requests
- sometime data change fast sometime not - but users have habit for repeat queries
- I am thinking about making data available on memory entirely but
-- Can we make sql on memory entirely and will this help my situation
-- If not using sql but create cache in memory - what would be the best way so web user can see the price rows and most up to date price rows without much cpu per calling

Many thanks,
0
Comment
Question by:JSW21
  • 2
4 Comments
 
LVL 65

Assisted Solution

by:Jim Horn
Jim Horn earned 250 total points
ID: 39876926
How often is this data updated?  

As long as we're not talking realtime, one solution would be to create a couple of materialized views, indexed on the columns that users regularly search.  

Then in the front end, have it interpret which term(s) the users entered as search criteria, and call the appropriate view.

Then after the nightly backup/restore create a job that drops and re-executes these views.
0
 
LVL 69

Accepted Solution

by:
Scott Pletcher earned 250 total points
ID: 39877086
The single best way to address is to create the best clustered index possible on the table, one that resolves the majority of common lookups.

Unfortunately, I don't know enough detail about your table to assist with that at this point.

But you need to look at the JOIN and WHERE conditions in the most common queries.  Then determine if those columns are appropriate for the clustered index, and in what order the key column(s) should be.


If you need historical stock info, an indexed view might indeed help.  SQL can, and often will, use that view even if you don't explicitly reference it in the query.
0
 

Author Comment

by:JSW21
ID: 39902539
You refer to View as feature in the sql right?
So by creating view and indexed it properly will increase performance for repeated queries.

"Then after the nightly backup/restore create a job that drops and re-executes these views."

If I create a view why do I have to drop and recreate it again nightly? Would the View update automatically when data changes?
0
 
LVL 65

Expert Comment

by:Jim Horn
ID: 39945034
Thanks for the split.  Good luck with your project.  -Jim
0

Featured Post

Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Creating and Managing Databases with phpMyAdmin in cPanel.
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Wufoo.com provides powerful tools for surveying targeted groups, and utilizing data from completed surveys to find trends, discover areas of demand or customer expectation, and make business decisions on products or services.

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question