?
Solved

What is best way to store high frequency changing and high frequency query data for web user?

Posted on 2014-02-21
4
Medium Priority
?
552 Views
Last Modified: 2014-03-21
I have stock price data that
- many users will queries quite often
- about 2000-3000 rows at times per symbol
- yet data appending pretty fast.

what would be the best way to tackle this problem.
- sql is currently has high cpu because users keep repeating requests
- sometime data change fast sometime not - but users have habit for repeat queries
- I am thinking about making data available on memory entirely but
-- Can we make sql on memory entirely and will this help my situation
-- If not using sql but create cache in memory - what would be the best way so web user can see the price rows and most up to date price rows without much cpu per calling

Many thanks,
0
Comment
Question by:JSW21
  • 2
4 Comments
 
LVL 66

Assisted Solution

by:Jim Horn
Jim Horn earned 500 total points
ID: 39876926
How often is this data updated?  

As long as we're not talking realtime, one solution would be to create a couple of materialized views, indexed on the columns that users regularly search.  

Then in the front end, have it interpret which term(s) the users entered as search criteria, and call the appropriate view.

Then after the nightly backup/restore create a job that drops and re-executes these views.
0
 
LVL 70

Accepted Solution

by:
Scott Pletcher earned 500 total points
ID: 39877086
The single best way to address is to create the best clustered index possible on the table, one that resolves the majority of common lookups.

Unfortunately, I don't know enough detail about your table to assist with that at this point.

But you need to look at the JOIN and WHERE conditions in the most common queries.  Then determine if those columns are appropriate for the clustered index, and in what order the key column(s) should be.


If you need historical stock info, an indexed view might indeed help.  SQL can, and often will, use that view even if you don't explicitly reference it in the query.
0
 

Author Comment

by:JSW21
ID: 39902539
You refer to View as feature in the sql right?
So by creating view and indexed it properly will increase performance for repeated queries.

"Then after the nightly backup/restore create a job that drops and re-executes these views."

If I create a view why do I have to drop and recreate it again nightly? Would the View update automatically when data changes?
0
 
LVL 66

Expert Comment

by:Jim Horn
ID: 39945034
Thanks for the split.  Good luck with your project.  -Jim
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

By, Vadim Tkachenko. In this article we’ll look at ClickHouse on its one year anniversary.
This shares a stored procedure to retrieve permissions for a given user on the current database or across all databases on a server.
Viewers will learn how to use the SELECT statement in SQL and will be exposed to the many uses the SELECT statement has.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
Suggested Courses

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question