We are developing a database solution that will run on SQL Server 2014/2016 and wanted to see if any experts could pass comment or suggest a better/different approach.
The solution will involve a collection of tables (we'll call these the 'data' tables) that will be constantly updated by an external multi-threaded application running on a collection of external servers, probably around 20 data tables. The data in these tables is updated using SQL MERGE and performs well already.
The part we need to design is regularly (minimum of every minute) moving that data into the search tables which will be constantly searched on from a public website.
Our plan is to create a view/query to perform some basic calculations to create a subset of relevant data from each of the data tables, and use the output of that query to push the data into our search tables every minute. We plan to use a stored procedure that will run every minute for each data table. Each stored procedure will select the relevant data from the data table and use SQL MERGE to insert/update/delete the records into the search tables.
We then plan to use the (NOLOCK) hint when querying the search tables in case any locking slows down the searches. We'll probably create a view that UNIONS all the search tables to enable us to search across them.
This is our basic plan, along with optimizing the indexes to try and find a balance between optimizing the searches without degrading the time it takes to perform each merge.
Key points to consider are...
* There are around 20 'data' tables
* A subset of the data in each 'data' table is (after some basic calculations are used select the relevant subset) merged into the search tables
* Queries on search tables need to be as fast as possible with no delays (due to locking etc)
* Queries on search tables CAN allow for dirty reads
* Total amount of records in the search tables will be around 500,000 to 1m
Think that's the basics - please ask if you need any more information, but would love to hear any potential issues you see with our plan, or a better suggestion on how to approach this to put the most efficient solution in place.
Thank you :)