ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.
Course Of The Month
2 days, 11 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Handling Huge Data insert and retrieval from MS SQL Server 2005
Posted on 2014-04-03
Hii,
I have developed a Software for that inserts large data in to a SQL Server 2005 table. I am facing issue when extracting the data from the table using queries, it takes hours to retrieve data from the table. My table columns are
TABLE [history_data](
[INT_DATETIME] [bigint] NOT NULL,
[INT_POINTID] [int] NOT NULL,
[REAL_VALUE] [float] NOT NULL,
[INT_QUALITY] [smallint] NULL,
CONSTRAINT [PK_history_data] PRIMARY KEY CLUSTERED
(
[INT_DATETIME] ASC,
[INT_POINTID] ASC
)
I have 15000 Points changing values every second. So I am inserting these 15000 records every second. Hence there will be more than 1 GB data in 1 hour and hence every day more than 20GB data will be there in the table. Is there any issue in storing such huge data with 1 table ?
Please help me how I can retrieve data fast from this table. I tried BCP command which seems to be faster than the direct select sql query, but that also takes time in hours to get the values. What should be the way to fetch data in seconds ?
I have developed a Software for that inserts large data in to a SQL Server 2005 table. I am facing issue when extracting the data from the table using queries, it takes hours to retrieve data from the table. My table columns are
TABLE [history_data](
[INT_DATETIME] [bigint] NOT NULL,
[INT_POINTID] [int] NOT NULL,
[REAL_VALUE] [float] NOT NULL,
[INT_QUALITY] [smallint] NULL,
CONSTRAINT [PK_history_data] PRIMARY KEY CLUSTERED
(
[INT_DATETIME] ASC,
[INT_POINTID] ASC
)
I have 15000 Points changing values every second. So I am inserting these 15000 records every second. Hence there will be more than 1 GB data in 1 hour and hence every day more than 20GB data will be there in the table. Is there any issue in storing such huge data with 1 table ?
Please help me how I can retrieve data fast from this table. I tried BCP command which seems to be faster than the direct select sql query, but that also takes time in hours to get the values. What should be the way to fetch data in seconds ?
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
3-
2
6 Comments
Active 5 days ago
Please provide more details. How many records are there in your table now? What is(are) your SELECT query/queries?
Hi,
15000 records in one second is inserted. That means 129,60,00,000 records in a day. And we need to store data for 90 days :(
My query is : select REAL_VALUE, INT_QUALITY from HISTORY_DATA where INT_POINTID=9999 and INT_DATETIME>2014033014502
15000 records in one second is inserted. That means 129,60,00,000 records in a day. And we need to store data for 90 days :(
My query is : select REAL_VALUE, INT_QUALITY from HISTORY_DATA where INT_POINTID=9999 and INT_DATETIME>2014033014502
Active 1 day ago
The query looks perfect overall. You probably want to add "WITH (NOLOCK)" if you are dealing only with historical data to reduce overhead from the SELECT.
Also, make sure you have a large auto-grow amount on the data files. (Make sure you have IFI turned on, although that affects only INSERT speed).
If possible, make sure the log never needs to dynamically grow but has already been pre-allocated.
Also, make sure you have a large auto-grow amount on the data files. (Make sure you have IFI turned on, although that affects only INSERT speed).
If possible, make sure the log never needs to dynamically grow but has already been pre-allocated.
Active 5 days ago
1.2 Billion records per day.. 116 billion for 90 days.. that's really huge (atleast for me!). I have not dealt with such huge data (or anywhere near to that). Okay, so you have 15000 points which will insert 1 record each per point per second.. which means 3600*24 = 86400 records per point per day, that means 7,776,000 records for 90 days. And you always query for a single Point at a time.
And yes, as @ScottPletcher suggested, use "WITH (NOLOCK)" in your FROM clause (FROM HISTORY_DATA WITH (NOLOCK)).
-Harish
select REAL_VALUE, INT_QUALITY from HISTORY_DATA where INT_POINTID=9999 and INT_DATETIME>20140330145022968 and INT_DATETIME<20140403145022968
CREATE NONCLUSTERED INDEX IDX_HISTORY_DATA_POINTID
ON HISTORY_DATA (INT_POINTID, INT_DATETIME)
INCLUDE (REAL_VALUE, INT_QUALITY)
And yes, as @ScottPletcher suggested, use "WITH (NOLOCK)" in your FROM clause (FROM HISTORY_DATA WITH (NOLOCK)).
-Harish
Thank you. This has improved the performance. I think this is the better optimization I can do with the existing table design. For a requirement like retrieving data in some seconds or atleast less than 5 minute time, I need to redesign the History table structure. May be storing data in more than one table in the database. Or depending on data stores other than RDBMS !
Active 5 days ago
Yes, using multiple tables may also help. Can you explain what did you do now and how much was the improvement in performance. I think, you can keep one table for current day, which will have data appending every second, and another history table for data for past 90 days till yesterday. As long as you have a good index, you may not see considerable increase in performance by splitting data across multiple tables or even doing table partitioning. But that will be helpful to distribute your storage requirement across multiple drives, if that is a concern.
-Harish
-Harish
Featured Post
Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat. The purpose of this eBook is to educate the reader about ransomware attacks.
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
Suggested Solutions
| Title | # Comments | Views | Activity |
|---|---|---|---|
| Format Output of Select Statement | 2 | 38 | |
| Creating a View from a CTE | 15 | 50 | |
| SQL Server 2012 and core licensing | 5 | 32 | |
| SQL Server Error: 4060 | 8 | 33 |
This article explains how to reset the password of the sa account on a Microsoft SQL Server. The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
- Windows OS
- Windows XP
- Windows 7
- Mac OS X
- Windows 10
- Windows 8, Software-Other, System Utilities, Vulnerabilities, Web Browsers
Via a live example, show how to extract insert data into a SQL Server database table using the Import/Export option and Bulk Insert.
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Suggested Courses
Course of the Month2 days, 11 hours left to enroll
752 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.