Solved

SQL 2005 Delete rows

Posted on 2014-02-13
17
559 Views
Last Modified: 2014-02-17
I have a 2005 sql table that has 250,000 rows.  We only work with the last 1000 or so rows.    When it gets over 100,000 rows things start to slow down.  I have been going in and deleting the rows by opening the table in  sql management studio.  If I delete more than 1000 at a time it takes forever and most of the time just locks up.  

I read some stuff about truncating, but that seems to erase the whole table.  Is there a query I can run that will delete the rows in chunks without locking up sql?  Any other advice?

Thanks!
0
Comment
Question by:Tom_Hickerson
  • 7
  • 6
  • 2
  • +2
17 Comments
 
LVL 142

Expert Comment

by:Guy Hengel [angelIII / a3]
ID: 39856948
Does the table have a clustured index? If not create one.
Do you select the rows in gui? Don't, use a DELETE tablename WHERE .. statement.
Further tuning would be to index and eventually partition the table according to the fields based on which you want to decide which rows to keep/delete
0
 
LVL 34

Expert Comment

by:Paul MacDonald
ID: 39856991
As noted by [Guy Hengel], an index may go a long way toward addressing your performance problems, if you're not using one already.  If you have an index column, you could - again as already suggested...
   DELETE FROM tablename WHERE indexcolumn < somevalue

You could do something similar if you have a date column.
0
 
LVL 1

Author Comment

by:Tom_Hickerson
ID: 39857110
I have a date time field.  Here is the table

...GO
/****** Object:  Table [dbo].[opt_logs]    Script Date: 02/13/2014 13:56:24 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[opt_logs](
      [id] [int] IDENTITY(1,1) NOT NULL,
      [tag] [varchar](200) NOT NULL,
      [offset] [float] NOT NULL,
      [front_trim] [float] NOT NULL,
      [end_trim] [float] NOT NULL,
      [end_of_log] [float] NOT NULL,
      [butt_first] [bit] NOT NULL,
      [allow_export] [bit] NOT NULL,
      [selected_solution_id] [int] NULL,
      [settings_id] [int] NOT NULL,
      [datetime] [datetime] NOT NULL,
      [sort_solution] [varchar](200) NULL,
      [tag_first] [bit] NULL,
 CONSTRAINT [PK_opt_logs] PRIMARY KEY CLUSTERED
(
      [id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[opt_logs]  WITH CHECK ADD  CONSTRAINT [FK_opt_logs_opt_settings] FOREIGN KEY([settings_id])
REFERENCES [dbo].[opt_settings] ([id])
GO
ALTER TABLE [dbo].[opt_logs] CHECK CONSTRAINT [FK_opt_logs_opt_settings]
0
 
LVL 69

Expert Comment

by:ScottPletcher
ID: 39857116
You definitely do need to add a clustered index to the table if it doesn't have one.  To determine that, run the commands below and post the results:

USE <your_db_name>
EXEC sp_helpindex '<your_table_name>'


>> I have been going in and deleting the rows by opening the table in  sql management studio. <<

Because of the way the SSMS interface works, you definitely should not do that .


>> Is there a query I can run that will delete the rows in chunks without locking up sql?  Any other advice? <<

Not a single query but a script.  Once we know the clustering key, I can provide the script for you.
0
 
LVL 69

Expert Comment

by:ScottPletcher
ID: 39857131
OK, just saw the table definition.

How do you reference this table?  Is it by datetime or by id?

Often the clustered index on built on an identity column just by default, for no reason at all (except that some "experts" strongly push that idea).

If you reference the table by datetime, then change the clustered index to datetime.  You will get vastly better performance out of the table, whether it is 20,000 rows or 500,000 rows.


ALTER TABLE opt_logs DROP CONSTRAINT PK_opt_logs

CREATE CLUSTERED INDEX CI_opt_logs ON dbo.opt_logs ( datetime ) WITH ( FILLFACTOR = 99, ONLINE = ON /*if Enterprise Edition*/ ) ON [PRIMARY]
0
 
LVL 1

Author Comment

by:Tom_Hickerson
ID: 39857170
There is a vb app that access the database by id.  The date time is used by the vb app.

So do I run the snippet that scott suggests?
0
 
LVL 69

Expert Comment

by:ScottPletcher
ID: 39857303
If you actually do lookups by id, you could add a nonclustered index on id also:


ALTER TABLE opt_logs DROP CONSTRAINT PK_opt_logs

CREATE CLUSTERED INDEX CI_opt_logs ON dbo.opt_logs ( datetime ) WITH ( FILLFACTOR = 99, ONLINE = ON /*if Enterprise Edition*/ ) ON [PRIMARY]

CREATE UNIQUE NONCLUSTERED INDEX CX_opt_logs ON dbo.opt_logs ( id ) WITH ( FILLFACTOR = 99, ONLINE = ON /*if Enterprise Edition*/ ) ON [PRIMARY]
0
 
LVL 1

Author Comment

by:Tom_Hickerson
ID: 39860339
Scott,  I couldn't run what you had posted.  I got these errors.

Msg 1712, Level 16, State 1, Line 3
Online index operations can only be performed in Enterprise edition of SQL Server.
Msg 1712, Level 16, State 1, Line 5
Online index operations can only be performed in Enterprise edition of SQL Server.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 34

Expert Comment

by:Paul MacDonald
ID: 39860423
Thus his caveat about /*if Enterprise Edition*/.
0
 
LVL 69

Expert Comment

by:ScottPletcher
ID: 39860427
Sorry about that.  If it's not Enterprise Edition, we need to remove the ONLINE = ON option:


ALTER TABLE opt_logs DROP CONSTRAINT PK_opt_logs

CREATE CLUSTERED INDEX CI_opt_logs ON dbo.opt_logs ( datetime ) WITH ( FILLFACTOR = 99 ) ON [PRIMARY]

CREATE UNIQUE NONCLUSTERED INDEX CX_opt_logs ON dbo.opt_logs ( id ) WITH ( FILLFACTOR = 99 ) ON [PRIMARY]
0
 
LVL 1

Author Comment

by:Tom_Hickerson
ID: 39860456
More errors...
Msg 3725, Level 16, State 0, Line 1
The constraint 'PK_opt_logs' is being referenced by table 'opt_solutions', foreign key constraint 'FK_opt_solutions_opt_logs'.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
0
 
LVL 69

Expert Comment

by:ScottPletcher
ID: 39860482
Rats, FK(s) will be a real pain here.

You'll either have to live with the poor/bad performance of the table clustered as it currently is, or:

1) Script out all related FKs.
2) Temporarily drop the FKs.
3) Run the commands above to: drop the PK, create the new clus index, recreate the PK.
4) Recreate all the FKs.

This is a common situation, and there's no easy solution once the wrong clus index is in place.
0
 
LVL 1

Author Comment

by:Tom_Hickerson
ID: 39860544
What about deleting the old info without choking down sql?  Is there something I could do here besides going in and deleting 500 rows at a time?
0
 
LVL 69

Accepted Solution

by:
ScottPletcher earned 500 total points
ID: 39860599
OOPS, good point.  Here's a script to do that, 1000 rows at a time -- naturally adjust that number to whatever works best for you:



USE [<your_db_name>]

DECLARE @total_rows_in_table int
DECLARE @rows_deleted int
DECLARE @total_rows_deleted int
SELECT @total_rows_in_table = COUNT(*)
FROM dbo.opt_logs WITH (NOLOCK)
SET @rows_deleted = 0
SET @total_rows_deleted = 0

WHILE 1 = 1
BEGIN
    DELETE FROM ol
    FROM dbo.opt_logs ol
    INNER JOIN (
        SELECT TOP (1000) id
        FROM dbo.opt_logs WITH (NOLOCK)
        ORDER BY id
    ) AS ol_top ON
        ol_top.id = ol.id
    SET @rows_deleted = @@ROWCOUNT
    SET @total_rows_deleted = @total_rows_deleted + @rows_deleted
    -- keep DELETEing until the table reaches 20000 or fewer rows
    IF @total_rows_in_table - @total_rows_deleted <= 20000
        BREAK
    IF @total_rows_deleted % 100000 = 0
    BEGIN
        CHECKPOINT
    END --IF
    WAITFOR DELAY '00:00:00.150'
END --WHILE

PRINT 'Total Rows Deleted = ' + CAST(@total_rows_deleted AS varchar(10))
0
 
LVL 75

Expert Comment

by:Anthony Perkins
ID: 39862059
FK(s) will be a real pain here.
The author could also use the SSMS table designer to do this and it should take care of all the Foreign Keys.
0
 
LVL 1

Author Closing Comment

by:Tom_Hickerson
ID: 39865784
Awesome.  It only took 30 seconds to delete all the extra rows.

Thanks!
0
 
LVL 69

Expert Comment

by:ScottPletcher
ID: 39865805
You're welcome!

The key tip here is to use the clustered index whenever possible, because that results in the least I/O for SQL (and SQL has a natural bias to using the clustered index anyway).
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
MS SQL Inner Join - Multiple Join Parameters 2 23
Help Extract Specific in SQL 8 25
recover sqlserver db 8 55
ebay table structure 2 14
Introduction SQL Server Integration Services can read XML files, that’s known by every BI developer.  (If you didn’t, don’t worry, I’m aiming this article at newcomers as well.) But how far can you go?  When does the XML Source component become …
Slowly Changing Dimension Transformation component in data task flow is very useful for us to manage and control how data changes in SSIS.
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…

896 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now