[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Sql 2008 Remove unused / duplicate indexes

Posted on 2014-08-12
13
Medium Priority
?
233 Views
Last Modified: 2014-08-18
I've inherited a 600GB database that has several large (1 million+ records) tables that have 100+ indexes on each table. I've been reading online about index tuning but everything seems to apply to starting from nothing and creating indexes not reduction. I've tried out a couple scripts that identify unused and duplicate indexes but when I begin removing indexes identified as duplicate the software that uses the database grinds to a halt. Outside of hiring a dba to handle this, can anyone point a programmer struggling with a beast of a database in the right direction?
0
Comment
Question by:CAHFS
  • 5
  • 4
  • 2
  • +1
13 Comments
 
LVL 35

Assisted Solution

by:David Todd
David Todd earned 600 total points
ID: 40257151
Hi,

To start the discussion off

Are you deleting any clustered indexes? I'd avoid that on big tables for starters.

Do you update the statistics on tables after you delete a duplication index?

Are you doing this out of hours or during business hours?

Regards
   David
0
 

Author Comment

by:CAHFS
ID: 40257169
David,

No, avoiding deletion of clustered indexes until I feel like I have a better understanding of what I'm doing.

I have not updated statistics on any table after deleting an index. This, this is probably what I need to do most right?

I'm working on a dev system that is running on its own server so no issues interrupting users.

Scott
0
 
LVL 43

Expert Comment

by:Eugene Z
ID: 40257192
it is not clear what is the issue: please clarify if you have any performance issues?
was the DB restored on this server or was here before?
when did you run reindex of this DB last time?
check how many  Free GB in mdf\ldf files: maybe you just need to shrink files if there are too much free space?
---

if you really plan to delete:
before you start deleting anything from this database:
   you must identify what are the largest DBs \indexes there
what tables can be deleted or truncated
what indexes can be deleted
all these activities are normally coordinated with application developers\BAs\DAs
--

you can start from checking standard SSMS reports to see the top tables sizes \records:
SQL Server Management Studio and right-clicking a database name. From the menu that appears, click the Standard Reports, and then select  one of reports: Disk Usage by Table/Top Tables

and always make sure you have backup before you delete anything
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 35

Expert Comment

by:David Todd
ID: 40257366
Hi,

What version of SQL are you running on? Is dev the same as production? Is it running the same _edition_ as production?

HTH
  David
0
 

Author Comment

by:CAHFS
ID: 40258397
Running Sql server 10.50.2550, both dev and prod are the same. It generally runs with ~20GB free space, manually grown via script as needed. I also run another script nightly that checks for index fragmentation and will reorganize if fragmentation is greater than 10%, it will rebuild if greater than 30%.

The reason I am wanting to delete some of the indexes is because many of them appear to be duplicates and/or are very similar to other indexes on the same table. Also, should any single table have over 100 indexes? It seems to me that indexes have been created by blindly following the profilers recommendations.
0
 
LVL 70

Expert Comment

by:Scott Pletcher
ID: 40258640
I've been tuning dbs for decades.  In general, the wrong clustered index -- i.e., one on identity -- is the major culprit leading to bad performance and requiring gazillions of extra indexes.  

But, the starting point is always the same.  We first need to look at the missing index and index usage stats that SQL provides (at an absolute minimum; there are other things we can look at, but this will provide the core).  Then we'll have real numbers to guide us.  Run the script below to get that info.

[Yes, it's a pain changing the clustered index, but it's often the only way to get best performance overall.  Otherwise, getting even moderately good performance requires doing an intense analysis and rewrite of every query, one by one ... yikes!]



USE [<your_db_name_here>]

SET DEADLOCK_PRIORITY LOW --probably irrelevant, but just in case

DECLARE @list_missing_indexes bit
DECLARE @table_name_pattern sysname

--NOTE: showing missing indexes can take some time; set to 0 if you don't want to wait.
SET @list_missing_indexes = 1
SET @table_name_pattern = '%' --'%'=all tables.
--SET @table_name_pattern = '%'

PRINT 'Started @ ' + CONVERT(varchar(30), GETDATE(), 120)

IF @list_missing_indexes = 1
BEGIN
    SELECT
        GETDATE() AS capture_date,
        DB_NAME(mid.database_id) AS Db_Name,
        dps.row_count,
        OBJECT_NAME(mid.object_id /*, mid.database_id*/) AS Table_Name,
        mid.equality_columns, mid.inequality_columns, mid.included_columns,
        user_seeks, user_scans, ca1.max_days_active, unique_compiles,
        last_user_seek, last_user_scan, avg_total_user_cost, avg_user_impact,
        system_seeks, system_scans, last_system_seek, last_system_scan, avg_total_system_cost, avg_system_impact,
        mid.statement, mid.object_id, mid.index_handle
    FROM sys.dm_db_missing_index_details mid WITH (NOLOCK)
    CROSS APPLY (
        SELECT DATEDIFF(DAY, create_date, GETDATE()) AS max_days_active FROM sys.databases WHERE name = 'tempdb'
    ) AS ca1
    LEFT OUTER JOIN sys.dm_db_missing_index_groups mig WITH (NOLOCK) ON
        mig.index_handle = mid.index_handle
    LEFT OUTER JOIN sys.dm_db_missing_index_group_stats migs WITH (NOLOCK) ON
        migs.group_handle = mig.index_group_handle
    LEFT OUTER JOIN sys.dm_db_partition_stats dps WITH (NOLOCK) ON
        dps.object_id = mid.object_id AND
        dps.index_id IN (0, 1)
    --order by
        --DB_NAME, Table_Name, equality_columns
    WHERE
        1 = 1
        AND mid.database_id = DB_ID() --only current db
        AND OBJECT_NAME(mid.object_id) LIKE @table_name_pattern
        --AND mid.object_id IN (OBJECT_ID('<table_name_1>'), OBJECT_ID('<table_name_2>'))
    ORDER BY
        --avg_total_user_cost * (user_seeks + user_scans) DESC,
        Db_Name, Table_Name, equality_columns, inequality_columns
END --IF

PRINT 'Midpoint @ ' + CONVERT(varchar(30), GETDATE(), 120)
-- list index usage stats (seeks, scans, etc.)
SELECT
    ius2.row_num, DB_NAME() AS db_name,
    CASE WHEN i.name LIKE ca2.table_name + '%'
         THEN '~' + SUBSTRING(i.name, LEN(ca2.table_name) + 1 +
                  CASE WHEN SUBSTRING(i.name, LEN(ca2.table_name) + 1, 1) = '_' THEN
                      CASE WHEN SUBSTRING(i.name, LEN(ca2.table_name) + 2, 1) = '_' THEN 2 ELSE 1 END
                  ELSE 0 END, 200)
         ELSE i.name END AS index_name,
    CASE WHEN i.is_unique = 0 THEN 'N' ELSE 'Y' END + '.' +
    CASE WHEN i.is_primary_key = 0 AND i.is_unique_constraint = 0 THEN 'N' ELSE 'Y' END AS [uniq?],
    ca2.table_name,
    i.index_id, --ius.user_seeks + ius.user_scans AS total_reads,
    dps.row_count,
    SUBSTRING(key_cols, 3, 8000) AS key_cols, SUBSTRING(nonkey_cols, 3, 8000) AS nonkey_cols,
    ius.user_seeks, ius.user_scans, ius.user_lookups, ius.user_updates,
    ius.last_user_seek, ius.last_user_scan, ius.last_user_lookup, ius.last_user_update,
    fk.Reference_Count AS fk_ref_count,
    DATEDIFF(DAY, CASE WHEN o.create_date > ca1.sql_startup_date THEN o.create_date
        ELSE ca1.sql_startup_date END, GETDATE()) AS max_days_active,
    FILEGROUP_NAME(i.data_space_id) AS filegroup_name,
    ius.system_seeks, ius.system_scans, ius.system_lookups, ius.system_updates,
    ius.last_system_seek, ius.last_system_scan, ius.last_system_lookup, ius.last_system_update
FROM sys.indexes i WITH (NOLOCK)
INNER JOIN sys.objects o WITH (NOLOCK) ON
    o.object_id = i.object_id
CROSS JOIN (
    SELECT create_date AS sql_startup_date FROM sys.databases WHERE name = 'tempdb'
) AS ca1
CROSS APPLY (
    SELECT OBJECT_NAME(i.object_id/*, DB_ID()*/) AS table_name
) AS ca2
OUTER APPLY (
    SELECT
        ', ' + COL_NAME(object_id, ic.column_id)
    FROM sys.index_columns ic
    WHERE
        ic.key_ordinal > 0 AND
        ic.object_id = i.object_id AND
        ic.index_id = i.index_id
    ORDER BY
        ic.key_ordinal
    FOR XML PATH('')
) AS key_cols (key_cols)
OUTER APPLY (
    SELECT
        ', ' + COL_NAME(object_id, ic.column_id)
    FROM sys.index_columns ic
    WHERE
        ic.key_ordinal = 0 AND
        ic.object_id = i.object_id AND
        ic.index_id = i.index_id
    ORDER BY
        COL_NAME(object_id, ic.column_id)
    FOR XML PATH('')
) AS nonkey_cols (nonkey_cols)
LEFT OUTER JOIN sys.dm_db_partition_stats dps WITH (NOLOCK) ON
    dps.object_id = i.object_id AND
    dps.index_id = i.index_id
LEFT OUTER JOIN sys.dm_db_index_usage_stats ius WITH (NOLOCK) ON
    ius.database_id = DB_ID() AND
    ius.object_id = i.object_id AND
    ius.index_id = i.index_id
LEFT OUTER JOIN (
    SELECT
        database_id, object_id, MAX(user_scans) AS user_scans,
        ROW_NUMBER() OVER (ORDER BY MAX(user_scans) DESC) AS row_num --user_scans|user_seeks+user_scans
    FROM sys.dm_db_index_usage_stats WITH (NOLOCK)
    WHERE
        database_id = DB_ID()
        --AND index_id > 0
    GROUP BY
        database_id, object_id
) AS ius2 ON
    ius2.database_id = DB_ID() AND
    ius2.object_id = i.object_id
LEFT OUTER JOIN (
    SELECT
        referenced_object_id, COUNT(*) AS Reference_Count
    FROM sys.foreign_keys
    WHERE
        is_disabled = 0
    GROUP BY
        referenced_object_id
) AS fk ON
    fk.referenced_object_id = i.object_id
WHERE
    i.object_id > 100 AND
    i.is_hypothetical = 0 AND
    i.type IN (0, 1, 2) AND
    o.type NOT IN ( 'IF', 'IT', 'TF', 'TT' ) AND
    (
     o.name LIKE @table_name_pattern AND
     o.name NOT LIKE 'dtprop%' AND
     o.name NOT LIKE 'filestream[_]' AND
     o.name NOT LIKE 'MSpeer%' AND
     o.name NOT LIKE 'MSpub%' AND
     --o.name NOT LIKE 'queue[_]%' AND
     o.name NOT LIKE 'sys%'
    )
    --AND OBJECT_NAME(i.object_id /*, DB_ID()*/) IN ('tbl1', 'tbl2', 'tbl3')
ORDER BY
    --row_count DESC,
    --ius.user_scans DESC,
    --ius2.row_num, --user_scans&|user_seeks
    db_name, table_name,
    -- list clustered index first, if any, then other index(es)
    CASE WHEN i.index_id IN (0, 1) THEN 1 ELSE 2 END,
    key_cols

PRINT 'Ended @ ' + CONVERT(varchar(30), GETDATE(), 120)

SET DEADLOCK_PRIORITY NORMAL
0
 

Author Comment

by:CAHFS
ID: 40258885
I checked for clustered indexes on identity fields and did not find any so that's a good thing.

I ran the script but I don't understand what I'm looking at. Where to go from here?
0
 
LVL 70

Expert Comment

by:Scott Pletcher
ID: 40258895
Please post the results.  You can change the table/index names if you need to.
0
 

Author Comment

by:CAHFS
ID: 40258982
Results are attachedindex-data-08132014.xlsx
0
 
LVL 70

Accepted Solution

by:
Scott Pletcher earned 900 total points
ID: 40259106
Wow, that's a lot.

But, overall, the clustered indexes do look good.

As to indexes to remove, look at the second spreadsheet, and those that have no user seeks or scans should be able to be safely removed, esp. if they a similar index already exists.  I don't have much time now, but can look at this in more detail later if you want.

Also, these changes might also help overall performance:

----------------------------------------------------------------------------------------------------

DROP INDEX [IX_BATCHREPORTLOG02] ON BATCHREPORTLOG
ALTER TABLE BATCHREPORTLOG DROP CONSTRAINT PK_BATCHREPORTLOG
ALTER TABLE BATCHREPORTLOG ADD
    CONSTRAINT PK_BATCHREPORTLOG
    PRIMARY KEY ( REPORTGIUD, REQUESTSOURCE )

----------------------------------------------------------------------------------------------------

Change clustering index on::
C_GP_INVOICEDETAILS == ( INVOICEID )
    --naturally you won't recreate the existing nonclustered index on ( INVOICEID, STATUS )
    [ORIGREC can still be the PK, but it should not be the clus index]
C_GP_INVOICES == ( INVOICEID )
    --naturally you won't recreate the existing nonclustered index on ( INVOICEID )
    Create a new nonclustered index on ( FOLDERNO, VERSION )
COC == ( FROMLAB )
    --don't recreate ( STATUS, FROMLAB )
INVOICES = ( INVOICEID )
--Review SP_ANALYTES table for best clustered index

----------------------------------------------------------------------------------------------------
0
 

Author Comment

by:CAHFS
ID: 40259129
@ScottPletcher - Thank you for taking the time to review it in the first place. I'll get to work and let you know if I have any questions.
0
 
LVL 43

Expert Comment

by:Eugene Z
ID: 40259787
CAHFS: it is good that you try to do cleanup
the overall your indexes are "ok" ..those that are 100% identical - delete.
if it is a prod db- all actions must be tested on dev server via application report:
you need to know your system very well.
some "unused indexes" may be used by some EOY process...


please read my above post and check tables that you can delete, maybe compress to save space
0
 
LVL 70

Expert Comment

by:Scott Pletcher
ID: 40260593
You don't need to maintain indexes all year for an "EOY" process(es).

Either let that process(es) scan the table, or create the index(es) before the process, and remove them afterward.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
MSSQL DB-maintenance also needs implementation of multiple activities. However, unprecedented errors can hamper the database management. In that case, deploying Stellar SQL Database Toolkit ensures fast and accurate database and backup repair as wel…
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…
Suggested Courses

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question