Sql 2008 Remove unused / duplicate indexes

I've inherited a 600GB database that has several large (1 million+ records) tables that have 100+ indexes on each table. I've been reading online about index tuning but everything seems to apply to starting from nothing and creating indexes not reduction. I've tried out a couple scripts that identify unused and duplicate indexes but when I begin removing indexes identified as duplicate the software that uses the database grinds to a halt. Outside of hiring a dba to handle this, can anyone point a programmer struggling with a beast of a database in the right direction?
CAHFSAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David ToddSenior DBACommented:
Hi,

To start the discussion off

Are you deleting any clustered indexes? I'd avoid that on big tables for starters.

Do you update the statistics on tables after you delete a duplication index?

Are you doing this out of hours or during business hours?

Regards
   David
0
CAHFSAuthor Commented:
David,

No, avoiding deletion of clustered indexes until I feel like I have a better understanding of what I'm doing.

I have not updated statistics on any table after deleting an index. This, this is probably what I need to do most right?

I'm working on a dev system that is running on its own server so no issues interrupting users.

Scott
0
Eugene ZCommented:
it is not clear what is the issue: please clarify if you have any performance issues?
was the DB restored on this server or was here before?
when did you run reindex of this DB last time?
check how many  Free GB in mdf\ldf files: maybe you just need to shrink files if there are too much free space?
---

if you really plan to delete:
before you start deleting anything from this database:
   you must identify what are the largest DBs \indexes there
what tables can be deleted or truncated
what indexes can be deleted
all these activities are normally coordinated with application developers\BAs\DAs
--

you can start from checking standard SSMS reports to see the top tables sizes \records:
SQL Server Management Studio and right-clicking a database name. From the menu that appears, click the Standard Reports, and then select  one of reports: Disk Usage by Table/Top Tables

and always make sure you have backup before you delete anything
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

David ToddSenior DBACommented:
Hi,

What version of SQL are you running on? Is dev the same as production? Is it running the same _edition_ as production?

HTH
  David
0
CAHFSAuthor Commented:
Running Sql server 10.50.2550, both dev and prod are the same. It generally runs with ~20GB free space, manually grown via script as needed. I also run another script nightly that checks for index fragmentation and will reorganize if fragmentation is greater than 10%, it will rebuild if greater than 30%.

The reason I am wanting to delete some of the indexes is because many of them appear to be duplicates and/or are very similar to other indexes on the same table. Also, should any single table have over 100 indexes? It seems to me that indexes have been created by blindly following the profilers recommendations.
0
Scott PletcherSenior DBACommented:
I've been tuning dbs for decades.  In general, the wrong clustered index -- i.e., one on identity -- is the major culprit leading to bad performance and requiring gazillions of extra indexes.  

But, the starting point is always the same.  We first need to look at the missing index and index usage stats that SQL provides (at an absolute minimum; there are other things we can look at, but this will provide the core).  Then we'll have real numbers to guide us.  Run the script below to get that info.

[Yes, it's a pain changing the clustered index, but it's often the only way to get best performance overall.  Otherwise, getting even moderately good performance requires doing an intense analysis and rewrite of every query, one by one ... yikes!]



USE [<your_db_name_here>]

SET DEADLOCK_PRIORITY LOW --probably irrelevant, but just in case

DECLARE @list_missing_indexes bit
DECLARE @table_name_pattern sysname

--NOTE: showing missing indexes can take some time; set to 0 if you don't want to wait.
SET @list_missing_indexes = 1
SET @table_name_pattern = '%' --'%'=all tables.
--SET @table_name_pattern = '%'

PRINT 'Started @ ' + CONVERT(varchar(30), GETDATE(), 120)

IF @list_missing_indexes = 1
BEGIN
    SELECT
        GETDATE() AS capture_date,
        DB_NAME(mid.database_id) AS Db_Name,
        dps.row_count,
        OBJECT_NAME(mid.object_id /*, mid.database_id*/) AS Table_Name,
        mid.equality_columns, mid.inequality_columns, mid.included_columns,
        user_seeks, user_scans, ca1.max_days_active, unique_compiles,
        last_user_seek, last_user_scan, avg_total_user_cost, avg_user_impact,
        system_seeks, system_scans, last_system_seek, last_system_scan, avg_total_system_cost, avg_system_impact,
        mid.statement, mid.object_id, mid.index_handle
    FROM sys.dm_db_missing_index_details mid WITH (NOLOCK)
    CROSS APPLY (
        SELECT DATEDIFF(DAY, create_date, GETDATE()) AS max_days_active FROM sys.databases WHERE name = 'tempdb'
    ) AS ca1
    LEFT OUTER JOIN sys.dm_db_missing_index_groups mig WITH (NOLOCK) ON
        mig.index_handle = mid.index_handle
    LEFT OUTER JOIN sys.dm_db_missing_index_group_stats migs WITH (NOLOCK) ON
        migs.group_handle = mig.index_group_handle
    LEFT OUTER JOIN sys.dm_db_partition_stats dps WITH (NOLOCK) ON
        dps.object_id = mid.object_id AND
        dps.index_id IN (0, 1)
    --order by
        --DB_NAME, Table_Name, equality_columns
    WHERE
        1 = 1
        AND mid.database_id = DB_ID() --only current db
        AND OBJECT_NAME(mid.object_id) LIKE @table_name_pattern
        --AND mid.object_id IN (OBJECT_ID('<table_name_1>'), OBJECT_ID('<table_name_2>'))
    ORDER BY
        --avg_total_user_cost * (user_seeks + user_scans) DESC,
        Db_Name, Table_Name, equality_columns, inequality_columns
END --IF

PRINT 'Midpoint @ ' + CONVERT(varchar(30), GETDATE(), 120)
-- list index usage stats (seeks, scans, etc.)
SELECT
    ius2.row_num, DB_NAME() AS db_name,
    CASE WHEN i.name LIKE ca2.table_name + '%'
         THEN '~' + SUBSTRING(i.name, LEN(ca2.table_name) + 1 +
                  CASE WHEN SUBSTRING(i.name, LEN(ca2.table_name) + 1, 1) = '_' THEN
                      CASE WHEN SUBSTRING(i.name, LEN(ca2.table_name) + 2, 1) = '_' THEN 2 ELSE 1 END
                  ELSE 0 END, 200)
         ELSE i.name END AS index_name,
    CASE WHEN i.is_unique = 0 THEN 'N' ELSE 'Y' END + '.' +
    CASE WHEN i.is_primary_key = 0 AND i.is_unique_constraint = 0 THEN 'N' ELSE 'Y' END AS [uniq?],
    ca2.table_name,
    i.index_id, --ius.user_seeks + ius.user_scans AS total_reads,
    dps.row_count,
    SUBSTRING(key_cols, 3, 8000) AS key_cols, SUBSTRING(nonkey_cols, 3, 8000) AS nonkey_cols,
    ius.user_seeks, ius.user_scans, ius.user_lookups, ius.user_updates,
    ius.last_user_seek, ius.last_user_scan, ius.last_user_lookup, ius.last_user_update,
    fk.Reference_Count AS fk_ref_count,
    DATEDIFF(DAY, CASE WHEN o.create_date > ca1.sql_startup_date THEN o.create_date
        ELSE ca1.sql_startup_date END, GETDATE()) AS max_days_active,
    FILEGROUP_NAME(i.data_space_id) AS filegroup_name,
    ius.system_seeks, ius.system_scans, ius.system_lookups, ius.system_updates,
    ius.last_system_seek, ius.last_system_scan, ius.last_system_lookup, ius.last_system_update
FROM sys.indexes i WITH (NOLOCK)
INNER JOIN sys.objects o WITH (NOLOCK) ON
    o.object_id = i.object_id
CROSS JOIN (
    SELECT create_date AS sql_startup_date FROM sys.databases WHERE name = 'tempdb'
) AS ca1
CROSS APPLY (
    SELECT OBJECT_NAME(i.object_id/*, DB_ID()*/) AS table_name
) AS ca2
OUTER APPLY (
    SELECT
        ', ' + COL_NAME(object_id, ic.column_id)
    FROM sys.index_columns ic
    WHERE
        ic.key_ordinal > 0 AND
        ic.object_id = i.object_id AND
        ic.index_id = i.index_id
    ORDER BY
        ic.key_ordinal
    FOR XML PATH('')
) AS key_cols (key_cols)
OUTER APPLY (
    SELECT
        ', ' + COL_NAME(object_id, ic.column_id)
    FROM sys.index_columns ic
    WHERE
        ic.key_ordinal = 0 AND
        ic.object_id = i.object_id AND
        ic.index_id = i.index_id
    ORDER BY
        COL_NAME(object_id, ic.column_id)
    FOR XML PATH('')
) AS nonkey_cols (nonkey_cols)
LEFT OUTER JOIN sys.dm_db_partition_stats dps WITH (NOLOCK) ON
    dps.object_id = i.object_id AND
    dps.index_id = i.index_id
LEFT OUTER JOIN sys.dm_db_index_usage_stats ius WITH (NOLOCK) ON
    ius.database_id = DB_ID() AND
    ius.object_id = i.object_id AND
    ius.index_id = i.index_id
LEFT OUTER JOIN (
    SELECT
        database_id, object_id, MAX(user_scans) AS user_scans,
        ROW_NUMBER() OVER (ORDER BY MAX(user_scans) DESC) AS row_num --user_scans|user_seeks+user_scans
    FROM sys.dm_db_index_usage_stats WITH (NOLOCK)
    WHERE
        database_id = DB_ID()
        --AND index_id > 0
    GROUP BY
        database_id, object_id
) AS ius2 ON
    ius2.database_id = DB_ID() AND
    ius2.object_id = i.object_id
LEFT OUTER JOIN (
    SELECT
        referenced_object_id, COUNT(*) AS Reference_Count
    FROM sys.foreign_keys
    WHERE
        is_disabled = 0
    GROUP BY
        referenced_object_id
) AS fk ON
    fk.referenced_object_id = i.object_id
WHERE
    i.object_id > 100 AND
    i.is_hypothetical = 0 AND
    i.type IN (0, 1, 2) AND
    o.type NOT IN ( 'IF', 'IT', 'TF', 'TT' ) AND
    (
     o.name LIKE @table_name_pattern AND
     o.name NOT LIKE 'dtprop%' AND
     o.name NOT LIKE 'filestream[_]' AND
     o.name NOT LIKE 'MSpeer%' AND
     o.name NOT LIKE 'MSpub%' AND
     --o.name NOT LIKE 'queue[_]%' AND
     o.name NOT LIKE 'sys%'
    )
    --AND OBJECT_NAME(i.object_id /*, DB_ID()*/) IN ('tbl1', 'tbl2', 'tbl3')
ORDER BY
    --row_count DESC,
    --ius.user_scans DESC,
    --ius2.row_num, --user_scans&|user_seeks
    db_name, table_name,
    -- list clustered index first, if any, then other index(es)
    CASE WHEN i.index_id IN (0, 1) THEN 1 ELSE 2 END,
    key_cols

PRINT 'Ended @ ' + CONVERT(varchar(30), GETDATE(), 120)

SET DEADLOCK_PRIORITY NORMAL
0
CAHFSAuthor Commented:
I checked for clustered indexes on identity fields and did not find any so that's a good thing.

I ran the script but I don't understand what I'm looking at. Where to go from here?
0
Scott PletcherSenior DBACommented:
Please post the results.  You can change the table/index names if you need to.
0
CAHFSAuthor Commented:
Results are attachedindex-data-08132014.xlsx
0
Scott PletcherSenior DBACommented:
Wow, that's a lot.

But, overall, the clustered indexes do look good.

As to indexes to remove, look at the second spreadsheet, and those that have no user seeks or scans should be able to be safely removed, esp. if they a similar index already exists.  I don't have much time now, but can look at this in more detail later if you want.

Also, these changes might also help overall performance:

----------------------------------------------------------------------------------------------------

DROP INDEX [IX_BATCHREPORTLOG02] ON BATCHREPORTLOG
ALTER TABLE BATCHREPORTLOG DROP CONSTRAINT PK_BATCHREPORTLOG
ALTER TABLE BATCHREPORTLOG ADD
    CONSTRAINT PK_BATCHREPORTLOG
    PRIMARY KEY ( REPORTGIUD, REQUESTSOURCE )

----------------------------------------------------------------------------------------------------

Change clustering index on::
C_GP_INVOICEDETAILS == ( INVOICEID )
    --naturally you won't recreate the existing nonclustered index on ( INVOICEID, STATUS )
    [ORIGREC can still be the PK, but it should not be the clus index]
C_GP_INVOICES == ( INVOICEID )
    --naturally you won't recreate the existing nonclustered index on ( INVOICEID )
    Create a new nonclustered index on ( FOLDERNO, VERSION )
COC == ( FROMLAB )
    --don't recreate ( STATUS, FROMLAB )
INVOICES = ( INVOICEID )
--Review SP_ANALYTES table for best clustered index

----------------------------------------------------------------------------------------------------
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
CAHFSAuthor Commented:
@ScottPletcher - Thank you for taking the time to review it in the first place. I'll get to work and let you know if I have any questions.
0
Eugene ZCommented:
CAHFS: it is good that you try to do cleanup
the overall your indexes are "ok" ..those that are 100% identical - delete.
if it is a prod db- all actions must be tested on dev server via application report:
you need to know your system very well.
some "unused indexes" may be used by some EOY process...


please read my above post and check tables that you can delete, maybe compress to save space
0
Scott PletcherSenior DBACommented:
You don't need to maintain indexes all year for an "EOY" process(es).

Either let that process(es) scan the table, or create the index(es) before the process, and remove them afterward.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server 2008

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.