?
Solved

SQL Distinct List for fields in very large tables

Posted on 2014-11-02
4
Medium Priority
?
172 Views
Last Modified: 2014-11-03
Hi

I need to create a distinct list of values for columns in tables that are between 5 and 10 million records.
These lists can be up to 200,000 records long. At the moment I am creating them while my app runs and
it takes a long time. How do I get around this? Should I create views or tables that have a distinct list?
0
Comment
Question by:Murray Brown
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
4 Comments
 
LVL 49

Accepted Solution

by:
PortletPaul earned 1000 total points
ID: 40418824
There is no detail information to work with, so a generic answer would be:

Views will probably be faster than what you do now; Tables would be faster than views.

Indexes on the source table will have a big influence on performance.
0
 
LVL 11

Expert Comment

by:HuaMinChen
ID: 40418866
Hi,
1. put "distinct" to the statement
2. You can then select the records into other table, on which there is table constraint that is ensuring uniqueness of all records.
0
 
LVL 9

Assisted Solution

by:Valliappan AN
Valliappan AN earned 1000 total points
ID: 40418920
You can use (NOLOCK) attribute to your select query something like this, to avoid locking of all those records while fetching, which will dramatically give good performance. Something like this:

Select DISTINCT field1, field2,...,fieldn FROM [yourtable] (NOLOCK)
JOIN [anothertable] (NOLOCK) ON <join condition>
 
HTH.
0
 

Author Closing Comment

by:Murray Brown
ID: 40419367
Thanks
0

Featured Post

Get proactive database performance tuning online

At Percona’s web store you can order full Percona Database Performance Audit in minutes. Find out the health of your database, and how to improve it. Pay online with a credit card. Improve your database performance now!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.
Suggested Courses

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question