Solved

SQL Distinct List for fields in very large tables

Posted on 2014-11-02
4
156 Views
Last Modified: 2014-11-03
Hi

I need to create a distinct list of values for columns in tables that are between 5 and 10 million records.
These lists can be up to 200,000 records long. At the moment I am creating them while my app runs and
it takes a long time. How do I get around this? Should I create views or tables that have a distinct list?
0
Comment
Question by:murbro
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
4 Comments
 
LVL 48

Accepted Solution

by:
PortletPaul earned 250 total points
ID: 40418824
There is no detail information to work with, so a generic answer would be:

Views will probably be faster than what you do now; Tables would be faster than views.

Indexes on the source table will have a big influence on performance.
0
 
LVL 10

Expert Comment

by:HuaMinChen
ID: 40418866
Hi,
1. put "distinct" to the statement
2. You can then select the records into other table, on which there is table constraint that is ensuring uniqueness of all records.
0
 
LVL 9

Assisted Solution

by:Valliappan AN
Valliappan AN earned 250 total points
ID: 40418920
You can use (NOLOCK) attribute to your select query something like this, to avoid locking of all those records while fetching, which will dramatically give good performance. Something like this:

Select DISTINCT field1, field2,...,fieldn FROM [yourtable] (NOLOCK)
JOIN [anothertable] (NOLOCK) ON <join condition>
 
HTH.
0
 

Author Closing Comment

by:murbro
ID: 40419367
Thanks
0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Introduction SQL Server Integration Services can read XML files, that’s known by every BI developer.  (If you didn’t, don’t worry, I’m aiming this article at newcomers as well.) But how far can you go?  When does the XML Source component become …
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.

735 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question