finding duplicate rows using SQL loader (Bulk Load)

I want to load records into table from a file.
I want to find out the duplicate records in a table where in i have defined primary key.
now while loading the data into table it will reject duplicate records because of the constraint and
put all those duplicate recrods in .dsc file...

but if i use BULK Load option in SQL*Loader.. as i understand it will first disable table constraints and then load data..
because of this i cannot find out the duplcate rows...

i have tried out that using BULK load option performance is more than 60% higher than conventional loading method using SQL*Loader ..

now any body can please help me out finding duplicate rows with same performance what i am getting with bulk load??

it is very urgent..
thanks in advance




NagsAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

andrewstCommented:
By "BULK load" do you mean direct path load?

If so, UNIQUE and PRIMARY KEY constraints are NOT disabled during direct path load, only CHECK and FOREIGN KEY constraints are disabled - and they will be automatically re-enabled afterwards if you use the REENABLE clause.

So you can use direct path load and still find duplicates.

0
i014354Commented:
Use SQL*Loader to load into a temporary table first, then delete the duplicates from the temp table.  You can then either SELECT INTO...  or CREATE TABLE AS SELECT from the temp table.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Mark GeerlingsDatabase AdministratorCommented:
Yes, the "direct path" load is much faster than a conventional data load (which does an insert for each row or set of rows) but there are some limitations with the "direct path" load.  You may have to decide what is more important to you:
row-by-row processing
or
speed of the load.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

NagsAuthor Commented:
thanks for the solutions. i am worried about the speed of the load so i  use "direct path".

i still have some problem..
ie. in the input file two fields are null and in the table
those fields are number not null ..

now whiell loading i want to translate null to 0..
for this i can use DECODE in control file.. but with "direct path". this will not be allowed..
how will i do this?????
0
andrewstCommented:
So you want all the speed of the direct path load, but without any of the limitations?  Think about it.  If that was possible, Oracle would make direct load work that way, wouldn't they?

You have a few choices:

1) Use direct path load into temporary table, then have a program to move the data from the temporary table into the real table, checking for constraint violations.

2) Use direct path load into the real table, then sort out the constraint violations before re-enabling the constraints.

3) Don't use direct path load.  You said the performance difference was "more than 60%".  But after fixing the constraint violations in options (1) and (2), you may well find all that gain has been lost, and more.

Why not experiment with all 3 approaches and see which is fastest in fact to load and validate the data?
0
Mark GeerlingsDatabase AdministratorCommented:
Here are a couple more options that may work:
1. Use a text editor on the data file to replace the nulls (or spaces) with 0.

2. Use direct-path load into a work table, clean up the data, then use SQL*Plus to spool it out to another ASCII file that you load into your target table with direct path.
0
anand_2000vCommented:
No comment has been added lately, so it's time to clean up this TA.
I will leave the following recommendation for this question in the Cleanup topic area:

Split: i014354 {http:#8121042} & andrewst {http:#8126624} & markgeer {http:#8136682}

Please leave any comments here within the next seven days.
PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

anand_2000v
EE Cleanup Volunteer
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Oracle Database

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.