Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2656
  • Last Modified:

finding duplicate rows using SQL loader (Bulk Load)

I want to load records into table from a file.
I want to find out the duplicate records in a table where in i have defined primary key.
now while loading the data into table it will reject duplicate records because of the constraint and
put all those duplicate recrods in .dsc file...

but if i use BULK Load option in SQL*Loader.. as i understand it will first disable table constraints and then load data..
because of this i cannot find out the duplcate rows...

i have tried out that using BULK load option performance is more than 60% higher than conventional loading method using SQL*Loader ..

now any body can please help me out finding duplicate rows with same performance what i am getting with bulk load??

it is very urgent..
thanks in advance




0
Nags
Asked:
Nags
3 Solutions
 
andrewstCommented:
By "BULK load" do you mean direct path load?

If so, UNIQUE and PRIMARY KEY constraints are NOT disabled during direct path load, only CHECK and FOREIGN KEY constraints are disabled - and they will be automatically re-enabled afterwards if you use the REENABLE clause.

So you can use direct path load and still find duplicates.

0
 
i014354Commented:
Use SQL*Loader to load into a temporary table first, then delete the duplicates from the temp table.  You can then either SELECT INTO...  or CREATE TABLE AS SELECT from the temp table.
0
 
Mark GeerlingsDatabase AdministratorCommented:
Yes, the "direct path" load is much faster than a conventional data load (which does an insert for each row or set of rows) but there are some limitations with the "direct path" load.  You may have to decide what is more important to you:
row-by-row processing
or
speed of the load.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
NagsAuthor Commented:
thanks for the solutions. i am worried about the speed of the load so i  use "direct path".

i still have some problem..
ie. in the input file two fields are null and in the table
those fields are number not null ..

now whiell loading i want to translate null to 0..
for this i can use DECODE in control file.. but with "direct path". this will not be allowed..
how will i do this?????
0
 
andrewstCommented:
So you want all the speed of the direct path load, but without any of the limitations?  Think about it.  If that was possible, Oracle would make direct load work that way, wouldn't they?

You have a few choices:

1) Use direct path load into temporary table, then have a program to move the data from the temporary table into the real table, checking for constraint violations.

2) Use direct path load into the real table, then sort out the constraint violations before re-enabling the constraints.

3) Don't use direct path load.  You said the performance difference was "more than 60%".  But after fixing the constraint violations in options (1) and (2), you may well find all that gain has been lost, and more.

Why not experiment with all 3 approaches and see which is fastest in fact to load and validate the data?
0
 
Mark GeerlingsDatabase AdministratorCommented:
Here are a couple more options that may work:
1. Use a text editor on the data file to replace the nulls (or spaces) with 0.

2. Use direct-path load into a work table, clean up the data, then use SQL*Plus to spool it out to another ASCII file that you load into your target table with direct path.
0
 
anand_2000vCommented:
No comment has been added lately, so it's time to clean up this TA.
I will leave the following recommendation for this question in the Cleanup topic area:

Split: i014354 {http:#8121042} & andrewst {http:#8126624} & markgeer {http:#8136682}

Please leave any comments here within the next seven days.
PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

anand_2000v
EE Cleanup Volunteer
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now