Link to home
Start Free TrialLog in
Avatar of amukta
amukta

asked on

Look up transformation slowing down the load in SSIS

I’m loading data from a Csv file using SSIS(SSDT 2015.).I’m first loading the data into a stage table and then converting  the column data to the data types in the target table using  data conversion transformation, then, using a look up transformation in partial cache mode to compare the values in 1 column ( this could be null or not null)+ the load date( It’s the current date time that I’m converting to date data type while comparing) in stage table with the ones in destination.Both these columns are non primary key columns. Any row that matches the stage and final destination table should not be loaded but the non matching ones can be loaded to the final destination.

I do have a column store clustered index on the destination table. There is no primary key on the table.

in the lookup Transformation,I’m  using the following query:

select distinct colA , ColB from Stgtbl A
Left join destbl B
On A.colA= B.colA and A.colB=B.ColB
Where b. colA Is Null And b.colB is null

I tried to avoid using look up and tried using left join and where not exists as well in the Oledb source
But it was running really slow and sometimes even getting stuck after loading a few rows.
I thought the above look up might resolve the issue, but seeing the same issue.

Can someone please help me resolve this ASAP.

Thanks a million in advance!
Avatar of Raja Jegan R
Raja Jegan R
Flag of India image

Hi amukta,

I'm able to see this question opened up a long time before..
Kindly let us know whether you are still looking out for this issue so that I can assist accordingly.

Regards,
Raja Jegan R.
EE Solution Guide.
Avatar of amukta
amukta

ASKER

I'm closing this question, as no one answered and I figured myself.
ASKER CERTIFIED SOLUTION
Avatar of amukta
amukta

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial