SSIS Lookup Error- Help needed

Hi EE,

Back again with the lookup issue in SSIS,
SSIS lookup failing with the following errors
[ClientRefLkup [296]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "ClientRefLkup" failed because error code 0xC020901E occurred, and the error row disposition on "ClientRefLkup.Outputs[Lookup Match Output]" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.  The ProcessInput method on component "ClientRefLkup" (296) failed with error code 0xC0209029 while processing input "Lookup Input" (310). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure.

[Booking Source [2]] Error: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.  The PrimeOutput method on Booking Source returned error code 0xC02020C4.  The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.  There may be error messages posted before this with more information about the failure.

Open in new window


I know its related to underlying data but i am just baffled on how to sort this out.
I have attached the sql sheet with the source data and the dimension i am trying to lookup.
However i am unable to upload .dtx file with the project. But its a simple package containing the source data and then a lookup to the dimension data.
Please help.

Regards,

Ali
Data.xlsx
Ali ShahSQL DeveloperAsked:
Who is Participating?
 
Nakul VachhrajaniConnect With a Mentor Technical Architect, Capgemini IndiaCommented:
Hello!

Glad to know that you  now have partial success with the lookup.

Note that comparisons in SSIS can be case sensitive. Hence, if you already have the value "Unknown" in the lookup cache, it should work provided that the cases match (you can explore converting values to upper or lower case, as the business need may be).

As for using UNION ALL in lookups, I have never personally tried to do this. However, at least in your case, the UNION is not required. You can use the following:

SELECT ClientRefSK, ClientRef
FROM dbo.DimClientReferences
WHERE (ClientRefType = 'ClientOptionalRef' AND ClientRef <> 'Unknown')
   OR (ClientRefSK = 0)

Open in new window

0
 
Ali ShahSQL DeveloperAuthor Commented:
Sorry but here's the dimension table structre
CREATE TABLE [dbo].[DimClientReferences](
	[ClientRefSK] [INT] IDENTITY(1,1) NOT NULL,
	[ClientRef] [VARCHAR](200) NULL,
	[ClientRefType] [VARCHAR](50) NULL,
 CONSTRAINT [PK_DimClientReferences] PRIMARY KEY CLUSTERED 
(
	[ClientRefSK] ASC)

Open in new window

0
 
Nakul VachhrajaniTechnical Architect, Capgemini IndiaCommented:
Without looking at the actual LookUp transformation and the associated keys, it is slightly difficult to comment on this one. But, here's something I noticed in your "Data.xlsx" file.

My understanding is that you are performing a lookup for SourceData.ClientOptionalRef, SourceData.Clientref1, SourceData.Clientref2 and so on. The lookup is being made on the dimension DimClient for a particular "ClientRefType" which corresponds to the column in the SourceData that you are working on.

If that is true, it appears that you do not have all the possible values in your dimension. For example:
- The value "Unknown" is required for all "ClientRefType"
- For "ClientRefType" = Clientref2, the following values do not exist in the dimension table, but are present in your inbound data: Client Ref 4, Client Ref 1, Client Ref 2

Similarly, you would need to check for all other "ClientRefType".

If this is an expected pattern, I would explore handing this as a late-arriving dimension in your ETL.
Also, in the look-up transform, you would need to choose what needs to happen when a look-up fails (i.e. value is not found in the look-up dimension table). Please choose the appropriate option. Typically, you would want to set this to "Redirect rows to no match output" to allow you to handle late-arriving dimensions appropriately rather than having the rows go to the Error Output (default behaviour).
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

 
Ali ShahSQL DeveloperAuthor Commented:
Hi Nakul,
 Thanks a lot for your reply,
Yes you are right i have clientref1 , 2 and so on. I have attached the image here LookupThis explains the lookup.
one thing is quite bizarre, i have converted the stored procedure into a user defined function and selected the Distinct ClientOptionalRef (only this column) from the function and it worked. I have also tested it by putting Unknown against every clientref but still fails.

well another question, I am using SK0 and Unknown for the table and then using Union all in my lookup as you can see to bring back the Unknown value. should it not work?

regards,

regards
0
 
Ali ShahSQL DeveloperAuthor Commented:
Hi Nakul,

thanks again for your kind help. Yes its the case sensitivity issue which i was not aware of.
So do i need to save my dimensions data in Upper / lower case and do the same with the source data?

regards.

Ali
0
 
Nakul VachhrajaniConnect With a Mentor Technical Architect, Capgemini IndiaCommented:
Yes, ideally the data in a transactional system would simply reference the values via an ID (e.g. you would have all Student information stored as master data and in all other places, you would simply use StudentId - thereby removing the dependency on the case).

But, if that is not possible, you should ensure consistency in (a) storage of data OR (b) during comparison by using TOUPPER() and TOLOWER() methods to ensure that comparison is happening using the same case.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.