SSIS Lookup Error- Help needed

Hi EE,

Back again with the lookup issue in SSIS,
SSIS lookup failing with the following errors
[ClientRefLkup [296]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "ClientRefLkup" failed because error code 0xC020901E occurred, and the error row disposition on "ClientRefLkup.Outputs[Lookup Match Output]" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.  The ProcessInput method on component "ClientRefLkup" (296) failed with error code 0xC0209029 while processing input "Lookup Input" (310). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure.

[Booking Source [2]] Error: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.  The PrimeOutput method on Booking Source returned error code 0xC02020C4.  The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.  There may be error messages posted before this with more information about the failure.

Open in new window

I know its related to underlying data but i am just baffled on how to sort this out.
I have attached the sql sheet with the source data and the dimension i am trying to lookup.
However i am unable to upload .dtx file with the project. But its a simple package containing the source data and then a lookup to the dimension data.
Please help.


Ali ShahSQL DeveloperAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ali ShahSQL DeveloperAuthor Commented:
Sorry but here's the dimension table structre
CREATE TABLE [dbo].[DimClientReferences](
	[ClientRefSK] [INT] IDENTITY(1,1) NOT NULL,
	[ClientRef] [VARCHAR](200) NULL,
	[ClientRefType] [VARCHAR](50) NULL,
	[ClientRefSK] ASC)

Open in new window

Nakul VachhrajaniTechnical Architect, Capgemini IndiaCommented:
Without looking at the actual LookUp transformation and the associated keys, it is slightly difficult to comment on this one. But, here's something I noticed in your "Data.xlsx" file.

My understanding is that you are performing a lookup for SourceData.ClientOptionalRef, SourceData.Clientref1, SourceData.Clientref2 and so on. The lookup is being made on the dimension DimClient for a particular "ClientRefType" which corresponds to the column in the SourceData that you are working on.

If that is true, it appears that you do not have all the possible values in your dimension. For example:
- The value "Unknown" is required for all "ClientRefType"
- For "ClientRefType" = Clientref2, the following values do not exist in the dimension table, but are present in your inbound data: Client Ref 4, Client Ref 1, Client Ref 2

Similarly, you would need to check for all other "ClientRefType".

If this is an expected pattern, I would explore handing this as a late-arriving dimension in your ETL.
Also, in the look-up transform, you would need to choose what needs to happen when a look-up fails (i.e. value is not found in the look-up dimension table). Please choose the appropriate option. Typically, you would want to set this to "Redirect rows to no match output" to allow you to handle late-arriving dimensions appropriately rather than having the rows go to the Error Output (default behaviour).
Ali ShahSQL DeveloperAuthor Commented:
Hi Nakul,
 Thanks a lot for your reply,
Yes you are right i have clientref1 , 2 and so on. I have attached the image here LookupThis explains the lookup.
one thing is quite bizarre, i have converted the stored procedure into a user defined function and selected the Distinct ClientOptionalRef (only this column) from the function and it worked. I have also tested it by putting Unknown against every clientref but still fails.

well another question, I am using SK0 and Unknown for the table and then using Union all in my lookup as you can see to bring back the Unknown value. should it not work?


Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Nakul VachhrajaniTechnical Architect, Capgemini IndiaCommented:

Glad to know that you  now have partial success with the lookup.

Note that comparisons in SSIS can be case sensitive. Hence, if you already have the value "Unknown" in the lookup cache, it should work provided that the cases match (you can explore converting values to upper or lower case, as the business need may be).

As for using UNION ALL in lookups, I have never personally tried to do this. However, at least in your case, the UNION is not required. You can use the following:

SELECT ClientRefSK, ClientRef
FROM dbo.DimClientReferences
WHERE (ClientRefType = 'ClientOptionalRef' AND ClientRef <> 'Unknown')
   OR (ClientRefSK = 0)

Open in new window


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ali ShahSQL DeveloperAuthor Commented:
Hi Nakul,

thanks again for your kind help. Yes its the case sensitivity issue which i was not aware of.
So do i need to save my dimensions data in Upper / lower case and do the same with the source data?


Nakul VachhrajaniTechnical Architect, Capgemini IndiaCommented:
Yes, ideally the data in a transactional system would simply reference the values via an ID (e.g. you would have all Student information stored as master data and in all other places, you would simply use StudentId - thereby removing the dependency on the case).

But, if that is not possible, you should ensure consistency in (a) storage of data OR (b) during comparison by using TOUPPER() and TOLOWER() methods to ensure that comparison is happening using the same case.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server

From novice to tech pro — start learning today.