Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!
SSIS Lookup Error- Help needed
Hi EE,
Back again with the lookup issue in SSIS,
SSIS lookup failing with the following errors
I know its related to underlying data but i am just baffled on how to sort this out.
I have attached the sql sheet with the source data and the dimension i am trying to lookup.
However i am unable to upload .dtx file with the project. But its a simple package containing the source data and then a lookup to the dimension data.
Please help.
Regards,
Ali
Data.xlsx
Back again with the lookup issue in SSIS,
SSIS lookup failing with the following errors
[ClientRefLkup [296]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "ClientRefLkup" failed because error code 0xC020901E occurred, and the error row disposition on "ClientRefLkup.Outputs[Lookup Match Output]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "ClientRefLkup" (296) failed with error code 0xC0209029 while processing input "Lookup Input" (310). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
[Booking Source [2]] Error: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.
[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on Booking Source returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
I know its related to underlying data but i am just baffled on how to sort this out.
I have attached the sql sheet with the source data and the dimension i am trying to lookup.
However i am unable to upload .dtx file with the project. But its a simple package containing the source data and then a lookup to the dimension data.
Please help.
Regards,
Ali
Data.xlsx
Who is Participating?
Experts Exchange Solution brought to you by
Enjoy your complimentary solution view.
Get every solution instantly with Premium.
Start your 7-day free trial.
I wear a lot of hats...
"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.
Without looking at the actual LookUp transformation and the associated keys, it is slightly difficult to comment on this one. But, here's something I noticed in your "Data.xlsx" file.
My understanding is that you are performing a lookup for SourceData.ClientOptionalR
If that is true, it appears that you do not have all the possible values in your dimension. For example:
- The value "Unknown" is required for all "ClientRefType"
- For "ClientRefType" = Clientref2, the following values do not exist in the dimension table, but are present in your inbound data: Client Ref 4, Client Ref 1, Client Ref 2
Similarly, you would need to check for all other "ClientRefType".
If this is an expected pattern, I would explore handing this as a late-arriving dimension in your ETL.
Also, in the look-up transform, you would need to choose what needs to happen when a look-up fails (i.e. value is not found in the look-up dimension table). Please choose the appropriate option. Typically, you would want to set this to "Redirect rows to no match output" to allow you to handle late-arriving dimensions appropriately rather than having the rows go to the Error Output (default behaviour).
My understanding is that you are performing a lookup for SourceData.ClientOptionalR
If that is true, it appears that you do not have all the possible values in your dimension. For example:
- The value "Unknown" is required for all "ClientRefType"
- For "ClientRefType" = Clientref2, the following values do not exist in the dimension table, but are present in your inbound data: Client Ref 4, Client Ref 1, Client Ref 2
Similarly, you would need to check for all other "ClientRefType".
If this is an expected pattern, I would explore handing this as a late-arriving dimension in your ETL.
Also, in the look-up transform, you would need to choose what needs to happen when a look-up fails (i.e. value is not found in the look-up dimension table). Please choose the appropriate option. Typically, you would want to set this to "Redirect rows to no match output" to allow you to handle late-arriving dimensions appropriately rather than having the rows go to the Error Output (default behaviour).
Hi Nakul,
Thanks a lot for your reply,
Yes you are right i have clientref1 , 2 and so on. I have attached the image here
This explains the lookup.
one thing is quite bizarre, i have converted the stored procedure into a user defined function and selected the Distinct ClientOptionalRef (only this column) from the function and it worked. I have also tested it by putting Unknown against every clientref but still fails.
well another question, I am using SK0 and Unknown for the table and then using Union all in my lookup as you can see to bring back the Unknown value. should it not work?
regards,
regards
Thanks a lot for your reply,
Yes you are right i have clientref1 , 2 and so on. I have attached the image here
This explains the lookup. one thing is quite bizarre, i have converted the stored procedure into a user defined function and selected the Distinct ClientOptionalRef (only this column) from the function and it worked. I have also tested it by putting Unknown against every clientref but still fails.
well another question, I am using SK0 and Unknown for the table and then using Union all in my lookup as you can see to bring back the Unknown value. should it not work?
regards,
regards
Hello!
Glad to know that you now have partial success with the lookup.
Note that comparisons in SSIS can be case sensitive. Hence, if you already have the value "Unknown" in the lookup cache, it should work provided that the cases match (you can explore converting values to upper or lower case, as the business need may be).
As for using UNION ALL in lookups, I have never personally tried to do this. However, at least in your case, the UNION is not required. You can use the following:
Glad to know that you now have partial success with the lookup.
Note that comparisons in SSIS can be case sensitive. Hence, if you already have the value "Unknown" in the lookup cache, it should work provided that the cases match (you can explore converting values to upper or lower case, as the business need may be).
As for using UNION ALL in lookups, I have never personally tried to do this. However, at least in your case, the UNION is not required. You can use the following:
SELECT ClientRefSK, ClientRef
FROM dbo.DimClientReferences
WHERE (ClientRefType = 'ClientOptionalRef' AND ClientRef <> 'Unknown')
OR (ClientRefSK = 0)
Experts Exchange Solution brought to you by
Your issues matter to us.
Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.
Start your 7-day free trial
Hi Nakul,
thanks again for your kind help. Yes its the case sensitivity issue which i was not aware of.
So do i need to save my dimensions data in Upper / lower case and do the same with the source data?
regards.
Ali
thanks again for your kind help. Yes its the case sensitivity issue which i was not aware of.
So do i need to save my dimensions data in Upper / lower case and do the same with the source data?
regards.
Ali
Yes, ideally the data in a transactional system would simply reference the values via an ID (e.g. you would have all Student information stored as master data and in all other places, you would simply use StudentId - thereby removing the dependency on the case).
But, if that is not possible, you should ensure consistency in (a) storage of data OR (b) during comparison by using TOUPPER() and TOLOWER() methods to ensure that comparison is happening using the same case.
But, if that is not possible, you should ensure consistency in (a) storage of data OR (b) during comparison by using TOUPPER() and TOLOWER() methods to ensure that comparison is happening using the same case.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server
From novice to tech pro — start learning today.
Experts Exchange Solution brought to you by
Enjoy your complimentary solution view.
Get every solution instantly with Premium.
Start your 7-day free trial.
Open in new window