SSIS is skipping random rows

Hi Guys,

I am transferring 1000 rows from a SQL Data Source to SQL Destination using OLEDB Command control. I am using OLEDB Command because i need to execute the stored procedure for each row.
I have found that 974 rows were transferred. I figured out which rows were skipped and then executed stored procedure manually to see if any errors but manually stored procedure ran successfully and it inserted those records.

I have also created a flow task to include only those records which missed during the first load
SELECT TOP ( 1000 )
        AddressStringId ,
        HASHBYTES('SHA1', CONVERT(VARCHAR(12), AddressStringId)) AS Guid ,
        HASHBYTES('SHA1', SearchString) AS UniqueHashString ,
        OrganisationName AS LocationName ,
        OrganisationName AS AddressLine1 ,
        SubBuildingName AS AddressLine2 ,
        '' AS AddressLine3 ,
        BuildingNumber ,
        BuildingName ,
        StreetFirstPart + ' ' + StreetSecondPart AS Street ,
        '' AS Locality ,
        'England' AS AdministrativeArea ,
        '' AS SubAdministrativeArea ,
        PostTown AS Municipality ,
        PostCode ,
        'GB' AS Iso3166MinusOneCountryAlpha2Code ,
        STR(Longitude, 25, 5) AS Longitude ,
        STR(Latitude, 25, 5) Latitude ,
        SearchString ,
        3.0 AS PointBufferDistanceInMeters ,
		--below are the output parameters in the stored procedure however SSIS doesn't seem to handle this. so passing 0 or null values for these parameters
        ISNULL(ERROR_NUMBER(), 0) AS ErrorNumber ,
        ISNULL(ERROR_SEVERITY(), 0) AS ErrorSeverity ,
        ISNULL(ERROR_STATE(), 0) AS ErrorState ,
        ISNULL(ERROR_PROCEDURE(), 0) AS ErrorProcedure ,
        ISNULL(ERROR_LINE(), 0) AS ErrorLine ,
        ISNULL(ERROR_MESSAGE(), '') AS ErrorMessage ,
        0 AS InsertedRecords
FROM    AddressStringSearch
WHERE AddressStringId IN (16855935,16855951,16856022,16856056,16856155,16856162,16856207,16856243,16856257,16856301,16856362,16856406,16856430,16856444,16856453,16856453,16856491,16856585
,16856602,16856623,16856650,16856680,16856687,16856701,16856703,16856758,16856809)

Open in new window


The task ran successfully without any error but no records got transferred.
That's how i manually executed the stored procedure
USE [Spaces]
GO

DECLARE	@return_value int,
		@ErrorNumber int,
		@ErrorSeverity int,
		@ErrorState int,
		@ErrorProcedure nvarchar(128),
		@ErrorLine int,
		@ErrorMessage nvarchar(4000),
		@NumberOfInsertedLocations INT
        
				DECLARE @Wgs84Longitude1 AS FLOAT =CAST(LTRIM(RTRIM('                 -2.15637')) AS FLOAT)
		DECLARE @Wgs84Latitude1 AS FLOAT =  CAST(LTRIM(RTRIM('                 57.26038')) AS FLOAT)

EXEC	@return_value = [CrossContext].[USP_Locations_AddPointLocations]
		@LocationGuid = 0x9F24EC0E32CD4ED6FC96B737D9CBD8FD4DFEA8E2,
		@AddressStringId=16855951,
		@LocationName = N'',
		@AddressLine1 = N'',
		@AddressLine2 = N'',
		@AddressLine3 = N'',
		@BuildingNumbers = N'10',
		@BuildingName = N'',
		@Street = N'BROOMHILL WAY',
		@Locality = N'',
		@Municipality = N'ABERDEEN',
		@AdministrativeArea = N'England',
		@SubAdministrativeArea = N'',
		@PostalCode = N'AB21 0AA',
		@Iso3166MinusOneCountryAlpha2Code = N'GB',
		@Wgs84Longitude =@Wgs84Longitude1,
		@Wgs84Latitude =  @Wgs84Latitude1,
		@PointBufferDistanceInMeters = 3.0,
		@UniqueHashString = N'0xC6566F541E3CC5B2066541379CCA27F9EB522901',
		@ErrorNumber = @ErrorNumber OUTPUT,
		@ErrorSeverity = @ErrorSeverity OUTPUT,
		@ErrorState = @ErrorState OUTPUT,
		@ErrorProcedure = @ErrorProcedure OUTPUT,
		@ErrorLine = @ErrorLine OUTPUT,
		@ErrorMessage = @ErrorMessage OUTPUT,
		@NumberOfInsertedLocations = @NumberOfInsertedLocations OUTPUT

SELECT	@ErrorNumber as N'@ErrorNumber',
		@ErrorSeverity as N'@ErrorSeverity',
		@ErrorState as N'@ErrorState',
		@ErrorProcedure as N'@ErrorProcedure',
		@ErrorLine as N'@ErrorLine',
		@ErrorMessage as N'@ErrorMessage',
		@NumberOfInsertedLocations as N'@NumberOfInsertedLocations'

SELECT	'Return Value' = @return_value

GO

Open in new window


I manually passed the values for two Ids 16855935 and 16855951. The stored procedure ran fine.
Now i am totally confused as why is it happening can you please help?
Ali ShahSQL DeveloperAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Pawan KumarDatabase ExpertCommented:
In clause doesn't work if you have NULL in the table.

I think below is the issue.

WHERE AddressStringId IN (16855935,16855951,16856022,16856056,16856155,16856162,16856207,16856243,16856257,16856301,16856362,16856406,16856430,16856444,16856453,16856453,16856491,16856585
,16856602,16856623,16856650,16856680,16856687,16856701,16856703,16856758,16856809)
Ali ShahSQL DeveloperAuthor Commented:
I don't have null values in my table. all of these id's do exist. Also these ids were skipped from the initial load and i can add them by running stored procedure manually as explained in my original question
Pawan KumarDatabase ExpertCommented:
What is the query for initial load ?
Why Diversity in Tech Matters

Kesha Williams, certified professional and software developer, explores the imbalance of diversity in the world of technology -- especially when it comes to hiring women. She showcases ways she's making a difference through the Colors of STEM program.

Ali ShahSQL DeveloperAuthor Commented:
it is the same query without where clause in it

SELECT TOP ( 1000 )
        AddressStringId ,
        HASHBYTES('SHA1', CONVERT(VARCHAR(12), AddressStringId)) AS Guid ,
        HASHBYTES('SHA1', SearchString) AS UniqueHashString ,
        OrganisationName AS LocationName ,
        OrganisationName AS AddressLine1 ,
        SubBuildingName AS AddressLine2 ,
        '' AS AddressLine3 ,
        BuildingNumber ,
        BuildingName ,
        StreetFirstPart + ' ' + StreetSecondPart AS Street ,
        '' AS Locality ,
        'England' AS AdministrativeArea ,
        '' AS SubAdministrativeArea ,
        PostTown AS Municipality ,
        PostCode ,
        'GB' AS Iso3166MinusOneCountryAlpha2Code ,
        STR(Longitude, 25, 5) AS Longitude ,
        STR(Latitude, 25, 5) Latitude ,
        SearchString ,
        3.0 AS PointBufferDistanceInMeters ,
		--below are the output parameters in the stored procedure however SSIS doesn't seem to handle this. so passing 0 or null values for these parameters
        ISNULL(ERROR_NUMBER(), 0) AS ErrorNumber ,
        ISNULL(ERROR_SEVERITY(), 0) AS ErrorSeverity ,
        ISNULL(ERROR_STATE(), 0) AS ErrorState ,
        ISNULL(ERROR_PROCEDURE(), 0) AS ErrorProcedure ,
        ISNULL(ERROR_LINE(), 0) AS ErrorLine ,
        ISNULL(ERROR_MESSAGE(), '') AS ErrorMessage ,
        0 AS InsertedRecords
FROM    AddressStringSearch

Open in new window

Pawan KumarDatabase ExpertCommented:
Have you defined batch size or something like that ... Query seems to be fine.
Ali ShahSQL DeveloperAuthor Commented:
no i have not defined anything perhaps using default settings for Data Source, and OLEDB Command component. The only thing i am using is Data Conversion component to convert some fields to unicode string.
The AMAZING thing is that i could not find any difference between inserted and skipped records. Also the strange is that i can manually execute the stored procedure which successfully operates. There must be something with OLEDB command control
Pawan KumarDatabase ExpertCommented:
Ok, Try this approach.

Create a sp and perform everything from there and see if any row is missed or not.

If not you can continue with the above approach. If not then we shall look into the SSIS code.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ali ShahSQL DeveloperAuthor Commented:
Thanks a lot. It did help. For some reasons stored procedure worked instead of normal select clause
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
SSIS

From novice to tech pro — start learning today.