Solved

SSIS is skipping random rows

Posted on 2016-09-27
8
63 Views
Last Modified: 2016-09-28
Hi Guys,

I am transferring 1000 rows from a SQL Data Source to SQL Destination using OLEDB Command control. I am using OLEDB Command because i need to execute the stored procedure for each row.
I have found that 974 rows were transferred. I figured out which rows were skipped and then executed stored procedure manually to see if any errors but manually stored procedure ran successfully and it inserted those records.

I have also created a flow task to include only those records which missed during the first load
SELECT TOP ( 1000 )
        AddressStringId ,
        HASHBYTES('SHA1', CONVERT(VARCHAR(12), AddressStringId)) AS Guid ,
        HASHBYTES('SHA1', SearchString) AS UniqueHashString ,
        OrganisationName AS LocationName ,
        OrganisationName AS AddressLine1 ,
        SubBuildingName AS AddressLine2 ,
        '' AS AddressLine3 ,
        BuildingNumber ,
        BuildingName ,
        StreetFirstPart + ' ' + StreetSecondPart AS Street ,
        '' AS Locality ,
        'England' AS AdministrativeArea ,
        '' AS SubAdministrativeArea ,
        PostTown AS Municipality ,
        PostCode ,
        'GB' AS Iso3166MinusOneCountryAlpha2Code ,
        STR(Longitude, 25, 5) AS Longitude ,
        STR(Latitude, 25, 5) Latitude ,
        SearchString ,
        3.0 AS PointBufferDistanceInMeters ,
		--below are the output parameters in the stored procedure however SSIS doesn't seem to handle this. so passing 0 or null values for these parameters
        ISNULL(ERROR_NUMBER(), 0) AS ErrorNumber ,
        ISNULL(ERROR_SEVERITY(), 0) AS ErrorSeverity ,
        ISNULL(ERROR_STATE(), 0) AS ErrorState ,
        ISNULL(ERROR_PROCEDURE(), 0) AS ErrorProcedure ,
        ISNULL(ERROR_LINE(), 0) AS ErrorLine ,
        ISNULL(ERROR_MESSAGE(), '') AS ErrorMessage ,
        0 AS InsertedRecords
FROM    AddressStringSearch
WHERE AddressStringId IN (16855935,16855951,16856022,16856056,16856155,16856162,16856207,16856243,16856257,16856301,16856362,16856406,16856430,16856444,16856453,16856453,16856491,16856585
,16856602,16856623,16856650,16856680,16856687,16856701,16856703,16856758,16856809)

Open in new window


The task ran successfully without any error but no records got transferred.
That's how i manually executed the stored procedure
USE [Spaces]
GO

DECLARE	@return_value int,
		@ErrorNumber int,
		@ErrorSeverity int,
		@ErrorState int,
		@ErrorProcedure nvarchar(128),
		@ErrorLine int,
		@ErrorMessage nvarchar(4000),
		@NumberOfInsertedLocations INT
        
				DECLARE @Wgs84Longitude1 AS FLOAT =CAST(LTRIM(RTRIM('                 -2.15637')) AS FLOAT)
		DECLARE @Wgs84Latitude1 AS FLOAT =  CAST(LTRIM(RTRIM('                 57.26038')) AS FLOAT)

EXEC	@return_value = [CrossContext].[USP_Locations_AddPointLocations]
		@LocationGuid = 0x9F24EC0E32CD4ED6FC96B737D9CBD8FD4DFEA8E2,
		@AddressStringId=16855951,
		@LocationName = N'',
		@AddressLine1 = N'',
		@AddressLine2 = N'',
		@AddressLine3 = N'',
		@BuildingNumbers = N'10',
		@BuildingName = N'',
		@Street = N'BROOMHILL WAY',
		@Locality = N'',
		@Municipality = N'ABERDEEN',
		@AdministrativeArea = N'England',
		@SubAdministrativeArea = N'',
		@PostalCode = N'AB21 0AA',
		@Iso3166MinusOneCountryAlpha2Code = N'GB',
		@Wgs84Longitude =@Wgs84Longitude1,
		@Wgs84Latitude =  @Wgs84Latitude1,
		@PointBufferDistanceInMeters = 3.0,
		@UniqueHashString = N'0xC6566F541E3CC5B2066541379CCA27F9EB522901',
		@ErrorNumber = @ErrorNumber OUTPUT,
		@ErrorSeverity = @ErrorSeverity OUTPUT,
		@ErrorState = @ErrorState OUTPUT,
		@ErrorProcedure = @ErrorProcedure OUTPUT,
		@ErrorLine = @ErrorLine OUTPUT,
		@ErrorMessage = @ErrorMessage OUTPUT,
		@NumberOfInsertedLocations = @NumberOfInsertedLocations OUTPUT

SELECT	@ErrorNumber as N'@ErrorNumber',
		@ErrorSeverity as N'@ErrorSeverity',
		@ErrorState as N'@ErrorState',
		@ErrorProcedure as N'@ErrorProcedure',
		@ErrorLine as N'@ErrorLine',
		@ErrorMessage as N'@ErrorMessage',
		@NumberOfInsertedLocations as N'@NumberOfInsertedLocations'

SELECT	'Return Value' = @return_value

GO

Open in new window


I manually passed the values for two Ids 16855935 and 16855951. The stored procedure ran fine.
Now i am totally confused as why is it happening can you please help?
0
Comment
Question by:shah36
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
8 Comments
 
LVL 29

Expert Comment

by:Pawan Kumar
ID: 41817602
In clause doesn't work if you have NULL in the table.

I think below is the issue.

WHERE AddressStringId IN (16855935,16855951,16856022,16856056,16856155,16856162,16856207,16856243,16856257,16856301,16856362,16856406,16856430,16856444,16856453,16856453,16856491,16856585
,16856602,16856623,16856650,16856680,16856687,16856701,16856703,16856758,16856809)
0
 

Author Comment

by:shah36
ID: 41817605
I don't have null values in my table. all of these id's do exist. Also these ids were skipped from the initial load and i can add them by running stored procedure manually as explained in my original question
0
 
LVL 29

Expert Comment

by:Pawan Kumar
ID: 41817608
What is the query for initial load ?
0
On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

 

Author Comment

by:shah36
ID: 41817613
it is the same query without where clause in it

SELECT TOP ( 1000 )
        AddressStringId ,
        HASHBYTES('SHA1', CONVERT(VARCHAR(12), AddressStringId)) AS Guid ,
        HASHBYTES('SHA1', SearchString) AS UniqueHashString ,
        OrganisationName AS LocationName ,
        OrganisationName AS AddressLine1 ,
        SubBuildingName AS AddressLine2 ,
        '' AS AddressLine3 ,
        BuildingNumber ,
        BuildingName ,
        StreetFirstPart + ' ' + StreetSecondPart AS Street ,
        '' AS Locality ,
        'England' AS AdministrativeArea ,
        '' AS SubAdministrativeArea ,
        PostTown AS Municipality ,
        PostCode ,
        'GB' AS Iso3166MinusOneCountryAlpha2Code ,
        STR(Longitude, 25, 5) AS Longitude ,
        STR(Latitude, 25, 5) Latitude ,
        SearchString ,
        3.0 AS PointBufferDistanceInMeters ,
		--below are the output parameters in the stored procedure however SSIS doesn't seem to handle this. so passing 0 or null values for these parameters
        ISNULL(ERROR_NUMBER(), 0) AS ErrorNumber ,
        ISNULL(ERROR_SEVERITY(), 0) AS ErrorSeverity ,
        ISNULL(ERROR_STATE(), 0) AS ErrorState ,
        ISNULL(ERROR_PROCEDURE(), 0) AS ErrorProcedure ,
        ISNULL(ERROR_LINE(), 0) AS ErrorLine ,
        ISNULL(ERROR_MESSAGE(), '') AS ErrorMessage ,
        0 AS InsertedRecords
FROM    AddressStringSearch

Open in new window

0
 
LVL 29

Expert Comment

by:Pawan Kumar
ID: 41817618
Have you defined batch size or something like that ... Query seems to be fine.
0
 

Author Comment

by:shah36
ID: 41817625
no i have not defined anything perhaps using default settings for Data Source, and OLEDB Command component. The only thing i am using is Data Conversion component to convert some fields to unicode string.
The AMAZING thing is that i could not find any difference between inserted and skipped records. Also the strange is that i can manually execute the stored procedure which successfully operates. There must be something with OLEDB command control
0
 
LVL 29

Accepted Solution

by:
Pawan Kumar earned 500 total points
ID: 41817626
Ok, Try this approach.

Create a sp and perform everything from there and see if any row is missed or not.

If not you can continue with the above approach. If not then we shall look into the SSIS code.
0
 

Author Closing Comment

by:shah36
ID: 41819608
Thanks a lot. It did help. For some reasons stored procedure worked instead of normal select clause
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

From time to time, for debugging and troubleshooting your flow at run time you’ll need to check if the variable has the correct value or not, there’re several ways to check for the value of the variables inside the flow, You can add Break Points, a …
My client sends a request to me that they want me to load data, which will be returned by Web Service APIs, and do some transformation before importing to database. In this article, I will provide an approach to load data with Web Service Task and X…
Michael from AdRem Software outlines event notifications and Automatic Corrective Actions in network monitoring. Automatic Corrective Actions are scripts, which can automatically run upon discovery of a certain undesirable condition in your network.…
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: …

689 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question