Link to home
Start Free TrialLog in
Avatar of JP_TechGroup
JP_TechGroupFlag for United States of America

asked on

Simple SQL Query Problem on nested statement.

Consider the following MS SQL query:

DECLARE @po_length as bigint
SET @po_length = 12192000001;

SELECT
 Round(Left(@po_length,(len(@po_length)-6)) / 304.8,0);

It generates this error:
Msg 8115, Level 16, State 8, Line 4
Arithmetic overflow error converting varchar to data type numeric.

And I am at a loss at to why. Anyone?
Avatar of Ryan Simmons
Ryan Simmons

Your function is converting the number into a string.

This works:

DECLARE @po_length as bigint
SET @po_length = 12192000001

SELECT
   ROUND(CONVERT(NUMERIC, Left(@po_length,(len(@po_length)-6))) / 304.8,0);

Open in new window

Avatar of ste5an
SQL Server internally uses numeric(), but it takes the precision and scale from the constant 304.8 in the expression, hence it is internally working with NUMERIC(4,1). Thus the overflow.

Two ways to solve it. Math or helping SQL Server determine the correct internal numeric type:

DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT @po_length ,
       LEFT(@po_length, ( LEN(@po_length) - 6 )) ,
       @po_length / 1000000 ,
       ROUND(@po_length / 1000000 / 304.8, 0) ,
       ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 0);

Open in new window


See also the Caution at https://docs.microsoft.com/en-us/sql/t-sql/data-types/int-bigint-smallint-and-tinyint-transact-sql.
Avatar of JP_TechGroup

ASKER

ste5an- Thank you.
The last solution would seem to work best in our scenario (as the @po_length is often of different lengths in the column)
However, the result is producing numerous 0000 after the result, even though round is set to 0.
Do I need to do something like this to clean it up? This whole solution feels very 'clunky"!
FORMAT(CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(5, 0)), 0)  AS DECIMAL(9,6)),'g15')
Don't mix rounding with cutting decimal places, thus you need to change the results data type. So as far as I understand you, your result should be an integer, no decimal places:

DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 0) AS INT);

Open in new window

Much cleaner.
If I wanted it to display two places after the decimal, then:
DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS BIGINT);
Correct?
ASKER CERTIFIED SOLUTION
Avatar of ste5an
ste5an
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Makes sense, thank you.
Moving the query into parsing data on the table, I am now getting this error:

Msg 537, Level 16, State 2, Procedure Peachtree_Export, Line 18
Invalid length parameter passed to the LEFT or SUBSTRING function.

CAST(ROUND(LEFT(polineview.[length], ( LEN(polineview.[length]) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS numeric(19,2))

Open in new window

What data type is it?

Maybe you should consider posting a concise and complete example. Include table DDL.
BIGINT
Here is a sample of the relevant data and the SELECT Statement

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[tbl_demo](
	[pokey] [int] IDENTITY(1,1) NOT NULL,
	[width] [bigint] NULL,
	[length] [bigint] NULL
) ON [PRIMARY]

GO
SET IDENTITY_INSERT [dbo].[tbl_demo] ON 

INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (1, 0, 12192000001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (2, 2438400001, 16865600001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (3, 0, 13716000001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (4, 0, 3048000001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (5, 0, 7315200001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (6, 0, 0)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (7, 0, 6096000001)
SET IDENTITY_INSERT [dbo].[tbl_demo] OFF

Open in new window


SELECT
CAST(ROUND(LEFT(tbl_demo.[length], ( LEN(tbl_demo.[length]) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS numeric(19,2))
FROM tbl_demo
WHERE
Width=0;

Open in new window

Figured it out. When the length column is zero, it creates an impossible LEN solution.
Thank you for your help sir. I has been some time since I have had to convert numbers in MS SQL. This was educational.