Simple SQL Query Problem on nested statement.

Consider the following MS SQL query:

DECLARE @po_length as bigint
SET @po_length = 12192000001;

SELECT
 Round(Left(@po_length,(len(@po_length)-6)) / 304.8,0);

It generates this error:
Msg 8115, Level 16, State 8, Line 4
Arithmetic overflow error converting varchar to data type numeric.

And I am at a loss at to why. Anyone?
LVL 1
JP_TechGroupAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ryan SimmonsBusiness Analyst IIICommented:
Your function is converting the number into a string.

This works:

DECLARE @po_length as bigint
SET @po_length = 12192000001

SELECT
   ROUND(CONVERT(NUMERIC, Left(@po_length,(len(@po_length)-6))) / 304.8,0);

Open in new window

Ryan SimmonsBusiness Analyst IIICommented:
ste5anSenior DeveloperCommented:
SQL Server internally uses numeric(), but it takes the precision and scale from the constant 304.8 in the expression, hence it is internally working with NUMERIC(4,1). Thus the overflow.

Two ways to solve it. Math or helping SQL Server determine the correct internal numeric type:

DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT @po_length ,
       LEFT(@po_length, ( LEN(@po_length) - 6 )) ,
       @po_length / 1000000 ,
       ROUND(@po_length / 1000000 / 304.8, 0) ,
       ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 0);

Open in new window


See also the Caution at https://docs.microsoft.com/en-us/sql/t-sql/data-types/int-bigint-smallint-and-tinyint-transact-sql.
Become a Certified Penetration Testing Engineer

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

JP_TechGroupAuthor Commented:
ste5an- Thank you.
The last solution would seem to work best in our scenario (as the @po_length is often of different lengths in the column)
However, the result is producing numerous 0000 after the result, even though round is set to 0.
Do I need to do something like this to clean it up? This whole solution feels very 'clunky"!
FORMAT(CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(5, 0)), 0)  AS DECIMAL(9,6)),'g15')
ste5anSenior DeveloperCommented:
Don't mix rounding with cutting decimal places, thus you need to change the results data type. So as far as I understand you, your result should be an integer, no decimal places:

DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 0) AS INT);

Open in new window

JP_TechGroupAuthor Commented:
Much cleaner.
If I wanted it to display two places after the decimal, then:
DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS BIGINT);
Correct?
ste5anSenior DeveloperCommented:
When you need decimal places, then use the correct NUMERIC() type:

DECLARE @po_length AS BIGINT;
SET @po_length = 12192000001;

SELECT CAST(ROUND(LEFT(@po_length, ( LEN(@po_length) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS NUMERIC(19, 2));

Open in new window


btw, use the CODE button or the [code][/code] tags to embed code. It's much more reader friendly that way.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JP_TechGroupAuthor Commented:
Makes sense, thank you.
Moving the query into parsing data on the table, I am now getting this error:

Msg 537, Level 16, State 2, Procedure Peachtree_Export, Line 18
Invalid length parameter passed to the LEFT or SUBSTRING function.

CAST(ROUND(LEFT(polineview.[length], ( LEN(polineview.[length]) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS numeric(19,2))

Open in new window

ste5anSenior DeveloperCommented:
What data type is it?

Maybe you should consider posting a concise and complete example. Include table DDL.
JP_TechGroupAuthor Commented:
BIGINT
Here is a sample of the relevant data and the SELECT Statement

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[tbl_demo](
	[pokey] [int] IDENTITY(1,1) NOT NULL,
	[width] [bigint] NULL,
	[length] [bigint] NULL
) ON [PRIMARY]

GO
SET IDENTITY_INSERT [dbo].[tbl_demo] ON 

INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (1, 0, 12192000001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (2, 2438400001, 16865600001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (3, 0, 13716000001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (4, 0, 3048000001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (5, 0, 7315200001)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (6, 0, 0)
INSERT [dbo].[tbl_demo] ([pokey], [width], [length]) VALUES (7, 0, 6096000001)
SET IDENTITY_INSERT [dbo].[tbl_demo] OFF

Open in new window


SELECT
CAST(ROUND(LEFT(tbl_demo.[length], ( LEN(tbl_demo.[length]) - 6 )) / CAST(304.8 AS NUMERIC(19, 5)), 2) AS numeric(19,2))
FROM tbl_demo
WHERE
Width=0;

Open in new window

JP_TechGroupAuthor Commented:
Figured it out. When the length column is zero, it creates an impossible LEN solution.
Thank you for your help sir. I has been some time since I have had to convert numbers in MS SQL. This was educational.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Query Syntax

From novice to tech pro — start learning today.