Solved

# Bulk INSERT with different field types?

Posted on 2008-11-09
450 Views
Hello,

I need to bulk insert a csv file with the following format:
Date,Open,High,Low,Close,Delta,Signal,SystemIdent

They have the following types
Date,Integer,Integer,Integer,Integer,Integer,Integer,Integer,

However with the code below I get an conversion error.
USE [Monday]

GO

/****** Object:  StoredProcedure [dbo].[Bulk_Insert]    Script Date: 11/09/2008 17:10:35 ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

ALTER procedure [dbo].[Bulk_Insert] as BULK insert Data From 'c:\structure\test.txt' with (Fieldterminator =',', ROWTERMINATOR = '\n')

0
Question by:Schuttend

LVL 51

Expert Comment

Could be a number of conversion problems - likely to be the date column in terms of how it is being formatted and then natively understood in the database.

You have a few choices, either use an  openrowset type function where you can cast/convert the incoming value, or, bulk insert into a staging table with varchars., then validate and load the "real" table from the staging table.

0

LVL 51

Expert Comment

Could also be the first line has column headers ? in which case use the FIRSTROW=2 qualifier

Got some sample data ?
0

Author Comment

Hello,

Please find attached sample data pic. Currently the data is imported all as string(varchar 50). But is not very good, because I have to convert data fields each time for calculations.

Regards,
Dennis
Carl3.jpg
0

Author Comment

sorry, wrong pic....
cast3.jpg
0

LVL 51

Expert Comment

OK, when going from DATA to TEMPDATA, then use convert(datetime, date,101)

Looks familiar - have we done something with this data before ?
0

LVL 51

Expert Comment

could also do something like :

SELECT convert(datetime,col1,101),col2,col3,col4,col5,col6,col7,case when isnumeric(col8) > 0 then convert(decimal(18,5),col8) else 0 end
FROM Openrowset(Bulk 'C:\ee\test.txt', formatfile = 'c:\ee\test.fmt.txt', firstrow=2) As A

where the format file is per attached... (have not considered quote encapsulation)...
test.fmt.txt
0

Author Comment

Mark,

What is your opinion about formatting everything first as floats. Import data and thereafter insert it into a destination table? This seems to work, but what is the negative aspect of using floats?

Regardsn
Dennis
0

LVL 51

Expert Comment

Well, floats are an approximation of a number, it is not a native number per se, so instead of 123.360 you can get 123.355555555559  in that case it rounds OK, but in other cases it might not. So, if the numbers do have a quantifiable scal (ie decimals) then far better to use something like decimal(18,6)  ie toal length 18 digits, 6 of which are decimals...
0

LVL 51

Expert Comment

s/b scale not scal
0

Author Comment

I have tried to format fields even with decimal (38,38), but Delta value and Signal value will not go in there. Would you mind take a look at the attached import file what is going wrong?
I though using float would be a option (importing does work), but as you said, it's not accurate.
test.txt
0

LVL 51

Accepted Solution

Well, delta and signal are both float - in fact signal has been exponeniated in some cases (in the raw data - you can see a vale 7.74743856259124E-02 )

which means, probably best to import those two as float, and can even do formatting for each column along the way (it does retrieve all rows):

SELECT convert(datetime,col1,101) as date,convert(decimal(18,5),col2) as [open],convert(decimal(18,5),col3) as high,convert(decimal(18,5),col4) as low,convert(decimal(18,5),col5) as [close],convert(decimal(18,5),convert(float,col6)) as delta,convert(decimal(24,18),convert(float,col7)) as signal,case when isnumeric(col8) > 0 then convert(decimal(18,5),col8) else 0 end as systemident
FROM Openrowset(Bulk 'C:\ee\test.txt', formatfile = 'c:\ee\test.fmt.txt', firstrow=1) A
0

## Featured Post

### Suggested Solutions

Hi all, It is important and often overlooked to understand “Database properties”. Often we see questions about "log files" or "where is the database" and one of the easiest ways to get general information about your database is to use “Database p…
SQL Server engine let you use a Windows account or a SQL Server account to connect to a SQL Server instance. This can be configured immediatly during the SQL Server installation or after in the Server Authentication section in the Server properties …
This video is in connection to the article "The case of a missing mobile phone (https://www.experts-exchange.com/articles/28474/The-Case-of-a-Missing-Mobile-Phone.html)". It will help one to understand clearly the steps to track a lost android phone.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…