Bulk INSERT with different field types?


I need to bulk insert a csv file with the following format:

They have the following types

However with the code below I get an conversion error.
USE [Monday]
/****** Object:  StoredProcedure [dbo].[Bulk_Insert]    Script Date: 11/09/2008 17:10:35 ******/
ALTER procedure [dbo].[Bulk_Insert] as BULK insert Data From 'c:\structure\test.txt' with (Fieldterminator =',', ROWTERMINATOR = '\n')

Open in new window

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mark WillsTopic AdvisorCommented:
Could be a number of conversion problems - likely to be the date column in terms of how it is being formatted and then natively understood in the database.

You have a few choices, either use an  openrowset type function where you can cast/convert the incoming value, or, bulk insert into a staging table with varchars., then validate and load the "real" table from the staging table.

Mark WillsTopic AdvisorCommented:
Could also be the first line has column headers ? in which case use the FIRSTROW=2 qualifier

Got some sample data ?
SchuttendAuthor Commented:

Please find attached sample data pic. Currently the data is imported all as string(varchar 50). But is not very good, because I have to convert data fields each time for calculations.

10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

SchuttendAuthor Commented:
sorry, wrong pic....
Mark WillsTopic AdvisorCommented:
OK, when going from DATA to TEMPDATA, then use convert(datetime, date,101)

Looks familiar - have we done something with this data before ?
Mark WillsTopic AdvisorCommented:
could also do something like :

SELECT convert(datetime,col1,101),col2,col3,col4,col5,col6,col7,case when isnumeric(col8) > 0 then convert(decimal(18,5),col8) else 0 end
FROM Openrowset(Bulk 'C:\ee\test.txt', formatfile = 'c:\ee\test.fmt.txt', firstrow=2) As A

where the format file is per attached... (have not considered quote encapsulation)...
SchuttendAuthor Commented:

What is your opinion about formatting everything first as floats. Import data and thereafter insert it into a destination table? This seems to work, but what is the negative aspect of using floats?

Mark WillsTopic AdvisorCommented:
Well, floats are an approximation of a number, it is not a native number per se, so instead of 123.360 you can get 123.355555555559  in that case it rounds OK, but in other cases it might not. So, if the numbers do have a quantifiable scal (ie decimals) then far better to use something like decimal(18,6)  ie toal length 18 digits, 6 of which are decimals...
Mark WillsTopic AdvisorCommented:
s/b scale not scal
SchuttendAuthor Commented:
I have tried to format fields even with decimal (38,38), but Delta value and Signal value will not go in there. Would you mind take a look at the attached import file what is going wrong?
I though using float would be a option (importing does work), but as you said, it's not accurate.
Mark WillsTopic AdvisorCommented:
Well, delta and signal are both float - in fact signal has been exponeniated in some cases (in the raw data - you can see a vale 7.74743856259124E-02 )

which means, probably best to import those two as float, and can even do formatting for each column along the way (it does retrieve all rows):

SELECT convert(datetime,col1,101) as date,convert(decimal(18,5),col2) as [open],convert(decimal(18,5),col3) as high,convert(decimal(18,5),col4) as low,convert(decimal(18,5),col5) as [close],convert(decimal(18,5),convert(float,col6)) as delta,convert(decimal(24,18),convert(float,col7)) as signal,case when isnumeric(col8) > 0 then convert(decimal(18,5),col8) else 0 end as systemident
FROM Openrowset(Bulk 'C:\ee\test.txt', formatfile = 'c:\ee\test.fmt.txt', firstrow=1) A

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server 2008

From novice to tech pro — start learning today.