Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

MS SQL Bulk Import UTF8 CSV data

Posted on 2014-03-11
5
Medium Priority
?
4,770 Views
Last Modified: 2014-04-05
We have to import UTF8 formated CSV data into a MS SQL 2008R2 DB table.

MS SQL don't support the codepage 65001.

Also the datafiletypes "widechar" and "widenative" stops with the error message that the first column is too long in the first line.

For convert the data file without lost information we try convert the file from UFT8 to UTF16, but UTF16 is not a known coding in .NET.  Only UTF7, UTF8 and UTF32 are knowing codings.

What is good working solution for this problem?
0
Comment
Question by:hpnix4ever
  • 3
4 Comments
 
LVL 143

Assisted Solution

by:Guy Hengel [angelIII / a3]
Guy Hengel [angelIII / a3] earned 2000 total points
ID: 39920803
for the time being, here some MS official statement about why the UTF-8 is not supported (yet):
https://connect.microsoft.com/SQLServer/feedback/details/321839/msft-edw-tapblock-bcp-doesnt-recognize-field-terminator-on-code-page-65001

and .net does support utf-16, namely unicode encoding:
http://msdn.microsoft.com/en-us/library/system.text.unicodeencoding%28v=vs.110%29.aspx
0
 

Accepted Solution

by:
hpnix4ever earned 0 total points
ID: 39923119
Thanks for answer. I will try to convert from UTF8 to UTF16 before SQL bulk import.
------------------------------
Now i had test some convert functions, but nothing of the results can be imported in SQL.  

For read the UTF8 file i had used a streamreader with encoding UFT8. For write as UTF16 i had used a streamwriter with unicode encoding.

For verification i had convert from UTF8 to UTF16 and back to UFT8 that i can load with Excel. The UTF16 file has double size as UTF8, but i can't show with my programs. The reconvert file i can load to Excel and it looks fine. Only the SQL import don't work. Anything by the UTF16 file looks wrong.

For the string conversation i had used all known .NET functions and also some native conversations, but nothing is ready for the bulk import.

The SQL Part is like this:

BULK INSERT tbl_Import
FROM 'C:\temp\test.csv'
WITH
(
  FIELDTERMINATOR=';',
  ROWTERMINATOR='\r\n',
  -- DATAFILETYPE = 'widechar',
  -- DATAFILETYPE= 'widenative',
  CODEPAGE = 1200
 -- CODEPAGE = 1201
)

See the other versions as SQL comments
0
 

Author Comment

by:hpnix4ever
ID: 39953109
The solution was to convert the data before import from UTF-8 to Unicode-BigEndian with a .NET-Programm. Unicode-BigEndian can be imported with BULK-Import.
0
 

Author Closing Comment

by:hpnix4ever
ID: 39979837
It's no solution posible direct inside MS SQL. Only a converison to UnicodeBigEndian with a external Script is working.
0

Featured Post

Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This month, Experts Exchange sat down with resident SQL expert, Jim Horn, for an in-depth look into the makings of a successful career in SQL.
Simulator games are perfect for generating sample realistic data streams, especially for learning data analysis. It is even useful for demoing offerings such as Azure stream analytics, PowerBI etc.
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
This video teaches viewers about errors in exception handling.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question