Solved

Import csv for SQL Server

Posted on 2016-09-29
17
36 Views
Last Modified: 2016-09-30
Good afternoon,

I would like your help again.

I have some files in csv format to be imported into SQL Server.

The files have the following format:

report_SLA08_20160601.csv

Where 01 would be the 06 month and 2016 year.

I have a file for each day of this, I need to import these files to the SQL SERVER automatically where each file would be a different table.

Could help me do this in powershell because the way I'm doing, it takes a long time.

Detail:

Each CSV file has a character | the end of the line, I have to take this character of each file to import does not cause error.

attached a file of the model.

Thank you.
REPORT_SLA08_20160601.csv
0
Comment
Question by:Alex
  • 8
  • 6
  • 3
17 Comments
 
LVL 16

Expert Comment

by:Pawan Kumar Khowal
Comment Utility
Here it is..

SELECT * INTO #pawan 
FROM OPENROWSET (BULK 'E:\pawan1.csv', SINGLE_CLOB) MyFile 

SELECT * FROM #pawan

Open in new window

0
 

Author Comment

by:Alex
Comment Utility
Hello,

The file is not being created in tabular format between the columns.

an attached print
select.png
0
 
LVL 16

Accepted Solution

by:
Pawan Kumar Khowal earned 333 total points
Comment Utility
Here it is .. Enjoy !!

Remove last | from the CSV file. It is a bug.

CREATE TABLE testCSV
(
	 Name NVARCHAR(1000)
	,Disponibilidade NVARCHAR(1000)
	,Data NVARCHAR(1000)
	,Report NVARCHAR(1000)
	,Country NVARCHAR(1000)
	,Last_refresh NVARCHAR(1000)
)
GO

BULK
INSERT testCSV
FROM 'E:\Pawan1.CSV'
WITH
(
FirstRow = 2,
FIELDTERMINATOR = ';',
ROWTERMINATOR = '|'
)
GO

SELECT * FROM testCSV

Open in new window

Pawan1.csv
1
 
LVL 26

Assisted Solution

by:Zberteoc
Zberteoc earned 167 total points
Comment Utility
1. Create this table in your database:
CREATE TABLE [dbo].[report_SLA08](
	[Name] [varchar](250) NULL,
	[Disponibilidade] [varchar](10) NULL,
	[Data] [varchar](50) NULL,
	[Report] [varchar](250) NULL,
	[Country] [varchar](250) NULL,
	[Last_refresh] [varchar](50) NULL
) ON [PRIMARY]

GO

Open in new window

2. Create a working folder on your computer that has BCP installed(any computer with SQL server on it) and copy all the CSV files into it. For example purposes I named it C:\TEMP\EE :

3. In the same folder create a batch file named _import_report_SLA08.bat with the following content:
@echo off
for /r C:\TEMP\EE %%f in (*.csv) do (
bcp [YourDB].[dbo].[report_SLA08] in %%f -T -c -t; -r0x7c0d0a -SDEVSQL03 > %%f.output
)

Open in new window

Of course you will replace YourDB with the real database name.

3. Open the folder and execute the batch file by double clicking on it.


This script will look in the folder and it will search all the .CSV files and loop through them and import them one by one into the table created at step 1. For every CSV file there will be created an output file with the same name plus the .output extension. Those files will keep the import results.

EDIT:

I forgot to mention that the BCP command uses as row separator the {|}{CR}{LF} sequence so you don't have to worry about the | (pipe) at the end of the rows.

Another thing is that the Disponibilidade column has comma as decimal separator instead of dot. For this reason the column is of varchar type and you will have to replace the comma with dot in the table after the import:
update [YourDB].[dbo].[report_SLA08] set [Disponibilidade] =replace([Disponibilidade] ,',','.')

Open in new window

Only then you will be able to convert it to number, I think it is a pecentage.
0
 

Author Comment

by:Alex
Comment Utility
I run the script, I am getting the below error:


Starting copy...
SQLState = 22001, NativeError = 0
Error = [Microsoft][SQL Server Native Client 10.0]String data, right truncation

0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1
0
 
LVL 26

Expert Comment

by:Zberteoc
Comment Utility
I used the sample from the question in 2 separate files and both were imported with no issues. Please post here the content of the batch file and the sample you tried to import. Is it possible that some columns are wider that the size in the table?
csv_import.JPGreport_SLA08_20160601.csv
0
 

Author Comment

by:Alex
Comment Utility
Hello

I'm using the same example you, but for me the error has already informed.

Below the contents of my .bat file

@echo off
for /r E:\temp\EE %%f in (*.csv) do (
bcp [NEW_SLA].[dbo].[REPORT_SLA08] in %%f -U -T -c -t; -r0x7c0d0a -SVBR001001-015\AUDITORIA > %%f.output
)
0
 
LVL 26

Expert Comment

by:Zberteoc
Comment Utility
What about the [NEW_SLA].[dbo].[REPORT_SLA08] table? Does it have the same structure like the one I posted?

If yes then it has to work. Use the file I attached. Just download it.
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 

Author Comment

by:Alex
Comment Utility
I created the Report Sla 08 table just with the script you sent.
I used the same file that I have attached, I do not know if it could be something related to my SQL client, but I think not.
0
 
LVL 26

Expert Comment

by:Zberteoc
Comment Utility
Remove -U from that line in the batch file. I don't use that. I just noticed it now.
0
 

Author Comment

by:Alex
Comment Utility
Unfortunately had the same error, try to run on another server.      :-(


Starting copy...
SQLState = 22001, NativeError = 0
Error = [Microsoft][SQL Server Native Client 11.0]String data, right truncation

23 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1      Average : (23000.00 rows per sec.)
0
 

Author Comment

by:Alex
Comment Utility
Now I work, I created a table with some varchar fields with higher values.

How do I leave only the first line with field name?

attached the result
Capturar.PNG
0
 

Author Comment

by:Alex
Comment Utility
I also had to remove the option -r0x7c0d0a
only then stopped display error
0
 
LVL 16

Assisted Solution

by:Pawan Kumar Khowal
Pawan Kumar Khowal earned 333 total points
Comment Utility
@Alex - Have you tried the code I have given you ? It was really easy to understand.
0
 
LVL 26

Expert Comment

by:Zberteoc
Comment Utility
I also had to remove the option -r0x7c0d0a
only then stopped display error
If you remove that you will have the  pipe, |, attached to the last date column. Like I said it worked for me and I am not sure why doesn't work for you.

To skip first row in BCP you can try -F1 switch, to ignore the first row. Again, it worked for me as well, the header row is not imported without using -F1 or any other switch than the ones I posted.
0
 

Author Comment

by:Alex
Comment Utility
@Pawan

Sorry about the delay in response, your solution worked perfectly so, now just need to tailor it to import all the files at once.
0
 
LVL 26

Expert Comment

by:Zberteoc
Comment Utility
You just run that batch file. It will import all the csv files from the folder one by one in one run to the same table.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

"Migrate" an SMTP relay receive connector to a new server using info from an old server.
This article will help you understand what HashTables are and how to use them in PowerShell.
Familiarize people with the process of utilizing SQL Server functions from within Microsoft Access. Microsoft Access is a very powerful client/server development tool. One of the SQL Server objects that you can interact with from within Microsoft Ac…
Via a live example, show how to extract insert data into a SQL Server database table using the Import/Export option and Bulk Insert.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now