Solved

Import csv for SQL Server

Posted on 2016-09-29
17
47 Views
Last Modified: 2016-09-30
Good afternoon,

I would like your help again.

I have some files in csv format to be imported into SQL Server.

The files have the following format:

report_SLA08_20160601.csv

Where 01 would be the 06 month and 2016 year.

I have a file for each day of this, I need to import these files to the SQL SERVER automatically where each file would be a different table.

Could help me do this in powershell because the way I'm doing, it takes a long time.

Detail:

Each CSV file has a character | the end of the line, I have to take this character of each file to import does not cause error.

attached a file of the model.

Thank you.
REPORT_SLA08_20160601.csv
0
Comment
Question by:Alex
  • 8
  • 6
  • 3
17 Comments
 
LVL 28

Expert Comment

by:Pawan Kumar
ID: 41821915
Here it is..

SELECT * INTO #pawan 
FROM OPENROWSET (BULK 'E:\pawan1.csv', SINGLE_CLOB) MyFile 

SELECT * FROM #pawan

Open in new window

0
 

Author Comment

by:Alex
ID: 41821976
Hello,

The file is not being created in tabular format between the columns.

an attached print
select.png
0
 
LVL 28

Accepted Solution

by:
Pawan Kumar earned 333 total points
ID: 41822055
Here it is .. Enjoy !!

Remove last | from the CSV file. It is a bug.

CREATE TABLE testCSV
(
	 Name NVARCHAR(1000)
	,Disponibilidade NVARCHAR(1000)
	,Data NVARCHAR(1000)
	,Report NVARCHAR(1000)
	,Country NVARCHAR(1000)
	,Last_refresh NVARCHAR(1000)
)
GO

BULK
INSERT testCSV
FROM 'E:\Pawan1.CSV'
WITH
(
FirstRow = 2,
FIELDTERMINATOR = ';',
ROWTERMINATOR = '|'
)
GO

SELECT * FROM testCSV

Open in new window

Pawan1.csv
1
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 26

Assisted Solution

by:Zberteoc
Zberteoc earned 167 total points
ID: 41822087
1. Create this table in your database:
CREATE TABLE [dbo].[report_SLA08](
	[Name] [varchar](250) NULL,
	[Disponibilidade] [varchar](10) NULL,
	[Data] [varchar](50) NULL,
	[Report] [varchar](250) NULL,
	[Country] [varchar](250) NULL,
	[Last_refresh] [varchar](50) NULL
) ON [PRIMARY]

GO

Open in new window

2. Create a working folder on your computer that has BCP installed(any computer with SQL server on it) and copy all the CSV files into it. For example purposes I named it C:\TEMP\EE :

3. In the same folder create a batch file named _import_report_SLA08.bat with the following content:
@echo off
for /r C:\TEMP\EE %%f in (*.csv) do (
bcp [YourDB].[dbo].[report_SLA08] in %%f -T -c -t; -r0x7c0d0a -SDEVSQL03 > %%f.output
)

Open in new window

Of course you will replace YourDB with the real database name.

3. Open the folder and execute the batch file by double clicking on it.


This script will look in the folder and it will search all the .CSV files and loop through them and import them one by one into the table created at step 1. For every CSV file there will be created an output file with the same name plus the .output extension. Those files will keep the import results.

EDIT:

I forgot to mention that the BCP command uses as row separator the {|}{CR}{LF} sequence so you don't have to worry about the | (pipe) at the end of the rows.

Another thing is that the Disponibilidade column has comma as decimal separator instead of dot. For this reason the column is of varchar type and you will have to replace the comma with dot in the table after the import:
update [YourDB].[dbo].[report_SLA08] set [Disponibilidade] =replace([Disponibilidade] ,',','.')

Open in new window

Only then you will be able to convert it to number, I think it is a pecentage.
0
 

Author Comment

by:Alex
ID: 41822314
I run the script, I am getting the below error:


Starting copy...
SQLState = 22001, NativeError = 0
Error = [Microsoft][SQL Server Native Client 10.0]String data, right truncation

0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1
0
 
LVL 26

Expert Comment

by:Zberteoc
ID: 41822326
I used the sample from the question in 2 separate files and both were imported with no issues. Please post here the content of the batch file and the sample you tried to import. Is it possible that some columns are wider that the size in the table?
csv_import.JPGreport_SLA08_20160601.csv
0
 

Author Comment

by:Alex
ID: 41822517
Hello

I'm using the same example you, but for me the error has already informed.

Below the contents of my .bat file

@echo off
for /r E:\temp\EE %%f in (*.csv) do (
bcp [NEW_SLA].[dbo].[REPORT_SLA08] in %%f -U -T -c -t; -r0x7c0d0a -SVBR001001-015\AUDITORIA > %%f.output
)
0
 
LVL 26

Expert Comment

by:Zberteoc
ID: 41822528
What about the [NEW_SLA].[dbo].[REPORT_SLA08] table? Does it have the same structure like the one I posted?

If yes then it has to work. Use the file I attached. Just download it.
0
 

Author Comment

by:Alex
ID: 41822553
I created the Report Sla 08 table just with the script you sent.
I used the same file that I have attached, I do not know if it could be something related to my SQL client, but I think not.
0
 
LVL 26

Expert Comment

by:Zberteoc
ID: 41822573
Remove -U from that line in the batch file. I don't use that. I just noticed it now.
0
 

Author Comment

by:Alex
ID: 41822661
Unfortunately had the same error, try to run on another server.      :-(


Starting copy...
SQLState = 22001, NativeError = 0
Error = [Microsoft][SQL Server Native Client 11.0]String data, right truncation

23 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1      Average : (23000.00 rows per sec.)
0
 

Author Comment

by:Alex
ID: 41822679
Now I work, I created a table with some varchar fields with higher values.

How do I leave only the first line with field name?

attached the result
Capturar.PNG
0
 

Author Comment

by:Alex
ID: 41822708
I also had to remove the option -r0x7c0d0a
only then stopped display error
0
 
LVL 28

Assisted Solution

by:Pawan Kumar
Pawan Kumar earned 333 total points
ID: 41822822
@Alex - Have you tried the code I have given you ? It was really easy to understand.
0
 
LVL 26

Expert Comment

by:Zberteoc
ID: 41823378
I also had to remove the option -r0x7c0d0a
only then stopped display error
If you remove that you will have the  pipe, |, attached to the last date column. Like I said it worked for me and I am not sure why doesn't work for you.

To skip first row in BCP you can try -F1 switch, to ignore the first row. Again, it worked for me as well, the header row is not imported without using -F1 or any other switch than the ones I posted.
0
 

Author Comment

by:Alex
ID: 41823986
@Pawan

Sorry about the delay in response, your solution worked perfectly so, now just need to tailor it to import all the files at once.
0
 
LVL 26

Expert Comment

by:Zberteoc
ID: 41824161
You just run that batch file. It will import all the csv files from the folder one by one in one run to the same table.
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Why is this different from all of the other step by step guides?  Because I make a living as a DBA and not as a writer and I lived through this experience. Defining the name: When I talk to people they say different names on this subject stuff l…
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Viewers will learn how the fundamental information of how to create a table.

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question