T-SQL Inport/Export data to SQL/CSV without using SSIS : T-SQL

May 17, 2008 10:55am pdt

1. chapmandew 1,124,487
2. angelIII 797,044
3. aneeshat... 383,966
4. ee_rlee 334,902
5. dtodd 258,729
6. mark_wills 251,810
7. ScottPle... 190,791
8. acperkins 173,198
9. ptjcb 115,237
10. brejk 104,465
11. EugeneZ 104,309
12. asvforce 103,642
13. bhess1 97,730
14. dportas 80,581
15. BrandonG... 80,437

1. angelIII 9,752,157
2. aneeshat... 4,441,515
3. acperkins 4,161,658
4. ScottPle... 4,004,843
5. arbert 2,245,195
6. Lowfatsp... 2,162,122
7. rafrancisco 1,460,132
8. imran_fast 1,450,838
9. EugeneZ 1,380,471
10. Hilaire 1,221,775
11. Nightman 1,136,404
12. chapmandew 1,124,487
13. Mr_Peerapol 1,087,815
14. ptjcb 1,048,696
15. nmcdermaid 1,035,158

03.12.2008 at 07:34AM PDT, ID: 23235175

	[x] Attachment Details

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

9.1

T-SQL Inport/Export data to SQL/CSV without using SSIS

Zone: MS SQL Server

Tags: T-SQL

Hi,
I have a situation where I'm transferring data between two MS SQL 2005 servers and one of the tasks is to retrieve data from one SQL server [S1] (it is also a Linked Server) and export it to a CSV file on a second server with SQL 2005 running [S2].
Rather simple would be to use SSIS and that is what i have been trying to use.
I have few of these SSIS packages that export data to csv file (Text Qualifier set as " !!!) in a SQL Job that runs nightly.
The problem I continuously run into is the connection to S1 cannot be established. I know that the connection is solid as when use a query in that same job as one of the steps it works. To fix this I go into the job, open the offending step, click on the "Data Sources" tab and enter a password into the Source Connection String. The reason for doing this is when I'm creating the SSIS package I choose the option to "Do not save sensitive data". Every now and then I have to either reapply the password or recreate the SSIS package and it's just a pain.

I'm thinking of using the BCP utility to do this but found some super extensive scripts when using it placed in stored procvedures which really seems like an overkill. I would rather not use a SSIS package at all and do this via SQL if at all possible. There's got to be a simple way to export data to a CSV file and in the same token to import data from a CSV file into a SQL 2005 tables.

I would appreciate specific examples on how to achieve both the import and export whatever the method. Few important things to note are that the columns are separated by a comma and all columns are within " (double quotes). This goes for data being exported to file from SQL 2005 table as well as provided in a file for import to a SQL 2005 table.

Sorry for the long post. Thanks a lot.

Start your free trial to view this solution

Question Stats

Zone: Microsoft

Question Asked By: ttepandi

Solution Provided By: Zberteoc

Participating Experts: 1

Solution Grade: A

Translate:

Loading Advertisement...

Microsoft

Internet Protocols
Applications

Development
OS

Hardware
Windows Security

Apple

Operating Systems
Hardware

Programming
Networking

Software

Internet

Search Engines
File Sharing
WebTrends / Stats
Spy / Ad Blockers

Web Browsers
New Net Users
Web Development
Chat / IM

Anti Spam
Web Servers
Anti-Virus
Email Clients

Gamers

Tips
Online / MMORPG
Puzzle
Emulators

Action / Adventure
Role Playing
Consoles
Game Programming

Strategy
Sports
Misc
Computer Games

Digital Living

Hardware
New Net Users
New Users

Software
Digital Music
Gaming World

Home Security
Apple
Networking Hardware

Virus & Spyware

Vulnerabilities
IDS
Encryption
Anti-Virus

Operating Systems Security
Software Firewalls
WebApplications
Cell Phones

Operating Systems
Internet
Hardware Firewalls

Hardware

Handhelds / PDAs
Displays / Monitors
Components
Networking Hardware

Peripherals
Laptops/Notebooks
Storage
Servers

Desktops
New Users
Misc
Apple

Software

System Utilities
Industry Specific
Network Management
Photos / Graphics
Page Layout
VMWare
Misc
Web Development

OS
CYGWIN
Voice Recognition
Message Queue
Quality Assurance
Security
Firewalls
MultiMedia Applications

Development
Database
Office / Productivity
Business Management
OS/2 Apps
Server Software
Internet / Email

ITPro

OS
Storage
Encryption
Operating Systems Security
Apple Hardware
Laptops & Notebooks
Servers
Networking Hardware
Peripherals

Devices
Displays / Monitors
WebTrends / Stats
Search Engines
Firewalls
WebApplications
IDS
Vulnerabilities
Email Clients

File Sharing
Spy / Ad Blockers
Web Browsers
Web Servers
Networking
Anti-Virus
Chat / IM
Anti Spam

Developer

Web Servers
Web Browsers
Game Programming
Dev Tools
Industry Specific
Office / Productivity

Database
CYGWIN
Web Development
Search Engines
File Sharing
WebTrends / Stats

Programming
Content Management
Application Servers
Protocols

Storage

Removable Backup Media
Storage Technology
Servers

Grid
Remote Access
Backup / Restore

Misc
Hard Drives

Miscellaneous
Security
Development
Linux
VMWare

MainFrame OS
Unix
Apple
OS / 2
AS / 400

BeOS
Microsoft
VMS / OpenVMS

Database

Oracle
Miscellaneous
MySQL
Software
Sybase
Contact Management
PostgreSQL
Data Manipulation
Clarion
InterSystems Cache

Siebel
MUMPS
OLAP
SQLBase
SAS
GIS & GPS
4GL
Berkeley DB
DB2
Informix

Interbase / Firebird
FoxPro
Reporting
LDAP
Filemaker Pro
MS SQL Server
dBase
MS Access

Security

Misc
Web Browsers
Software Firewalls
Operating Systems Security
File Sharing

Spy / Ad Blockers
Vulnerabilities
WebApplications
IDS
Anti-Virus

Encryption
Anti Spam
Email Clients
VPN
Chat / IM

Programming

Editors IDEs
Installation
Handhelds / PDAs
Multimedia Programming
System / Kernel

Algorithms
Game
Signal Processing
Project Management
Open Source

Database
Misc
Languages
Processor Platforms
Theory

Web Development

Scripting
Blogs
Web Servers
Software
Search Engines
Web Graphics
Images

Internet Marketing
Images and Photos
Components
Document Imaging
Web Languages/Standards
Illustration
WebApplications

Fonts
WebTrends / Stats
Authoring
Digital Camera Software
Miscellaneous

Networking

Protocols
Apple Networking
Network Management
Message Queue
Application Servers
Content Management
File Servers
Email Servers
Misc
Java Editors & IDEs

Wireless
Networking Hardware
Backup / Restore
System Utilities
ISPs & Hosting
Web Servers
Storage Technology
Removable Backup Media
Servers
Broadband

Grid
OS / 2
Novell Netware
Unix Networking
Windows Networking
Security
Telecommunications
Operating Systems
Linux Networking

Other

Community Advisor
Lounge
Community Support
New Net Users

Philosophy / Religion
Math / Science
Miscellaneous
URLs

Expert Lounge
Politics
Puzzles / Riddles

Community Support

Suggestions
New to EE
New Topics
Community Advisor

CleanUp
Announcements
General

Feedback
Input
EE Bugs

03.12.2008 at 07:49AM PDT, ID: 21106542

Rank: Guru

Zberteoc:

You can try to use OPENROWSET:

SELECT
*
FROM
OPENROWSET('MSDASQL',--provider name (ODBC)
'Driver={Microsoft Text Driver (*.txt; *.csv)};
DEFAULTDIR=C:\Temp\;Extensions=CSV;',--data source
'SELECT * yourfile.CSV')

if it's not enable run:
sp_configure 'show advanced options',1
RECONFIGURE WITH override
go
sp_configure 'Ad Hoc Distributed Queries',1
RECONFIGURE WITH override
go

details on how to use OPENROWSET:

http://www.databasejournal.com/features/mssql/article.php/10894_3331881_2

in regards with the delimiter, there is an option in the openrowset parameters that will take care of it but i can't remember now. do some search.

03.12.2008 at 08:31AM PDT, ID: 21107080

ttepandi:

OK, So this would be for quering a CSV or TXT file and I can then just do

INSERT INTO mytable
SELECT * FROM OPENROWSET (...)

That's great! I just have one problem here, the CSV file does not have column rows specified and it's got four date columns, two of which may be identical, so I'm gettign an error: The column name "2003-05-27 00:00:00" is a duplicate.

Is there a way to specify column names?

03.12.2008 at 08:34AM PDT, ID: 21107116

Rank: Guru

Zberteoc:

You can specify that the file has no headers, the switch is like NHR or something but I don't remember exactly how you specify that within the oprnrowset function.

03.12.2008 at 08:42AM PDT, ID: 21107194

Rank: Guru

Zberteoc:

Here I think I found an example. The HDR=NO speciffies that there are no column headers.

                       
                                SELECT *
FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0',
'Text;Database=C:\TEST\;HDR=NO',
'SELECT * FROM test1.csv')

Open in New Window

03.12.2008 at 08:42AM PDT, ID: 21107200

Rank: Guru

Zberteoc:

It should work with the MSDASQL provider as well.

03.12.2008 at 09:17AM PDT, ID: 21107587

ttepandi:

Perfect! Thank you! I could not get the MSDASQL to work with the no header option but using the Jet driver works. Do you know which one would be faster? Are there other reasons to consider?
This is a monster file of 27 MB and about 80K rows. using SSIS it took about 7 minutes. using this method it took about... well it's 14 minutes and still running. :)

03.12.2008 at 09:23AM PDT, ID: 21107651

Rank: Guru

Zberteoc:

You would have to try. You could add column headers to your csv file and try to import it.

The fastest way to get a csv file in is with BULK INSERT:

BULK INSERT pubs..publishers2 FROM 'c:\newpubs.dat'
WITH (
DATAFILETYPE = 'char',
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)

if you need the no headers you will need to use it with format.

Accepted Solution

03.12.2008 at 09:26AM PDT, ID: 21107693

ttepandi:

Thank you. I'll experiment with this.

What about exporting to a CSV file? Export as columns separated by comma, rows terminated by \r\t and values encolsed in double quotes.

03.12.2008 at 09:36AM PDT, ID: 21107819

Rank: Guru

Zberteoc:

for that I recommend you the bcp utility is the most powerfull for that, the only "problem" is that it is an external excutable so you can only invoke it from SQL using the xp_cmdshell extended stored procedure and that need permissisons to be used. If you use it in a secure context like form a job is OK but if you need to grant permissions to users not so OK. You can still use it though.

examples here:

http://www.simple-talk.com/sql/database-administration/creating-csv-files-using-bcp-and-stored-procedures/

Assisted Solution

03.12.2008 at 09:49AM PDT, ID: 21107991

ttepandi:

OK, thanks!
This is what I also found earlier where if the data needs to be in quotes it actually would need to be manually formatted.

So there is no option to just do this using a switch similar to the -t?

03.12.2008 at 09:58AM PDT, ID: 21108103

Rank: Guru

Zberteoc:

I'm not sure I understand. You can set delimiters with BCP as well.

You could also use OPENROWSET to export your data, however the CSV file has to exist as is not created on the fly like when you use BCP:

                       
                                INSERT INTO 
	OPENROWSET('Microsoft.Jet.OLEDB.4.0',
	'Text;Database=C:\TEST\;HDR=NO',
	'SELECT * FROM test1.csv')
SELECT 
	<columns_list_here>
FROM 
	(...)

Open in New Window

03.12.2008 at 10:11AM PDT, ID: 21108281

ttepandi:

What I meant was I need to "quote encapsulate" character data for all columns exported into the CSV file. And there seems to be no way to do this using a switch. Instead the solution I have come across and the one you also provided suggest doing this in a stored procedure and "add" the quotes to the data:
SELECT
column1 = '"' + convert(varchar(8), column1, 112) + '"' ,
column2 = '"' + convert(varchar(8), column2, 112) + '"'
FROM mytable

and then do the bcp trick.

I'm just surprised that there is no simpler way to encolse fields/columns in quotes when exporting to CSV.

I came across something like using a format file. Would you suggest using this?

20080236-EE-VQP-29 / EE_QW_2_20070628