Read comma delimited file using fso : file, sql, delimited, comma, read

May 17, 2008 11:31am pdt

1. chapmandew 73,360
2. acperkins 41,384
3. nmcdermaid 29,225
4. brejk 25,200
5. cs97jjm3 20,048
6. PedroCGD 19,230
7. angelIII 17,110
8. dready 12,500
9. bhess1 9,700
10. Zberteoc 9,350
11. JimFive 8,375
12. mark_wills 8,000
13. dbbishop 7,300
14. ptjcb 7,120
15. paulop1975 7,100

1. acperkins 309,633
2. angelIII 157,526
3. arbert 149,684
4. nmcdermaid 141,942
5. chapmandew 73,360
6. EugeneZ 57,168
7. matthews... 42,304
8. aneeshat... 39,799
9. ptjcb 38,638
10. rboyd56 34,850
11. bhess1 29,280
12. Sirees 27,902
13. twoboats 27,515
14. jltoops 27,275
15. cs97jjm3 26,578

10.31.2007 at 04:06PM PDT, ID: 22931253

	[x] Attachment Details

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

7.4

Read comma delimited file using fso

Zones: MS SQL DTS, MS SQL Server

Tags: file, sql, delimited, comma, read

Running Sql Server 2000 on SBS 2003 premium fully patched.

I have a fairly large text file filled with comma delimited data. I need to read it as fast as possible.
I am not an fso expert.

Yes, i have tried using DTS (data file pump) to open the text file and write the data to a table. Lots of problems. So thought I could write a little VBScrpt job which could read the text file using fso.

Who has some sample code I could view showing:
1.) how to open the text file

2.) How to load it into an array so I can read each field

3.) How to determine when I have read a line of data

4.) How to determine when I have finished reading the entire file.

If you have better ideas I am eager to learn.

I am assigning 500 points because I need a solution soon.

Start your free trial to view this solution

Question Stats

Zone: Microsoft

Question Asked By: donpick

Solution Provided By: lexiflex

Participating Experts: 3

Solution Grade: A

Translate:

Loading Advertisement...

Microsoft

Internet Protocols
Applications

Development
OS

Hardware
Windows Security

Apple

Operating Systems
Hardware

Programming
Networking

Software

Internet

Search Engines
File Sharing
WebTrends / Stats
Spy / Ad Blockers

Web Browsers
New Net Users
Web Development
Chat / IM

Anti Spam
Web Servers
Anti-Virus
Email Clients

Gamers

Tips
Online / MMORPG
Puzzle
Emulators

Action / Adventure
Role Playing
Consoles
Game Programming

Strategy
Sports
Misc
Computer Games

Digital Living

Hardware
New Net Users
New Users

Software
Digital Music
Gaming World

Home Security
Apple
Networking Hardware

Virus & Spyware

Vulnerabilities
IDS
Encryption
Anti-Virus

Operating Systems Security
Software Firewalls
WebApplications
Cell Phones

Operating Systems
Internet
Hardware Firewalls

Hardware

Handhelds / PDAs
Displays / Monitors
Components
Networking Hardware

Peripherals
Laptops/Notebooks
Storage
Servers

Desktops
New Users
Misc
Apple

Software

System Utilities
Industry Specific
Network Management
Photos / Graphics
Page Layout
VMWare
Misc
Web Development

OS
CYGWIN
Voice Recognition
Message Queue
Quality Assurance
Security
Firewalls
MultiMedia Applications

Development
Database
Office / Productivity
Business Management
OS/2 Apps
Server Software
Internet / Email

ITPro

OS
Storage
Encryption
Operating Systems Security
Apple Hardware
Laptops & Notebooks
Servers
Networking Hardware
Peripherals

Devices
Displays / Monitors
WebTrends / Stats
Search Engines
Firewalls
WebApplications
IDS
Vulnerabilities
Email Clients

File Sharing
Spy / Ad Blockers
Web Browsers
Web Servers
Networking
Anti-Virus
Chat / IM
Anti Spam

Developer

Web Servers
Web Browsers
Game Programming
Dev Tools
Industry Specific
Office / Productivity

Database
CYGWIN
Web Development
Search Engines
File Sharing
WebTrends / Stats

Programming
Content Management
Application Servers
Protocols

Storage

Removable Backup Media
Storage Technology
Servers

Grid
Remote Access
Backup / Restore

Misc
Hard Drives

Miscellaneous
Security
Development
Linux
VMWare

MainFrame OS
Unix
Apple
OS / 2
AS / 400

BeOS
Microsoft
VMS / OpenVMS

Database

Oracle
Miscellaneous
MySQL
Software
Sybase
Contact Management
PostgreSQL
Data Manipulation
Clarion
InterSystems Cache

Siebel
MUMPS
OLAP
SQLBase
SAS
GIS & GPS
4GL
Berkeley DB
DB2
Informix

Interbase / Firebird
FoxPro
Reporting
LDAP
Filemaker Pro
MS SQL Server
dBase
MS Access

Security

Misc
Web Browsers
Software Firewalls
Operating Systems Security
File Sharing

Spy / Ad Blockers
Vulnerabilities
WebApplications
IDS
Anti-Virus

Encryption
Anti Spam
Email Clients
VPN
Chat / IM

Programming

Editors IDEs
Installation
Handhelds / PDAs
Multimedia Programming
System / Kernel

Algorithms
Game
Signal Processing
Project Management
Open Source

Database
Misc
Languages
Processor Platforms
Theory

Web Development

Scripting
Blogs
Web Servers
Software
Search Engines
Web Graphics
Images

Internet Marketing
Images and Photos
Components
Document Imaging
Web Languages/Standards
Illustration
WebApplications

Fonts
WebTrends / Stats
Authoring
Digital Camera Software
Miscellaneous

Networking

Protocols
Apple Networking
Network Management
Message Queue
Application Servers
Content Management
File Servers
Email Servers
Misc
Java Editors & IDEs

Wireless
Networking Hardware
Backup / Restore
System Utilities
ISPs & Hosting
Web Servers
Storage Technology
Removable Backup Media
Servers
Broadband

Grid
OS / 2
Novell Netware
Unix Networking
Windows Networking
Security
Telecommunications
Operating Systems
Linux Networking

Other

Community Advisor
Lounge
Community Support
New Net Users

Philosophy / Religion
Math / Science
Miscellaneous
URLs

Expert Lounge
Politics
Puzzles / Riddles

Community Support

Suggestions
New to EE
New Topics
Community Advisor

CleanUp
Announcements
General

Feedback
Input
EE Bugs

10.31.2007 at 04:15PM PDT, ID: 20189881

Rank: Master

lexiflex:

The fastest and safest way to import a delimited text file in SQL Server is by using the BCP utility that is installed with your client tools.

This Microsoft page describes how it works: http://msdn2.microsoft.com/en-us/library/ms162802.aspx

10.31.2007 at 04:33PM PDT, ID: 20189964

Rank: Master

lexiflex:

But if you're set on using fso (wich can take a while!), here's an example of how to open a file, read the lines and closing it after you've read everything (steps 1, 3 and 4).

' Constants used for file manipulation
Const ForReading = 1, ForWriting = 2, ForAppending = 8

' Instantiate global FileSystemObject
Set fso = CreateObject("Scripting.FileSystemObject")

' Which file to read
Set oFile = fso.GetFile("C:\MyFile.txt")
Set fSource = oFile.OpenAsTextStream(ForReading)

Do While fSource.AtEndOfStream <> True
' Read a line
strLine = fSource.ReadLine()

' Proces the line
MsgBox strLine
Loop

' Close file
fSource.Close()

MsgBox "End batch. "

Accepted Solution

10.31.2007 at 04:52PM PDT, ID: 20190034

donpick:

lexiflex:

Thank you for your help. How can I read each "field" in the comma delimited file?
How would split help me?

10.31.2007 at 05:35PM PDT, ID: 20190168

drjimcook:

Although BCP and BULK INSERT are much faster than DTS, they're much less sophisticated as well (e.g., re implicit data-type conversions). Also, some of your problems may represent defects in the CSV file. For example, Excel and Excel-based export tools are notorious for stripping leading zeros from numbers in text-formatted cells. I find it easiest to BCP import into an intermediate table with nvarchar fields, and then write the final table using CAST, after repairing defects as needed.

10.31.2007 at 06:52PM PDT, ID: 20190437

Rank: Genius

nmcdermaid:

As already mentioned..... if speed is a requirement then I wouldn't recommend FSO or any looping/scripting solution.

Assuming the file is clean, you just run this on a command line

BCP <importtable> IN <YourFile> -t , -c -r \n -S<YouSQLServer> -T

.. no coding whatsoever, and waaaaaaaay faster to import. Give it a go and if it works straight off, there's your solution.

You can also use the -h "TABLOCK" command line parameter to exclusive lock the table
If the data is in the same order as the tables clustered index, you can used the ORDER hint to speed things up

Regardless of your load method, on the database side you can also:

Drop indexes/triggers before loading the data and recreate them after.
Set the recovery model to simple

10.31.2007 at 11:28PM PDT, ID: 20191075

Rank: Master

lexiflex:

Indeed split can be used:
strLine = "a, ab, d"

MyArray = Split(strLine, ",", -1, 1)

MyArray(0) contains "a".
MyArray(1) contains "ab".
MyArray(2) contains "d".

The -1 parameter means you want all parts of the split string and the 1 means comparison should be text based.

11.01.2007 at 10:44AM PDT, ID: 20194976

donpick:

nmcdermaid

Yes, agreed. However, the text file comes from a legacy application. Some of the data is numeric with E's in it (ex: 1.60422438e+09) I tried loading the data into a SQL table having all fields defined as varchar a while ago and SQL complained of data mismatches. SQL just cannot seem to handle these kinds of numbers. I am not a SQL expert so may be you have ideas on how to handle data like this?

Right now I am using DTS to load the text table into Excel and then using Access to read the Excel file. It's very slow but Access can handle numbers like 1.60422438e+09 . It was the only way I could think of handling this.

I studied FSO and it seemed I could read the text file directly, not use Excel, etc, and handle the 1.60422438e+09 numbers directly in VBScript because vbscript understands cdec() etc.

May be you have better ideas? I am all ears.

11.01.2007 at 12:36PM PDT, ID: 20195916

Rank: Master

lexiflex:

nmcdermaid's solution of using an import table with all varchar fields is the most stable way of importing data from a text file. After it is loaded into the table you can process the data row by row and column by column to convert the data and move it to a table with the right datatypes. When importing into varchar fields it doesn't matter if SQL can handle the numbers, it sees all as just text.

You said you got errors when you tried to import it into such a table. Mostly this means one (or more) of three things:
1) The varchar fields are not big enough for some of the data
2) The text file is not accoording to specs and some delimiters are missing or some rows have less or more columns than you expext
3) The specification of the delimiters and/or terminators isn't right

Assisted Solution

11.01.2007 at 02:15PM PDT, ID: 20196746

drjimcook:

I agree with lexiflex that errors importing into a working table with varchar fields indicate either values longer than your specified field length(s) or errors in the data. You could use nvarchar in case there are non-ASCII characters. Also, you could create the working table in advance, and specify all of the fields as nvarchar(255). Although you may get a warning that the maximum row size exceeds the maximum number of bytes per row, there won't be any problems unless you actually attempt to load a row that's too long.

This is just a wild guess, but perhaps there are values containing commas that aren't quote-delimited. If the file is small enough for your version of Excel, you could import each row as a single cell, and count the number of commas and quotes in each row. Any extra comma should come with two quotes.

Assisted Solution

donpick

01.22.2008 at 08:55PM PST, ID: 20720912

Here is what I discovered (to save others countless hours of frustration):
1.) The numbers like 1.799945e+10 are displayed this way by Notepad. They are displayed this way in Excel if the cell is formatted as "general" . Format the cell to numeric and watch the real number appear.
I worked really hard to get VBSCRIPT to recognize numbers like 1.799945e+10. What a stupid waste of time! VBSCRIPT already understands these numbers and converts them so they have no exponent.

My DTS routine was converting a text file to Excel and then an Access routine to read the Excel file because I couldn't think of any other way to handle numbers like 1.799945e+10. Once I discovered VBSCRIPT handles these numbers it was quite easy to get fso to read the text file. Now the text file data is loaded directly into the SQL table (no need for all string variables). I was told this was going to a slow process. The Access method took 2 hours. The FSO method takes 3 minutes. An improvement of something like 400%.

nmcdermaid

01.23.2008 at 12:49AM PST, ID: 20721738

If you loaded it into a table of varchars it would probably take half a minute... thats why I said that FSO would be a slow process.

Anyway I'm glad you found a solution!

N

20080236-EE-VQP-29 / EE_QW_2_20070628