Advertisement

09.02.2007 at 05:37PM PDT, ID: 22802923
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

automated import from fixed width flat file

Tags: fixed, file, width, import, flat
I have about 50 separate fixed width text files that I need to pull into SQL Server 2000, so I want to automate the process.  They are BIG files, with 731 fields and a sum of 250 gigs.  

I have a list of the field names and lengths, so I want to use that to instruct the transfer.  I already have created a template table in SQL Server with the corresponding fields so I only need to append the records.  

I would love to do this in VBA using an Access project (.adp) with stored procedures if that can be done.  But as long as there is a not-too-crazy solution to this, that would be great.  I've been looking at the questions on this and there are bulk inserts but I'm only getting more confused.  

Thanks.

The table with field lengths looks like this with field name and length:
F1      18
F2      15
F3      1
F4      20
F5      2
etc........

Start your free trial to view this solution
Question Stats
Zone: Microsoft
Question Asked By: dougfosterNYC
Solution Provided By: nmcdermaid
Participating Experts: 2
Solution Grade: A
Views: 120
Translate:
Loading Advertisement...
09.02.2007 at 06:33PM PDT, ID: 19818649

Rank: Genius

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.02.2007 at 06:33PM PDT, ID: 19818652

Rank: Genius

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.02.2007 at 07:03PM PDT, ID: 19818709

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.03.2007 at 03:22AM PDT, ID: 19819960

Rank: Wizard

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.03.2007 at 02:15PM PDT, ID: 19822150

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.04.2007 at 12:34AM PDT, ID: 19823639

Rank: Wizard

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.10.2007 at 11:19AM PDT, ID: 19862983

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.11.2007 at 01:06AM PDT, ID: 19866772

Rank: Genius

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.11.2007 at 05:24AM PDT, ID: 19867881

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
09.12.2007 at 02:24AM PDT, ID: 19874809

Rank: Genius

All comments and solutions are available to Premium Service Members only.

Start your 7 day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
Loading Advertisement...
Microsoft
  • Internet Protocols
  • Applications
  • Development
  • OS
  • Hardware
  • Windows Security
Apple
  • Operating Systems
  • Hardware
  • Programming
  • Networking
  • Software
Internet
  • Search Engines
  • File Sharing
  • WebTrends / Stats
  • Spy / Ad Blockers
  • Web Browsers
  • New Net Users
  • Web Development
  • Chat / IM
  • Anti Spam
  • Web Servers
  • Anti-Virus
  • Email Clients
Gamers
  • Tips
  • Online / MMORPG
  • Puzzle
  • Emulators
  • Action / Adventure
  • Role Playing
  • Consoles
  • Game Programming
  • Strategy
  • Sports
  • Misc
  • Computer Games
Digital Living
  • Hardware
  • New Net Users
  • New Users
  • Software
  • Digital Music
  • Gaming World
  • Home Security
  • Apple
  • Networking Hardware
Virus & Spyware
  • Vulnerabilities
  • IDS
  • Encryption
  • Anti-Virus
  • Operating Systems Security
  • Software Firewalls
  • WebApplications
  • Cell Phones
  • Operating Systems
  • Internet
  • Hardware Firewalls
Hardware
  • Handhelds / PDAs
  • Displays / Monitors
  • Components
  • Networking Hardware
  • Peripherals
  • Laptops/Notebooks
  • Storage
  • Servers
  • Desktops
  • New Users
  • Misc
  • Apple
Software
  • System Utilities
  • Industry Specific
  • Network Management
  • Photos / Graphics
  • Page Layout
  • VMWare
  • Misc
  • Web Development
  • OS
  • CYGWIN
  • Voice Recognition
  • Message Queue
  • Quality Assurance
  • Security
  • Firewalls
  • MultiMedia Applications
  • Development
  • Database
  • Office / Productivity
  • Business Management
  • OS/2 Apps
  • Server Software
  • Internet / Email
ITPro
  • OS
  • Storage
  • Encryption
  • Operating Systems Security
  • Apple Hardware
  • Laptops & Notebooks
  • Servers
  • Networking Hardware
  • Peripherals
  • Devices
  • Displays / Monitors
  • WebTrends / Stats
  • Search Engines
  • Firewalls
  • WebApplications
  • IDS
  • Vulnerabilities
  • Email Clients
  • File Sharing
  • Spy / Ad Blockers
  • Web Browsers
  • Web Servers
  • Networking
  • Anti-Virus
  • Chat / IM
  • Anti Spam
Developer
  • Web Servers
  • Web Browsers
  • Game Programming
  • Dev Tools
  • Industry Specific
  • Office / Productivity
  • Database
  • CYGWIN
  • Web Development
  • Search Engines
  • File Sharing
  • WebTrends / Stats
  • Programming
  • Content Management
  • Application Servers
  • Protocols
Storage
  • Removable Backup Media
  • Storage Technology
  • Servers
  • Grid
  • Remote Access
  • Backup / Restore
  • Misc
  • Hard Drives
OS
  • Miscellaneous
  • Security
  • Development
  • Linux
  • VMWare
  • MainFrame OS
  • Unix
  • Apple
  • OS / 2
  • AS / 400
  • BeOS
  • Microsoft
  • VMS / OpenVMS
Database
  • Oracle
  • Miscellaneous
  • MySQL
  • Software
  • Sybase
  • Contact Management
  • PostgreSQL
  • Data Manipulation
  • Clarion
  • InterSystems Cache
  • Siebel
  • MUMPS
  • OLAP
  • SQLBase
  • SAS
  • GIS & GPS
  • 4GL
  • Berkeley DB
  • DB2
  • Informix
  • Interbase / Firebird
  • FoxPro
  • Reporting
  • LDAP
  • Filemaker Pro
  • MS SQL Server
  • dBase
  • MS Access
Security
  • Misc
  • Web Browsers
  • Software Firewalls
  • Operating Systems Security
  • File Sharing
  • Spy / Ad Blockers
  • Vulnerabilities
  • WebApplications
  • IDS
  • Anti-Virus
  • Encryption
  • Anti Spam
  • Email Clients
  • VPN
  • Chat / IM
Programming
  • Editors IDEs
  • Installation
  • Handhelds / PDAs
  • Multimedia Programming
  • System / Kernel
  • Algorithms
  • Game
  • Signal Processing
  • Project Management
  • Open Source
  • Database
  • Misc
  • Languages
  • Processor Platforms
  • Theory
Web Development
  • Scripting
  • Blogs
  • Web Servers
  • Software
  • Search Engines
  • Web Graphics
  • Images
  • Internet Marketing
  • Images and Photos
  • Components
  • Document Imaging
  • Web Languages/Standards
  • Illustration
  • WebApplications
  • Fonts
  • WebTrends / Stats
  • Authoring
  • Digital Camera Software
  • Miscellaneous
Networking
  • Protocols
  • Apple Networking
  • Network Management
  • Message Queue
  • Application Servers
  • Content Management
  • File Servers
  • Email Servers
  • Misc
  • Java Editors & IDEs
  • Wireless
  • Networking Hardware
  • Backup / Restore
  • System Utilities
  • ISPs & Hosting
  • Web Servers
  • Storage Technology
  • Removable Backup Media
  • Servers
  • Broadband
  • Grid
  • OS / 2
  • Novell Netware
  • Unix Networking
  • Windows Networking
  • Security
  • Telecommunications
  • Operating Systems
  • Linux Networking
Other
  • Community Advisor
  • Lounge
  • Community Support
  • New Net Users
  • Philosophy / Religion
  • Math / Science
  • Miscellaneous
  • URLs
  • Expert Lounge
  • Politics
  • Puzzles / Riddles
Community Support
  • Suggestions
  • New to EE
  • New Topics
  • Community Advisor
  • CleanUp
  • Announcements
  • General
  • Feedback
  • Input
  • EE Bugs
 
09.02.2007 at 06:33PM PDT, ID: 19818649

Rank: Genius

See here for a stored procedure that will BULK insert all files into a table:

http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/Q_21700652.html

Note that you will need to set up a format file and test it first. This is really only useful it you want to automate the import.


If its a one off then you could append all of the files together into a couple of larger files using

COPY File1.TXT + File2.TXT + File3.TXT OutputFile.TXT

and just bulk insert OutputFile.TXT in one go.
 
09.02.2007 at 06:33PM PDT, ID: 19818652

Rank: Genius

.... oh just realised that SP is for importing CSV files. It will need to be changed to import fixed width.
 
09.02.2007 at 07:03PM PDT, ID: 19818709
This looks great, but yes, I need it for a flat file. If you can give me code that deals with fixed width that would be great.  Once again, they will all have the same layout and I have a list with the field lengths for all 731 fields.

Thanks.

Doug
 
09.03.2007 at 03:22AM PDT, ID: 19819960

Rank: Wizard

you will need to have a format file, which you can use along with the BULK INSERT, which specifies the columns and their lengths

"To import data from data files with fixed-length or fixed-width fields, you must use a format file"

Preparing Data for Bulk Export or Import
http://msdn2.microsoft.com/en-us/library/ms188609.aspx
 
09.03.2007 at 02:15PM PDT, ID: 19822150
Thanks.  I am feeling very slow.  I would love a basic description of how to set this up.  Am I creating a couple of stored procedures, and is the bulk insert a command for that.?  I see what I think is a format for defining the fixed width tables (copied below).  

I could use a short, sample code on this, like an actual SP.  I'm not a SQL Server person, but rather a general developer that uses SQL Server for programming db apps.  

Thanks.


<?xml version="1.0"?>
<BCPFORMAT
       xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <RECORD>
    <FIELD ID="1" xsi:type="CharFixed" LENGTH="10"/>
    <FIELD ID="2" xsi:type="CharFixed" LENGTH="6"/>
    <FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n"
  </RECORD>
  <ROW>
    <COLUMN SOURCE="1" NAME="C1" xsi:type="SQLINT" />
    <COLUMN SOURCE="2" NAME="C2" xsi:type="SQLINT" />
  </ROW>
</BCPFORMAT>
 
09.04.2007 at 12:34AM PDT, ID: 19823639

Rank: Wizard

you need to create a .fmt Format file and reference that file in the BULK INSERT statement thus:

BULK INSERT myTbl
FROM 'c:\myFile.txt'
WITH (FORMATFILE='c:\FormatFile.fmt')

see these links for info on creating a fmt file:

Using the bcp Format File
http://doc.ddart.net/mssql/sql70/impt_bcp_16.htm

Creating a Format File
http://msdn2.microsoft.com/en-us/library/ms191516.aspx
 
09.10.2007 at 11:19AM PDT, ID: 19862983
Ok, just to be clear, since I'm confused about the difference between bcp and bulk insert.  The "creating a format file" directions is about creating a bcp format file (see below).  But I can creat an xml type file as below and name it fileformat.fmt and that will be the file I use for your simple bulk insert command you have above?  

Sorry for my slowness on this......

<?xml version="1.0"?>
<BCPFORMAT
xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <RECORD>
    <FIELD ID="1" xsi:type="CharTerm" TERMINATOR="\t"
      MAX_LENGTH="12"/>
    <FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\t"
      MAX_LENGTH="20" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
    <FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n"
      MAX_LENGTH="30"
      COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  </RECORD>
  <ROW>
    <COLUMN SOURCE="1" NAME="age" xsi:type="SQLINT"/>
    <COLUMN SOURCE="2" NAME="firstname" xsi:type="SQLVARYCHAR"/>
    <COLUMN SOURCE="3" NAME="lastname" xsi:type="SQLVARYCHAR"/>
  </ROW>
</BCPFORMAT>
 
09.11.2007 at 01:06AM PDT, ID: 19866772

Rank: Genius

I'm not aware of a format file ever being XML. All the format files I have used are straight text files.

If you look up format files in books online, and choose the first entry, you'll find this example of a format file:


8.0
9
1  SQLCHAR  0  11 ","      1  au_id       SQL_Latin1_General_Cp437_BIN
2  SQLCHAR  0  40 ","      2  au_lname    SQL_Latin1_General_Cp437_BIN
3  SQLCHAR  0  20 ","      3  au_fname    SQL_Latin1_General_Cp437_BIN
4  SQLCHAR  0  12 ","      4  phone       SQL_Latin1_General_Cp437_BIN
5  SQLCHAR  0  0  ""       5  address     SQL_Latin1_General_Cp437_BIN
6  SQLCHAR  0  20 ","      6  city        SQL_Latin1_General_Cp437_BIN
7  SQLCHAR  0  2  ","      7  state       SQL_Latin1_General_Cp437_BIN
8  SQLCHAR  0  0  ""       8  zip         SQL_Latin1_General_Cp437_BIN
9  SQLCHAR  0  1  "\r\n"   9  contract    SQL_Latin1_General_Cp437_BIN



Thats the format you are aiming for. The format file tells BULK INSERT or BCP.EXE where each fixed width field sits, and what data type it is.

This is in SQL 2000 right?



BULK INSERT and BCP are almost exactly the same.

However BULK INSERT is a T-SQL command - you submit it over a database connection

BCP.EXE is an EXE, run from the command line.


They both use exactly the same kind of format file though.


Once you have successfuly manually imported one of your files using BULK INSERT, you can either append them all together and insert in one go, or use the link I provided for a SP that will loop through a folder and BULK INSERT the files.

Accepted Solution
 
09.11.2007 at 05:24AM PDT, ID: 19867881
Thanks nmcdermaid for the detailed explanation.

I did finally do more reading on it and figured out to use the text format.  The MS documentation really is not as robust as I would like.  I had one head banging moment where I couldn't get it to work and it turned out that I didn't put a CR at the end of the last line of the text file.  Somebody on some user group mentioned that and that was the problem.

I'm still having issues getting it in correctly.  It's hard because they are HUGE files with 731 fields and a total of 130 million records.  I'm going to break it up though.

Thanks again.
 
09.12.2007 at 02:24AM PDT, ID: 19874809

Rank: Genius

Yeah the whole format file thing is pretty painful but its the only way to get a fast dataload
 
 
20080236-EE-VQP-29 / EE_QW_1_20070628