asked on

Bulk Insert Fails Error

I am getting this error when trying to do a bulk insert of a text file

Bulk Insert fails. Column is too long in the data file for row 1, column 4.
Make sure the field terminator and row terminator are specified correctly.

Here is my SQL statement:

BULK INSERT MPI
FROM 'd:\inetpub\wwwroot\mpi\load\data\unit.fil'
WITH (FIELDTERMINATOR = '|', ROWTERMINATOR = '\n', DATAFILETYPE = 'CHAR', BATCHSIZE = 800000, ORDER( unit_num ASC), TABLOCK)

What's wierd is that if I go an open the file in wordpad and save it, then try the bulk insert again it succeeds with no problems. The file is initially transferred from a remote server via ftp. I'm thinking that the rowterminator is different than \n initially before I open it in wordpad, but I don't know how to determine what it is.

ptjcb

I would suspect that someone is using the '|' in the third column - that would terminate the column, and probably make the fourth column too large. Just a guess.

Aneesh

post the table structure and the some of the file entries

dspent

ASKER

Here's the first few lines from the text file (*'s are there because it is hospital patient data). In total there are approx 700,000 rows.

A000000001 |*******,PETER R |10-08-**|***-**-****
A000000002 |*******,LINDA S | |***-**-****
A000000003 |*******,TIM | |
A000000004 |**********,BETTY L |12-29-**|***-**-****
A000000006 |*******,JOHN L |02-25-**|***-**-****
A000000008 |******,JANIS D |07-15-**|***-**-****

here is the create table sql:

CREATE TABLE MPI
(unit_num varchar(11) NOT NULL PRIMARY KEY, pat_name varchar(28) NOT NULL, dob varchar(8), ssn varchar(12))

The reason ssn is set to 12 instead of 11 is that many rows include a trailing space after the 11 digit ssn and others don't. I have no control over the creation of the text file.

ASKER CERTIFIED SOLUTION

ptjcb

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

dspent

ASKER

well I am basically updating this process to work in a coldfusion template.....
previously it was done in a batch file using BCP.

In the BCP statement they used \n as the row terminator and it works without a hitch. So I can't understand why it would not work with bulk insert.

In windows/dos , do I include the < and > characters???

ptjcb

No...

In VB - if I remember (and it's been awhile) vbCrLf = Chr(13) & Chr(10) = Carriage return/linefeed combination

Use char(13)+ char(10) in SQL.

dspent

ASKER

no go.... however if I use \CR or \LF

I get a different error....
for example using ROWTERMINATOR = '\LF' I now get:

Bulk insert data conversion error (truncation) for row 1, column 4 (ssn).

dspent

ASKER

Well I have a workaround... Coldfusion has a cffile tag that allows you to read the contents of a file into a variable, and then output that variable into a new file.
In doing so, you can add an attribute called fixnewline which will replace the line end of the file with the OS specific line end (in this case windows).

So what I have done is after the FTP transfer from the remote server, I read the file contents into a variable, delete the file then rewrite it using the fixnewline attriubute. This now allows me to use the ROWTERMINATOR = '\n' and all is now at peace with the world.

Seeing as the problem lies with rowterminator, I will award the points to you as everything you mentioned did lead me in the direction of either getting the right terminator to use or fixing the file so I could use the default.

Thanks for your assistance.

ptjcb

No problem. I just wrestled with the same problem last month (it was a Unix file) so I knew to look in that direction.