Solved

Trouble Importing Flat File Into MS SQL 2008

Posted on 2014-02-27
2
568 Views
Last Modified: 2014-02-27
Good morning!

I've got a very large pipe delimited flat file that I'm having difficulty importing into SQL.

Here's the situation:

The file is about 650MB (a few million records
The file is pipe delimited
I don't know the max lengths of the values in the fields
I tried importing the file into a table in temdb with all of the fields set to varchar(max) and I still get truncation errors in SQL.
I'm getting truncation errors on a specific column when I tell the import/export wizard to ignore truncation and even when I tell it to ignore the entire column

I've been able to open the file in a program called "Stat Transfer" (something we purchased along with a program called STATA - an SPSS competitor).  That reads about 4,400 records but, it seems to think those are all the records in the file.  

I suspect there's some kind of odd character around the end of that 4,400th record that's causing these issues.  Typically, I would search around in the text file and manually edit it to fix something like that but, this file is too big to open in notepad or notepad++.  

So what to do?  Any ideas as to how I might go about opening or editing the source file?  

Thanks in advance!
0
Comment
Question by:ttist25
2 Comments
 
LVL 11

Accepted Solution

by:
John_Vidmar earned 500 total points
ID: 39891801
There are free file-splitting utilities.  Break the file into smaller pieces, and then open the smaller file using Notepad++ to find out what's going on around that 4,400th record.
0
 
LVL 1

Author Comment

by:ttist25
ID: 39893243
Thanks John.  

Your method would definitely work so I'm awarding points but, I ended up using something called "File Query" by AgileSoftware.  It's a free trial and it was pretty useful so I may actually pay for it.  It opened the file right up and loaded all of the records.  I was able to query min.max lengths etc.  I'm still working to get it into MSSQL but, this worked well for opening and allowing edits.  

Thanks!
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SQL Command Tool comes with APEX under SQL Workshop. It helps us to make changes on the database directly using a graphical user interface. This helps us writing any SQL/ PLSQL queries and execute it on the database and we can create any database ob…
How to leverage one TLS certificate to encrypt Microsoft SQL traffic and Remote Desktop Services, versus creating multiple tickets for the same server.
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

26 Experts available now in Live!

Get 1:1 Help Now