Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Importing  a verly large CSV file into RDBMS

Posted on 2011-09-29
4
367 Views
Last Modified: 2012-06-21
I have a 4gb data file that has 4 million rows and 314 colums/fields. I am trying to import this either into a MYSQL or SQL SERVER dbase. I have tried using the Import/Export and MIgration tool kits on SQL server. I encounter an error each time, when i amtrying to import it. Usually In the sql server it is truncation errosrs and In MYSQL the error is  the length of the row is too large.

Any help in this matter is greatly appreciated.
0
Comment
Question by:m3mdicl
4 Comments
 
LVL 17

Expert Comment

by:Garry-G
ID: 36882946
Length of the row in reference to a certain column? Have you checked the char sets used, as e.g. a string with the length of 10 utf-8 chars might be something like 30 chars long ...
Also, on the import, unless you use transactions, you should end up with "n" rows inserted, so you could check the CSV after those n lines and see if the next line is in fact malformed/defective ... maybe delete it, try importing the rest of the data, than manually fix the corrupt lines ...
On Unix, splitting the csv up into smaller packets, like e.g. 100k lines a piece, might also help getting it in ...
0
 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 36889933
0
 
LVL 7

Accepted Solution

by:
luani earned 500 total points
ID: 36890290
The first thing that comes in mind is to split the file......
0
 
LVL 1

Expert Comment

by:ob2s
ID: 36896129
Hi,

Assuming there aren't fixed length rows in your data file, check if your import tool's expectation of line terminators matches what is actually used in the data file (e.g. \n vs \r\n).

If you're using mysqlimport, see the --lines-terminated-by option.

Hope this helps,.
0

Featured Post

NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Creating and Managing Databases with phpMyAdmin in cPanel.
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Viewers will learn how the fundamental information of how to create a table.

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question