Link to home
Start Free TrialLog in
Avatar of Jason C. Levine
Jason C. LevineFlag for United States of America

asked on

How to handle a 3 gigabyte CSV file

Hi folks,

We've been given a 3.5 gig CSV file that I need to get into a database somehow.  We have access to MySQL and Access 2003 and 2007 but I'm stumped on how to convert a file of this size to a MySQL query and Access seems to not want to open it at all.  

What tools do I need to do this?
ASKER CERTIFIED SOLUTION
Avatar of hernst42
hernst42
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Jason C. Levine

ASKER

Thanks, hernst.   One follow-up, if I may:

Does that command require the table exist or will it create it based on the header row?
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
>> Access limit is 2gb, so if you have 3gb file, you are better off with mysql

Yep, that's exactly what we're discovering today.  Thanks for the additional info, ralmada.  
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Yeah, well.

3,207,489 rows with 255 columns.  Access is not rated at those sizes.
So yeah.  It helps to RTFM.  But knowing which part of the FM to R was crucial.  Thanks to hernst for the fast answer and ralmada and boag2000 for filling in some details.
<3,207,489 rows with 255 columns.  Access is not rated at those sizes.>
There is no theoretical limit to the number of rows in Access.
(The Autonumber field (Long Integer) limit is set to 2 billion)

Access can handle up to, ...but not beyond 255 columns.
But here again, because of the datatypes and other properties, it is not practical to come near this limit as other "strangeness" will probably occur.
;-)
jason1178,

Nice to see you taking a break from dominating the Web zones and actually posting a question...
;-)

Jeff
Heh.

Hardly "dominating" anything (except for Dreamweaver) these days.  And this one stumped my in-house guys so the "old man" had to step up and show off a little.