mySQL - How to dump & upload text files

Oracle uses SQLoader, SQL server uses BCP to load text files with millions of records. What technology mySQL have if I need to load around 10 million records every single day ?
Who is Participating?
arnoldConnect With a Mentor Commented:
Lad data in file deals with CSV, TSV I.e. exported data only.

Load data in file "filename "
Into table
Defines what the separator, terminator and the columns of the table.

Your best approach might be to use a perl or similar script that will go through the file and perform tests while inserting the data into mysql.
l'm not a MYSQL expert but in my last job we did this by "mysql -u <user_name> -p <password> <database> < file_to_load.sql". Note that the file_to_load.sql was in sql format.
crazywolf2010Author Commented:
I don't have insert statements here.

I have tonnes of 3rd party text files in varying sizes (KB to GB)  in comma, pipe delimited format.

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Guy Hengel [angelIII / a3]Billing EngineerCommented:
In mysql you can use the LOAD DATA INFILE statement. There is also a command line version for this
Mysqldump is the backup utility.

To load/restore the database/data
You would cat the backup file into mysql -u root databasename

With a backup, you can also set the inclusion of use statements.
crazywolf2010Author Commented:
Can LOAD DATA INFILE  manage files in size of GB? Does anyone have a good example of how it should be used and any best practises.

Effectively I need to have details of rejected records while loading and any error trapping done automatically will be very useful.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.