MySQL RDS Instance taking forever to load CSV files.

Hello all,

I had a large file e.g. 85 million records that I needed to import into an Amazon Mysql RDS instance (t2.medium). I took the file from our server, zipped and moved it to S3. I then copied the file from S3 to a t2.micro instance and split the file into multiple parts e.g. 1 million rows.

I then have the following shell script running:

#!/bin/bash

for f in file_part*
do
    echo $f
    mysql -e "load data local infile '"$f"' into table TABLENAME FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES    TERMINATED BY '\n'" -u user --password=password database -h hostname
done

Select allOpen in new window

I am running the following and seeing its taken over 12 hours to load nearly 12 million records. Seems extremely slow any ideas?

SELECT TABLE_NAME,TABLE_ROWS, round(((data_length + index_length) / (1024*1024)), 2) "Size in MB"
FROM information_schema.TABLES 
WHERE table_schema = "database"

Select allOpen in new window

Wayne