asked on

Speed Up MySQL Import

Hi Experts.

I'm having a sql file about 3.5GB with some tables about 7 millions records , and a Ubuntu 16.04 server configured to run on 32GB RAM. (Boot To RAM)
When importing that sql file, its took about 3 hours and a half to finish.
I have try using some parameters change as we can find on the Internet like max_allowed_packet... but the time reduce not so much.
So i would like to know if there's any way to improve performance when importing such a large sql file.

Thanks and Best Regards.

ASKER CERTIFIED SOLUTION

Ganesh Gurudu

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

David Favor

Likely you've already figured out to wrap your INSERT statements inside a TRANSACTION. If not, this will dramatically speed up your load.

Run mysqltuner to ensure your database is well tuned.

A few tricks I use for hosting very high traffic, high speed, WordPress sites.

1) Filesystem setup, where /var/lib/mysql lives. Here's my /etc/fstab for this.

/dev/md4	/	ext4	errors=remount-ro,noatime,dioread_nolock	0	1

Open in new window

Note - noatime (which implies nodiratime also), which unloads your disk from updating access times by all other processes.

Note - dioread_nolock which fixes a long standing problem when running MariaDB/MySQL with O_DIRECT, which is required for good throughput.

2) Change to innodb_flush_log_at_trx_commit = 2 if your hosting/provisioning company provides UPS backup support for all machines. Most do these days.

3) Be sure to set my.cnf O_DIRECT.

4) Be sure to set my.cnf to use InnoDB as your default storage engine.

5) Deinstall MySQL + Install MariaDB - provides anywhere from 30%-50%+ speed increase.

Many sensible Distros now install MariaDB instead of MySQL, so this will likely already be done if your running recent Debian/Ubuntu.

6) Make sure you're target table is actually running InnoDB as the storage engine, as InnoDB allows many additional tunings over MyISAM.

7) If your records have many variable length fields, then consider setting PAGE COMPRESSION + LZ4 ALGO on your database create statement.

arnold

where are you exporting the data from?
trying to see whether backup/restore would be a better option.
import means a csv file of data?