Speed Up MySQL Import

Hi Experts.

I'm having a sql file about 3.5GB with some tables about 7 millions records , and a Ubuntu 16.04 server configured to run on 32GB RAM. (Boot To RAM)
When importing that sql file, its took about 3 hours and a half to finish.
I have try using some parameters change as we can find on the Internet like max_allowed_packet... but the time reduce not so much.
So i would like to know if there's any way to improve performance when importing such a large sql file.

Thanks and Best Regards.
LVL 2
ldvhaiAsked:
Who is Participating?
 
Ganesh GuruduSenior ConsultantCommented:
hi there are multiple ways to speed up the operation.

check the below parameters
innodb_buffer_pool_size = 4G
innodb_log_buffer_size = 256M
innodb_log_file_size = 1G
innodb_write_io_threads = 16
innodb_flush_log_at_trx_commit = 0

check the below solutions as well.
https://stackoverflow.com/questions/29643714/improve-speed-of-mysql-import
https://dbahire.com/testing-the-fastest-way-to-import-a-table-into-mysql-and-some-interesting-5-7-performance-results/
https://serverfault.com/questions/146525/how-can-i-speed-up-a-mysql-restore-from-a-dump-file
1
 
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Likely you've already figured out to wrap your INSERT statements inside a TRANSACTION. If not, this will dramatically speed up your load.

Run mysqltuner to ensure your database is well tuned.

A few tricks I use for hosting very high traffic, high speed, WordPress sites.

1) Filesystem setup, where /var/lib/mysql lives. Here's my /etc/fstab for this.

/dev/md4	/	ext4	errors=remount-ro,noatime,dioread_nolock	0	1

Open in new window


Note - noatime (which implies nodiratime also), which unloads your disk from updating access times by all other processes.

Note - dioread_nolock which fixes a long standing problem when running MariaDB/MySQL with O_DIRECT, which is required for good throughput.

2) Change to innodb_flush_log_at_trx_commit = 2 if your hosting/provisioning company provides UPS backup support for all machines. Most do these days.

3) Be sure to set my.cnf O_DIRECT.

4) Be sure to set my.cnf to use InnoDB as your default storage engine.

5) Deinstall MySQL + Install MariaDB - provides anywhere from 30%-50%+ speed increase.

Many sensible Distros now install MariaDB instead of MySQL, so this will likely already be done if your running recent Debian/Ubuntu.

6) Make sure you're target table is actually running InnoDB as the storage engine, as InnoDB allows many additional tunings over MyISAM.

7) If your records have many variable length fields, then consider setting PAGE COMPRESSION + LZ4 ALGO on your database create statement.
1
 
arnoldCommented:
where are you exporting the data from?
trying to see whether backup/restore would be a better option.
import means a csv file of data?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.