Speed Up MySQL Import

Hi Experts.

I'm having a sql file about 3.5GB with some tables about 7 millions records , and a Ubuntu 16.04 server configured to run on 32GB RAM. (Boot To RAM)
When importing that sql file, its took about 3 hours and a half to finish.
I have try using some parameters change as we can find on the Internet like max_allowed_packet... but the time reduce not so much.
So i would like to know if there's any way to improve performance when importing such a large sql file.

Thanks and Best Regards.
LVL 2
ldvhaiAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ganesh GuruduSenior ConsultantCommented:
hi there are multiple ways to speed up the operation.

check the below parameters
innodb_buffer_pool_size = 4G
innodb_log_buffer_size = 256M
innodb_log_file_size = 1G
innodb_write_io_threads = 16
innodb_flush_log_at_trx_commit = 0

check the below solutions as well.
https://stackoverflow.com/questions/29643714/improve-speed-of-mysql-import
https://dbahire.com/testing-the-fastest-way-to-import-a-table-into-mysql-and-some-interesting-5-7-performance-results/
https://serverfault.com/questions/146525/how-can-i-speed-up-a-mysql-restore-from-a-dump-file
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Likely you've already figured out to wrap your INSERT statements inside a TRANSACTION. If not, this will dramatically speed up your load.

Run mysqltuner to ensure your database is well tuned.

A few tricks I use for hosting very high traffic, high speed, WordPress sites.

1) Filesystem setup, where /var/lib/mysql lives. Here's my /etc/fstab for this.

/dev/md4	/	ext4	errors=remount-ro,noatime,dioread_nolock	0	1

Open in new window


Note - noatime (which implies nodiratime also), which unloads your disk from updating access times by all other processes.

Note - dioread_nolock which fixes a long standing problem when running MariaDB/MySQL with O_DIRECT, which is required for good throughput.

2) Change to innodb_flush_log_at_trx_commit = 2 if your hosting/provisioning company provides UPS backup support for all machines. Most do these days.

3) Be sure to set my.cnf O_DIRECT.

4) Be sure to set my.cnf to use InnoDB as your default storage engine.

5) Deinstall MySQL + Install MariaDB - provides anywhere from 30%-50%+ speed increase.

Many sensible Distros now install MariaDB instead of MySQL, so this will likely already be done if your running recent Debian/Ubuntu.

6) Make sure you're target table is actually running InnoDB as the storage engine, as InnoDB allows many additional tunings over MyISAM.

7) If your records have many variable length fields, then consider setting PAGE COMPRESSION + LZ4 ALGO on your database create statement.
1
arnoldCommented:
where are you exporting the data from?
trying to see whether backup/restore would be a better option.
import means a csv file of data?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.