troubleshooting Question

MySQL best way to merge two large tables

Avatar of Jim Youmans
Jim YoumansFlag for United States of America asked on
* importMySQL ServerLinux* CentOS
3 Comments1 Solution14 ViewsLast Modified:
MySQL 5.6 on Centos 7
I have two very large tables (table A has 65 million rows and table B has 135 million rows) that I need to merge into a third table (lets call it table C).  Table C is partitioned on the CREATED_TS column.  A and B are identical except that they come from different "sources" and so in table C there is an extra column call OrigSource that has a code in it.
I have created a stored proc that cycles through each table and inserts 50K to 100K rows at a time but my issue is that it takes forever to run.  For Table A it took 10-12 hours and for table B it is still running after about 18 hours.
Is there a better, faster way to do this?  Can I export the tables with MySQLDump or something and then reinsert them into the new table?  Is there some kind of utility that will do this in a faster way?
Any suggestions would be greatly appreciated .  Thank you.
Jim
ASKER CERTIFIED SOLUTION
Tomas Helgi Johannsson
Database Administrator / Software Engineer

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Log in to continue reading
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform for $9.99/mo
View membership options
Unlock 1 Answer and 3 Comments.
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
The Value of Experts Exchange in My Daily IT Life

Experts Exchange (EE) has become my company's go-to resource to get answers. I've used EE to make decisions, solve problems and even save customers. OutagesIO has been a challenging project and... Keep reading >>

Mike

Owner of Outages.IO
Phoenix, Arizona, United States
Member Since 2016
Join a full scale community that combines the best parts of other tools into one platform.
Unlock 1 Answer and 3 Comments.
View membership options
“All of life is about relationships, and EE has made a virtual community a real community. It lifts everyone's boat.”
William Peck

Member since 2004