Solved

MySQL Query with Inner Join to small table suddenly takes FOREVER

Posted on 2014-03-06
7
512 Views
Last Modified: 2014-03-17
Below are 2 queries.  They're exactly the same except for the inner join to the t_lob table (which has 14 rows in it and an index on account and skill).  The query with the join used to run in a matter of seconds before I upgraded to MySQL v5.6 (from 5.5).  Now it crawls.  

Why? What can I do to fix it?  

Here are the explain results:

Query with Join:
id,select_type,table,type,possible_keys,key,key_len,ref,rows,Extra
1,PRIMARY,<derived2>,ALL,NULL,NULL,NULL,NULL,6066,"Using temporary; Using filesort"
2,DERIVED,l,ref,"Account,Skill",Account,11,const,6,"Using index condition; Using where"
2,DERIVED,t,ref,"StartDate,Skill,Account",Skill,103,att.l.Skill,1011,"Using where"


Query with Join Commented out:
id,select_type,table,type,possible_keys,key,key_len,ref,rows,Extra
1,PRIMARY,<derived2>,ALL,NULL,NULL,NULL,NULL,445155,"Using temporary; Using filesort"
2,DERIVED,t,range,"StartDate,Account",StartDate,4,NULL,593540,"Using index condition; Using where"



.....So, it seems to me that the inclusion of the inner join causes fewer rows to be evaluated.  However, as I've explained it actually takes longer. Not even sure if it will complete.  I've let it go for about 30 min before killing it.  The version without the join takes 10.8 seconds

To clarify.  t_transcripts has about 10M rows in it.  t_lob has 14.


select distinct
#lob,
URL_Cleaned, count(URL_Cleaned) as URL_Count, sum(convcounter) as Conversions

from (select  
t.account,  
#l.lob,
#l.sortseq,
t.sessionid,
t.skill,  
t.startdate,  
t.chat_button_name,
t.chatreferer as URL,
substring_index(substring_index(substring_index(t.chatreferer, '://',-1),'#', 1), '?',1) as URL_Cleaned,
if(isnull(conversion), 0, 1) as convcounter

from  
t_transcripts t  

#inner join t_lob l
#on t.account = l.account
#and t.skill = l.skill  

where  
t.account = '12345678'
and t.startdate between '2014-02-12' and '2014-02-28'
)q  

group by
#lob,
url_cleaned



-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SLOW QUERY (inner join is not commented out):




select distinct
#lob,
URL_Cleaned, count(URL_Cleaned) as URL_Count, sum(convcounter) as Conversions

from (select  
t.account,  
#l.lob,
#l.sortseq,
t.sessionid,
t.skill,  
t.startdate,  
t.chat_button_name,
t.chatreferer as URL,
substring_index(substring_index(substring_index(t.chatreferer, '://',-1),'#', 1), '?',1) as URL_Cleaned,
if(isnull(conversion), 0, 1) as convcounter

from  
t_transcripts t  

inner join t_lob l
on t.account = l.account
and t.skill = l.skill  

where  
t.account = '12345678'
and t.startdate between '2014-02-12' and '2014-02-28'
)q  

group by
#lob,
url_cleaned
0
Comment
Question by:ducky801
  • 4
  • 3
7 Comments
 
LVL 48

Expert Comment

by:PortletPaul
ID: 39911233
quick comment

there is absolutely no point in asking for distinct & group by...
drop use of distinct in both of your queries

select distinct (i.e. make a unique list of) X
...
group by (i.e. make a unique list of) X

i.e. you making a unique list twice
0
 
LVL 48

Expert Comment

by:PortletPaul
ID: 39911246
do both tables have indexes on account & skill?
0
 
LVL 5

Author Comment

by:ducky801
ID: 39911340
Yea.  Muscle memory on the distinct.  It flows from the fingertips.....

Yes.  Both tables have and index on Account and Skill.
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 5

Author Comment

by:ducky801
ID: 39911344
One other thing to note-

I'm also having this problem:
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Windows/Q_28382342.html

I read somewhere today that long queries like the one i'm wresting with could be related to running out of space in your tmpdir.  Not sure if these could be related or how to go about testing that though.
0
 
LVL 48

Expert Comment

by:PortletPaul
ID: 39912018
>> "distinct.  It flows from the fingertips"
it most definitely should not flow, it should only ever be used when it is absolutely needed

I haven't more specific ideas on the actual question I'm afraid.  Are your statistics current? Do you need to analyse those tables perhaps?
see:
https://dev.mysql.com/doc/refman/5.6/en/innodb-performance.html#innodb-persistent-stats
(assuming innob)
0
 
LVL 5

Accepted Solution

by:
ducky801 earned 0 total points
ID: 39924680
I was able to get the query to run 180x faster by replacing the inner join with a straight_join.

Straight_join overrides the query optimizer when it makes a poor decision.

http://dev.mysql.com/doc/refman/5.0/en/join.html
0
 
LVL 5

Author Closing Comment

by:ducky801
ID: 39933757
I found this answer on a separate message board.  Other expert comments on this post didn't help the query to run faster
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Introduction Since I wrote the original article about Handling Date and Time in PHP and MySQL (http://www.experts-exchange.com/articles/201/Handling-Date-and-Time-in-PHP-and-MySQL.html) several years ago, it seemed like now was a good time to updat…
This article describes how to use the timestamp of existing data in a database to allow Tableau to calculate the prior work day instead of relying on case statements or if statements to calculate the days of the week.
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now