Solved

Help with select query speed

Posted on 2012-04-05
6
343 Views
Last Modified: 2012-04-09
I have a process that helps me organize data into groups.  Below is a select that helps pull out certain characteristics of the product listings.  I am averaginges 5-8 seconds per query, the problem is, is that there are about 10,000 products that it needs to search through another table of listings that has about 1.6 million records. So this process can take 10+ hours to complete.  All the columns here are indexed.  Any suggestions?

select item_id, item_title,end_date
      from productlistings
      where end_date > '2012-03-01 00:00:00'
    and end_date < '2012-03-31 23:59:59'
    and (item_title like '% 10114 %')
      and (item_title not like '% only %')
      and (item_title not like '% only')
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')
      and (item_title not like 'loose %')
      and (item_title not like '% loose')
      and (item_title not like '% sealed box %')
      and (item_title not like '% packs %')
      and (item_title not like '% boxcase %')
      and (item_title not like '% 60 %')
      and (item_title not like '% complete set %')
      and (item_title not like '% all 16 packs')
      and (item_title not like '% case')
      and (item_title not like '% set of 16 %')
      and (item_title not like '% set 16 %')
      and (item_title not like '% all 16 %')
      and (item_title not like '% set of 16')
      and (item_title not like '% loose %')
      and (item_title not like 'set of 16 %')
      and (item_title not like '% set of %')
      and (item_title not like '% custom %')
        and set_id2 = 0
      and cleared_ind = 'x'
0
Comment
Question by:theideabulb
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 59

Expert Comment

by:Darius Ghassem
ID: 37814660
Do the product listing have a product_listing_id for example. Can you filter a different way?
0
 

Author Comment

by:theideabulb
ID: 37814667
no not really, think of it like an ebay feed or craiglists feed. It is going through the titles and looking for specific keyword, and filtering out those words.  We get an update every month that adds an extra 70-100k in listings to filter.   I am trying to grab the most recent date, but I still need to leave all the old listings in for individual searches.
0
 
LVL 24

Expert Comment

by:johanntagle
ID: 37814728
LIKE or NOT LIKE '% keyword %' does not make use of indexes.  Consider using boolean full-text search.  See http://dev.mysql.com/doc/refman/5.5/en/fulltext-boolean.html

Note that this is only available for myisam tables, and from what I read it doesn't really scale very well for very large data sets.  If you hit a performance issue with it a good alternative is a 3rd-party search engine like sphinx (http://sphinxsearch.com/) or maybe you can just integrate google search to your app.
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 250 total points
ID: 37815090
If the date filter significantly reduces the recordset size, you can try a subquery with that as the only filter then apply the rest.

select item_id, item_title,end_date
      from (select * from productlistings
      where end_date between '2012-03-01' and  '2012-04-01') as Pl
    Where (item_title like '% 10114 %')
      and (item_title not like '% only %')
...

If any of your where filters take out a big chunk of stuff, a subquery can significantly speed things up. You can also use cte , I just find that harder to read.

Also you have duplication going on:
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')

% will match any OR no characters so the second and third statements above are pointless. Just removing all those may help.
0
 
LVL 25

Assisted Solution

by:Tomas Helgi Johannsson
Tomas Helgi Johannsson earned 250 total points
ID: 37819372
Hi!

Put this index on your table and see if that doesn't help you
decreasing the query time

create index prodidcleard_ix
using btree
on productlistings (set_id2, cleared_ind, end_date,item_title);

Regarads,
      Tomas Helgi
0
 

Author Closing Comment

by:theideabulb
ID: 37824491
Combined, both of these comments shaved off 2-3 seconds, which adds up over the 10K plus times this will run.  Thank you for your knowledge.
0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
FrontEnd tools to create web database application 7 116
How to fix Datetime in MySQL? 4 74
Very Large data in MYSQL 7 106
Inserting data into database 10 52
Foreword This is an old article.  Instead of using the MySQL extension that was used in the original code examples, please choose one of the currently supported database extensions instead.  More information is available here: MySQLi / PDO (http://…
When table data gets too large to manage or queries take too long to execute the solution is often to buy bigger hardware or assign more CPUs and memory resources to the machine to solve the problem. However, the best, cheapest and most effective so…
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…
Are you ready to implement Active Directory best practices without reading 300+ pages? You're in luck. In this webinar hosted by Skyport Systems, you gain insight into Microsoft's latest comprehensive guide, with tips on the best and easiest way…

737 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question