Solved

Help with select query speed

Posted on 2012-04-05
6
339 Views
Last Modified: 2012-04-09
I have a process that helps me organize data into groups.  Below is a select that helps pull out certain characteristics of the product listings.  I am averaginges 5-8 seconds per query, the problem is, is that there are about 10,000 products that it needs to search through another table of listings that has about 1.6 million records. So this process can take 10+ hours to complete.  All the columns here are indexed.  Any suggestions?

select item_id, item_title,end_date
      from productlistings
      where end_date > '2012-03-01 00:00:00'
    and end_date < '2012-03-31 23:59:59'
    and (item_title like '% 10114 %')
      and (item_title not like '% only %')
      and (item_title not like '% only')
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')
      and (item_title not like 'loose %')
      and (item_title not like '% loose')
      and (item_title not like '% sealed box %')
      and (item_title not like '% packs %')
      and (item_title not like '% boxcase %')
      and (item_title not like '% 60 %')
      and (item_title not like '% complete set %')
      and (item_title not like '% all 16 packs')
      and (item_title not like '% case')
      and (item_title not like '% set of 16 %')
      and (item_title not like '% set 16 %')
      and (item_title not like '% all 16 %')
      and (item_title not like '% set of 16')
      and (item_title not like '% loose %')
      and (item_title not like 'set of 16 %')
      and (item_title not like '% set of %')
      and (item_title not like '% custom %')
        and set_id2 = 0
      and cleared_ind = 'x'
0
Comment
Question by:theideabulb
6 Comments
 
LVL 59

Expert Comment

by:Darius Ghassem
ID: 37814660
Do the product listing have a product_listing_id for example. Can you filter a different way?
0
 

Author Comment

by:theideabulb
ID: 37814667
no not really, think of it like an ebay feed or craiglists feed. It is going through the titles and looking for specific keyword, and filtering out those words.  We get an update every month that adds an extra 70-100k in listings to filter.   I am trying to grab the most recent date, but I still need to leave all the old listings in for individual searches.
0
 
LVL 24

Expert Comment

by:johanntagle
ID: 37814728
LIKE or NOT LIKE '% keyword %' does not make use of indexes.  Consider using boolean full-text search.  See http://dev.mysql.com/doc/refman/5.5/en/fulltext-boolean.html

Note that this is only available for myisam tables, and from what I read it doesn't really scale very well for very large data sets.  If you hit a performance issue with it a good alternative is a 3rd-party search engine like sphinx (http://sphinxsearch.com/) or maybe you can just integrate google search to your app.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 38

Accepted Solution

by:
Aaron Tomosky earned 250 total points
ID: 37815090
If the date filter significantly reduces the recordset size, you can try a subquery with that as the only filter then apply the rest.

select item_id, item_title,end_date
      from (select * from productlistings
      where end_date between '2012-03-01' and  '2012-04-01') as Pl
    Where (item_title like '% 10114 %')
      and (item_title not like '% only %')
...

If any of your where filters take out a big chunk of stuff, a subquery can significantly speed things up. You can also use cte , I just find that harder to read.

Also you have duplication going on:
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')

% will match any OR no characters so the second and third statements above are pointless. Just removing all those may help.
0
 
LVL 25

Assisted Solution

by:Tomas Helgi Johannsson
Tomas Helgi Johannsson earned 250 total points
ID: 37819372
Hi!

Put this index on your table and see if that doesn't help you
decreasing the query time

create index prodidcleard_ix
using btree
on productlistings (set_id2, cleared_ind, end_date,item_title);

Regarads,
      Tomas Helgi
0
 

Author Closing Comment

by:theideabulb
ID: 37824491
Combined, both of these comments shaved off 2-3 seconds, which adds up over the 10K plus times this will run.  Thank you for your knowledge.
0

Featured Post

U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Foreword This is an old article.  Instead of using the MySQL extension that was used in the original code examples, please choose one of the currently supported database extensions instead.  More information is available here: MySQLi / PDO (http://…
Popularity Can Be Measured Sometimes we deal with questions of popularity, and we need a way to collect opinions from our clients.  This article shows a simple teaching example of how we might elect a favorite color by letting our clients vote for …
Along with being a a promotional video for my three-day Annielytics Dashboard Seminor, this Micro Tutorial is an intro to Google Analytics API data.
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now