Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Help with select query speed

Posted on 2012-04-05
6
Medium Priority
?
347 Views
Last Modified: 2012-04-09
I have a process that helps me organize data into groups.  Below is a select that helps pull out certain characteristics of the product listings.  I am averaginges 5-8 seconds per query, the problem is, is that there are about 10,000 products that it needs to search through another table of listings that has about 1.6 million records. So this process can take 10+ hours to complete.  All the columns here are indexed.  Any suggestions?

select item_id, item_title,end_date
      from productlistings
      where end_date > '2012-03-01 00:00:00'
    and end_date < '2012-03-31 23:59:59'
    and (item_title like '% 10114 %')
      and (item_title not like '% only %')
      and (item_title not like '% only')
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')
      and (item_title not like 'loose %')
      and (item_title not like '% loose')
      and (item_title not like '% sealed box %')
      and (item_title not like '% packs %')
      and (item_title not like '% boxcase %')
      and (item_title not like '% 60 %')
      and (item_title not like '% complete set %')
      and (item_title not like '% all 16 packs')
      and (item_title not like '% case')
      and (item_title not like '% set of 16 %')
      and (item_title not like '% set 16 %')
      and (item_title not like '% all 16 %')
      and (item_title not like '% set of 16')
      and (item_title not like '% loose %')
      and (item_title not like 'set of 16 %')
      and (item_title not like '% set of %')
      and (item_title not like '% custom %')
        and set_id2 = 0
      and cleared_ind = 'x'
0
Comment
Question by:theideabulb
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 59

Expert Comment

by:Darius Ghassem
ID: 37814660
Do the product listing have a product_listing_id for example. Can you filter a different way?
0
 

Author Comment

by:theideabulb
ID: 37814667
no not really, think of it like an ebay feed or craiglists feed. It is going through the titles and looking for specific keyword, and filtering out those words.  We get an update every month that adds an extra 70-100k in listings to filter.   I am trying to grab the most recent date, but I still need to leave all the old listings in for individual searches.
0
 
LVL 24

Expert Comment

by:johanntagle
ID: 37814728
LIKE or NOT LIKE '% keyword %' does not make use of indexes.  Consider using boolean full-text search.  See http://dev.mysql.com/doc/refman/5.5/en/fulltext-boolean.html

Note that this is only available for myisam tables, and from what I read it doesn't really scale very well for very large data sets.  If you hit a performance issue with it a good alternative is a 3rd-party search engine like sphinx (http://sphinxsearch.com/) or maybe you can just integrate google search to your app.
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 1000 total points
ID: 37815090
If the date filter significantly reduces the recordset size, you can try a subquery with that as the only filter then apply the rest.

select item_id, item_title,end_date
      from (select * from productlistings
      where end_date between '2012-03-01' and  '2012-04-01') as Pl
    Where (item_title like '% 10114 %')
      and (item_title not like '% only %')
...

If any of your where filters take out a big chunk of stuff, a subquery can significantly speed things up. You can also use cte , I just find that harder to read.

Also you have duplication going on:
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')

% will match any OR no characters so the second and third statements above are pointless. Just removing all those may help.
0
 
LVL 25

Assisted Solution

by:Tomas Helgi Johannsson
Tomas Helgi Johannsson earned 1000 total points
ID: 37819372
Hi!

Put this index on your table and see if that doesn't help you
decreasing the query time

create index prodidcleard_ix
using btree
on productlistings (set_id2, cleared_ind, end_date,item_title);

Regarads,
      Tomas Helgi
0
 

Author Closing Comment

by:theideabulb
ID: 37824491
Combined, both of these comments shaved off 2-3 seconds, which adds up over the 10K plus times this will run.  Thank you for your knowledge.
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When table data gets too large to manage or queries take too long to execute the solution is often to buy bigger hardware or assign more CPUs and memory resources to the machine to solve the problem. However, the best, cheapest and most effective so…
This article shows the steps required to install WordPress on Azure. Web Apps, Mobile Apps, API Apps, or Functions, in Azure all these run in an App Service plan. WordPress is no exception and requires an App Service Plan and Database to install
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…
Suggested Courses

604 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question