Help with select query speed

I have a process that helps me organize data into groups.  Below is a select that helps pull out certain characteristics of the product listings.  I am averaginges 5-8 seconds per query, the problem is, is that there are about 10,000 products that it needs to search through another table of listings that has about 1.6 million records. So this process can take 10+ hours to complete.  All the columns here are indexed.  Any suggestions?

select item_id, item_title,end_date
      from productlistings
      where end_date > '2012-03-01 00:00:00'
    and end_date < '2012-03-31 23:59:59'
    and (item_title like '% 10114 %')
      and (item_title not like '% only %')
      and (item_title not like '% only')
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')
      and (item_title not like 'loose %')
      and (item_title not like '% loose')
      and (item_title not like '% sealed box %')
      and (item_title not like '% packs %')
      and (item_title not like '% boxcase %')
      and (item_title not like '% 60 %')
      and (item_title not like '% complete set %')
      and (item_title not like '% all 16 packs')
      and (item_title not like '% case')
      and (item_title not like '% set of 16 %')
      and (item_title not like '% set 16 %')
      and (item_title not like '% all 16 %')
      and (item_title not like '% set of 16')
      and (item_title not like '% loose %')
      and (item_title not like 'set of 16 %')
      and (item_title not like '% set of %')
      and (item_title not like '% custom %')
        and set_id2 = 0
      and cleared_ind = 'x'
theideabulbAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Darius GhassemCommented:
Do the product listing have a product_listing_id for example. Can you filter a different way?
0
theideabulbAuthor Commented:
no not really, think of it like an ebay feed or craiglists feed. It is going through the titles and looking for specific keyword, and filtering out those words.  We get an update every month that adds an extra 70-100k in listings to filter.   I am trying to grab the most recent date, but I still need to leave all the old listings in for individual searches.
0
johanntagleCommented:
LIKE or NOT LIKE '% keyword %' does not make use of indexes.  Consider using boolean full-text search.  See http://dev.mysql.com/doc/refman/5.5/en/fulltext-boolean.html

Note that this is only available for myisam tables, and from what I read it doesn't really scale very well for very large data sets.  If you hit a performance issue with it a good alternative is a 3rd-party search engine like sphinx (http://sphinxsearch.com/) or maybe you can just integrate google search to your app.
0
Cloud Class® Course: Microsoft Windows 7 Basic

This introductory course to Windows 7 environment will teach you about working with the Windows operating system. You will learn about basic functions including start menu; the desktop; managing files, folders, and libraries.

Aaron TomoskySD-WAN SimplifiedCommented:
If the date filter significantly reduces the recordset size, you can try a subquery with that as the only filter then apply the rest.

select item_id, item_title,end_date
      from (select * from productlistings
      where end_date between '2012-03-01' and  '2012-04-01') as Pl
    Where (item_title like '% 10114 %')
      and (item_title not like '% only %')
...

If any of your where filters take out a big chunk of stuff, a subquery can significantly speed things up. You can also use cte , I just find that harder to read.

Also you have duplication going on:
      and (item_title not like '% lot %')
      and (item_title not like 'lot %')
      and (item_title not like '%lot')

% will match any OR no characters so the second and third statements above are pointless. Just removing all those may help.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tomas Helgi JohannssonCommented:
Hi!

Put this index on your table and see if that doesn't help you
decreasing the query time

create index prodidcleard_ix
using btree
on productlistings (set_id2, cleared_ind, end_date,item_title);

Regarads,
      Tomas Helgi
0
theideabulbAuthor Commented:
Combined, both of these comments shaved off 2-3 seconds, which adds up over the 10K plus times this will run.  Thank you for your knowledge.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
MySQL Server

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.