Better Approach for full text ranking

I have a program that builds a database of keywords to relate to a certain email thread.  Users can select sets of keywords that uniquely identify an email thread and then allow the program to run through the database of these "sets of keywords" against every email to create a ranking of possible threads the email is related to.

Now I notice that my searches are getting slower and slower.  My algorithim is very simple:  for each Keyword, do an instr function against the email text and count the numnber of times a set of keywords matches.  because the number of defined keyword sets is increasing, this is taking longer and longer.

Even if I branch this out to multiple threads or "mothball" old keyword sets it's clear to me that I am in need of a different approach if I expect performance to be acceptable for the long term.  I marvel at the searches that google does, quick and thourogh, with ranking, all on a few keywords.  How can I extend this kind of functionality to my application?  I am writting in Visual Studio, my keywords sets are stored in a disconnected ADO.NET dataset, so I don't have a real database backend with full text capabilities.
Who is Participating?
RobertRFreemanConnect With a Mentor Commented:
I'd use sql express to store the data.  Then you can just use a where clause with a LIKE filter.
Alternatively, you can put trace statements in your code to identify where the hangup is.
fmonroyConnect With a Mentor Commented:
Maybe you are reading records on a loop. If yes I think you can extract every word on the email's body, then insert them in a temporary table and join this data to the threads' list.

I think this will have good performance; to get better performance you can read entire thread's list into memory and compara there, but it can take a lot of memory for large lists.
Jai SConnect With a Mentor Tech ArchCommented:
cant you use a WHERE clause in your QUERY to the database rather than doing it in .NET ? or you can use FILTERS from your recordsets (datatable)
photowhizConnect With a Mentor Commented:
Google (and in fact all web search engines) use full-text search. Read about it here:
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.