How to diagnose exchange server indexing?

Hi experts!

Our exchange 2007 SP3 search index had worked, so far.
Its folder "catalog...someGUID" held about 750 MB.
Then, some people have complained, that they don't find certain certain mails any more although they know, they are there. Search results were indeed incomplete. That made me re-index the exchange server on friday following this procedure  How to Rebuild the Full-Text Index Catalog It went fine and the application event log reports "Exchange Search Indexer has created a new search index and will perform a full crawl for the Mailbox Database  First Storage Group\Mailbox Database GUID...".

Problem: the catalog folder now holds less than 1 MB, and indexing is complete, the server is idling. The searches are terribly slow now, the index does not seem to hold anything anymore or is not even used!

How can I diagnose this dilemma?
Points will go to people that help me solve it, or at least help me understand what is going on.
LVL 63
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Simon Butler (Sembee)ConsultantCommented:
There isn't really much you can do.
It either works, or it doesn't. If the index file is 1mb, then it obviously hasn't indexed properly, so I would try a reset of the index again.

However I have seen problems with the database cause problems with indexing.
In that case the only fix was to create a new database, move all users to the new database and then drop the original.

McKnifeAuthor Commented:
Hi Simon.

I have new info. I did the re-indexing a second time already yesterday after posting here. The index directory has grown some more files overnight. When testing the search now, it is indeed fast (tested on some large folders (>1000 mails) of my own mailbox). I am a little surprised, because the old index folder was 750 MB and the new is less than 15 MB.

Where does the search get its results from? Using procmon on the server while I search, I could reveal, that the folder ...\First Storage Group\CatalogData-...
was not even touched!?
Simon Butler (Sembee)ConsultantCommented:
The search service is running all the time, so the fact that procmon didn't show anything doesn't surprise me because it wouldn't be a new connection.

Whether a 15 mb file is normal I cannot say as there is no context. If the database is 100mb, then maybe, if the database is 500gb, then maybe not.

Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

McKnifeAuthor Commented:
The old index was working and the folder was 750 MB like I wrote. The whole database is about 90 GB.
Based on my experience with file server indexing, the relation of documents sizes (office/pdf) to index size would be between 100:1 and 10:1, so the index should be at least a few hundred MB if not GB.

Procmon should show file access, no matter if the search service is already running, the index files should be constantly touched. I don't know any index that could not be monitored with procmon. Are you sure about what you write? Did you try and monitor your catalog folder for read access while perfoming a search?
McKnifeAuthor Commented:
More info:
The eventlog reveals that indexing has started (event 109) but hasn't indicated that is has completed (event 110), yet. Following there would be a performance indicator that explicitly tells us if indexing is active, namely MSExchange Search Indices\Full Crawl Mode Status which should be "1" when performing a full crawl. But it is 0.
Simon Butler (Sembee)ConsultantCommented:
How much load is the server under?
Indexing can be quite conservative, so if the server is under load it may well have stopped - particularly if you haven't had the event log entry.
You may have to be patient, see if anything happens overnight when the load should be less.

Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Is the corresponding (indexing) service still running, or has it stopped?
McKnifeAuthor Commented:
@Simon the load is low, cpu and disk wise. It had time from friday evening to sunday evening while no one was even using it apart one or two vpn people. That should have been enough.
@Qlemo Hi. "Microsoft Exchange Search Indexer" and "Microsoft Search  (Exchange)" - both started.
McKnifeAuthor Commented:
Found something interesting which indicates this is a known problem of exchange 2007 SP3:
This holds a solution which I will try now.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
McKnifeAuthor Commented:
Oh yeah, this is it. Just one single host-file-entry
and there we go, the CatalogData folder is growing and growing. Too bad MS assumes we have internet access on any exchange server.
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
<sarcastic mode> ... and very logical we need certificate revocation servers for maintaining an index ... </sarcastic mode>
Simon Butler (Sembee)ConsultantCommented:
Does the setting in Internet Explorer to check the revocation list not apply here?
I have two or three sites where the mailbox servers do not have internet access and it is fine. However I have to set Internet Explorer to not check the revocation list for updates to install in a timely manner.

McKnifeAuthor Commented:
It could very well be that you are right and there are other workarounds. The link I got it from offered another. Anyway it has finished indexing now and case can be closed.
statistical info: indexing of a 94 GB database took almost 8 hours and the index weighs 1.1 GB
McKnifeAuthor Commented:
Found it.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.