asked on

Exchange 2013 - Event ID: 1010 Source: MSExchangeFastSearch

I have two relatively new Exchange 2013 servers running CU7, (no DAG, but both have CAS and mailbox roles). Both are throwing event ID 1010 in the application log with a source of MSExchangeFastSearch. I have googled, and my google-fu is coming up empty... it looks like there are at least three or four different variants. I'm getting these sometimes only a few seconds apart, and other times as much as 5 or 10 minutes apart. Any help in quieting these down would be much appreciated. I have highlighted the differences that I have found in each one.

Here's the fist one (chronologically):

An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.GetFlows()
at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
--- End of inner exception stack trace ---

approximately 6 seconds later:

An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Ceres.CoreServices.Admin.IAdminServiceManagementAgent.get_IsPendingReconfiguration()
at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
--- End of inner exception stack trace ---

Then, approximately 9 minutes later:

An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Ceres.SearchCore.Admin.IIndexManagement.get_IndexSystemNames()
at Microsoft.Exchange.Search.Fast.IndexManager.<GetCatalogs>b__1d()
at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
--- End of inner exception stack trace ---

Thanks,

Jonathan

Simon Butler (Sembee)

Have you reset the search index?
There is a script in the Scripts directory of the Exchange install to do that for you.

Simon.

Jonathan Raper

ASKER

Hi Simon,

I was hoping you would see this! I've looked in C:\Program Files\Microsoft\Exchange Server\V15\Scripts, and I don't see it. what's the script name? or am I looking in the wrong place?

Thanks!

Jonathan

Simon Butler (Sembee)

Late night posting strikes again... I was thinking of Exchange 2010.
On 2013 you have to do it manually.

Stop the "Microsoft Exchange Search service" and "Microsoft Exchange Host Controller."
Then delete the catalog for the database.
Restart the above services.

Simon.

Minecraft_ Enderman

This issue may occur if the search platform tries to check its membership in a security group that is named "ContentSubmitters." This group is not created by the search platform or by Exchange Server 2013 and is therefore not usually present. Although the check usually fails silently, without any consequences, an exception sometimes occurs. This causes the search component to fail.

To resolve the issue, do the following:
1.      Create a new Active Directory group that is named "ContentSubmitters" and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.
2.      Force or wait for Active Directory replication.
3.      Restart the following services:
•      Microsoft Exchange Search
•      Microsoft Exchange Search Host Controller

Jonathan Raper

ASKER

Thanks to both of you for posting.

Simon - the "Microsoft Exchange Host Controller" is set to disabled. Is that an issue? And by "delete the catalog for the database", just to be clear, I need to delete the folder in this directory:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 0609142006

named REALLYLONGUID.Single, correct?

Kiss - it looks like you pulled that straight from this MS KB: http://support.microsoft.com/en-us/kb/2807668

I found that one in my searching, but essentially dismissed it because I'm not seeing event ID 1009 - only 1010

However, for kicks, I ran the command....here is the result...

[PS] C:\>Get-MailboxDatabaseCopyStatus | FL *index*,ma*ser*,*e*

ContentIndexState : Disabled
ContentIndexErrorMessage : Content indexing is not enabled for the database.
ContentIndexErrorCode : 10

I saw that and thought, "hmmm...perhaps it is failing because indexing was disabled....", but alas, that was not it.

I have not created the AD group as you outlined just yet, because I only want to do one thing at a time, and will give Simon's advice a try first. I'll post back with my results. I'm working late tonight (I'm in the US on the East Coast) so I'll be working on this (and other things) for at least the next 5 hours.

Thanks,

Jonathan

Jonathan Raper

ASKER

Interestingly enough, I am now seeing event Id 1006, whereas I do not believe I was before....

Log Name: Application
Source: MSExchangeFastSearch
Date: 4/1/2015 10:59:34 AM
Event ID: 1006
Task Category: General
Level: Warning
Keywords: Classic
User: N/A
Computer: Ex01.contoso.CORP
Description:
The FastFeeder component received a connection exception from FAST. Error details: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0469322. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult result)
--- End of inner exception stack trace ---

Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.CommunicationObject.EndOpen(IAsyncResult result)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.EndOpen(IAsyncResult result)
at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.CreateProxy()
at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.AcquireProxy()
at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel& serviceProxy, Action`1 call, Int32 retryCount)
at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteAndReadPage(QueryParameters parameters, String outputName)
at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.GetHitCount(QueryParameters parameters)
at Microsoft.Exchange.Search.Fast.ExchangeQueryExecutor.<>c__DisplayClass20.<GetHitCount>b__1f()
at Microsoft.Exchange.Search.Fast.ExchangeQueryExecutor.RunUnderExceptionHandler[T](Func`1 call, IDiagnosticsSession session, String flowName)

Kiss - your suggestion did not resolve the issue.

Simon - I will try to reseed the index this afternoon, or tomorrow morning.

Thanks,

Jonathan

Jonathan Raper

ASKER

Simon - if reseeding the search index should have solved it immediately....that didn't resolve it :-(

Other thoughts?

Thanks,

Jonathan

ASKER CERTIFIED SOLUTION

Simon Butler (Sembee)

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Jonathan Raper

ASKER

Thanks Simon. I am pretty sure it is the Exchange Host Controller, but I will double check this afternoon or tonight. teaching my son how to install new brakes on the family car right now. :-)

Jonathan Raper

ASKER

Simon - Apologies - I just realized I did not respond as intended.

It is in fact "Microsoft Exchange Search Host Controller" that is disabled. That is the ONLY one of the "Microsoft exchange..." services that is disabled.

I just set it to Auto, started it, and we'll see what happens.

Thanks,

Jonathan

Jonathan Raper

ASKER

Its been almost two hours, and ever since I set the service to auto and started it, I have not had any more of these events! I will monitor for the next 24 hours and report back tomorrow. It looks like this may be solved!

Thanks,

Jonathan

Jonathan Raper

ASKER

Setting the "Microsoft Exchange Search Host Controller" to Automatic and starting the service made the event ID 1010 with source MSExchangeFastSearch stop completely.