Exchange 2013 - Event ID: 1010 Source: MSExchangeFastSearch

I have two relatively new Exchange 2013 servers running CU7, (no DAG, but both have CAS and mailbox roles). Both are throwing event ID 1010 in the application log with a source of MSExchangeFastSearch. I have googled, and my google-fu is coming up empty... it looks like there are at least three or four different variants. I'm getting these sometimes only a few seconds apart, and other times as much as 5 or 10 minutes apart. Any help in quieting these down would be much appreciated. I have highlighted the differences that I have found in each one.

Here's the fist one (chronologically):

An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace:
   at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.GetFlows()
   at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
   --- End of inner exception stack trace ---


approximately 6 seconds later:

An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace:
   at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Ceres.CoreServices.Admin.IAdminServiceManagementAgent.get_IsPendingReconfiguration()
   at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
   --- End of inner exception stack trace ---

Then, approximately 9 minutes later:

An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.

Server stack trace:
   at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Ceres.SearchCore.Admin.IIndexManagement.get_IndexSystemNames()
   at Microsoft.Exchange.Search.Fast.IndexManager.<GetCatalogs>b__1d()

   at Microsoft.Exchange.Search.Fast.FastManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)
   --- End of inner exception stack trace ---

Thanks,

Jonathan
LVL 3
JonathanSpitfireSenior Solutions EngineerAsked:
Who is Participating?
 
Simon Butler (Sembee)ConsultantCommented:
Do you mean Microsoft SEARCH host controller?
None of the Exchange services should be disabled. Some of them would be set to manual, but none disabled. If it is disabled then that could be a problem.

Simon.
0
 
Simon Butler (Sembee)ConsultantCommented:
Have you reset the search index?
There is a script in the Scripts directory of the Exchange install to do that for you.

Simon.
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Hi Simon,

I was hoping you would see this! I've looked in C:\Program Files\Microsoft\Exchange Server\V15\Scripts, and I don't see it. what's the script name? or am I looking in the wrong place?

Thanks!

Jonathan
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

 
Simon Butler (Sembee)ConsultantCommented:
Late night posting strikes again... I was thinking of Exchange 2010.
On 2013 you have to do it manually.

Stop the "Microsoft Exchange Search service" and "Microsoft Exchange Host Controller."
Then delete the catalog for the database.
Restart the above services.

Simon.
0
 
Minecraft_ EndermanCommented:
This issue may occur if the search platform tries to check its membership in a security group that is named "ContentSubmitters." This group is not created by the search platform or by Exchange Server 2013 and is therefore not usually present. Although the check usually fails silently, without any consequences, an exception sometimes occurs. This causes the search component to fail.

To resolve the issue, do the following:
1.      Create a new Active Directory group that is named "ContentSubmitters" and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.
2.      Force or wait for Active Directory replication.
3.      Restart the following services:
•      Microsoft Exchange Search
•      Microsoft Exchange Search Host Controller
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Thanks to both of you for posting.

Simon - the "Microsoft Exchange Host Controller" is set to disabled. Is that an issue? And by "delete the catalog for the database", just to be clear, I need to delete the folder in this directory:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database 0609142006

named REALLYLONGUID.Single, correct?

Kiss - it looks like you pulled that straight from this MS KB: http://support.microsoft.com/en-us/kb/2807668

I found that one in my searching, but essentially dismissed it because I'm not seeing event ID 1009 - only 1010

However, for kicks, I ran the command....here is the result...

[PS] C:\>Get-MailboxDatabaseCopyStatus | FL *index*,ma*ser*,*e*


ContentIndexState                     : Disabled
ContentIndexErrorMessage              : Content indexing is not enabled for the database.
ContentIndexErrorCode                 : 10


I saw that and thought, "hmmm...perhaps it is failing because indexing was disabled....", but alas, that was not it.

I have not created the AD group as you outlined just yet, because I only want to do one thing at a time, and will give Simon's advice a try first. I'll post back with my results. I'm working late tonight (I'm in the US on the East Coast) so I'll be working on this (and other things) for at least the next 5 hours.

Thanks,

Jonathan
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Interestingly enough, I am now seeing event Id 1006, whereas I do not believe I was before....

Log Name:      Application
Source:        MSExchangeFastSearch
Date:          4/1/2015 10:59:34 AM
Event ID:      1006
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Ex01.contoso.CORP
Description:
The FastFeeder component received a connection exception from FAST. Error details: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0469322. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult result)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.CommunicationObject.EndOpen(IAsyncResult result)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.EndOpen(IAsyncResult result)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.CreateProxy()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.AcquireProxy()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel& serviceProxy, Action`1 call, Int32 retryCount)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteAndReadPage(QueryParameters parameters, String outputName)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.GetHitCount(QueryParameters parameters)
   at Microsoft.Exchange.Search.Fast.ExchangeQueryExecutor.<>c__DisplayClass20.<GetHitCount>b__1f()
   at Microsoft.Exchange.Search.Fast.ExchangeQueryExecutor.RunUnderExceptionHandler[T](Func`1 call, IDiagnosticsSession session, String flowName)


Kiss - your suggestion did not resolve the issue.

Simon - I will try to reseed the index this afternoon, or tomorrow morning.

Thanks,

Jonathan
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Simon - if reseeding the search index should have solved it immediately....that didn't resolve it :-(

Other thoughts?

Thanks,

Jonathan
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Thanks Simon. I am pretty sure it is the Exchange Host Controller, but I will double check this afternoon or tonight. teaching my son how to install new brakes on the family car right now. :-)
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Simon - Apologies - I just realized I did not respond as intended.

It is in fact "Microsoft Exchange Search Host Controller" that is disabled. That is the ONLY one of the "Microsoft exchange..." services that is disabled.

I just set it to Auto, started it, and we'll see what happens.

Thanks,

Jonathan
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Its been almost two hours, and ever since I set the service to auto and started it, I have not had any more of these events! I will monitor for the next 24 hours and report back tomorrow. It looks like this may be solved!

Thanks,

Jonathan
0
 
JonathanSpitfireSenior Solutions EngineerAuthor Commented:
Setting the "Microsoft Exchange Search Host Controller" to Automatic and starting the service made the event ID 1010 with source MSExchangeFastSearch stop completely.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.