Link to home
Create AccountLog in
Avatar of dave558
dave558

asked on

Exchange DAG environment - fails to mount passive copy

Hi

Running in test environment with a DAG group consisting of two DAG members.

Replication of the database works fine but is having issues when simulating failure. I have disconnected the server with the active copy of the database but the redundant server fails to mount the passive copy.

Here's the error message I get in event viewer:

PID 3920, Thread 15) Task Mount-Database writing error when processing record of index 0. Error: System.InvalidOperationException: Couldn't mount the database that you specified. Specified database: Mailbox Database 2143096883; Error code: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1105)
 [Database: Mailbox Database 2143096883, Server: MAILSTORE-A.example.info]. ---> Microsoft.Exchange.Cluster.Replay.AmDbActionWrapperException: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1105)
 ---> Microsoft.Exchange.Data.Storage.AmOperationFailedException: An Active Manager operation failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1105)
 ---> Microsoft.Mapi.MapiExceptionCallFailed: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1105)

   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown)
   at Microsoft.Mapi.ExRpcAdmin.MountDatabase(Guid guidStorageGroup, Guid guidMdb, Int32 ulFlags)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmStoreHelper.Mount(Guid mdbGuid, MountFlags flags)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.ActiveManagerCore.MountDatabaseDirect(Guid mdbGuid, MountFlags flags, AmDbActionCode actionCode)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmRpcServer.<>c__DisplayClass19.<MountDatabaseDirect>b__18()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
   --- End of inner exception stack trace (Microsoft.Mapi.MapiExceptionCallFailed) ---
   --- End of stack trace on server (MAILSTORE-A.example.info) ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.RunRpcOperationWithAuth(AmRpcOperationHint rpcOperationHint, String serverName, String databaseName, NetworkCredential networkCredential, Nullable`1 timeoutMs, AmRpcClient& rpcClient, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.MountDatabaseDirectEx(String serverToRpc, Guid dbGuid, AmMountArg mountArg)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbAction.MountDatabaseDirect(AmServerName serverName, AmServerName lastMountedServerName, Guid dbGuid, MountFlags flags, AmDbActionCode actionCode)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbPamAction.RunMountDatabaseDirect(AmServerName serverToMount, MountFlags mountFlags, Boolean fLossyMountEnabled)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbPamAction.<>c__DisplayClass3.<AttemptMountOnServer>b__1(Object , EventArgs )
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.HandleKnownExceptions(EventHandler ev)
   --- End of inner exception stack trace (Microsoft.Exchange.Data.Storage.AmOperationFailedException) ---
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbOperation.Wait(TimeSpan timeout)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.ActiveManagerCore.MountDatabase(Guid mdbGuid, MountFlags flags, DatabaseMountDialOverride mountDialOverride, AmDbActionCode actionCode)
   at Microsoft.Exchange.Cluster.ActiveManagerServer.AmRpcServer.<>c__DisplayClass4.<MountDatabase>b__3()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
   --- End of stack trace on server (MAILSTORE-A.example.info) ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.RunDatabaseRpcWithReferral(AmRpcOperationHint rpcOperationHint, Database database, String targetServer, AmRpcClient& rpcClient, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.MountDatabase(Database database, Int32 flags, Int32 mountDialOverride)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase.InternalProcessRecord()
   --- End of inner exception stack trace ---
Avatar of R--R
R--R
Flag of India image

Check the event viewer and find the error events related to Exchange
what was the details of the passive copy.
i.e. 1.status of the content index state
      2.status of the database copy
   
If the content index state and status of the database copy is healthy it should mount automatically
Avatar of dave558
dave558

ASKER

Status of the database copy is healthy.

How do I find out the content index state? Thanks.
Avatar of dave558

ASKER

I've ran the following command "Get-MailboxDatabaseCopyStatus –Server EX2 | fl Name,*Index*" and I get the content index state as failed.

Here's the details of what's happened:
SERVER-A - Passive copy (healthy), content index state (healthy)
SERVER-B - Active copy (mounted), content index state (healthy)

After disconnect SERVER-B, server goes in to the follow state:
SERVER-A - Dismounted (should now be active), content index state (failed)
SERVER-B - Server down

It doesn't show an error messages about the reason why failed content index state failed. Since content indexing should only occur when a database is mount, I'm thinking this failure message is as a result of the dismounted database.

Is this correct? Thanks.
Avatar of dave558

ASKER

Here's a few other messages I get when attempting to mount the DB:

Event 9519
While starting database Mailbox Database 2143096883, the following error occurred: 0x451.  
Start DB failed..

Event 9519
While starting database Mailbox Database 2143096883, the following error occurred: 0x451.  
Failed to configure MDB.

Event 1197
Database "Mailbox Database 2143096883" is too new to be upgraded.
please use this command to make content index state healthy

Update-MailboxDatabaseCopy "databasename\servername" -CatalogOnly
How Many NIC's do you have on both servers ?
Is the Catalog health before the Failure you do ?
What if you suspend the Catalog ? Is the search service and others running after the Failure exercise done ?

- Rancy
Avatar of dave558

ASKER

There are two NICs on each server but the Mailbox server are on Virtual Machines with only a single virtual NIC attached.

Prior to simulating the failure, the catalog is in a healthy for the passive server (SERVER-A). It's only after I've simulated the failure that the catalog goes in the "failed" state.

Also as the passive server has now become the active server, I am no longer able to suspend or update the Catalog feature. Seems like it flagged SERVER-A as the active copy even without be able to mount the DB.

The previous active server (SERVER-B) is now stuck on "Disconnected and Resynchronizing" and has an a Copy Queue length that appears to be invalid.

Seems like I'd need to get the current failed server (SERVER-A) active again before the SERVER-B can get resychronised.
How many Databases do you have ?

- Rancy
Avatar of dave558

ASKER

Two databases
ASKER CERTIFIED SOLUTION
Avatar of dave558
dave558

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of dave558

ASKER

Okay. Forced the server to mount the previous database using the following cmdlet:
Move-ActiveMailboxDatabase <DB> -ActivateOnServer <MB> -MountDialOverride BestEffort -SkipLagChecks -SkipClientExperienceChecks

Good thing this is only on a test environment. Thank you all for your help.
I would say get them both to possibly SP2 or SP3 .... I guess there are many emails stuck in the Queue .... HUB or Copy\Replay queue

- Rancy
Avatar of dave558

ASKER

SP1 mailboxes are incompatible with standard Exchange