Link to home
Start Free TrialLog in
Avatar of SysUA
SysUAFlag for Belgium

asked on

Exchange 2010 to 2016 moves fail in completing stage

Hi all,
Since I upgraded a couple of our Exchange 2016 servers to CU 5 I am unable to do mailbox move requests properly. They all end up "relinquished" due to source mailbox lock. I made myself a test mailbox and tried moving it from 2010 to 2016 but that one has the same issue. Even moving mailboxes from one 2016 server to another 2016 server won't go, same message. Moving them to CU 4 servers does work, but we need those CU5 servers too. The move report goes like this:

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​4/27/2017 4:08:55 PM [EXCH-01] '' created move request.
4/27/2017 4:09:00 PM [EXCH-01] The Microsoft Exchange Mailbox Replication service 'EXCH-01.domain.com' (15.1.845.34 caps:07FFFF) is examining the request.
4/27/2017 4:09:00 PM [EXCH-01] Connected to target mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)', database 'MDB-01', Mailbox server 'EXCH-01.domain.com' Version 15.1 (Build 845.0).
4/27/2017 4:09:01 PM [EXCH-01] Connected to source mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)', database 'MSTORE06', Mailbox server 'XMAIL35.domain.com' Version 14.3 (Build 319.0).
4/27/2017 4:09:01 PM [EXCH-01] Request processing started.
4/27/2017 4:09:01 PM [EXCH-01] Source mailbox information:
Regular Items: 1, 4.343 KB (4,447 bytes)
Regular Deleted Items: 0, 0 B (0 bytes)
FAI Items: 9, 0 B (0 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
4/27/2017 4:09:01 PM [EXCH-01] Cleared sync state for request 4fd2c52e-a05e-44d4-9832-e9a7163d5d2c due to 'CleanupOrphanedMailbox'.
4/27/2017 4:09:01 PM [EXCH-01] Mailbox signature will not be preserved for mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)'. Outlook clients will need to restart to access the moved mailbox.
4/27/2017 4:09:02 PM [EXCH-01] Stage: CreatingFolderHierarchy. Percent complete: 10.
4/27/2017 4:09:02 PM [EXCH-01] Initializing folder hierarchy from mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)': 33 folders total.
4/27/2017 4:09:02 PM [EXCH-01] Folder creation progress: 0 folders created in mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)'.
4/27/2017 4:09:03 PM [EXCH-01] Folder hierarchy initialized for mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)': 32 folders created.
4/27/2017 4:09:03 PM [EXCH-01] Stage: CreatingFolderHierarchy. Percent complete: 10.
4/27/2017 4:09:03 PM [EXCH-01] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
4/27/2017 4:09:03 PM [EXCH-01] Initial sync checkpoint progress: 0/33 folders processed. Currently processing mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)'.
4/27/2017 4:09:03 PM [EXCH-01] Initial sync checkpoint completed: 29 folders processed.
4/27/2017 4:09:03 PM [EXCH-01] Stage: LoadingMessages. Percent complete: 20.
4/27/2017 4:09:04 PM [EXCH-01] Stage: LoadingMessages. Percent complete: 20.
4/27/2017 4:09:07 PM [EXCH-01] Copying messages is complete. Copying rules and security descriptors.
4/27/2017 4:09:08 PM [EXCH-01] Initial seeding completed, 9 items copied, total size 4.212 KB (4,313 bytes).
4/27/2017 4:09:08 PM [EXCH-01] Stage: LoadingMessages. Percent complete: 95.
4/27/2017 4:09:09 PM [EXCH-01] Folder hierarchy changes reported in source '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)': 0 changed folders, 0 deleted folders.
4/27/2017 4:09:09 PM [EXCH-01] Incremental Sync '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)' completed: 0 hierarchy updates, 0 content changes.
4/27/2017 4:09:09 PM [EXCH-01] Stage: IncrementalSync. Percent complete: 95.
4/27/2017 4:09:09 PM [EXCH-01] Job has reached synced state. This job will be auto-resumed at 4/28/2017 4:09:09 PM.
4/27/2017 4:10:27 PM [EXCH-01] '' modified move request.
4/27/2017 4:10:31 PM [EXCH-01] The Microsoft Exchange Mailbox Replication service 'EXCH-01.domain.com' (15.1.845.34 caps:07FFFF) is examining the request.
4/27/2017 4:10:31 PM [EXCH-01] Connected to target mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)', database 'MDB-01', Mailbox server 'EXCH-01.domain.com' Version 15.1 (Build 845.0).
4/27/2017 4:10:31 PM [EXCH-01] Connected to source mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)', database 'MSTORE06', Mailbox server 'XMAIL35.domain.com' Version 14.3 (Build 319.0).
4/27/2017 4:10:31 PM [EXCH-01] Request processing continued, stage IncrementalSync.
4/27/2017 4:10:31 PM [EXCH-01] Folder hierarchy changes reported in source '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)': 0 changed folders, 0 deleted folders.
4/27/2017 4:10:31 PM [EXCH-01] Incremental Sync '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)' completed: 0 hierarchy updates, 0 content changes.
4/27/2017 4:10:31 PM [EXCH-01] Stage: IncrementalSync. Percent complete: 95.
4/27/2017 4:10:31 PM [EXCH-01] Final sync has started.
4/27/2017 4:10:31 PM [EXCH-01] Folder hierarchy changes reported in source '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)': 0 changed folders, 1 deleted folders.
4/27/2017 4:10:31 PM [EXCH-01] Incremental Sync '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)' completed: 1 hierarchy updates, 0 content changes.
4/27/2017 4:10:31 PM [EXCH-01] Source mailbox information:
Regular Items: 1, 4.343 KB (4,447 bytes)
Regular Deleted Items: 0, 0 B (0 bytes)
FAI Items: 9, 0 B (0 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
4/27/2017 4:10:31 PM [EXCH-01] Stage: FinalIncrementalSync. Percent complete: 95.
4/27/2017 4:10:32 PM [EXCH-01] Mailbox store finalization is complete.
4/27/2017 4:10:32 PM [EXCH-01] SessionStatistics updated.
4/27/2017 4:10:32 PM [EXCH-01] Verifying mailbox contents...
4/27/2017 4:10:32 PM [EXCH-01] Mailbox contents verification complete: 29 folders, 8 items, 3.871 KB (3,964 bytes).
4/27/2017 4:10:32 PM [EXCH-01] Waiting for mailbox changes to replicate.
4/27/2017 4:11:41 PM [EXCH-01] Waiting for mailbox changes to replicate.
4/27/2017 4:12:18 PM [EXCH-01] The Microsoft Exchange Mailbox Replication service 'EXCH-01.domain.com' (15.1.845.34 caps:07FFFF) is examining the request.
4/27/2017 4:12:18 PM [EXCH-01] Connected to target mailbox '4fd2c52e-a05e-44d4-9832-e9a7163d5d2c (Primary)', database 'MDB-01', Mailbox server 'EXCH-01.domain.com' Version 15.1 (Build 845.0).
4/27/2017 4:12:19 PM [EXCH-01] Relinquishing job because the mailbox is locked. The job will attempt to continue again after 4/27/2017 4:17:19 PM.

And then if I just wait it keeps repeating the last part over and over again, saying the mailbox is locked.

Get-MoveRequestStatistics shows:
DisplayName: test mbx
StatusDetail: StalledDueToSource_MailboxLock
PercentComplete: 95

And it generally stays that way.

I have tried upping the MRS config settings for MaxActiveJobsPerSourceMailbox and MaxActiveJobsPerTargetMailbox tot 20, not sure if that would help, restarted the replication and mailbox replication services, checked the databasecopystatus (specifically content indexing), uninstalled antivirus that was there first on all servers that have copies of the databases in question, rebooted the EXCH-01 server twice... Nothing helped. Is this a problem with CU5 (since our CU4 machine still works) or is there something else going on here?

Thanks for your time!
Avatar of Scott C
Scott C
Flag of United States of America image

Yep.  Looks like an issue with Exchange 2016 CU5.

Take a look here.

https://technet.microsoft.com/en-us/library/jj150489(v=exchg.160).aspx

Moving mailboxes from earlier versions of Exchange to Exchange 2016 CU5 or later can fail When you attempt to move a mailbox from an earlier version of Exchange to Exchange CU5 or later using a migration batch request, the move might fail. This can happen if the migration system mailbox isn't located on an Exchange 2016 server with CU5 or later installed.
The pertinent fix command from Scott's link is this:
New-MoveRequest "Migration.8f3e7716-2011-43e4-96b1-aba62d229136" -TargetDatabase "<mailbox database name>"

Open in new window

Which should move the migration system mailbox to the new database, which will allow it to properly manage the mailboxes that are created on that DB for the move.

If, for some reason, that doesn't fix your problem, though https://blogs.technet.microsoft.com/mikehall/2013/08/06/managing-failed-mailbox-move-requests/ has a lot of helpful information on troubleshooting move failures.
Avatar of SysUA

ASKER

Thanks for your comments! I did however see those pages and 1) the Migration system mailbox already is in a CU5 mailbox database and 2) the moverequests don't actually fail as far as I've seen (I haven't waited for 5 hours or whatever it takes to finale fail), they simply continue on starting over and over again OR simply saying "mailbox locked" and trying this again every 10 minutes or so... I can show you a log where it did so 10 times or so, each time causing user interrupts with Outlook disconnecting while it tries to finalize the move. So frustrating...

I'm now running a move of my test mailbox without stopping it, just to see if it actually fails or somehow completes correctly, but I doubt that. Again, thanks for your time, I appreciate it!
Avatar of SysUA

ASKER

I noticed in application log that while it retries the move this error shows up twice:

Replication for database MDB-01 is not flushed yet. Constraint: SecondCopy, number of copies: 3, minimum replay time: 27/04/2017 17:08:01, commit time: 27/04/2017 17:13:50. Failure reason: Database 84e022bd-d75b-4bee-a0a9-e37afc69d8e5 doesn't satisfy the constraint SecondCopy because the commit time 27/04/2017 17:13:50 isn't guaranteed by replication time 27/04/2017 17:08:01.

Not sure if this is of any help...

There's also this warning (twice):

Service MSExchangeMailboxAssistants.  An exception has been thrown: Microsoft.Exchange.Assistants.SkipException: MailboxNotFound:c9603ad7-df15-4c49-a5ce-a4edee0e91d5 ---> Microsoft.Exchange.Data.Storage.ObjectNotFoundException: The Active Directory user wasn't found.
   at Microsoft.Exchange.Data.Storage.ExchangePrincipalFactory.<>c__DisplayClass2f.<FromLocalServerMailboxGuid>b__2e()
   at Microsoft.Exchange.Data.Storage.ExchangePrincipalFactory.FromLocalServerMailboxGuid(IRecipientSession recipientSession, Guid mdbGuid, Guid mailboxGuid, IMailboxLocation mbxLocation, Boolean isContentIndexing)
   at Microsoft.Exchange.Data.Storage.ExchangePrincipalFactory.<>c__DisplayClass2c.<FromLocalServerMailboxGuid>b__2b()
   at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.OpenMailboxSession(MailboxQueueItem item)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Assistants.Util.TraceAndThrow(Action function, AIException aiException, String nonLocalizedAssistantName)
   at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)
   at Microsoft.Exchange.Assistants.Base.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)
Ok. Are you moving mailboxes to or from a DAG? If so, disable replication (and enable circular logging) on the databases. It looks like you're having some issues with the databases replicating the mailbox move data across. Also look at this: http://www.itprocentral.com/solving-datamovereplicationconstraint-when-moving-large-mailboxes/ since you may need to disable the requirement that the mailboxes be replicated to the other database and verified before being completed. It's better to migrate the mailboxes with replication disabled, then seed the database copies after all the mailboxes are moved, particularly if your DAGs aren't configured with a dedicated MAPI network. There's a lot of activity going on with a mailbox move, and having the mailbox move data and replication data on the same network can slow things down significantly.
Avatar of SysUA

ASKER

Setting DataMoveReplicationContraint to "None" on both sides (source and target servers) didn't help, mailbox move goes to finalization step, then says "StalledDueToSource_MailboxLock". I now tried to move a mailbox from a 2010 database to a 2016 CU4 database which isn't replicating (single copy) and am getting the same message: "StalledDueToSource_MailboxLock". So I don't think it's a CU5 related issue but rather something in our 2010 environment or in the communication between 2010 and 2016, since it says "Source_MailboxLock", not something like "Target_MailboxLock". So basically, I am now unable to move any mailbox at all.
Avatar of SysUA

ASKER

Moving a test mailbox from 2010 database 1 to an other 2010 database does work.
Moving from 2016 CU4 to CU5 works as well. For a small test mailbox anyway.
Moving from Exchange 2010 SP3 databases to 2016 CU5 still gives me the "StalledDueToSource_MailboxLock" message over and over again.
Avatar of SysUA

ASKER

Further troubleshooting reveals the follow warning in the source server:

(EventID 9660, MSExchangeIS) "User TESTUSERX (/o=ORG/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=TESTUSERX797) failed to sign in because their mailbox is in the process of being moved."

However, my test user is definitely not signing in anywhere. Based on the article over at https://www.experts-exchange.com/questions/28157054/Warning-during-mailbox-move-on-2010.html I ran:

Get-mailboxstatistics -Identity "TESTUSERX" -IncludeMoveReport | select movehistory

Which gave:
MapiExceptionMailboxInTransit: Unable to open message store. (hr=0x80004005, ec=1292)
Diagnostic context:
    Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=193]
    Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=256][latency=0]
    Lid: 23226   --- ROP Parse Start ---
    Lid: 27962   ROP: ropLogon [254]
    Lid: 17082   ROP Error: 0x50C
    Lid: 26937
    Lid: 21921   StoreEc: 0x50C
    Lid: 27962   ROP: ropExtendedError [250]
    Lid: 1494    ---- Remote Context Beg ----
    Lid: 26426   ROP: ropLogon [254]
    Lid: 22787   Error: 0x0
    Lid: 13032   StoreEc: 0x8004010F
    Lid: 25848
    Lid: 7588    StoreEc: 0x8004010F
    Lid: 25840
    Lid: 6564    StoreEc: 0x8004010F
    Lid: 48729
    Lid: 24316
    Lid: 2199    StoreEc: 0x50C
    Lid: 56415
    Lid: 48223   StoreEc: 0x50C
    Lid: 17097   StoreEc: 0x50C
    Lid: 8620    StoreEc: 0x50C
    Lid: 1750    ---- Remote Context End ----
    Lid: 26849
    Lid: 21817   ROP Failure: 0x50C
    Lid: 26297
    Lid: 16585   StoreEc: 0x50C
    Lid: 32441
    Lid: 1706    StoreEc: 0x50C
    Lid: 24761
    Lid: 20665   StoreEc: 0x50C
    Lid: 25785
    Lid: 29881   StoreEc: 0x50C
    + CategoryInfo          : ResourceUnavailable: (4fd2c52e-a05e-44d4-9832-e9a7163d5d2c:MailboxId) [Get-MailboxStatis
   tics], MapiExceptionMailboxInTransit
    + FullyQualifiedErrorId : B5FCF035,Microsoft.Exchange.Management.MapiTasks.GetMailboxStatistics

Might be normal during the move of the mailbox, so I stopped the migration and tried again, but the message remained the same. The same command for a random user who wasn't moved yet doesn't yield such errors...
Avatar of SysUA

ASKER

It's me again. I got some things in order by re-running setup.exe /prepareAD. After that I could move (small) mailboxes to Exchange 2016. Larger mailboxes still get the "StalledDueToSourceMailboxLocked" state indefinitely. Now, even a smaller mailbox gets this status after 20% moving. I did however notice this yellow warning popping up when quickly repeating Get-MoveRequestStatistics:

WARNING: Call to MRS to get up-to-date request info has failed. An exception occured: The call to 'net.tcp://OURSERVER/Microsoft.Exchange.MailboxReplicationService' failed because no service was listening on the specified endpoint. Error details: There was no endpoint listening at net.tcp://OURSERVER/Microsoft.Exchange.MailboxReplicationService that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details..

What is going on...? Could it be our load balancing (F5 bigIP)?
Avatar of SysUA

ASKER

This silence is eerie...

I did some testing and research and after restarting the MSExchangeMRS service on all Exchange servers I was able to get a move going from one 2016 database to another, only to have it stop with statusdetail "StalledDueToUnknownReason". I've attached the moverequest report with the "localizedstring" only for readability, the full report and the "Get-ExchangeDiagnosticInfo" output.

Thanks for your time.
report-localizedstringonly.txt
report-full.txt
diagnosticsinfo.txt
Avatar of Michel van Loon
Michel van Loon

Hi all, maybe a bit late, but i did not find a clear answer on the internet on this, so i'll put here what our solution was.

We ran into the same problem (exchange 2016 CU 5 with server 2016, moving from exchange 2010 and server 2008R2)

We do not have NLB in this case.

The logs point to network connection issues, IIS, MRSProxy, giving a nice tour of the 2016 (exchange)server when you follow the breadcrums.

It turned out to be an active directory rights issue. (at 95 percent the move process wants to put the new mailbox location in the AD for the user object).
It gives the net.tcp: error very brief in powershell and causing the mailbox to lock.

After some tests we solved this by creating an new OU at the top level and moving all users to the new OU (linking the GPO's there)
Here the exchange server could do it's thing and the move was completed successfully.

It is probably an inheritance issue on one of the OU's causing the Exchange server to have no access to edit the AD objects. (it could read the objects)
 moving the users was the quickest way for us to go on with the migration.

I hope this will save you all some time when you run into this issue.

Cheers,
Avatar of SysUA

ASKER

Thank you Michel for your reply. In our case it seems we have an issue with address policies containing special characters, we are working with Microsoft support to find out what exactly goes wrong.
Thanks for the information,

It could actually be related,

The address policies are based on OU's in our situation. The address policy were not applied when the users were in the new OU.

I've tried to move some users again (the next batch) in a OU where the mail address policy is applied, and got the source mailbox lock.

I now have 10 users who are stuck again, I will run some tests this weekend to see if i can get them moving again.

I've tried to disable all mailaddress policies to see if that would help, but that had no results.

The 10 users do not have special characters in their names, but I'll have a look in the ADUC attribute editor.

After moving the users to the new OU they got their USERNAME@ADNAME.LOCAL mailaddress as primary e-mail address.

Maybe the e-mail address policy is locking an attribute in the userobject in AD somehow (just a thought) this could also cause an access denied to exchange.

I notice the mailbox move getting stuck at 2 places:

at 20%  and at 95%

The 95% is an ADchange, but the 20% is unknown to me what exchange is doing at that point.

I will get back to you as soon as i find something.
ASKER CERTIFIED SOLUTION
Avatar of SysUA
SysUA
Flag of Belgium image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Disabling the Email Address Policy before the move worked, thanks for the information. All users are migrated now.

thank you.
Avatar of SysUA

ASKER

Fixed this together with MS support.