Exchange 2013 Migration from Exchange 2010

After migrating all our users from the old server to the new one, email does not seem to flow internally.  users are able to send and receive external emails though.

The new server is setup with Mailbox and Client Access roles.  And has the following receive connectors:

Client Frontend "servername"
Client Proxy "servername"
Default "servername"
Default SMTP Internal Relay
Outbound Proxy Frontend "servername"

Where should are start looking to solve this issue?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AmitIT ArchitectCommented:
User on same server and same database also unable to send or rec emails?
Amit KumarCommented:
anything you can see in queue if pending, error event ID related to Transport service  on Exchange 2013 mailbox server?
AmitIT ArchitectCommented:
If you have multiple server and if service pack or rollups are not matching on all server, then you will see this kind of behavior. Can you run below command  and check version on all your 2013 and 2010 server. Post it , if you are not clear.

GCM exsetup |%{$_.Fileversioninfo}
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

modrynaAuthor Commented:
All of my users are on the same server now.  I have created new users and they cannot send to each other either.

In an attempt to solve the issue I have installed CU9 and now I do not have access to the Shell or Exchange tool to view the queue.  But can still access the EAC,
Amit KumarCommented:
Please check eventviwer and share critical or warning events if there are something.

Also have you tried restart your box after installing CU9, make sure it is installed on all servers.
modrynaAuthor Commented:
I see quite a few of these in the event log

The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database.
Database: Mailbox Database 1041719559
Error: Couldn't find system mailbox '4505e70c-d3e2-4421-9346-deabc24b4f53' in Active Directory.

Different databases and different mailboxes.  

This is the only server in the org at this time.  I have removed the old 2010 exchange server from the network.
Amit KumarCommented:
Please share the results of below powershell:

Get-Mailbox –Arbitration

Also check if any more errors are there in Event viewer.
AmitIT ArchitectCommented:
Run this and post the result

modrynaAuthor Commented:
I have not been able to access the shell yet but this was a recent log entry.

Performance counter updating error. Counter name is Current Requests Pending, category name is MSExchange LiveIdBasicAuthentication. Optional code: 2. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
   at System.Diagnostics.PerformanceCounter.InitializeImpl()
   at System.Diagnostics.PerformanceCounter.get_RawValue()
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.get_RawValue()
Last worker process info : System.ArgumentException: Process with an Id of 5868 is not running.
   at System.Diagnostics.Process.GetProcessById(Int32 processId)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetLastWorkerProcessInfo()
Processes running while Performance counter failed to update:
2060 sqlwriter
2164 updateservice
1148 svchost
5904 scanningprocess
1372 certsrv
7116 Microsoft.Exchange.Imap4
5504 msdtc
4320 umservice
8456 svchost
12732 winlogon
3328 SMSvcHost
372 csrss
12388 explorer
7352 Microsoft.Exchange.Store.Worker
2728 mqsvc
1348 svchost
14296 ngentask
5876 scanningprocess
5584 conhost
9152 taskhostex
1536 fms
6852 msexchangerepl
7836 conhost
2908 ocspsvc
3892 Microsoft.Exchange.Search.Service
1852 ismserv
9800 svchost
2904 noderunner
9584 svchost
5428 Microsoft.Exchange.Imap4
7036 Microsoft.Exchange.RpcClientAccess.Service
1752 MSExchangeTransportLogSearch
8216 Microsoft.Exchange.UM.CallRouter
11676 ngentask
924 svchost
528 lsass
4664 rundll32
1904 Microsoft.Exchange.Diagnostics.Service
17204 w3wp
10176 WmiPrvSE
1704 hostcontrollerservice
7416 Microsoft.Exchange.Store.Worker
520 services
7808 svchost
5640 scanningprocess
1108 w3wp
9972 conhost
712 svchost
4200 taskhost
2676 ForefrontActiveDirectoryConnector
1296 Microsoft.ActiveDirectory.WebServices
1500 dns
10380 svchost
7400 Microsoft.Exchange.Store.Worker
4444 ngen
2276 noderunner
12484 dwm
9872 ServerManager
1484 svchost
7392 MSExchangeTransport
2860 SMSvcHost
5420 MSExchangeSubmission
888 svchost
688 svchost
5808 Microsoft.Exchange.Imap4Service
15036 WsusService
16928 mscorsvw
1272 spoolsv
856 ngen
4204 conhost
2252 dfssvc
672 svchost
3232 MSExchangeFrontendTransport
3428 Microsoft.Exchange.AntispamUpdateSvc
12648 TiWorker
4876 MSExchangeMailboxAssistants
8348 svchost
9332 VSSVC
10008 rdpclip
464 winlogon
3024 sftracing
1644 svchost
264 smss
2232 Microsoft.Exchange.Directory.TopologyService
4792 conhost
4988 Microsoft.Exchange.EdgeSyncSvc
11684 mmc
12076 w3wp
4784 MSExchangeHMWorker
10496 rdpinput
9924 svchost
6368 MSExchangeThrottling
6676 MSExchangeMailboxReplication
7340 Microsoft.Exchange.Store.Worker
13052 csrss
1428 dfsrs
836 svchost
7532 vds
7768 EdgeTransport
832 MSExchangeHMHost
8908 UMWorkerProcess
436 wininit
14212 Microsoft.Exchange.Imap4Service
3200 noderunner
1972 TabTip
9380 WmiPrvSE
1808 inetinfo
428 csrss
9624 TabTip32
3736 conhost
3772 w3wp
816 dwm
14208 InetMgr
2584 noderunner
4356 rundll32
6432 MAPILab.Pop3Connector.Service
7308 Microsoft.Exchange.Store.Worker
13284 MSExchangeDelivery
10852 Microsoft.IdentityServer.ServiceHost
804 LogonUI
1000 svchost
1984 Microsoft.Exchange.ServiceHost
9108 w3wp
3160 sqlservr
9716 TrustedInstaller
4 System
16000 wsmprovhost
3548 MSExchangeDagMgmt
6108 Microsoft.Exchange.Store.Service
0 Idle
Performance Counters Layout information: FileMappingNotFoundException for category MSExchange LiveIdBasicAuthentication : Microsoft.Exchange.Diagnostics.FileMappingNotFoundException: Cound not open File mapping for name Global\netfxcustomperfcounters.1.0msexchange liveidbasicauthentication. Error Details: 2
   at Microsoft.Exchange.Diagnostics.FileMapping.OpenFileMapping(String name, Boolean writable)
   at Microsoft.Exchange.Diagnostics.PerformanceCounterMemoryMappedFile.Initialize(String fileMappingName, Boolean writable)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetAllInstancesLayout(String categoryName)
modrynaAuthor Commented:
Welcome to the Exchange Management Shell!

Full list of cmdlets: Get-Command
Only Exchange cmdlets: Get-ExCommand
Cmdlets that match a specific string: Help *<string>*
Get general help: Help
Get help for a cmdlet: Help <cmdlet name> or <cmdlet name> -?
Exchange team blog: Get-ExBlog
Show full output for a command: <command> | Format-List

Show quick reference guide: QuickRef
Tip of the day #36:

Want to move your database path to another location? Type:

 Move-DatabasePath -EdbFilePath DestFileName

To change the file path setting without moving data, use this command together with the ConfigurationOnly parameter. This command is especially useful for disaster recovery. Caution: Misuse of this cmdlet will cause data loss.

VERBOSE: Connecting to
VERBOSE: Connected to
[PS] C:\Windows\system32>Get-Mailbox -Arbitration

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
FederatedEmail.4c1f4d8... FederatedEmail.4c... ferarri          1 MB (1,048,576 bytes)
WARNING: The object has been corrupted,
 and it's in an inconsistent state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
SystemMailbox{1f05a927... SystemMailbox{1f0... ferarri          Unlimited
WARNING: The object{1f05a927-7d9e-49e0-89bc-d53f1ed0abd5} has been corrupted,
 and it's in an inconsistent state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.
SystemMailbox{bb558c35... SystemMailbox{bb5... bugatti          Unlimited
Migration.8f3e7716-201... Migration.8f3e771... bugatti          300 MB (314,572,800 bytes)
SystemMailbox{e0dc1c29... SystemMailbox{e0d... bugatti          Unlimited

[PS] C:\Windows\system32>

Ferarri is the old server that has been removed.
modrynaAuthor Commented:
[PS] C:\Windows\system32>Test-ServiceHealth

Role                    : Mailbox Server Role
RequiredServicesRunning : True
ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeDelivery, MSExchangeIS, MSExchangeMailboxAssistants, MSExchangeRepl, MSExchangeRPC, MSExchangeServiceHost, MSExchangeSubmission, MSExchangeThrottling, MSExchangeTransportLogSearch, W3Svc, WinRM}
ServicesNotRunning      : {}

Role                    : Client Access Server Role
RequiredServicesRunning : True
ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeIMAP4, MSExchangeMailboxReplication, MSExchangeRPC, MSExchangeServiceHost, W3Svc, WinRM}
ServicesNotRunning      : {}

Role                    : Unified Messaging Server Role
RequiredServicesRunning : True
ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeServiceHost, MSExchangeUM, W3Svc, WinRM}
ServicesNotRunning      : {}

Role                    : Hub Transport Server Role
RequiredServicesRunning : True
ServicesRunning         : {IISAdmin, MSExchangeADTopology, MSExchangeEdgeSync, MSExchangeServiceHost, MSExchangeTransport, MSExchangeTransportLogSearch, W3Svc, WinRM}
modrynaAuthor Commented:
Just ran PrepareAD again and this is what I get now.

[PS] C:\Windows\system32>get-mailbox -arbitration

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
SystemMailbox{bb558c35... SystemMailbox{bb5... bugatti          Unlimited
Migration.8f3e7716-201... Migration.8f3e771... bugatti          300 MB (314,572,800 bytes)
SystemMailbox{e0dc1c29... SystemMailbox{e0d... bugatti          Unlimited
modrynaAuthor Commented:
Problem Solved.  

I had to delete and recreate the Federation accounts via the PrepareAD command.

What I found about mail flow, was a combination of the Federated account being corrupted and  Backflow kicking in on the server.  I backed up the Databases and freed approximately 35GB of space.  Then email started flowing again.


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
modrynaAuthor Commented:
After following several recommendations I found several issues related to my problems.  Once resolved the server started behaving as designed.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.