Exchange 2016 CU9 Event 3003 and 3002 Repeating

Exchange 2016 CU 9 DAG 2 Nodes

Getting this two events over and over

[Log Name:      Application
Source:        MsExchange BackEndRehydration
Date:          9/13/2018 10:38:27 PM
Event ID:      3003
Task Category: Requests
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TGCS021-N1.our.network.tgcsnet.com
Description:
Protocol /mapi/nspi failed to perform token rehydration because source identity NT AUTHORITY\SYSTEM does not have token serialization permission.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MsExchange BackEndRehydration" />
    <EventID Qualifiers="49156">3003</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-09-14T02:38:27.433555100Z" />
    <EventRecordID>1152866</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TGCS021-N1.our.network.tgcsnet.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>/mapi/nspi</Data>
    <Data>NT AUTHORITY\SYSTEM</Data>
  </EventData>
</Event>/code]


[code]Log Name:      Application
Source:        MsExchange BackEndRehydration
Date:          9/13/2018 10:38:27 PM
Event ID:      3002
Task Category: Requests
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TGCS021-N1.our.network.tgcsnet.com
Description:
Protocol /mapi/nspi failed to process request from identity NT AUTHORITY\SYSTEM. Exception: Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission. 
   at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.TryGetCommonAccessToken(HttpContext httpContext, Stopwatch stopwatch, CommonAccessToken& token)
   at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.ProcessRequest(HttpContext httpContext)
   at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.OnAuthenticateRequest(Object source, EventArgs args).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MsExchange BackEndRehydration" />
    <EventID Qualifiers="49156">3002</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-09-14T02:38:27.434551600Z" />
    <EventRecordID>1152867</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TGCS021-N1.our.network.tgcsnet.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>/mapi/nspi</Data>
    <Data>NT AUTHORITY\SYSTEM</Data>
    <Data>Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission. 
   at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.TryGetCommonAccessToken(HttpContext httpContext, Stopwatch stopwatch, CommonAccessToken&amp; token)
   at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.ProcessRequest(HttpContext httpContext)
   at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.OnAuthenticateRequest(Object source, EventArgs args)</Data>
  </EventData>
</Event>

Open in new window



Any ideas?
LVL 23
Thomas GrassiSystems AdministratorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Edward van BiljonMessaging and Collaboration Technical Lead (Exchange MVP)Commented:
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Thomas GrassiSystems AdministratorAuthor Commented:
ED

I verified this


to vote  




I solved it differently, with the exact same symptoms.  I did a recovery install of Exchange 2016 and eventually tripped across the solution while trying to manually set internal and external URLs for the virtual directories, hoping it would help. When I tried to do that I got permission errors.  After tracking down how to solve those, I did this:


1. Verify that your Exchange Server is a member of the following groups:
• Exchange Install Domain Servers
• Exchange Servers
• Exchange Trusted Subsystem
• Managed Availability Servers

2. Verify that the local Administrators group has these members:
• Exchange Trusted Subsystem
• Organization Management


The install process did not put the exchange server into the proper groups.  (the stuff in step 2 was good on mine.)  It probably has to do with replacing the server with an identical server with the same name or something.  Either way, Ijust fixed the above, rebooted, and everything instantly magically worked.  I didn't even need to mess with URLs

I hope this helps.  I sure know if I'd have been able to find this answer, it would've saved me 5 hours of extremely intense stress.

Good Luck!


My exchange servers are a member of those groups

Those members are a member of the local administrator group

Any commands to verify or that might help solve this problem?
0
Thomas GrassiSystems AdministratorAuthor Commented:
Not Happy

No one on this site ever answers a question with a proper fix

This error is filing up the logs every day after restarting the servers

Does anyone on here have any idea?
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

SysToolsData Expert - Recovery,Backup,MigrationCommented:
0
Thomas GrassiSystems AdministratorAuthor Commented:
Systools

I saw that posting before and it was not helpful

Has to be a setting somewhere that is missing of something like that


His fix

PS] G:\>Test-WebServicesConnectivity -ClientAccessServer TGCS021-n2
Unable to find the client accesss monitoring user. Please run G:\Program Files\Microsoft\Exchange
Server\V15\Scripts\New-TestCasConnectivityUser.ps1. Exception:
Microsoft.Exchange.Monitoring.MailboxNotFoundException: Failed to find the mailbox. Mailbox =
'extest_2416929445214@our.network.tgcsnet.com'.
   at Microsoft.Exchange.Monitoring.CommonTestTasks.GetDefaultTestAccount(ClientAccessContext context)
   at Microsoft.Exchange.Monitoring.TestWebServicesTaskBase.GetMonitoringAccount()
    + CategoryInfo          : InvalidData: (:) [Test-WebServicesConnectivity], TestWebServicesTaskException
    + FullyQualifiedErrorId : [Server=TGCS021-N1,RequestId=791e59ae-a0ca-42f1-a226-77e656d9d2b1,TimeStamp=9/17/2018 12
   :19:21 PM] [FailureCategory=Cmdlet-TestWebServicesTaskException] 8598D61B,Microsoft.Exchange.Monitoring.TestWebSer
  vicesConnectivity
    + PSComputerName        : tgcs021-n1.our.network.tgcsnet.com

[PS] G:\>Test-WebServicesConnectivity -ClientAccessServer TGCS021-n1
Unable to find the client accesss monitoring user. Please run G:\Program Files\Microsoft\Exchange
Server\V15\Scripts\New-TestCasConnectivityUser.ps1. Exception:
Microsoft.Exchange.Monitoring.MailboxNotFoundException: Failed to find the mailbox. Mailbox =
'extest_2416929445214@our.network.tgcsnet.com'.
   at Microsoft.Exchange.Monitoring.CommonTestTasks.GetDefaultTestAccount(ClientAccessContext context)
   at Microsoft.Exchange.Monitoring.TestWebServicesTaskBase.GetMonitoringAccount()
    + CategoryInfo          : InvalidData: (:) [Test-WebServicesConnectivity], TestWebServicesTaskException
    + FullyQualifiedErrorId : [Server=TGCS021-N1,RequestId=bd1d8dd4-aed8-480d-a80f-095494a89d0f,TimeStamp=9/17/2018 12
   :19:24 PM] [FailureCategory=Cmdlet-TestWebServicesTaskException] 8598D61B,Microsoft.Exchange.Monitoring.TestWebSer
  vicesConnectivity
    + PSComputerName        : tgcs021-n1.our.network.tgcsnet.com

[PS] G:\>



PS] G:\>Test-OutlookWebServices -Identity:admin03@tgcsnet.com
The MailboxCredential parameter is required for mailbox 'admin03@tgcsnet.com'.
    + CategoryInfo          : InvalidArgument: (:) [Test-OutlookWebServices], TestWebServicesTaskException
    + FullyQualifiedErrorId : [Server=TGCS021-N1,RequestId=62e6f83c-f804-46c4-b7c2-275348fb96d8,TimeStamp=9/17/2018 12
   :22:16 PM] [FailureCategory=Cmdlet-TestWebServicesTaskException] 2327F914,Microsoft.Exchange.Monitoring.TestOutloo
  kWebServicesTask
    + PSComputerName        : tgcs021-n1.our.network.tgcsnet.com
0
Thomas GrassiSystems AdministratorAuthor Commented:
ED,

removed .net 4.7.2 restarted the servers and now no client can access email they outlook clients always asking for password over and over

Events 3002 and 3003 still filling up my event log

Email me
0
Thomas GrassiSystems AdministratorAuthor Commented:
Thanks Ed for your help
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.