Exchange 2016 Event 4127 Cannot run certain commandsin EMS Error 403

Thomas Grassi
Thomas Grassi used Ask the Experts™
on
Windows 2016 Datacenter
Exchange 2016 Enterprise CU9 DAG 2 Node

Getting Event 4127 every day over and over


Log Name:      Application
Source:        MSExchange ADAccess
Date:          6/26/2018 12:16:18 PM
Event ID:      4127
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      serv021-N1.mynet.com
Description:
Process powershell.exe (PID=9520). Component: Microsoft.Exchange.Data.Directory.ConfigurationSettingsADNotificationException: Error running AD operation. ---> Microsoft.Exchange.Data.Directory.ADTopologyUnexpectedException: Unexpected error when calling the Microsoft Exchange Active Directory Topology service on server 'TopologyClientTcpEndpoint (localhost)'. Error details: Access is denied.. ---> System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.

Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Exchange.Data.Directory.TopologyDiscovery.ITopologyClient.GetServersForRole(String partitionFqdn, List`1 currentlyUsedServers, ADServerRole role, Int32 serversRequested, Boolean forestWideAffinityRequested)
   at Microsoft.Exchange.Data.Directory.ServiceTopologyProvider.<>c__DisplayClass33_0.<InternalServiceProviderGetServersForRole>b__0(IPooledServiceProxy`1 proxy)
   at Microsoft.Exchange.Net.ServiceProxyPool`1.TryCallServiceWithRetry(Action`1 action, String debugMessage, WCFConnectionStateTuple proxyToUse, Int32 numberOfRetries, Boolean doNotReturnProxyOnSuccess, Exception& exception)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Data.Directory.ServiceTopologyProvider.GetConfigDCInfo(String partitionFqdn, Boolean throwOnFailure)
   at Microsoft.Exchange.Data.Directory.TopologyProvider.PopulateConfigNamingContexts(String partitionFqdn)
   at Microsoft.Exchange.Data.Directory.TopologyProvider.GetConfigurationNamingContext(String partitionFqdn)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.GetRootOrgContainer(String partitionFqdn, String domainController, NetworkCredential credential)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ConfigurationSettings.ADConfigDriver.<>c__DisplayClass16_0.<LoadSettings>b__0()
   at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.RunADOperation(ADOperation adOperation, Int32 retryCount)
   at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.TryRunADOperation(ADOperation adOperation, Int32 retryCount)
   --- End of inner exception stack trace ---. Unable to load application settings. Exception: '%4'
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange ADAccess" />
    <EventID Qualifiers="49152">4127</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-06-26T16:16:18.764049500Z" />
    <EventRecordID>215045</EventRecordID>
    <Channel>Application</Channel>
    <Computer>serv021-N1.mynet.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>powershell.exe</Data>
    <Data>9520</Data>
    <Data>Microsoft.Exchange.Data.Directory.ConfigurationSettingsADNotificationException: Error running AD operation. ---&gt; Microsoft.Exchange.Data.Directory.ADTopologyUnexpectedException: Unexpected error when calling the Microsoft Exchange Active Directory Topology service on server 'TopologyClientTcpEndpoint (localhost)'. Error details: Access is denied.. ---&gt; System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.

Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc&amp; rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
   at Microsoft.Exchange.Data.Directory.TopologyDiscovery.ITopologyClient.GetServersForRole(String partitionFqdn, List`1 currentlyUsedServers, ADServerRole role, Int32 serversRequested, Boolean forestWideAffinityRequested)
   at Microsoft.Exchange.Data.Directory.ServiceTopologyProvider.&lt;&gt;c__DisplayClass33_0.&lt;InternalServiceProviderGetServersForRole&gt;b__0(IPooledServiceProxy`1 proxy)
   at Microsoft.Exchange.Net.ServiceProxyPool`1.TryCallServiceWithRetry(Action`1 action, String debugMessage, WCFConnectionStateTuple proxyToUse, Int32 numberOfRetries, Boolean doNotReturnProxyOnSuccess, Exception&amp; exception)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Data.Directory.ServiceTopologyProvider.GetConfigDCInfo(String partitionFqdn, Boolean throwOnFailure)
   at Microsoft.Exchange.Data.Directory.TopologyProvider.PopulateConfigNamingContexts(String partitionFqdn)
   at Microsoft.Exchange.Data.Directory.TopologyProvider.GetConfigurationNamingContext(String partitionFqdn)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ADSystemConfigurationSession.GetRootOrgContainer(String partitionFqdn, String domainController, NetworkCredential credential)
   at Microsoft.Exchange.Data.Directory.SystemConfiguration.ConfigurationSettings.ADConfigDriver.&lt;&gt;c__DisplayClass16_0.&lt;LoadSettings&gt;b__0()
   at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.RunADOperation(ADOperation adOperation, Int32 retryCount)
   at Microsoft.Exchange.Data.Directory.ADNotificationAdapter.TryRunADOperation(ADOperation adOperation, Int32 retryCount)
   --- End of inner exception stack trace ---</Data>
  </EventData>
</Event>

Open in new window


When I start EMS Exchange Management Shell  I get this


         Welcome to the Exchange Management Shell!

Full list of cmdlets: Get-Command
Only Exchange cmdlets: Get-ExCommand
Cmdlets that match a specific string: Help *<string>*
Get general help: Help
Get help for a cmdlet: Help <cmdlet name> or <cmdlet name> -?
Exchange team blog: Get-ExBlog
Show full output for a command: <command> | Format-List

Show quick reference guide: QuickRef
VERBOSE: Connecting to serv021-N1.mynet.com.
New-PSSession : [serv021-n1.mynet.com] Connecting to remote server serv021-n1.mynet.com
failed with the following error message : The WinRM client received an HTTP status code of 403 from the remote
WS-Management service. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Micr ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108273,PSSessionOpenFailed
VERBOSE: Connecting to serv021-N1.mynet.com.
New-PSSession : [serv021-n1.our.network.tgcsnet.com] Connecting to remote server serv021-n1.mynet.com
failed with the following error message : The WinRM client received an HTTP status code of 403 from the remote
WS-Management service. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Micr ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108273,PSSessionOpenFailed
VERBOSE: Connecting to SERV021-N2.our.network.tgcsnet.com.
VERBOSE: Connected to SERV021-N2.our.network.tgcsnet.com.
[PS] C:\Windows\system32>

I ran this the other day
      Get-PowerShellVirtualDirectory -Server SERV021-N1| Remove-PowerShellVirtualDirectory

Then tried to run this

      New-PowerShellVirtualDirectory -Server <Server> -Name Powershell -RequireSSL $false -BasicAuthentication $false -WindowsAuthentication $false -InternalUrl http://SERV021-N1/powershell 
      
      
So I ran this from this article

https://social.technet.microsoft.com/wiki/contents/articles/51374.exchange-2016-troubleshooting-event-id-4127.aspx

Followed the instructions but have this problem
[PS] C:\Windows\system32>Get-PowerShellVirtualDirectory -server SERV021-N1 | Remove-PowerShellVirtualDirectory
Confirm
Are you sure you want to perform this action?
Removing the Windows PowerShell virtual directory "PowerShell (Default Web Site)" on server "SERV021-N1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
[PS] C:\Windows\system32>New-PowerShellVirtualDirectory -Server TGCS021-N1 -Name Powershell -RequireSSL $false -Internal
Url http://SERV021-N1.mynet.com/powershell
Creating a new session for implicit remoting of "New-PowerShellVirtualDirectory" command...
New-PSSession : [SERV021-n1.mynet.com] Connecting to remote server serv021-n1.mynet.com
failed with the following error message : The WinRM client received an HTTP status code of 403 from the remote
WS-Management service. For more information, see the about_Remote_Troubleshooting Help topic.
At C:\Users\exchadmin\AppData\Roaming\Microsoft\Exchange\RemotePowerShell\serv021-n1.mynet.com\serv021-n1
.mynet.com.psm1:137 char:17
+                 & $script:NewPSSession `
+                 ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108273,PSSessionOpenFailed
Exception calling "GetSteppablePipeline" with "1" argument(s): "No session has been associated with this implicit
remoting module."
At C:\Users\exchadmin\AppData\Roaming\Microsoft\Exchange\RemotePowerShell\serv021-n1.mynet.com\serv021-n1
.mynet.com.psm1:46620 char:13
+             $steppablePipeline = $scriptCmd.GetSteppablePipeline($myI ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : RuntimeException

I went to the other DAG and they ran successfully

Any ideas need help on this one

Thank you
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Thomas GrassiSystems Administrator
Commented:
Sunil

Yes tried that no luck

after further checking I found that the powershell virtual directory was missing from the default web site in IIS.

With all the trouble of the process that was given by MS the first command removes that folder.
Get-PowerShellVirtualDirectory -Server <Server>| Remove-PowerShellVirtualDirectory

Then the second command is suppose to recreate it.
      New-PowerShellVirtualDirectory -Server <Server> -Name Powershell -RequireSSL $false -BasicAuthentication $false -WindowsAuthentication $false -InternalUrl http://<server.fqdn>/powershell


But EMS needs remote power shelling to work which I find very strange that they would require this especially since I am on the server with the problem

So the trick I found on my own was to go to the other DAG and run the second command. So I thought.
I did that a few days ago and I still had the problem
It want until I saw the article you posted early this week that lead me to the problem
I compared both DAG's IIS and I also went to Exchange Admin Center and found the virtual directory was missing.
So late last night after I posted this I said what the heck Let me try running the second command again on DAG 2
And to my surprise it worked the powershell virtual directory appeared and all starting working again.

Hope this helps someone else along the line
Thomas GrassiSystems Administrator

Author

Commented:
Thank for the help
Thomas GrassiSystems Administrator

Author

Commented:
Thank you

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial