Exchange 2016 Virtual Directories and PowerShell Inaccessible

We have an Exchange 2016 (CU5) DAG environment with 4 nodes available on Windows Server 2016. Each node has sufficient storage of its own. They are Dell Rx730 PowerEdge servers with a system-storage drive (for OS) of 1TB and multiple data storage drives (for DBs) with 8TB size per drive.

Due to some specific reasons, we had to reboot one of the nodes. However, when the server came back, it was not working fine. We noticed some of the following issues:

1. Exchange PowerShell is not connecting on the concerned node.
2. Exchange ECP and OWA are no more accessible on this server.
3. The MSExchange Transport and Replication services are stuck on starting/stopping states. Moreover, the MSExchange FrontEnd Transport and MSExchange Information Store services are also not starting.

When we check the event-viewer, the following logs can be seen:

For OWA:

Log Name:      Application
Source:        MSExchange OWA
Date:          12/18/2017 8:57:52 AM
Event ID:      157
Task Category: Diagnostics
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MAIN-EX02.domain.local
Description:
Failed to register OWA online diagnostic RPC server, error code: 1721, exception: Microsoft.Exchange.Rpc.RpcException: RpcServerUseProtseq TCP/IP
   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)
   at Microsoft.Exchange.Clients.Owa2.Server.Core.OwaApplication.ExecuteApplicationSpecificStart()
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange OWA" />
    <EventID Qualifiers="49152">157</EventID>
    <Level>2</Level>
    <Task>12</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-12-18T05:57:52.020511000Z" />
    <EventRecordID>826641</EventRecordID>
    <Channel>Application</Channel>
    <Computer>MAIN-EX02.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>1721</Data>
    <Data>Microsoft.Exchange.Rpc.RpcException: RpcServerUseProtseq TCP/IP
   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)
   at Microsoft.Exchange.Clients.Owa2.Server.Core.OwaApplication.ExecuteApplicationSpecificStart()</Data>
  </EventData>
</Event>

For MSExchange FrontEnd Transport:

Log Name:      Application
Source:        MSExchange Front End HTTP Proxy
Date:          12/18/2017 8:59:02 AM
Event ID:      1003
Task Category: Core
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MAIN-EX02.domain.local
Description:
[Ews] An internal server error occurred. The unhandled exception was: Microsoft.Exchange.Rpc.RpcException: RpcServerUseProtseq TCP/IP
   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)
   at Microsoft.Exchange.HttpProxy.ProxyApplication.<Application_Start>b__0()
   at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange Front End HTTP Proxy" />
    <EventID Qualifiers="49152">1003</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-12-18T05:59:02.308339600Z" />
    <EventRecordID>826896</EventRecordID>
    <Channel>Application</Channel>
    <Computer>MAIN-EX02.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Ews</Data>
    <Data>Microsoft.Exchange.Rpc.RpcException: RpcServerUseProtseq TCP/IP
   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)
   at Microsoft.Exchange.HttpProxy.ProxyApplication.&lt;Application_Start&gt;b__0()
   at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)</Data>
  </EventData>
</Event>

For MSExchange Transport:

Log Name:      Application
Source:        MSExchange TransportService
Date:          12/18/2017 8:54:38 AM
Event ID:      1016
Task Category: ProcessManager
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MAIN-EX02.domain.local
Description:
The worker process crashes continuously on startup: C:\Program Files\Microsoft\Exchange Server\V15\Bin\edgetransport.exe. The service will be stopped.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange TransportService" />
    <EventID Qualifiers="49156">1016</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2017-12-18T05:54:38.830682700Z" />
    <EventRecordID>825911</EventRecordID>
    <Channel>Application</Channel>
    <Computer>MAIN-EX02.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C:\Program Files\Microsoft\Exchange Server\V15\Bin\edgetransport.exe</Data>
  </EventData>
</Event>

IIS Virtual Directories:

When we try to check the IIS virtual-directories in ECP against the concerned server, it shows the below error message:

An IIS directory entry couldn't be created. The error message is Not enough resources are available to complete this operation. . HResult = -2147023175


Can someone please let me know what is going wrong here?


Regards,

Waqas Ahmad
Waqas AhmadSenior Cloud EngineerAsked:
Who is Participating?
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Jose [MCSE] Ortega C and Amit Kumar

Thanks for all the support during this problem. We have now managed to resolve it.

Reference Article:

During the search for a resolution, we stumbled upon this Technet post: LINK

Resolution Steps:

There was an invalid registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Rpc\Internet which shouldn't be there at all. This key was also not available on the rest of our DAG nodes. I don't know how it got there. In any case, we removed it, rebooted the server and bang!!! Everything seems to be working fine now.
2
 
Jose Gabriel Ortega CEE Solution Guide - CEO Faru Bonon ITCommented:
Lack of resources.
Make sure that you have enough drive space in that concerned node, 1st of all, especially C: drive.
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Jose [MCSE] Ortega C

Thanks for the reply. As mentioned above, there are sufficient resources available. The C: drive still have 0.99TB available.
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

 
Jose Gabriel Ortega CEE Solution Guide - CEO Faru Bonon ITCommented:
Ok. Said that.
Let's go to the next step.

If the service like  "information store" is stuck, it means that the databases in those nodes in that server are  corrupted or not in a "clean shutdown"

Here's a procedure: (you need to use eseutil tool)
https://technet.microsoft.com/en-us/library/bb123919(v=exchg.80).aspx
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Jose [MCSE] Ortega C

Thanks, I will be following the steps mentioned by you in a short while using the eseutil tool.

Just to confirm, as the Exchange Management Shell is not connecting on this server, can this also be due to the stuck services like Information Store? Is there a possibility all these issues are linked to one reason? Or do we need to troubleshoot them separately?
0
 
Jose Gabriel Ortega CEE Solution Guide - CEO Faru Bonon ITCommented:
All the services are linked to Exchange Server (Mail system).
But they can be troubleshoot them separately.

MSExchange Transport     (old hub transport 2010, manage  internal emails and route external emails)
MSExchange Replication services   (replication of mailboxes between databases and outlook client services).
MSExchange FrontEnd Transport     (receive email)
MSExchange Information Store service  (Databases)

Each service has a very specific functionality and can be working on the troubleshooting separately.
0
 
Amit KumarCommented:
Can you check below two things:

1. Match affected Server time with running server and try to login with a new account as your existing account may have cache and getting logon on server. It is possible your server is out of domain.
2. Check certificates in IIS if they are assigned or unassigned.
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Jose [MCSE] Ortega C

Thanks for providing the details, I really appreciate it.

@Amit Kumar

Thanks for the answer. The timing on all the servers are correct. We have synced all the additional DCs with the PDC and same is the case with the Exchange nodes. We also resynced them to ensure that there isn't any time sync issues causing this problem.

In regards to the suggestion on accessing the server using a newly created AD account, I just did it and was able to access the affected Exchange node.

For the certificate part, we have the certificate installed on the Exchange. The "Default Web Site" in IIS has the certificate which we acquired from the CA, while the "Exchange Back End" site has the default self-signed certificate assigned on it.
0
 
Amit KumarCommented:
Any recent changes done or any patch installed on this server? Can you check actual memory in use as of now and see how much is allocated using My computer properties.

Seems very strange issue as not much KBs are available for Exchange 2016 yet.
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Amit Kumar

Yes, there were some modifications done on this server. We had executed IISCrypto tool (https://www.nartac.com/Products/IISCrypto) with best practices on this server to correct the weak cipher issues. This tool requires a reboot after its execution, and thus we rebooted the server. However, since then the services are not working fine. I have applied IISCrypto on several other servers in the past but never got such an issue.

In regards to the memory allocation, the server has 256GB RAM, and only 7-9% of it is in use, whereas the available storage on system drive is 0.99TB (approximately 1TB) and on data drive we have about 7.99TB available.
0
 
Jose Gabriel Ortega CEE Solution Guide - CEO Faru Bonon ITCommented:
In that case, just rerun the IIS Crypto tool. or just remove the registry things it creates
Details in: https://www.nartac.com/Blog/post/2013/04/19/IIS-Crypto-Explained.aspx

(reboot)  
And use my script will create just the required ones:  https://gallery.technet.microsoft.com/scriptcenter/Solve-SWEET32-Birthday-d2df9cf1

And reboot again.
1
 
Amit KumarCommented:
Agree with Jose, it seems it messed with registries and disabled required ciphers which is creating problem.
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Jose [MCSE] Ortega C

If I execute your script, would it set everything up for me? Or would I still need to remove the IIS Crypto tool's registry entries first and then execute your script?
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
@Jose [MCSE] Ortega C and Amit Kumar

I have reverted back all the changes IIS Crypto made in the registry, but still the services are stuck. I have again rebooted the server but got the same problem.
0
 
Amit KumarCommented:
I would suggest instead messing more with this server contact Microsoft, perhaps it needs more work.
0
 
Jose Gabriel Ortega CEE Solution Guide - CEO Faru Bonon ITCommented:
Hi waqas my script will secure the server removing TLS1.0 and the SWEET32 vulnerabilities in the server, that's all it will do.
The script is documented in the link I provided. Please check it out.
0
 
Jose Gabriel Ortega CEE Solution Guide - CEO Faru Bonon ITCommented:
Great news! glad to help
0
 
Waqas AhmadSenior Cloud EngineerAuthor Commented:
We managed to resolve it ourselves.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.