Exchange Server 2010 Service Buttons Greyed Out - Exchange Down

Exchange ActiveSync has encountered repeated failures when it tries to access data on Mailbox server [ODC-MAIL1.odc.local]. It will temporarily stop making requests to the Mailbox server for [60] seconds to reduce load on that server. This delay may occur if the Mailbox server is overloaded. If this event is logged frequently, review the Application log on this server and the Mailbox server noted above for other events that could indicate the root cause of performance problems.
Additional information:
"serverFQDN=ODC-MAIL1.odc.local
Error 0:

ErrorTimeStamp:
3/11/2012 7:16:39 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Thomas J. Holdren.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.<CreateMailboxSession>b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: ODC-MAIL1.odc.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273  
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=156]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081  
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065  
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805  
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583  
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778  
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730  
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, String mailboxDn, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 1:

ErrorTimeStamp:
3/11/2012 7:16:37 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Brad Parks.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.<CreateMailboxSession>b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: ODC-MAIL1.odc.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273  
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=149]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081  
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065  
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805  
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583  
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778  
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730  
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, String mailboxDn, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 2:

ErrorTimeStamp:
3/11/2012 7:16:37 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Brad Parks.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.<CreateMailboxSession>b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: ODC-MAIL1.odc.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273  
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=149]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081  
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065  
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805  
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583  
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778  
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730  
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, String mailboxDn, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
errorCount=3, backingOff=True".

Active Manager failed to mount database Public Folder Database 1302213288 on server odc-mail1.odc.local. Error: An Active Manager operation failed with a transient error. Please retry the operation. Error: Database action failed with transient error. Error: A transient error occurred during a database operation. Error: MapiExceptionNetworkError: Unable to make admin interface connection to server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 12696   dwParam: 0x6D9      Msg: EEInfo: Generation Time: 2012-03-12 00:16:54:57
    Lid: 10648   dwParam: 0x6D9      Msg: EEInfo: Generating component: 2
    Lid: 14744   dwParam: 0x6D9      Msg: EEInfo: Status: 1753
    Lid: 9624    dwParam: 0x6D9      Msg: EEInfo: Detection location: 501
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncalrpc
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string:
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -1988875570
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 24060   StoreEc: 0x80040115
    Lid: 23746  
    Lid: 31938   StoreEc: 0x80040115
    Lid: 19650  
    Lid: 27842   StoreEc: 0x80040115
    Lid: 20866  
    Lid: 29058   StoreEc: 0x80040115

Process w3wp.exe () (PID=3812). An remote procedure call (RPC) request to the Microsoft Exchange Active Directory Topology service failed with error 1753 (Error 0x6d9 (There are no more endpoints available from the endpoint mapper) from HrGetTopologyVersion). Make sure that the Remote Procedure Call (RPC) service is running. In addition, make sure that the network ports that are used by RPC are not blocked by a firewall.

Could you please provide some additional information on your Exchange environemnt and AD environment

Product: Microsoft Exchange Server -- Error 1719. The Windows Installer Service could not be accessed. This can occur if you are running Windows in safe mode, or if the Windows Installer is not correctly installed. Contact your support personnel for assistance.

my last two comments are the most recent errors since the server actually stopped sending/receiving mail

here is the latest:

Process powershell.exe () (PID=1040). An remote procedure call (RPC) request to the Microsoft Exchange Active Directory Topology service failed with error 1753 (Error 0x6d9 (There are no more endpoints available from the endpoint mapper) from HrGetServersForRole). Make sure that the Remote Procedure Call (RPC) service is running. In addition, make sure that the network ports that are used by RPC are not blocked by a firewall.

Ok, step by step. first windows installer:
http://www.out-of-warranty.com/windows-installer-error-1720-1722/

And:
Could you please provide some additional information on your Exchange environemnt and AD environment

I tried to run the Windows Installer and here is the bizarre error I get:

however, I have a ton of free space

you read my posts bevore ?!

we have exchange 2010 running on server 2008.  I have two domain controllers on 2008 as well.  One of them is my AD.  All of these servers are on VMware.

All I did this morning was reboot after installing all of those updates.  I also changed the OUTSIDE DNS on my Domain controllers to a different numbers since our ISP changed.

ok, in exchange server you have the internal dns configured too, not only the external dns?

my mail servers DNS is pointing to my two Domain controllers

can you verify on exchange server there are no quotas aktive in file system?

try this and install actual windows installer...
http://winhlp.com/node/40

Windows Server 2008 does not have a registry key called "IRPStackSize" so that last link doesn't work for me.

Ok ignore my last few posts.  I was trying to installed the 2003 WIndows Installer.  I have 2008.  my bad.  

I tried installing the correct one and it said it's not applicable to me.  I was able to start the windows installer server now.  For some reason it wasnt running and it was marked as "manual"

My Exchange server is now installing the updates.  I have a feeling that it tried installing the updates and they failed which is why I couldnt uninstall them and my Exchange stopped working.  We'll see if this works after they install and I restart.

why would my exchange services all be disabled and greyed out?
disabled.jpg

I did a few google searches on the latest updates that Server 2008 was trying to install and figured out that a problem with one of the updates caused all of my Exchange services to become disabled.