troubleshooting Question

Exchange Server 2007 - All messages from legacy Exchange 2003 moved to poison queue

Avatar of squidgy
squidgy asked on
Email SoftwareExchangeEmail Servers
10 Comments1 Solution1557 ViewsLast Modified:
Hi there.  I am in the process of performing some coexistence testing between Exchange 2003 SP2 and Exchange 2007 SP1.  I have an existing E2K3 org with all servers running SP2 Enterprise.  I have installed one each of E2K7 CAS, HT, MB (CCR) & PF servers into the existing Org and they are all running E2K7 SP1 plus SP1 rollup.  All servers are virtual.  OS being run is either W2K3 2003 SP2 Standard or Enterprise.  All E2K3 and E2K7 servers are in the same AD Site.  The site configuration is a little dodgy due to the lack of available subnets meaning we create a subnet for each server by using the IP address of the server with a subnet mask of 255.255.255.255.  This subnet is then associated with the required AD site.

Servernames, for the purpose of this question, can be:
CAS - E2K7CAS01
HT - E2K7HT01
MB - E2K7CCR01
PF - E2K7PF01
E2K3 bridgehead - E2K3BH01
E2K3 MB - E2K3MB01

Problem: All messages, originating in the E2K3 Org that are delivered to E2K7 recipients, seem to crash the transport service on E2K7HT01.  The message that caused the crash is then moved to the Poison queue on the HT.  If I resubmit or replay the message in the poison queue the transport service crashes again and the message is moved back to the poison queue.

The order of Error events that are logged on the HT at the time of the crash are 10003 then 4999.
Details of these events:

Source:MsExchangeTransport
Category:PoisonMessage
EventID:10003
Description:
The transport process failed during message processing with the following call stack: System.TypeInitializationException: The type initializer for 'Microsoft.Mapi.ExRpcPerf' threw an exception. ---> Microsoft.Mapi.MapiExceptionLowLevelInitializationFailure: MapiExceptionLowLevelInitializationFailure: ec=-2147467259 (0x80004005)
   at Microsoft.Mapi.ExRpcModule.Bind()
   at Microsoft.Mapi.ExRpcPerf..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.Mapi.ExRpcPerf.ConnectionCacheBirth(Int32 maxConnections)
   at Microsoft.Exchange.Data.Storage.ConnectionCachePool.GetConnectionCache(String server, ConnectFlag connectFlags)
   at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)
   at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsTransport(ExchangePrincipal mailboxOwner, OpenTransportSessionFlags flags)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.DeliveryItem.CreateItem(MailRecipient recipient, DeliveryItem previousItem, OpenTransportSessionFlags deliveryFlags)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.MailItemDeliver.<>c__DisplayClass1.<DeliverToRecipient>b__0()
   at Microsoft.Exchange.MailboxTransport.StoreDriver.StorageExceptionHandler.RunUnderExceptionHandler(IMessageConverter converter, StoreDriverDelegate workerFunction)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.MailItemDeliver.DeliverToRecipient(MailRecipient recipient)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.MailItemDeliver.DeliverToRecipients()
   at Microsoft.Exchange.MailboxTransport.StoreDriver.StoreDriver.DeliverMailItem(NextHopConnection connection, TransportMailItem mailItem)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.StoreDriver.LocalDeliveryCallback(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)

Source:MSExchange Common
Category:General
EventID:4999
Description:
Watson report about to be sent to dw20.exe for process id: 1188, with parameters: E12, c-RTL-AMD64, 08.01.0263.000, edgetransport, M.E.StoreProvider, M.M.ExRpcModule.Bind, M.Mapi.MapiExceptionLowLevelInitializationFailure, 970b, 08.01.0263.000.  ErrorReportingEnabled: False


Things I have checked so far:
- Googled the interweb.
- Examined the poison messages.  They can be either messages from my test mailboxes and even PF replication related messages.
 - If I resubmit or replay the message in the poison queue the transport service crashes again and the message is moved back to the poison queue.
- I replayed the message on a standalone E2K7 HT (no coexistence) and the service did not crash.  This system is not running the post SP1 rollup so will test it again after upgrading.

Any help would be appreciated.  If worse comes to worse I'll submit a case to Microsoft PSS.
ASKER CERTIFIED SOLUTION
squidgy

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 10 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 10 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros