Avatar of squidgy
squidgy
 asked on

Exchange Server 2007 - All messages from legacy Exchange 2003 moved to poison queue

Hi there.  I am in the process of performing some coexistence testing between Exchange 2003 SP2 and Exchange 2007 SP1.  I have an existing E2K3 org with all servers running SP2 Enterprise.  I have installed one each of E2K7 CAS, HT, MB (CCR) & PF servers into the existing Org and they are all running E2K7 SP1 plus SP1 rollup.  All servers are virtual.  OS being run is either W2K3 2003 SP2 Standard or Enterprise.  All E2K3 and E2K7 servers are in the same AD Site.  The site configuration is a little dodgy due to the lack of available subnets meaning we create a subnet for each server by using the IP address of the server with a subnet mask of 255.255.255.255.  This subnet is then associated with the required AD site.

Servernames, for the purpose of this question, can be:
CAS - E2K7CAS01
HT - E2K7HT01
MB - E2K7CCR01
PF - E2K7PF01
E2K3 bridgehead - E2K3BH01
E2K3 MB - E2K3MB01

Problem: All messages, originating in the E2K3 Org that are delivered to E2K7 recipients, seem to crash the transport service on E2K7HT01.  The message that caused the crash is then moved to the Poison queue on the HT.  If I resubmit or replay the message in the poison queue the transport service crashes again and the message is moved back to the poison queue.

The order of Error events that are logged on the HT at the time of the crash are 10003 then 4999.
Details of these events:

Source:MsExchangeTransport
Category:PoisonMessage
EventID:10003
Description:
The transport process failed during message processing with the following call stack: System.TypeInitializationException: The type initializer for 'Microsoft.Mapi.ExRpcPerf' threw an exception. ---> Microsoft.Mapi.MapiExceptionLowLevelInitializationFailure: MapiExceptionLowLevelInitializationFailure: ec=-2147467259 (0x80004005)
   at Microsoft.Mapi.ExRpcModule.Bind()
   at Microsoft.Mapi.ExRpcPerf..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.Mapi.ExRpcPerf.ConnectionCacheBirth(Int32 maxConnections)
   at Microsoft.Exchange.Data.Storage.ConnectionCachePool.GetConnectionCache(String server, ConnectFlag connectFlags)
   at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)
   at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsTransport(ExchangePrincipal mailboxOwner, OpenTransportSessionFlags flags)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.DeliveryItem.CreateItem(MailRecipient recipient, DeliveryItem previousItem, OpenTransportSessionFlags deliveryFlags)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.MailItemDeliver.<>c__DisplayClass1.<DeliverToRecipient>b__0()
   at Microsoft.Exchange.MailboxTransport.StoreDriver.StorageExceptionHandler.RunUnderExceptionHandler(IMessageConverter converter, StoreDriverDelegate workerFunction)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.MailItemDeliver.DeliverToRecipient(MailRecipient recipient)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.MailItemDeliver.DeliverToRecipients()
   at Microsoft.Exchange.MailboxTransport.StoreDriver.StoreDriver.DeliverMailItem(NextHopConnection connection, TransportMailItem mailItem)
   at Microsoft.Exchange.MailboxTransport.StoreDriver.StoreDriver.LocalDeliveryCallback(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)

Source:MSExchange Common
Category:General
EventID:4999
Description:
Watson report about to be sent to dw20.exe for process id: 1188, with parameters: E12, c-RTL-AMD64, 08.01.0263.000, edgetransport, M.E.StoreProvider, M.M.ExRpcModule.Bind, M.Mapi.MapiExceptionLowLevelInitializationFailure, 970b, 08.01.0263.000.  ErrorReportingEnabled: False


Things I have checked so far:
- Googled the interweb.
- Examined the poison messages.  They can be either messages from my test mailboxes and even PF replication related messages.
 - If I resubmit or replay the message in the poison queue the transport service crashes again and the message is moved back to the poison queue.
- I replayed the message on a standalone E2K7 HT (no coexistence) and the service did not crash.  This system is not running the post SP1 rollup so will test it again after upgrading.

Any help would be appreciated.  If worse comes to worse I'll submit a case to Microsoft PSS.
ExchangeEmail SoftwareEmail Servers

Avatar of undefined
Last Comment
GeminiISG

8/22/2022 - Mon
Arkware

You don't mention, so I thought I would ask...  Are you running Exchange 2007 x64 or 32-bit within the Virtual Environment?  I only ask because if you are running Exchange 2007 32-bit, this isn't supported (nor designed for production) by Microsoft, etc.   If you could confirm, that would be helpful.

If you are running x64, do you have an SMTP Connection setup within Exchange 2003's routing group for your Exchange 2003 server (setup via Exchange System Manager)?
joeljlarkin

I'm assuming your running the 64-bit code?  Can E2k7 users send to other e2k7 users?  I would also take a look at site design in the future, sounds a bit scary....
squidgy

ASKER
Arkware: Duh!  That information would have been handy.  Yes, all E2K7 servers are running x64 code.
The legacy RGC was created during the installation of the E2K7HT01.  Messages ARE flowing over this connector as messages originating in E2K3 are hitting the HT.

joeljlarkin: x64 As above.  Ahhhh, you've just reminded me of the other issue I've encountered.  No, E2K7 users cannot e-mail each other.  Whenever an E2K7 message is submitted for delivery the following event is logged on that users mailbox server.

Source:MSExchangeMailSubmission
Category:MSExchangeMailSubmission
Event ID:1009
Description:
The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.

I have confirmed that all the E2K7 servers are in the same AD site.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
squidgy

ASKER
And there are no no obvious back pressure events being logged on the HT (event id 15002).
joeljlarkin

Can you post details about your AD Site Configuration?  I believe that may be an issue as AD sites have relevance in E2k7 (they didn't have as much relevance in E2k3).
squidgy

ASKER
joeljlarkin:  Yep, mail routing is based on AD site topolgy.

Well, we have two AD sites which I'll refer to as Local and Remote.  These sites are dedicated to our Exchange testing.
Both AD sites have multiple subnets associated to it.  We'll say that all of our E2K7, E2K3 and Exchnage dedicated DC's are on the 192.168.1.0/24 network.
The network 192.168.1.0/24 is associated with a DIFFERENT AD site - one that was already existing in the Forest and hosts no Exchange servers.
Due to a lack of available networks we created a subnet for each indivudual E2K7 and E2K3 server and associated that to the Local and Remote AD sites as required,
Each of these "subnets" has the IP address of the server with a subnet mask of 255.255.255.255 which is supposed to take precedence over the 192.168.1.0/24 subnet associated with the non Exchange AD site.  Crazy but it APPEARS to work.
As far as DOmains go we have 2 domains hosting Exchange servers, domain controllers and mail-enabled users and a third domain hosting mail enabled users only.  We'll call them:
DomainA - Hosts users and Exchange servers - in the Local AD site
DomainB - Hosts users and Exchange servers - in the Remote AD site
DomainC - Hosts users only - in the Local AD site

Hope that makes sense.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
joeljlarkin

I think your problem may be related to the AD sites, is there a way to remove your explicit entries (255.255.255.255) and test?
squidgy

ASKER
joeljlarkin:

I removed the explicit entries for my CAS, HT and MB cluster boxes effectively changing the AD site they were  a member of.
I rebooted all servers concerned.
The problem still occurred.

I can, however, state that the previoously stated error regarding the MSExchangeMailSubmission is not relevant.  When I looked a little closer I discovered that that error was logged whenever the MSExchangeTransport service crashed :-)

So, I am now left with the following:

- I believe I have ruled out any dodgy AD site configurations.
- A MSExchangeTransportService that crashes whenever messages from the legacy messaging system are delivered to it.  These messages are then moved to the poison queue.
- A MSExchangeTransportService that crashes whenever a message is submitted by an E2K7 mailbox.  This message is NOT delivered to the poison queue and remains in the MB server's submission queue.

I thought I would try adding another HT into the site to see if there is something peculiar about my first HT installation.  Initially I will not movidy the legacy RGC connector configuration.  I will only test mail submission from the E2K7 MB server.
ASKER CERTIFIED SOLUTION
squidgy

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
GeminiISG

How did you fix the permissions?  (steps, propagation/inheritance method, utilities uses, etc.)
What are the default permissions supposed to be that fixed this for you?

I have set at:
  Administrators: Full Control
  Authenticated Users: Read and Execute, List Folder Contents, Read
  Creator Owner: None
  System: Full Control

Thanks!!!
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck