Solved

Messages to remote AD site getting stuck in Exchange 2007 queue

Posted on 2009-04-06
39
7,645 Views
Last Modified: 2014-05-13
I have recently installed a second Exchange 2007 SP1 CAS, HUB and MB server (Server B) in a remote AD site from the first Exchange 2007 SP1 CAS, HUB and MB server (Server A). I have configured a second Internet Send Connector for Server B. Receive Connectors on both servers have the same settings.

Sending mail to and from the Internet from either server works as well as Internally appears to work although messages from mailboxes on Server B to mailboxes on Server A end up in Server B's queue for Next Hop Site A, Delivery Type SMTP Relay to Remote Active Directory Site. Last Error is 451 4.4.0 Primary Target IP address responded with: "421 4.4.2 Connection dropped." Attempted to failover to alternate host, but that did not succeed. Either there are no alternate hosts or delivery failed to all alternate hosts."  

The reason I say appears to work is because my test messages from a mailbox on Server B to a mailbox on Server A show up in mailbox A's inbox but also ends up in Server B's queue as mentioned above. The messages in the queue stay there, giving Delivery Delayed messages to the Sender and then eventually fail and an NDR message with the following get send to Sender on Server B #550 4.4.7 QUEUE.Expired; message expired ##.

No errors are showing the App log on either server.

Mail Flow Troubleshooter run on Server B gives the error that SMTP ports 25 and 587 are not responding and gives the message "It appears that the SMTP service and SMTP instance(s) on server FRG-RNO-EX01 are started but the port did not respond. Check if there are any network errors or hung services." Also gives "Error submitting mail." A restart of Server B makes no difference.

I tested via telnet from Server B to Server B and was able to connect and send a test message successfully. Did same test from Server A to Server B which was also successful so it appears that SMTP is working fine.

Confused....
0
Comment
Question by:MultiTrends
  • 23
  • 11
  • +4
39 Comments
 
LVL 10

Expert Comment

by:mcrossland
Comment Utility
Are you by chance using a smart host?
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
nope.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Can you telnet both ways, using all three combinations of IP address, netbios and FQDN? Once you have made the connection, can you then send a message to a user on that remote server?

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
I have used telnet to send test messages that were successful both ways although I will confirm with all three combos and get back to you.

Thanks,
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Interestingly enough, I can telnet and send mail using IP and FQDN from Server B to Server A but not using Netbios. Using Netbios I get 550 5.7.1 Unable to Relay. I can telnet and send mail using telnet from Server A to B using all three.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Also messages sent using telnet do not get stuck in the queue.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
So now I can telnet and send messages using all three from both sides. The only thing I changed was I noticed the Default Receive Connector on Server A was set to its Netbios name instead of FQDN.

Messages sent using OWA are still being recieved by the recipient but the messages continue to get 'stuck' in the SMTP Relay to Remote AD Site queue same as before.

On another note, since I've read a couple of posts that had similar issues and their resolution was related to certificates. Server A has one third party UCC SSL Cert that is assigned to all services except UM (which isn't installed). It also has a self signed cert that is valid but not assigned to any services. I swear I removed it a couple of months ago but it remains. I will be removing that cert shortly. Server B has only one SSL cert which is the default self signed and is valid.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
The self signed certificates will come back if you do not have a certificate with the server's real name in its additional name list. Exchange checks that on its reboot.
Certificates are a pain point for most deployments, but the use of self signed certificates shouldn't be a problem internally because Exchange should trust its own certificates.

I would have another look at the sites and services configuration - particularly subnet configuration. It may not be an Exchange issue at all - but Exchange uses the site and service configuration for routing.

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
I've reviewed the Sites and Services setup and I don't see any issues. There is only one IP Site Link between Site A and Site B. The other IP Site Link is between Site A and Site C. There are no SMTP Site Links as I think that is only required if you want to use SMTP as the Inter-Site Transport Protocol. (Please correct me if I am wrong). There are valid subnets configured for both Sites A and B where the Exchange servers are located and they are assigned to the correct Sites. Is there something else I should be looking for specific for Exchange?

Appreciate your help.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
That sounds fine. As you said, you don't need the SMTP bit (I don't think I have ever used SMTP for the inter-site protocol).

Have you run the Best Practises tool from the toolbox to ensure that it doesn't flag anything?
Any changes made to the connectors on either server away from the defaults?

Is there AV on the machines that is scanning SMTP traffic?

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
I am running an ExBPA - Connectivity test now...will post results when it completes. There are no AV programs on either machine, they are scanned by an email gateway on a separate machine. For connectors, I have an Internet Send Connector for each server, both set for all addresses and using DNS for routing. Receive Connectors, both Server A and B have the Client Connector unchanged, the Default Connector is the same settings on both - all are checked except for Mutual Auth TLS and Externally Secured. Permission Groups on the Default Receive Connectors have everything checked except Partners. Server A also has a LocalServerApp Connector that allows specific IP the ability to relay.

The Connectvitiy test was run from Server B and came back with the Performance Data on Server A cannot be accessed. I will go through the steps on why that is. What there a particular test you think is most useful in this case?

Thanks.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Lets wait to see what the tools flag, if anything.
Do you have protocol logging enabled? If the connection is being dropped, then the connection attempt should be seen in the protocol logs.

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
I do have protocol logging enabled on the Receive Connector of Server A....reviewing it now.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
It seems to be dropping the connection right after Server A gives its SSL Cert info.

SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders
220 Server A Microsoft ESMTP MAIL Service ready at Mon, 6 Apr 2009 19:22:49 -0700
EHLO Server B
250-Server A Hello [10.1.0.9]
250-SIZE
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-X-ANONYMOUSTLS
250-AUTH NTLM
250-X-EXPS GSSAPI NTLM
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250-XEXCH50
250 XRDST
X-ANONYMOUSTLS
220 2.0.0 SMTP server ready

OU=Domain Control Validated, CN=mail.domain.com, O=Org
SERIALNUMBER=10688435, CN=Starfield Secure Certification Authority, OU=http://certificates.starfieldtech.com/repository, O="Starfield Technologies, Inc.", L=Scottsdale, S=Arizona, C=US
00E26364
84A2A58614F80A5DFC231DEB8E6B3AD066C8AC9D
mail.domain.com;autodiscover.domain.com;Server A;Server A
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
That could be certificate acceptance.
Do you have the intermediate certificates installed correctly on both servers?

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
As far as I know the intermediate certificate install is only done when installing a third party cert which I did do on Server A when I installed its Cert but haven't on Server B since it's only using the default self signed cert. Was going to purchase a cert for Server B but since I'm using Server A to proxy OWA and Outlook Anywhere requests, I am thinking a commercial cert isn't needed.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
so the ExBPA Health Check test came back with a certificate principal mismatch for the default domain site https://domain.com which isn't Exchange related as that points to the company website which is hosted externally. Not sure why the ExBPA checked that? Outside of that, the only issues reported were to do with older than 2 yr NIC driver, TCPIP.SYS driver issue, MS Filter Pack not installed for both Servers and the self signed certificate error for Server B.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
possibly of note when running the Exch Troubleshooter from Server B for messages backed up in a queue, it reports that packets need to be fragmented on Server A.

> Ping (Don't fragment = 'True' and buffer size = 4096) from server Server B to remote server Server A was not successful as the packet needs to be fragmented.

I checked into this yesterday and was able to ping consistently with 1472 bytes between servers with no issues so not sure why it's reporting this error.

> It also reports the following Mail Acceptance Failure:
Remote server Server A failed the mail acceptance test. BDAT command: Respond = Remote socket is not available. Check for firewalls and applications that can possibly block the BDAT command.

0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
EXBPA has been doing that certificate check since release, no idea why it does it and I tend to ignore it.
Out of date NIC driver - that should be fixed. If the drive isn't dated 2008 then it needs an update.

With the Starfield certificates there is a known bug that can sometimes occur which can then cause a problem with certificate acceptance.
Look in the Certificates MMC tool, at the ROOT certificates. See if anything is listed Starfield. If it is, then open the properties and change the settings to Disable for Instances. Do not remove it though.

You will probably need to restart the MS Exchange Transport Service afterwards for the change to be seen.

Simon.
0
Do email signature updates give you a headache?

Do you feel like you are constantly making changes to email signatures? Are the images not formatting how you want them to? Want high-quality HTML signatures on all devices, including on mobiles and Macs? Then, let Exclaimer solve all your email signature problems today.

 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
There is one Startfield cert listed in Computer Account on Server A | Trusted Root Certificates but it is already currently disabled for all purposes.

Checked Server B for a Starfield Cert and it has one too which is enabled for Server Authentication, Client Authentication, Code Signing and Secure Email.

I also see that self signed cert that isn't assigned to any Exchange services from Server A listed in the root certificates and it is enabled for All Purposes on Server B.

Since you mentioned that Exchange will recreate itself a self signed cert on reboot if the existing cert doesn't have the server's real name listed, I double checked it and the commercial cert that is installed and enabled for all services does have the Netbios and FQDN (both internal and external) names for the server (A). The one that is self signed is valid from Feb, 2008 and may have not been removed as I thought. It currently just has the Netbios name of Server A listed.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Does the date match the time the server was installed?
I don't think it is the problem though because your log above shows the commercial certificate has being used.

Although get-exchangecertificate will show you what certificates are valid.

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
it's not dated the same as the server install as Server A has been around for close to 2 yrs. Get-ExchangeCertificates shows only the Starfield active.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
That rules that out then.
I presume it is still saying connection dropped?

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Yes, connection is still showing as dropped. I have found some nodes are unable to communicate to each other between these two sites due to some sort of network issue which may be part of the problem here although the two Exchange servers can communicate without issue.

I have only two mailboxes on Server B right now. One is a 'live' (used by a client) and the other is a test. My test mailbox messages do get stuck in the queue same as the live mailbox users' messages to Site A mailboxes but my test messages also get received in the inbox of the intended Site A recipient which makes it even more confusing as to why those messages are showing up in the SMTP Relay to Remote AD Site queue when they appear to go through ok. The live user is reporting that his intended recipients are not getting his messages. Just to add to the confusion.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
It shouldn't be this difficult - so I have to think the problem is outside of Exchange.

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
I agree, it shouldn't. I'm going to set up a new route between these two sites to see if I can remove any network issues....
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Ok just to give an update....I have created new network routes between Sites A and B which has resolved all existing network issues that I am aware of. The EXBPA | Connectivity test run from Server B is now able to access the performance data on Server A which it wasn't using the previous routes so that is an improvement.

Messages are still stuck in the queue and I have also confirmed that my previous thought that my test messages were going through ok to the Site A mailboxes is incorrect. All internal messages from Site B to Site A are getting stuck in the SMTP Relay to Remote AD Site queue and are NOT being recieved by any local mailboxes on Server A. The test mailboxes I was using in Site A are forwarded externally which does work and is why I thought they were being received by the test mailbox on Server A. The external forwarding works but the local mailbox on Server A does not receive messages from Server B.

I have restarted the transport service on Server B after the network changes as well as force AD replication between all DCs at both sites but they remain stuck in the queue including new messages sent.

I am currently running through the Exchange troubleshooting tools again with the 'new' network configuration to see if I can find anything new.....
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
nothing new. same issues reported from last time about packet needs to be fragmented, BDAT command and Mail Acceptance failures. I have noticed that with the new route setup which uses IPSEC VPNs between sites will only accept pings up to 1398 bytes. I'm wondering if I need to reduce the MaxMTU setting for the Exchange NIC?

0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
A couple of observations - the MTU might need to be changed.
I have also seen Exchange routing take a while to update itself, even after restarting the various services. I did a site a few weeks ago where things were not working correctly for 12 hours, despite me kicking it frequently to try and get it to work correctly.

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Ok. Thanks for the input. Since, I believe, changing the MTU requires a reboot anyway, I'll have to wait for it til end of day. So will wait and give my brain a rest from this topic.

I REALLY appreciate your assistance.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Just to update the post....I tried changing the MTU to 1398 which was the highest that I could ping with between Server B to A and afterwards, routing still didn't occur from Mailboxes on Server B to Server A. Plus Exchange mail flow troubleshooter then reported the MTU was too small and still gave the Mail Submission Error. The MTU is still set at 1398 but I will be changing it back later today.

I have also just tried sending another message now and it still gets stuck so the Exchange routing table must be updated by now. I'm out of ideas other than reinstalling or repairing the install on Server B.

Any suggestions?
0
 
LVL 65

Assisted Solution

by:Mestha
Mestha earned 500 total points
Comment Utility
If you have exhausted everything then it has to be a bad build of one of the servers, I don't have anything else to suggest other than calling Microsoft. I have done this lots of times and it worked fine. Where I have had problems it has been something in between the servers, firewall, routers etc.

Simon.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
Ok, thanks for the input. I'm going to try reinstalling the Hub role on Server B and go from there. I'll post my results for the benefit of anyone else that has the same issues in the future.
0
 
LVL 2

Author Comment

by:MultiTrends
Comment Utility
just to give an update and to let the board know I haven't abandoned the question. I did reinstall the hub role and tried a few other things to no avail. I am going to call Microsoft to get their assistance either tomorrow or the next day and will update again as soon as I get a solution to post.
0
 

Expert Comment

by:kedarroy
Comment Utility
what did Microsoft say? I am having somewhat the same issue
0
 
LVL 2

Accepted Solution

by:
MultiTrends earned 0 total points
Comment Utility
I actually just got to calling them today and just finished the call about 10 mins ago. The fix was creating a new, self signed, SSL cert for SMTP services. Restart the transport service on both servers and messages are flowing.
0
 

Expert Comment

by:o_b_c
Comment Utility
This post would have been more helpful to me if it outline exactly what steps were done: "creating a new, self signed, SSL cert for SMTP services"

I wasn't sure so I contacted Microsoft and after 2 hours of them screwing around they ran the following script and it fixed my problem: My outbound email was queueing up on all queues after I installed a new self signed SSL cert. I already had a slef signed SSL cert that I created using SELFSSL (from the Exchange 07 CD).

From the Exchange management shell on your hub transport, type:

New-ExchangeCertificate -DomainName YourDomain'sName -Services SMTP

Once you issue this command, reboot all of your exchange servers (If you have many, maybe just do the ones that are affected - I only have 2 of them).

Hope this helps someone...
0
 

Expert Comment

by:thsulliv
Comment Utility
This thread was helpful while troubleshooting a stuck Exchange 2010 queue. In my case, it was a miss-assigned self signed cert for the SMTP service, which I was using for testing. When I removed the cert, the queue emptied immediately.
0
 

Expert Comment

by:maxtexgr
Comment Utility
Also can be caused by an issue with time synchronization between the hub transport servers.  Once resolved restarting the hub transport service on the side that was invalid and retrying the queues on both sides got the mail flowing again.  Assuming the underlying issue was the difference in time caused the SSL certificates to be unable to be verified for SMTP.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Resolve Outlook connectivity issues after moving mailbox to new Exchange 2016 server
Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
In this video we show how to create a User Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Mailb…
In this video we show how to create an Accepted Domain in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Ac…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now