i would first look in the mail queue on the 2003 server and see what it says

I found the following error message on the Exchange 2003 server.  

Event Type:      Error
Event Source:      MSExchangeTransport
Event Category:      NDR
Event ID:      3017
Date:            3/21/2014
Time:            11:58:43 AM
User:            N/A
Computer:      APOLLO
Description:
A non-delivery report with a status code of 5.3.5 was generated for recipient rfc822;exch2010@fserver4.com (Message-ID <1671C220D949E24AAE10266CA8C5B70C645D2A@apollo.ch13.local>).  
Causes: A looping condition was detected. (The server is configured to route mail back to itself). If you have multiple SMTP Virtual Servers configured on your Exchange server, make sure they are defined by a unique incoming port and that the outgoing SMTP port configuration is valid to avoid looping between local virtual servers.    
Solution: Check the configuration of the virtual serverÆs connectors for loops and ensure each virtual server is defined by a unique incoming port.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: cf 02 04 c0               Ï..À

I also have GFI MailEssentials installed on the Exchange 2003 Server.  Wondering if that could be causing problems.

Primary reason for this is a problem with the SMTP virtual server. Ensure that you do NOT have a smart host on the SMTP virtual server. If you do, remove it. Also check the FQDN value is set to the server's real name, not any kind of internet name.
Did you have an SMTP connector on Exchange 2003? If not, create one, otherwise Exchange will attempt to route email through the new server. GFI ME is intelligent enough to know about internal traffic, so I don't think the problem would be there.

Simon.

why not routing all emails through exchange 2010?  you send an email to a recipient inside the organization it resolves to your exchange 2003 server and loops to nowhere.
make sure that you do not have a smarthost configured in the Virtual Default Server for SMTP (Delivery Tab -> Advanced button).  If you do, it will break mail flow from 2003 -> 2010.  

Hope this helps,

Gilad

On the Exchange 2003 server I have an SMTP connector defined with no smarthost defined.    The fully qualified domain name is mail.????.com and not the server name apollo.ch.local.   We have always used mail.????.com and mail has worked fine for years.  I really do not want to tamper with this setting since it could affecting mail flow.

I made a mistake with my previous reply, in that the fully qualified domain name is using the correct name mail.????.com, and not the server's internal name.  We've have this SMTP connector setup for years without issue.  Never need to use a smarthost, so one was never defined.

The fact that it has been working for years is completely immaterial.
You have now introduced a second server. For email to flow over the RGC between the two servers the FQDN on the SMTP virtual server must be the server's real name. It didn't matter when you had a single server, but now it does.

You need to change it, then restart the SMTP Server service.

Simon.

Sending a test message from my Exch2003 mailbox to a test user on Exch2010, I receive the following bounce message:

The following recipient(s) cannot be reached:

      exch2010 on 3/21/2014 2:35 PM
            A configuration error in the e-mail system caused the message to bounce between two servers or to be forwarded between two recipients.  Contact your administrator.
            <mail.????.com #5.3.5>

ok, let's do the other way,on exchange 2003  first Check the configuration of the server’s connectors for loops, and ensure that each connector is defined by a unique incoming port. If there are multiple virtual servers, ensure that none are set to “All Unassigned.”
exchange may have added another connector that collides with the old connectors settings so delete/disable to old connector in that case..

if that did not  help -
please use the following troubleshooting article step by step:
http://msexchangeguru.com/2013/07/29/troubleshooting-mail-flow-issues/

go step by step and post back for any questions..


p.s
totally forgot, what happens when you telnet??
Access the console of the 2003 server, and start a Telnet session via port 25 to the 2007 server. What happens when you attempt to create/send a message to a valid address on the 2007 server?
see here:
http://exchange.mvps.org/smtp_frames.htm


Gilad

I have a default SMTP Virtual Connector that is disabled.  When I setup the mail server many years ago, this connector would not work in that mail was not flowing.  So I disabled it, and created a new connector by the name of "SMTP VS".

SMTP VS Config:  

IP Address:  All Unassigned
Delivery/Advanced:  Hop Count: 30, Masquerade Domain:  blank, Fully-QualifiedDomainName:  mail.???.com, SmartHost: blank, Perform DNS lookup: unchecked.

telnet from the exchange 2003 server (telnet ajax 25) returns the mail server banner

220 ajax.ch.local microsoft esmtp mail service ready...

When in telnet, and I typed helo

the server responded:

250 ajax.ch.local Hello [172.16.2.6]

The ip address of this server (exch 2010) is 172.16.2.7 (ajax), and 172.16.2.6 is the ip address of the exchange 2003 server (apollo).

I changed the fully qualified name to the name of the server apollo.ch.local, and restarted the SMTP service.  Able to send/receive mail to the Internet.  I tried sending another test email to the exchange 2013 server, and its stuck in the queue.

go back and do a full telnet test
http://exchange.mvps.org/smtp_frames.htm

do all steps that are marked in red step by step

Gilad

This is what I discovered on the Exchange 2003 server.  

I have two smtp connectors.

The "Default SMTP Virtual Server" has been disabled for many years, and I had problems with it then so I disabled it and created "SMTP VS" connector.  I cannot delete it.  When I right click on it there is not Delete option.

With the SMTP VS connector enabled.  I can send and receive mail from the internet and internal Exchange 2003 users.  I cannot send mail to the sole Exchange 2010 user.

I tried disabling the "SMTP VS" Connector, and then enabled the "Default SMTP Virtual Connector".  Also, under routing connectors, I changed the General/Local Bridgehead servers from the SMTP VS to the Default SMTP Virtual Connector.

Now I can send mail to the sole test user on the Exchange 2010 server. However, all Internet mail for all users stopped, and I also cannot send mail out.

So, if you use the same SMTP VS connector that I have been using for years all mail flows fine.  If I disable it, and use the Default SMTP Virtual Server I can send mail to Exchange 2010 users but all other mail stops working.

I followed the steps marked red and it worked.

There must be some difference between the configuration of the two SMTP virtual servers, could be authentication, could be connection restrictions, something like that.

Simon.

you mean that the message did arrive from the old server to the new? 2003-->2010

I am showing our true mail information, and I prefer this information not made available to the public.  So please block this information.  Thanks.

I just tried the following telnet commands this morning, and it did "not" work.  I restarted the SMTP service several times beforehand, so I do not know if that made the difference.  

telnet rcsch13.com 25

220 apollo.ch13.local Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready
 at  Mon, 24 Mar 2014 12:47:10 -0400
helo
250 apollo.ch13.local Hello [24.248.27.2]
mail from:emutis@rcsch13.com
501 5.5.4 Invalid Address
mail from:emutis@rcsch13.com
250 2.1.0 emutis@rcsch13.com....Sender OK
rcpt to:exch2010@rcsch13.com
250 2.1.5 exch2010@rcsch13.com
data
354 Start mail input; end with <CRLF>.<CRLF>
To: exch201
From: erik
Subject:  test test
.
250 2.6.0 <APOLLOpCDbX8BHzpGtB00000007@apollo.ch13.local> Queued mail for delive
ry

Shortly afterward I received a bounce message stating:

Your message did not reach some or all of the intended recipients.

      Subject:      TEST TEST
      Sent:      3/24/2014 11:46 AM

The following recipient(s) cannot be reached:

      exch2010 on 3/24/2014 11:49 AM
            A configuration error in the e-mail system caused the message to bounce between two servers or to be forwarded between two recipients.  Contact your administrator.
            <apollo.ch13.local #5.3.5>

I am not sure what to do about this.  Since I moved one test mailbox, and I cannot send mail from the Existing Exchange 2003 to the Exchange 2010, but can send mail from the test mailbox on the Exchange 2010 to the existing Exchange 2003, I cannot say I can truly move forward and move all our mailboxes to the new server, etc.

That is classic misconfiguration of the SMTP virtual servers. For some reason it is bouncing back. The usual cause is back to what I outlined above, the wrong FQDN.

Run the BPA tool from the toolbox in Exchange 2010 and see what that flags. You should probably run the BPA tool for Exchange 2003 (free download from Microsoft) as well.

Simon.

I enabled the "Default SMTP Virtual Server", and I confirmed the FQDN is the server name apollo.ch13.local.  I also have other virtual server "SMTP VS" that has been working for years, and it also has the same FQDN name "apollo.ch13.local".  

They both use the IP address of the server and not "all unassigned".
They both use Authentication:  Anonymous, Basic, and Integrated Windows.
No smart host defined for either virtual server.
Neither uses reverse DNS lookup or configured external servers.

I also have a routing group connector that we have been using successfully for many years.  It uses DNS to route to each address space.
The local bridge server points to itself and the second virtual server "SMTP VS".
The Internet Address properties:  Email Domain:  "*", and cost of "1",
Connected routing groups is empty.

Before adding the Exchange 2010 server, I only had the SMTP VS virtual server enabled, and the default was disabled.  I believe it did so many years ago, because mail would not flow at all so it was disabled, and the SMTP VS virtual server was created and working since then.

I ran the BPA on Exchange 2003 server (Apollo), and no errors were visible.  I did have an error running the Exchange 2007 Readiness Check since it stated the default virtual server was not listening to port 25.  I had to do this because any mail going to sole test account on Exchange 2010 (Ajax) briefly sits in the queue and then disappears.  While they sit in the queue there are no errors displayed either.  

I set the default virtual server to listen to port 25, and the pending messages sitting in the queue are gone.  No NDR and the messages never made it exchange 2010 user mailbox.

I will run the BPA tests again in a minute, and I can forward the results to you if you wish.

I just ran BPA on Exchange 2003 once again, and it found some issues:

1.  Active Directory Forest:  Unknown Schema Extension are from unknown version:  Schema version detected is:  14734.
2.  Certificate principal mismatch:  https://rcsch13.com does not appear to match host address.  Host address:  rcsch13.com. Principal: O=mail.rcsch13.com, OU = Domain Control Validated, CN = mail.rcsch13.com
3.  Domain: ch13:  Unrecognized Exchange Signature:  Active directory domain "ch13" has an unrecognized exchange signature:  Current Domain Prep version 13040.

I believe items #1 and #3 might not be an issue since it may consider them as foreign since its a newer Exchange 2010 server, SP-3 that extended the schema during installation.  I do not think the certificate issue would pose mail flow problems.

When I installed Exchange 2010, I initiated the following commands beforehand:

setup.com /preparelegacyexchangepermissions
setup.com /prepareschema

Extending the schema initially failed, but was able to complete it successfully after installing RSAT via command "install-windowsfeature RSAT -ADD".  

I then ran the exchange 2010 setup of which installed without error.  My understanding is that there was no need to worry about running these commands since installing Exchange 2010 will automatically take care of it.  

Anyway, Exchange 2003 BPA did show (5) other error message for:

Admin Group:  First Administrative Group:

1.  System Pages too high:  current value 798720
2.  HeapDeCommitFreeBlock Threshold not set.
3.  Paging file larger than physical memory.
4.  Background Cleanup is missing.
5.  Background Cleanup is missing.

I attached a screen shot of the BPA results for each Exchange server.  Left side (Apollo) is the exchange 2003 server, right side (Ajax) is for exchange 2010.   I also attached ESM for Apollo.
Apollo-AjaxBPAResultsScreenshot.jpg
ApolloExchangeSystemManager.jpg

There is nothing in what you have posted that leads to concern. The first three errors are all to be expected, so can be ignored.
Do check that you do not have restrictions on the SMTP virtual server. I am sure that this problem is somewhere with the SMTP virtual server configuration - either away from default configuration, restriction etc. Something isn't right there.

Simon.

This morning when I arrived  at the office I rebooted the Exchange 2003 server, and after the reboot mail was working fine.  This afternoon around 1:15pm mail stopped flowing, and mail messages from all Exchange 2003 users bounced back indicating the same error message "A configuration error in the e-mail system caused the message to bounce between two servers or to be forwarded between two recipients."  

No one could send mail out, and mail was not coming in.  The SMTP server was stopped.  GFI mailessentials indicated problems.  I decided to completely remove the second virtual server "SMTP VS" altogether.  I set GFI ME to use the "Default Virtual Server".  Mail was still not working.  I rebooted the server since restarting GFI services, SMTP, IIS, WWW, etc. did not resolve the mail flow problem.  After the reboot mail (Spam) slowly started flowing to my mailbox.  I was able to send, but the process was slow.  I contacted GFI support and we could not get ME working, so we had to manually uninstall it.  I reinstalled a fresh copy of ME 2014, and now the mail flow displays in the console logs.  

So now I only have one SMTP virtual server, and one send connector.   The 2003-2010 connector is there as well.  Mail appears to be flowing in and out of Exchange 2003 now, but a bunch of mail (public folder replication) messages and the one test message I sent in the morning is sitting in the 2003-2010 queue.  This mail is slowly building.  I think there are about 50 messages now.  

I am starting to think a migration is not the best avenue here.  I do not know why I cannot send mail to exchange 2010 at all.  I am thinking of actually abandoning the migration altogether, and exporting all mailboxes to pst files, uninstalling exchange 2003, uninstalling exchange 2010, installing a clean new version of exchange 2010 on the same Windows Server 2012 VM, making sure mail flows fine, setup the mailboxes if necessary, and importing mail back in.  Not sure if its too late for this scenario or whether there will be more problems going this route.  

I simply do not believe I can move forward with this migration since I cannot send mail to exchange 2010 server either internally or externally.  I can send mail out from the this box, but not receive it.

What is your take on this matter.  I am concerned as to how many more problems I am going to have if I ever get past this coms issue between local exchange servers.  My original thought was a migration would be the easiest way to go.  I also do not know what is involved completely removing Exchange 2003 to a 2003 AD domain for a clean Exchange 2010 install.

I checked the Exchange 2010 server, and earlier in the week I changed the default receive connector ports from 25 to 26.  I just changed it back to port 25, and now mail is flowing between servers.  I will now move along moving all the mailboxes from Exchange 2003 to Exchange 2010.  Thanks so much.  I honestly believe the initial problem was an issue with GFI MailEssentials.  The fact the GFI installation became corrupt, and after reinstalling it, and then setting the smtp port back to port 25, indicates to me GFI was doing something with the virtual server that disrupted communications.