Message stuck between Exchange 2007 and Exchange 2010 (same site) in the hub version 14 queue

make sure that default receive connector on both servers have the Exchange server permissions group enabled.

take a look at the fqdn of the receive connector and ensure the name can be resolved
get-receiveconnector default* | ft name,fqdn

under the permissions group for the receive connector make sure that exchange servers is selects
under authentication ensure that exchange server is selected

Both default receive connectors have the exchange server permissions. I also check the fqdn for the target server as endital suggested and it's set and resolveable from the 2007 server. Any other ideas ?

I should mention that we host several domains and it doesn't matter what the target e-mail domain is. If the mailbox exists on the Exchange 2010 servers it is affected by this problem.

The "Last Error" status on the queue is as follows"
DNS query failed, The error was: SMTPSEND.DNS.NonExistentDomain;nonexistent domain
 
It would appear that the routing engine knows it's on the Exchange 2010 server by virtue of being in the Hub Version 14 queue but it when it sends the can't find the other server. I am assuming MX records have nothing to do with this since this is all internal routing between servers.

what about the dns server configuration settings for the transport server (not within windows but from the management console)
did you specify unique dns servers on the 2007 server?

Well I have not implemented Internal and External DNS servers at the transport level. I really have needed to since we use a smarthost for all outbound delivery. So its currently using the DNS servers for the adapter. U have tested all the various DNS names for the EX2010 server and they resolve.

I am currently testing a new configuration and for the moment it seems to be resolved. I have setup a receive connect or each machine using and Internal template and specifying each other as receiving mail servers.
I will keep you posted.

No go on the new receive connectors. We still have messages getting stuck in the queue.
The queue status is still:

The "Last Error" status on the queue is as follows"
DNS query failed, The error was: SMTPSEND.DNS.NonExistentDomain;nonexistent domain
 

let's just double check a few things

exchange 2007 is sp2 or later on the hub transport server
2007 server can telnet into the 2010 server on port 25

Exchange 2007 is at SP2 (8.2 Build 176.2) and I am able to telnet to the Exchange 2010 server from the Exchange 2007 server on port 25.

sorry, i re-read your original post and i was hoping you could clarify something for me

are messages getting stuck in the queue being sent to internal mailboxes on a 2010 server, or are these external messages

These are all internal mailboxes. Mail from the 2010 server to 2007 server go fine, its only when a user on the 2007 servers sends to a user on the 2010 server.
Thanks for your dillegence!

you have me scratching my head here...
the issue is obviously with the 2007 server
can you post
get-transportserver ex2007 | fl
get-receiveconnector "default 2010srv" | fl

Excahnge 2007 Transport

Name                                    : CIMS01
AntispamAgentsEnabled                   : False
ConnectivityLogEnabled                  : False
ConnectivityLogMaxAge                   : 30.00:00:00
ConnectivityLogMaxDirectorySize         : 250MB
ConnectivityLogMaxFileSize              : 10MB
ConnectivityLogPath                     : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\Connectivity
DelayNotificationTimeout                : 01:00:00
ExternalDelayDsnEnabled                 : True
ExternalDNSAdapterEnabled               : True
ExternalDNSAdapterGuid                  : 00000000-0000-0000-0000-000000000000
ExternalDNSProtocolOption               : Any
ExternalDNSServers                      : {64.58.4.2, 64.58.15.2, 205.171.2.65}
ExternalIPAddress                       :
ExternalDsnDefaultLanguage              : en-US
ExternalDsnLanguageDetectionEnabled     : True
ExternalDsnMaxMessageAttachSize         : 10MB
ExternalDsnReportingAuthority           : concentrics.local
ExternalDsnSendHtml                     : True
ExternalPostmasterAddress               :
InternalDelayDsnEnabled                 : True
InternalDNSAdapterEnabled               : True
InternalDNSAdapterGuid                  : 00000000-0000-0000-0000-000000000000
InternalDNSProtocolOption               : Any
InternalDNSServers                      : {172.100.1.30, 172.100.1.12}
InternalDsnDefaultLanguage              : en-US
InternalDsnLanguageDetectionEnabled     : True
InternalDsnMaxMessageAttachSize         : 10MB
InternalDsnReportingAuthority           : CIMS01.concentrics.local
InternalDsnSendHtml                     : True
MaxConcurrentMailboxDeliveries          : 7
MaxConcurrentMailboxSubmissions         : 20
MaxConnectionRatePerMinute              : 1200
MaxOutboundConnections                  : 1000
MaxPerDomainOutboundConnections         : 20
MessageExpirationTimeout                : 2.00:00:00
MessageRetryInterval                    : 00:01:00
MessageTrackingLogEnabled               : True
MessageTrackingLogMaxAge                : 30.00:00:00
MessageTrackingLogMaxDirectorySize      : 250MB
MessageTrackingLogMaxFileSize           : 10MB
MessageTrackingLogPath                  : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking
MessageTrackingLogSubjectLoggingEnabled : True
OutboundConnectionFailureRetryInterval  : 00:10:00
IntraOrgConnectorProtocolLoggingLevel   : None
PickupDirectoryMaxHeaderSize            : 64KB
PickupDirectoryMaxMessagesPerMinute     : 100
PickupDirectoryMaxRecipientsPerMessage  : 100
PickupDirectoryPath                     : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Pickup
PipelineTracingEnabled                  : False
ContentConversionTracingEnabled         : False
PipelineTracingPath                     : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing
PipelineTracingSenderAddress            :
PoisonMessageDetectionEnabled           : True
PoisonThreshold                         : 2
QueueMaxIdleTime                        : 00:03:00
ReceiveProtocolLogMaxAge                : 30.00:00:00
ReceiveProtocolLogMaxDirectorySize      : 250MB
ReceiveProtocolLogMaxFileSize           : 10MB
ReceiveProtocolLogPath                  : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive
RecipientValidationCacheEnabled         : False
ReplayDirectoryPath                     : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Replay
RootDropDirectoryPath                   :
RoutingTableLogMaxAge                   : 7.00:00:00
RoutingTableLogMaxDirectorySize         : 50MB
RoutingTableLogPath                     : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\Routing
SendProtocolLogMaxAge                   : 30.00:00:00
SendProtocolLogMaxDirectorySize         : 250MB
SendProtocolLogMaxFileSize              : 10MB
SendProtocolLogPath                     : C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpSend
TransientFailureRetryCount              : 6
TransientFailureRetryInterval           : 00:05:00
AntispamUpdatesEnabled                  : False
InternalTransportCertificateThumbprint  : B03FAC6E5A227AB9E8D95E0DEDF5290E41E6B98F
IsValid                                 : True
OriginatingServer                       : CIDC02.concentrics.local
ExchangeVersion                         : 0.1 (8.0.535.0)
DistinguishedName                       : CN=CIMS01,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=C
                                          onfiguration,DC=concentrics,DC=local
Identity                                : CIMS01
Guid                                    : 113b5086-9a63-4432-ae08-a9f13e9ddd66
ObjectCategory                          : concentrics.local/Configuration/Schema/ms-Exch-Exchange-Server
ObjectClass                             : {top, server, msExchExchangeServer}
WhenChanged                             : 9/7/2010 3:48:15 PM
WhenCreated                             : 6/10/2009 5:01:00 PM

Exchange 2010 Recieve Connector

RunspaceId                              : 28c29a9d-7aba-4274-9705-ae1b248802ca
AuthMechanism                           : Tls, BasicAuth, ExchangeServer
Banner                                  :
BinaryMimeEnabled                       : True
Bindings                                : {172.100.1.161:25}
ChunkingEnabled                         : True
DefaultDomain                           :
DeliveryStatusNotificationEnabled       : True
EightBitMimeEnabled                     : True
DomainSecureEnabled                     : False
EnhancedStatusCodesEnabled              : True
LongAddressesEnabled                    : False
OrarEnabled                             : False
SuppressXAnonymousTls                   : False
AdvertiseClientSettings                 : False
Fqdn                                    : cims02.concentrics.local
Comment                                 :
Enabled                                 : True
ConnectionTimeout                       : 00:10:00
ConnectionInactivityTimeout             : 00:05:00
MessageRateLimit                        : unlimited
MessageRateSource                       : IPAddress
MaxInboundConnection                    : 5000
MaxInboundConnectionPerSource           : unlimited
MaxInboundConnectionPercentagePerSource : 100
MaxHeaderSize                           : 64 KB (65,536 bytes)
MaxHopCount                             : 30
MaxLocalHopCount                        : 8
MaxLogonFailures                        : 3
MaxMessageSize                          : 100 MB (104,857,600 bytes)
MaxProtocolErrors                       : 5
MaxRecipientsPerMessage                 : 5000
PermissionGroups                        : AnonymousUsers, ExchangeUsers, ExchangeServers, ExchangeLegacyServers
PipeliningEnabled                       : True
ProtocolLoggingLevel                    : Verbose
RemoteIPRanges                          : {0.0.0.0-255.255.255.255}
RequireEHLODomain                       : False
RequireTLS                              : False
EnableAuthGSSAPI                        : False
LiveCredentialEnabled                   : False
Server                                  : CIMS02
SizeEnabled                             : EnabledWithoutValue
TarpitInterval                          : 00:00:05
MaxAcknowledgementDelay                 : 00:00:30
AdminDisplayName                        :
ExchangeVersion                         : 0.1 (8.0.535.0)
Name                                    : Default CIMS02
DistinguishedName                       : CN=Default CIMS02,CN=SMTP Receive Connectors,CN=Protocols,CN=CIMS02,CN=Server
                                          s,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups
                                          ,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=
                                          concentrics,DC=local
Identity                                : CIMS02\Default CIMS02
Guid                                    : 7df2a364-8d4e-42b5-a094-e3fc5e31b585
ObjectCategory                          : concentrics.local/Configuration/Schema/ms-Exch-Smtp-Receive-Connector
ObjectClass                             : {top, msExchSmtpReceiveConnector}
WhenChanged                             : 9/7/2010 5:36:19 PM
WhenCreated                             : 4/26/2010 8:00:08 PM
WhenChangedUTC                          : 9/7/2010 11:36:19 PM
WhenCreatedUTC                          : 4/27/2010 2:00:08 AM
OrganizationId                          :
OriginatingServer                       : ConcentricsHyperV.concentrics.local
IsValid                                 : True

Thanks !

does cims02.concentrics.local resolve to 172.100.1.161 on the 2007 server?

still looking for other potential issues

Definately. one of the first tests I did.

ok, i wasn't sure if you had multiple ip addresses assigned to the 2010
i just noticed that the binding was restricted to the one ip address

Enditall, well I think we resolved it. This server used to be a DNS server at one time. In closer inspection of the server I found a total of 3 DNS servers specified, the two listed on the adapter TCP setup and a third on the advanced DNS setup. The third DNS server pointed to itself (127.0.0.1) and that take priority over the other two. I removed that DNS entry and that seems to have solved the problem. Thanks for your persistence and direction towards the DNS.