HELP!! - Exchange 2003 Queues Building SMTP (SMTP CONNECTOR)

Additional info... have powered off Echange FrontEnd Server to see if any differences.  None.  

Also have an Exchange 2007 server, that is being tested.  All 3 servers (EXFE, EXBE, EX2007) are open on port 25 to the internet.

can you telnet back and forth between the email servers on port 25?

Ran the MS Exchange Trouble Shooting Assistance, designating the symptom of messages backing up in 1 or more (52 in our case and growing) queues.  Ran it and showed the queues status as: REMOTE DELIVERY QUEUE IS IN RETRY.   (I already knew that.).....have FORCED CONNECTION on almost every queue but no changes.

When the EXTSA program  tried to connect to the remote servers in the queues, many returned the error that PORT 25 DID NOT RESPOND.  Not sure if that is a real problem or not.

Unable to telnet back and forth.....for all mail servers

BACKEND to FRONTEND - YES
BACKEND to EX2007 - YES
BACKEND to TESTBACKEND -

FRONTEND to BACKEND - NO - COULD NOT OPEN CONNECTION TO THE HOST ON PORT 25
FRONTEND to EX2007 - NO - COULD NOT OPEN CONNECTION TO THE HOST ON PORT 25
FRONTEND to TESTBACKEND- NO - COULD NOT OPEN CONNECTION TO THE HOST ON PORT 25

EX2007 to BACKEND - NO - TELNET NOT RECOGNIZED AS AN INTERNAL OR EXTERNAL COMMAND
EX2007 to FRONTEND - NO - TELNET NOT RECOGNIZED AS AN INTERNAL OR EXTERNAL COMMAND
EX2007 TO TESTBACKEND - NO - TELNET NOT RECOGNIZED AS AN INTERNAL OR EXTERNAL COMMAND

OLDTESTBACKEND to BACKEND - NO - COULD NOT OPEN CONNECTION TO THE HOST ON PORT 25
OLDTESTBACKEND to FRONTEND - NO - COULD NOT OPEN CONNECTION TO THE HOST ON PORT 25
OLDTESTBACKEND to EX2007 - NO - COULD NOT OPEN CONNECTION TO THE HOST ON PORT 25

All have SMTP running.  Not sure what to make of this.  EX2007 is Windows 2008 Server.

Oops forgot one:

BACKEND to TESTBACKEND - YES

Troubleshooter showed failures connecting to remote servers, had comment "POLITE PEOPLE SAY HELO"  So I change the SMTP CONNECTOR to SAY HELO, instead of EHLO.  Queues continuing to build....nothing has worked so far....

Opened Virus Scan Software (McAfee) on other servers to allow mass mailings.  Could then TELENT between all Exchange 2003 Servers and Exchange 2007 server....so that part is working.

Ran the troubleshooting assistant again.....

Do you route email via a smart host? When you did that telnet test, what did you telnet to externally? That message you received isn't a standard message, it someone trying to be funny.

Simon.

I tried something else..... configured my SMTP connector to use a smart host.  Set the smart host as the frontend server.  Lots of messages in the PENDING WAITING SUBMISSION QUEUE, now.....waiting to see what happens.  I am at my wits end....

TELNET was done internally, between MS exchange servers.  I ran the Exchange Troubleshooting assistance and it reported TELNET errors to the outside servers listed in the SMTP CONNECTOR queue ....from the messages in the queue.

You should not have set the smart host as your frontend server. NEVER set another Exchange server as the smart host. That will cause email delivery to stop completely.
If you want email to go out through the frontend server then set the frontend server only as the bridgehead. The backend server will then send email to that server for delivery.

You need to confirm that you can connect to external SMTP servers. If you cannot then something is blocking the traffic.

Simon.

ok...have changed the smtp connector to us DNS.  Stopped the SMTP VS on each ex server and restarted it.  HAve set the bridgehead server to be the FE...

How should I test the connect to the external SMTP servers?  EX trouble shooting assistant?

Just attempt to telnet to an external host.

Your ISPs SMTP server for example.

telnet host.isp.net 25

If you get a response then it isn't being blocked inside your ISPs network.
Then attempt to telnet to port 25 on a remote server.
If that is blocked then your ISP is blocking port 25. You will need to call them and ask why. Expect them to say they are not and then for it to start working about 30 minutes later.

Simon.

I changed the bridgehead to the EXFE.  Stopped and started each SMTP VS.  Messages still in queues on a couple of servers PENDING SUBMISSION.  Running the EXTRA again, since it will test connectivity to outside SMTP servers......

Just FYI to outline what I have done...

Yesterday afternoon, when I changed the connector to use EXFE as the smart host, all the messages in the individual queues : SMTP - domainname.xxx (SMTP CONNECTOR) when to the EXFE queues in MESSAGES PENDING SUBMISSION.  THis morning all the messages had moved to the TESTBACKEND server, but were still in the MESSAGES PENDING SUBMISSION queue.  Then I change from SMARTHOST back to DNS and changed the bridgehead to the EXFE, restarted all the SMTP VS.  The 192 messages in the TESTBACKEND server remain, but now getting some added to the PRODBACKEND server MESSAGES PENDING SUBMISSION queue.

Now having messages on the EXFE go into SMTP - domainname.xxx (SMTP CONNECTOR) queues like was happening when the problem first started.

did the following from the server that has everying queued, using the some of the domain names specified.

TELNET YAHOO.COM 25
TELNET COMCAST.NET 25
TELNET SBCGLOBAL.NET 25
TELNET GMX.COM 25

All failed with this error:   COULD NOT OPEN A CONNECTION TO THE HOST ON PORT 25, CONNECT FAILED

I tried the TELNET for our domain:

TELNET CALVARYSPRINGFIELD.ORG 25
and it failed also.  Trying some from outside the network....

You need to find the actual MX records for the hosts, not just pick domains.
For example, yahoo.com's MX records are actually as shown below.

C:\>nslookup
Default Server:  host.example.co.uk
Address:  192.168.3.1

> set type=mx
> yahoo.com
Server:  host.example.co.uk
Address:  192.168.3.1

Non-authoritative answer:
yahoo.com       MX preference = 1, mail exchanger = d.mx.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = e.mx.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = f.mx.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = g.mx.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = a.mx.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = b.mx.mail.yahoo.com
yahoo.com       MX preference = 1, mail exchanger = c.mx.mail.yahoo.com

Simon.

Went to an outside network (my PC at home, using ATT DSL and did the same TELNET commands as done previously, from inside the work network.

Again, All failed with this error:   COULD NOT OPEN A CONNECTION TO THE HOST ON PORT 25, CONNECT FAILED

HOWEVER, I went to our DMZ and did the TELNETs from an APPLE MAC terminal session and it tried to connect to the remote servers, but indicated CONNECTION REFUSED for each IP address mapped to the DNS domain name, before it indicated CONNECT FAILED.

Not sure what this really tells me?????

Ok.... you will have to pardon my ignorance here....trying the TELNET Again...from server with messages in the queues...

TELNET MX1.COMCAST.NET 25
TELNET MX2.COMCAST.NET 25
TELNET F.MAIL.YAHOO.COM 25
TELNET A.MAIL.YAHOO.COM 25
TELNET SBCMX1.PRODIGY.NET 25
TELNET SBCMX7.PRODIGY.NET 25
TELNET xxxx 25 (xxx were our 4 exchange servers)

All connected with no problems.   So connectivety to external SMTP servers is working.

So if you did

telnet a.mx.mail.yahoo.com 25

You received connection refused?

What you should have got was this:

220 mta133.mail.ac4.yahoo.com ESMTP YSmtp service ready

Otherwise that would indicate that the port is being blocked somewhere. If all connections return the same thing then I would be looking at either your router/firewall or ISP as the cause of the block.

Simon.

Update,  we are not getting mail from outside our domain either.  It is getting into the system and thru spam filters but not arriving....perhaps these are the MESSAGES PENDING SUBMISSION queues.

Clarify.....when I say all connected, I meant all responded with the
220 <server name> ESMTP SERVER READY
No connection refused now, since using the MX servers to TELNET to.....
Appears no blocking exists internally or by our ISP.  So the problem must be in DNS or Exchange.???

Now here is an interesting thing.....no users is receiving outside email.  HOWEVER, some spam is scanned and forwarded to my email address where it is put into a SPAM folder, for later visual review.  Not sure all of it, but a lot of it is still getting thru to my SPAM folder, yet legitimate email is not getting to the proper recipients.  

Could the EXFE have a setting wrong that would cause this.  Seems this problem started shortly after the EXFE was actually made and EXFE.  Prior to that is was just a backend server that I was preparing to be EXFE.  Used as BACKEND as it was be setup for eventual use as EXFE.
Surely these problem are not just a coincidence.  That is the only real change to the exchange topology.  Comments, please?

One other symptom....DISK Writes on the primary BACKEND server have been running at almost a continual 100%, since this problem started evening before last.  The INETINFO.EXE process has gone crazy doing disk writes.  This is all tied together somehow....just not sure how....

Thanks for your help so far....I really appreciate it.....

Here is a summary of the symptoms I have identified:

Cannot send to  outside email addresses
Inbound email getting into system but it appears perhaps not all of it is being delivered to user inboxes (some users getting ok, but some tests emails I have sent did not get delivered)
MESSAGE PENDING SUBMISSION QUEUE - building
SMTP <external domain> (SMTP CONNECTOR) QUEUES building
Exchange ActiveSync for domain users working from outside network
Internal email working
100% disk write time on production BACKEND server
TELNET <external mail server> 25 getting 220 ESMTP SERVER READY responses

To clarify again, users are receiving outside email, but not all of it.

OK.,,, ran the Exchange Trouble Shooting Assistant again.....
The REMOTE DELIVERY QUEUES SMTP COMMANDS ANALYSIS report

Lost of Errors - here the 2 that show up a lot in the list of errors....

Mail Acceptance Failure
Remote server b.mx.mail.yahoo.com failed the mail acceptance test.  
MAIL FROM command:  Respond = 555.5.5.2 Syntax error

Mail Acceptance Failure
Remote server sbcmx2.prodify.net failed the mail acceptance test.
MAIL FROM command; Respond - 503.5.0.0 Polite people say HELO first

Yesterday I ran this and saw the HELO error, so I changed the SMTP CONNECTOR to do an EHLO, but none of the messages every left the queues.

What about the one server becoming an EXFE?  Could that cause this problem?

If you think the frontend server is the cause of the problem, bypass it. Change the SMTP forwarding on your firewall back to how it was, and set the backend server as the bridgehead on the SMTP connector.

Unless a DNS change has been made in the external DNS it is highly unusual for some email to come through and not all.

Simon.

I demoted the FXFE to a regular backend server and items in the SMTP CONNECTOR queues remain.

I am at my wits end.  

Did not change the bridgehead......have done that now....also removing exchange from the test backend server.  So far the only thing that has changed is that the queues moved from the former EXFE to the previous BACKEND server.  Queues still building and emails not being sent.

I am open to any suggestion at this point....

Removed test exchange server.  Still not working...have enabled SMTP LOGGING and seeing a lot of RCPT 550 errors in the log.  Need a real SMTP log analyzer

Can you telnet to your ISPs SMTP server?
If so, set the SMTP connector to use the ISPs SMTP server as a smart host.
If that doesn't work, then I would be looking at something between the server and the internet causing a problem. Firewall and AV being the prime candidates.

Simon.

Our ISP does not have a SMTP server I can use as a smart host.....  Ran the exch trouble shooting assistant again and getting these errors when it tries to connect to the domains, for which messages are queued

The remote server did not respond to a connection attempt

Domain xxxxxxx.xxx is a domain for which server 1730WEXCHGIIS is possibly having trouble routing messages thru an SMTP Connector.

Here is a test....

Ran a manual email session to my personal email, which fails when sent thru exchange&..

TELNET SBCMX1.PRODIGY.COM 25

220 flph261.prodigy.net ESMTP Sendmail 8.13.8 inb ipv6 jeff0203/8.13.8; Fri, 1
ay 2009 14:23:13 -0700
EHLO xx
250-flph261.prodigy.net Hello ctccmail.calvaryspringfield.org [66.99.103.247],
leased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE 31981568
250-DELIVERBY
250 HELP
MAIL FROM: rstuemke@calvaryspringfield.org
250 2.1.0 rstuemke@calvaryspringfield.org... Sender ok
RCPT TO: trumpet1503@sbcglobal.net
250 2.1.5 trumpet1503@sbcglobal.net... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
test from telnet session - manual
.
250 2.0.0 n41LNDwF014922 Message accepted for delivery

As you can see it works.  I received the email with no problem.  However, sending to the same address, thu Exchange fails.

I would be very surprised if your ISP doesn't have an SMTP server you can use. They don't call them smart hosts, but most ISPs have a server - look for Outlook Express instructions on their web site.

Was that a telnet test from the server itself?

Simon.

Yes the test was from both of the exchange servers.  However, here is a strange thing....when I run ExTRA, I get different results at times, when it runs the connectivity test to the remote SMTP servers.  Sometimes BAD SYNTAX, sometime POLITE PEOPLE SAY HELO, sometimes something like USER NOT FOUND, sometimes BDAT ERROR.  
This problems started Tuesday evening.  When I came into work Tuesday morning, the primary EX server was hosed up.  Had to do a power down.  When it came back up, could not get the Network Adapter to work.  Have 2 on this server, so connected the other one up and got it running, then the main one started working.  THen disabled the 2nd one, to use the primary we always use.  I am wondering if we could have a hardware error?

Our ISP is the Illinois Century Network...a government agency, that provides internet service for not-for-profit schools and agencies.  They have no SMTP server.  I also talked with the technician and this is something they have thought about doing, but nothing yet.,

I changed the bridgehead to the newer server and all the messages went to that  server's queues.  Then ran the ExTRA on that server, from that server.  Errors were the same as on the other server

Is it possible that I have been taken off track, by focusing on the ExTRA errors and may have lost sight of the real problem? ExTRA sends test emails to all the servers which have queued messages.  However, perhaps it is not sending messages the same way as Exchange.  Sureley this cannot be this difficult????

OH MY!  I moved all the queues back to the new server and they still do not send.....
Would SMTP be corrupt on all server?  I have an exchange 2007 server that I am configuring to send email.  Have moved a user over to it to test a send message, but cannot get the PC/Outlook to communicate witht the e2007 server.  It finds the mailbox, but cannot communicate....should I persue this line of thought...

Outlook not finding the mailbox isn't unusual - but that could be a replication delay. It is not unheard of for Outlook not to redirect immediately, but five or ten minutes later it will.

If it is following between machines, then it is unlikely to be an SMTP issue across servers. I wouldn't rule it out, but it is highly unlikely. It has to be something outside of the Exchange servers. So that could be third party apps, issues with the domain controllers, firewall, something like that. However I am well in to the straw clutching phase now.

You said you had an Exchange 2007 server - does that have a routing group connector to the Exchange 2003 server?
Does it have a Send Connector configured on its own? Exchange 2007 can upset mail routing if you aren't careful.

Simon.

GOOD NEWS!  Was able to fix it....here is what I did.....
did not want to due a software reload, so built a new administrative group and another server.
Got this working and was able to send to internet....set up a routing connector to new admin group and mail started working....looks like the problem was with the SMTP Internet Send Connector.
Thanks for all your help.  Your suggestion to do a reload, took me down the path that led to the resolution.  I appreciate your willingness to assist and your patience during these last few days.
I am awarding you the points.....

Suggestion led down path that developed into problem resolution.  Thanks.,