We help IT Professionals succeed at work.

sendmail not sending outbound email - but accepting inbound email

jvossler
jvossler asked
on
I have sendmail running on Solaris 10. It is correctly handling incoming email for 4 domains.

All mail that is attempting to send outbound form this server fails with "Deferred: Connection timed out" error connecting to itself.

All DNS entries are correct and the firewall is  not involved, and the firewall logs contain no indications of rejecting, denying or stopping connections.

I can telnet to port 25 using the fqdn, short name, "localhost" or 127.0.0.1.

All the mail is still queued up for delivery.

Here is a sample sendmail dialog with a test message along with mailq output

Spider:/etc/mail> echo "test 5" | mailx -v -s "test 5" -r john@vosslers.net john@vosslers.net
john@vosslers.net... Connecting to [127.0.0.1] via relay...
220 smtp.itinfrastructures.net ESMTP Sendmail 8.13.8+Sun/8.13.8; Wed, 2 Nov 2011 08:31:12 -0600 (MDT)
>>> EHLO Spider.itinfrastructures.net
250-smtp.itinfrastructures.net Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> VERB
250 2.0.0 Verbose mode
>>> MAIL From:<john@vosslers.net> SIZE=64
250 2.1.0 <john@vosslers.net>... Sender ok
>>> RCPT To:<john@vosslers.net>
>>> DATA
250 2.1.5 <john@vosslers.net>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
050 <john@vosslers.net>... Connecting to smtp.secureserver.net. via esmtp...
050 <john@vosslers.net>... Connecting to mailstore1.secureserver.net. via esmtp...
050 <john@vosslers.net>... Deferred: Connection timed out with mailhost.itinfrastructures.net
250 2.0.0 pA2EVCmr000595 Message accepted for delivery
john@vosslers.net... Sent (pA2EVCmr000595 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 smtp.itinfrastructures.net closing connection



Spider:/etc/mail> mailq
                /var/spool/mqueue (16 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
pA2EVCmr000595*       7 Wed Nov  2 08:31 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2DH7oO000052        7 Wed Nov  2 07:17 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2D8uB8029968        7 Wed Nov  2 07:08 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2CuNWa029833       29 Wed Nov  2 06:56 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3034375962@messaging.sprintpcs.com>
pA2CthLg029816       29 Wed Nov  2 06:55 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3038875478@messaging.sprintpcs.com>
pA2Cu3pZ029823       29 Wed Nov  2 06:56 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3038875413@messaging.sprintpcs.com>
pA2CkQQi029686        5 Wed Nov  2 06:46 <root@Spider.itinfrastructures.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2CrkqI029775        5 Wed Nov  2 06:53 <root@Spider.itinfrastructures.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2Ct2Im029800       29 Wed Nov  2 06:55 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2CtNEp029809       29 Wed Nov  2 06:55 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <nancy@vosslers.net>
pA2CgPhL029646       24 Wed Nov  2 06:42 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3038875413@messaging.sprintpcs.com>
pA2BQQoh029159       24 Wed Nov  2 05:26 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3034375962@messaging.sprintpcs.com>
pA2BQ6HW029152       24 Wed Nov  2 05:26 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3038875413@messaging.sprintpcs.com>
pA2BP5WZ029130       24 Wed Nov  2 05:25 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <john@vosslers.net>
pA2BPPtf029138       24 Wed Nov  2 05:25 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <nancy@vosslers.net>
pA2BPk0l029145       24 Wed Nov  2 05:25 <john@vosslers.net>
                 (Deferred: Connection timed out with mailhost.itinfrastructur)
                                         <3038875478@messaging.sprintpcs.com>
                Total requests: 16
Spider:/etc/mail>

Open in new window

Comment
Watch Question

CERTIFIED EXPERT

Commented:
Can you please do:
grep mailhost /etc/hosts

Open in new window

CERTIFIED EXPERT

Commented:
Also, please do:
grep ^DS /etc/mail/sendmail.cf

Open in new window

Author

Commented:
Below is the output you requested.

There are almost no entries in /etc/hosts - the sendmail server is also the secondary DNS server as well. I included the nslookup for "mailhost"


Spider:/etc/mail> grep mailhost /etc/hosts
Spider:/etc/mail> nslookup mailhost 
Server:         71.39.23.98
Address:        71.39.23.98#53

mailhost.itinfrastructures.net  canonical name = smtp.itinfrastructures.net.
Name:   smtp.itinfrastructures.net
Address: 71.39.23.109

Spider:/etc/mail> 
Spider:/etc/mail> 
Spider:/etc/mail> grep ^DS /etc/mail/sendmail.cf
DS
Spider:/etc/mail>

Open in new window

CERTIFIED EXPERT

Commented:
Okay, well you have avoided the 2 pitfalls I see most in Solaris 10 sendmail setups - the biggest mistake people make is changing ANYTHING, since it's ready to send mail by default.

Did you change anything other DaemonPortOptions when you were enabling it to receive inbound mail?

Better yet, please run a diff against the attached sendmail.cf, blot out anything sensitive, and post the results.
sendmail.cf.txt
CERTIFIED EXPERT

Commented:
There are a couple of things puzzling to me.  First is, line 29 above shows:

        john@vosslers.net... Sent (pA2EVCmr000595 Message accepted for delivery)

And then its followed by a timeout.  Did the message ever arrive?

Also, there is no reference anywhere to mailhost.itinfrastructures.net in the MX records for the domain or /etc/hosts, so the system should not be using it - I'm curious to see if its using this line in sendmail.cf:

        O FallbackSmartHost=mailhost$?m.$m$.

...and consequently, what would happen if you commented it out.  Because it looks like it's trying to use a smarthost when it doesn't need one.

Author

Commented:

Here are the differences - besides comment fields

It does not look like anything that would make outbound mail fail.

I did add the virtusertable, but I set that up in 2008 and it has been modified several times but continues to work as expected. The last update to it was Apr 2011



 
84a91
> Djsmtp.itinfrastructures.net
133a141,143
> # Virtual user table (maps incoming users)
> Kvirtuser hash -o /etc/mail/virtusertable.db
> 
241c251
< #O ClientPortOptions=Family=inet, Address=0.0.0.0
---
> O ClientPortOptions=Family=inet
695a705
> R$* < @ $={VirtHost} > $*     $: $1 < @ $2 . > $3
798a809,827
> # handle virtual users
> R$+                   $: <!> $1               Mark for lookup
> R<!> $+ < @ $={VirtHost} . >  $: < $(virtuser $1 @ $2 $@ $1 $: @ $) > $1 < @ $2 . >
> R<!> $+ < @ $=w . >   $: < $(virtuser $1 @ $2 $@ $1 $: @ $) > $1 < @ $2 . >
> R<@> $+ + $+ < @ $* . >
>                       $: < $(virtuser $1 + + @ $3 $@ $1 $@ $2 $@ +$2 $: @ $) > $1 + $2 < @ $3 . >
> R<@> $+ + $* < @ $* . >
>                       $: < $(virtuser $1 + * @ $3 $@ $1 $@ $2 $@ +$2 $: @ $) > $1 + $2 < @ $3 . >
> R<@> $+ + $* < @ $* . >
>                       $: < $(virtuser $1 @ $3 $@ $1 $@ $2 $@ +$2 $: @ $) > $1 + $2 < @ $3 . >
> R<@> $+ + $+ < @ $+ . >       $: < $(virtuser + + @ $3 $@ $1 $@ $2 $@ +$2 $: @ $) > $1 + $2 < @ $3 . >
> R<@> $+ + $* < @ $+ . >       $: < $(virtuser + * @ $3 $@ $1 $@ $2 $@ +$2 $: @ $) > $1 + $2 < @ $3 . >
> R<@> $+ + $* < @ $+ . >       $: < $(virtuser @ $3 $@ $1 $@ $2 $@ +$2 $: ! $) > $1 + $2 < @ $3 . >
> R<@> $+ < @ $+ . >    $: < $(virtuser @ $2 $@ $1 $: @ $) > $1 < @ $2 . >
> R<@> $+                       $: $1
> R<!> $+                       $: $1
> R< error : $-.$-.$- : $+ > $*         $#error $@ $1.$2.$3 $: $4
> R< error : $- $+ > $*         $#error $@ $(dequote $1 $) $: $2
> R< $+ > $+ < @ $+ >   $: $>Recurse $1

Open in new window

Author

Commented:
The messages have been accepted and are sitting in the local servers /var/spool/mqueue directory

Nothing has left the server and thus nothing from this server has been received. Other email sources continue to arrive successfully.

The MX record for this server is smtp.itinfrastructures.net with CNAME entries for mailhost.itinfrastructures.net and email.itinfrastructures.net

CERTIFIED EXPERT

Commented:
Okay, I don't know that you NEED the Djsmtp.itinfrastructures.net, but I can't imagine it's hurting anything.

Can you please see what happens if you comment out the fallback smarthost?  It's the only thing I can possibly see that references mailhost.
# O FallbackSmartHost=mailhost$?m.$m$.

Open in new window

Author

Commented:
It looks like it is using the smarthost as a fall back

Spider:/etc/mail> grep FallbackSmartHost sendmail.cf

O FallbackSmartHost=mailhost$?m.$m$.

Spider:/etc/mail>
CERTIFIED EXPERT

Commented:
Yes, I looked up your MX records at the start, and those are fine.  The CNAMES for the other hostnames shouldn't really be relevant to this - they are nice to have, but most people just put "mailhost" as an alias in /etc/hosts on the far right of the line that has the system IP address in order to appease Solaris' strange affinity for this name.

The crux of it is that it shouldn't be attempting to contact any external host except the remote mail server.  What's so weird is that your debug shows it relaying normally first, and only THEN trying mailhost.  Which really does in some way validate my SWAG of the FallbackSmartHost being an issue.

Author

Commented:
With the "FallbackSmartHost" line commented out and sendmail restarted I get a slightly different failure

 
Spider:/etc/mail> echo "test 6" | mailx -v -s "test 6" -r john@vosslers.net john@vosslers.net
john@vosslers.net... Connecting to [127.0.0.1] via relay...
220 smtp.itinfrastructures.net ESMTP Sendmail 8.13.8+Sun/8.13.8; Wed, 2 Nov 2011 09:55:58 -0600 (MDT)
>>> EHLO Spider.itinfrastructures.net
250-smtp.itinfrastructures.net Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP
>>> VERB
250 2.0.0 Verbose mode
>>> MAIL From:<john@vosslers.net> SIZE=64
250 2.1.0 <john@vosslers.net>... Sender ok
>>> RCPT To:<john@vosslers.net>
>>> DATA
250 2.1.5 <john@vosslers.net>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
050 <john@vosslers.net>... Connecting to smtp.secureserver.net. via esmtp...
050 <john@vosslers.net>... Connecting to mailstore1.secureserver.net. via esmtp...
050 <john@vosslers.net>... Deferred: Connection timed out with mailstore1.secureserver.net.
250 2.0.0 pA2Ftwi5000967 Message accepted for delivery
john@vosslers.net... Sent (pA2Ftwi5000967 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 smtp.itinfrastructures.net closing connection
Spider:/etc/mail>

Open in new window

CERTIFIED EXPERT
Commented:
Okay, so all 3 are failing.   I think you have an outbound port 25/tcp firewall in place either on the box or on your network.

Something tells me, this probably won't work for you as it does me:
[xterm@foo ~]$ telnet smtp.secureserver.net 25
Trying 72.167.238.201...
Connected to smtp.secureserver.net.
Escape character is '^]'.
220 p3pismtp01-021.prod.phx3.secureserver.net ESMTP
^]
telnet> quit
Connection closed.

Open in new window

Author

Commented:
Well I re-checked our firewall. The logs indicate the the traffic from the source address to the destination are explicitly accepted.

The "telnet smtp.secureserver.net 25" does fail. If I change it to "smtpout.secureserver.net" using port 80 I am able to get through.

My next stop is our ISP (we changed ISPs over the summer) and report back. I think that the ISP is blocking outbound traffic on port 25




CERTIFIED EXPERT

Commented:
Yep, it's common, we do it too (although I open it for customers by request.)

Keep in mind, when they open the port for you, that queue is going to empty in a hurry.  If those mails are no longer relevant, you can whack everything in /var/spool/mqueue and /var/spool/clientmqueue to prevent that.

Author

Commented:
It did turn out that the ISP (centurylink/qwest) be default blocks outbound traffic on port 25.

I could access the server "smtp.secureserver.net" only on port 80, not port 25 until the ISP unblocked port 25.

Once that was complete I was able to telnet to the server on port 25 with no issues.

I then manually cleared the mail queue "sendmail -v -q" and all messages were delivered.

Since we changed ISP's over the summer along with a new firewall and new static IPs I originally thought it might be DNS. Since I never requested port 25 be blocked from the ISP I assumed it was open (yes, I know about assuming).


Thanks for your efforts and quick responses.

Author

Commented:

I wondered why our equipment was working so well. Alerts are sent to a couple of our cell phones via email.

Now the real work begins.

Thanks again

CERTIFIED EXPERT

Commented:
The work never ends, but hey, that's one hurdle down - glad you're back in business now.