[Last Call] Learn about multicloud storage options and how to improve your company's cloud strategy. Register Now

x
?
Solved

How to solve "Deferred: Invalid argument" problems in sendmail (bounces bouncing)

Posted on 2004-10-27
15
Medium Priority
?
4,934 Views
Last Modified: 2013-12-17
I have a constant large number of items in my sendmail queue, most of them containing "Deferred: Invalid argument" in the "q" file.

I know most of these should be deliverable, but it seems sendmail is somehow sending something stupid to the receiving server (if it's even trying that at all) or something else, because these sit here for 5 days then hit my "postmaster" address.

It seems the majority of these emails are bounces.

In other words - my mail server is having problems delivering some bounces, so they sit for 5 days and then bounce into my postmaster account.

I notice that they all have a strange line: H?P?Return-Path: <g>

In case you can't see it - there's a strange character before the "g" (in "vi" it shows up as <<81>g>). This may or may not be a problem? I've no idea where it came from.

Below is an example file (domain names have been changed to protect against spam, but otherwise it's unedited).  The RPF line is a valid email address (well, it was before I obscured it to protect against spam).  It happens to be my own address, on a different server running the same version of sendmail as the server having the problem sending it.

Any help, clues, hints, advice, or anything you can provide would be most welcome!!!


[root@my mqueue]# pwd
/var/spool/mqueue
[root@my mqueue]# grep -l "Deferred: Invalid argument" * |wc
     39      39     663

[root@my mqueue]# dir *fi9PMqwvt019196  
-rw-------    1 root     smmsp        1414 Oct 25 23:59 dfi9PMqwvt019196
-rw-------    1 root     smmsp         896 Oct 27 08:42 qfi9PMqwvt019196
[root@my mqueue]#
[root@my mqueue]# cat qfi9PMqwvt019196
V6
T1098748757
K1098866550
N33
P2970000
I104/2/9831242
MDeferred: Invalid argument
Frns
$_localhost
$r
$slocalhost
${daemon_flags}b h
${if_addr}127.0.0.1
SMAILER-DAEMON@maindomain.com
MDeferred: Invalid argument
Ccnd:500:500:cnd
rRFC822; postmaster@otherdomain.com
RPF:"cnd@recipient.net.au"
H?P?Return-Path: <g>
H??Received: from localhost (localhost)
        by otherdomain.com (8.12.10/8.12.10) id i9PMqwvt019196;
        Mon, 25 Oct 2004 23:59:17 GMT
H?D?Date: Mon, 25 Oct 2004 23:59:17 GMT
H?F?From: Mail Delivery Subsystem <MAILER-DAEMON@maindomain.com>
H?x?Full-Name: Mail Delivery Subsystem
H?M?Message-Id: <200410252359.i9PMqwvt019196@otherdomain.com>
H??To: postmaster
H??MIME-Version: 1.0
H??Content-Type: multipart/report; report-type=delivery-status;
        boundary="i9PMqwvt019196.1098748757/otherdomain.com"
H??Subject: Postmaster notify: see transcript for details
H??Auto-Submitted: auto-generated (postmaster-notification)
.
[root@my mqueue]#
[root@my mqueue]# cat dfi9PMqwvt019196
This is a MIME-encapsulated message

--i9PMqwvt019196.1098748757/otherdomain.com

The original message was received at Wed, 20 Oct 2004 23:13:02 GMT
from localhost
with id i9KMWfDp019382

   ----- The following addresses had permanent fatal errors -----
<jamas@xyzwebmail.net>

   ----- Transcript of session follows -----
<jamas@xyzwebmail.net>... Deferred: Invalid argument
Message could not be delivered for 5 days
Message will be deleted from queue

--i9PMqwvt019196.1098748757/otherdomain.com
Content-Type: message/delivery-status

Reporting-MTA: dns; otherdomain.com
Arrival-Date: Wed, 20 Oct 2004 23:13:02 GMT

Final-Recipient: RFC822; jamas@xyzwebmail.net
Action: failed
Status: 4.4.7
Remote-MTA: DNS; ns3.xyzwebmail.com
Last-Attempt-Date: Mon, 25 Oct 2004 23:59:17 GMT

--i9PMqwvt019196.1098748757/otherdomain.com
Content-Type: text/rfc822-headers

Return-Path: <MAILER-DAEMON@maindomain.com>
Received: from localhost (localhost)
        by otherdomain.com (8.12.10/8.12.10) id i9KMWfDp019382;
        Wed, 20 Oct 2004 23:13:02 GMT
Date: Wed, 20 Oct 2004 23:13:02 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON@maindomain.com>
Message-Id: <200410202313.i9KMWfDp019382@otherdomain.com>
To: <jamas@xyzwebmail.net>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="i9KMWfDp019382.1098313982/otherdomain.com"
Subject: Return receipt
Auto-Submitted: auto-generated (return-receipt)

--i9PMqwvt019196.1098748757/otherdomain.com--

[root@my mqueue]#
0
Comment
Question by:ChrisDrake
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 4
  • 2
15 Comments
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12420674
My problem still remains, but I've investigated the "H?P?Return-Path: <g>" situation a bit more and it looks like that is a red herring: I've found some normal queue messages with the same thing that do go through OK (not "Invalid argument" one though).

I think the <<81>g> part is a sendmail macro, since my sendmail.cf file specifies "H?P?Return-Path: <$g>", and the queue file probably needs to replace the "$" so it know the expand that macro when the time comes to do the sending.

I also manually edited one of the queue files, replace the <<81>g> with a valid address, and tried "sendmail -q" (to reprocess my queue) and the queued message still remained there.

I remain stumped !
0
 
LVL 34

Expert Comment

by:PsiCop
ID: 12421685
Yes, the "H?x?" and "<<81>g>" strings are part of sendmail's internal message format. Don't worry about it.

This draws my attention:

Reporting-MTA: dns; otherdomain.com
Arrival-Date: Wed, 20 Oct 2004 23:13:02 GMT

This suggests to me that your mailserver's traffic is being rejected by the "otherdomain.com" mailserver. It either doesn't like something your mailserver is saying, or it is misconfigured and is therefore rejecting you.

If you are able to send E-Mail to other places without any problems, I would tend to think the problem is on the "otherdomain.com" mailserver.
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12427883
Sorry to not explain the domains better:

maindomain.com is the main domain my server sends mails out from; here's what's in my sendmail.cf file:

# my name for error messages
DnMAILER-DAEMON@maindomain.com

I specifically set this because some broken mailers won't accept "MAIL FROM: <>" and I absolutely must get bounces back to all senders.

otherdomain.com is the same server; it's the "main name" that sendmail runs under; what gets added on the end of "root@" if you just type the "Mail" command in a shell.  Here's my "hosts" file, which I guess is where this comes from (not my real IP tho):-

127.0.0.1       localhost otherdomain.com localhost.localdomain localhost
208.285.243.123 otherdomain.com

I run a bunch of other virtual domains, and it's necessary that my server uses "otherdomain" so that customers don't get confused by headers and such.

I think you're correct (in a roundabout way) in suggesting the problem is on the "otherdomain.com" mailserver... that's my mailserver, and I reccon the problem is likley there: If only we could work out what it is!!

Is there a debugging mode where sendmail can run and output a detailed log of everything it's trying to do?  Maybe I can shut down sendmail, restart it in this mode, tell it to process my mailqueue, and see what the heck it's trying to do with all those queued things.  Maybe if we had some idea what, exactly, this "argument" is that's invalid we'd be some ways to working out what to change :-)

Hmm. Maybe I could search the source code to see what kinds of things produce that error?
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 
LVL 2

Author Comment

by:ChrisDrake
ID: 12427921
Hmm. The mystery deepens - no "Invalid argument" in the binary nor source code.

[cnd@my sendmail-8.12.10]$ pwd
/usr/src/redhat/BUILD/sendmail-8.12.10
[cnd@my sendmail-8.12.10]$ grep -r  'Invalid argument' /usr/sbin/sendmail
[cnd@my sendmail-8.12.10]$ grep -r 'Invalid argument' .
./RELEASE_NOTES:                like "Cannot create qfXXXXXX: Invalid argument"
./libsmutil/lockfile.c: **  previous fcntl will fail with "Invalid argument" errors.
./sendmail/conf.c:      **  previous fcntl will fail with "Invalid argument" errors.
./libsmdb/smdb.c:       **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-4.ELsmp.i686/libsmutil/lockfile.c:   **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-4.ELsmp.i686/sendmail/conf.c:        **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-4.ELsmp.i686/libsmdb/smdb.c: **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-15.ELsmp.i686/sendmail/conf.c:       **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-15.ELsmp.i686/libsmutil/lockfile.c:  **  previous fcntl will fail with "Invalid argument" errors.
[cnd@my sendmail-8.12.10]$
0
 
LVL 34

Expert Comment

by:PsiCop
ID: 12433714
Hmmm.... I'm thinking it might have to do not with the format of the E-Mail, but with how sendmail is called when it is given the E-mail.

How is this E-Mail being submitted to sendmail?
0
 
LVL 6

Expert Comment

by:anfi
ID: 12438044
Could you post the relevant entries from the log file?
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12439909
OK - Here's another one.  The bounce has been sitting in my mailq for some time now.  The original email was sent like this:-

/usr/sbin/sendmail -i -N delay,failure -R full -f "k***@***teams.com" -- "y***@scrr.com"

However, the scrr.com mailserver is not working, so the original sat for 5 days until sendmail gave up on it, and since then, the bounce has been stuck in my queue.

Here's the log file from the original.

Oct 22 03:46:57 my sendmail[5379]: i9M3kuf3005379: to=y***@scrr.com, ctladdr=k***@***teams.com (8/12), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=39575, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (i9M3kvDn005383 Message accepted for delivery)
Oct 22 03:52:57 my sendmail[5385]: i9M3kvDn005383: to=<y***@scrr.com>, delay=00:06:00, xdelay=00:06:00, mailer=esmtp, pri=39649, relay=scrr.com. [66.218.65.224], dsn=4.0.0, stat=Deferred: Connection timed out with scrr.com.
Oct 22 04:42:51 my sendmail[12228]: i9M3kvDn005383: to=<y***@scrr.com>, delay=00:55:54, xdelay=00:06:00, mailer=esmtp, pri=129649, relay=scrr.com. [66.218.65.226], dsn=4.0.0, stat=Deferred: Connection timed out with scrr.com.
[ lots of entries skipped ]

Then here's what appears to be the bounce that's now stuck in the queue

Oct 28 02:16:41 my sendmail[30551]: i9R3r5vu021108: to=<k***@***teams.com>, delay=20:59:17, xdelay=00:00:01, mailer=esmtp, pri=1844510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument
Oct 28 05:16:03 my sendmail[30813]: i9R3r5vu021108: to=<k***@***teams.com>, delay=23:58:39, xdelay=00:00:00, mailer=esmtp, pri=1934510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument

By coincidence, something unrelated was actually mailled directly to k***@***teams.com - so here's the log entries showing that this email address is receiving OK, followed by sendmail failing to deliver the bounce again:-

Oct 28 21:27:18 my sendmail[21800]: i9SLRGPH021800: to=k***@***teams.com, ctladdr=help@***mydomain.com (0/0), delay=00:00:02, xdelay=00:00:00, mailer=relay, pri=51754, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (i9SLRIvq021809 Message accepted for delivery)
Oct 28 21:27:24 my sendmail[21811]: i9SLRIvq021809: to=<k***@***teams.com>, delay=00:00:06, xdelay=00:00:06, mailer=esmtp, pri=51951, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=2.0.0, stat=Sent (4-0828838842 Message accepted for delivery)
Oct 28 21:46:25 my sendmail[13895]: i9R3r5vu021108: to=<k***@***teams.com>, delay=1+16:29:01, xdelay=00:00:00, mailer=esmtp, pri=3284510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument

I'm wondering if maybe it's some kind of obscure bug inside of the sendmail I'm running perhaps?

[root@my mqueue]# Oct 28 23:34:56 my sm-msp-queue[15030]: starting daemon (8.12.10):
0
 
LVL 6

Accepted Solution

by:
anfi earned 2000 total points
ID: 12441830
Make sendmail deliver the stalled message in verbode mode. Execute:
   sendmail -v -qI_queueu_id_
e.g.
   sendmail -v -qIi9R3r5vu021108

One possible explanation: "***teams.com" server is misconfigure (or broken) and does not accept bounce messages (messages with envelope sender set to <>).
It is rare but happens. Such doamins are listed in DSN list at http://rfc-ignorant.org/
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12503370
Hi anfi - great idea... it's given me a clue to try later (modifying my DNS so I can force sendmail to connect instead to a proxy so I can see what is being said by whom).

Here's what I just tried (verbatim - no alterations) - but I've told my sendmail never to use <> already because I need my DSN's to get back to the sender more than I care about my postmaster getting bounced DSN's :-)

[root@my root]# sendmail -v -qIiA4Mpfao009544

Running /var/spool/mqueue/iA4Mpfao009544 (sequence 1 of 1)
<team@eigenbaaszijn.nl>... Connecting to mail.eigenbaaszijn.nl. via esmtp...
<team@eigenbaaszijn.nl>... Deferred: Invalid argument
[root@my root]# telnet mail.eigenbaaszijn.nl 25
Trying 212.123.1.45...
Connected to hosted.by.hostbasket.com (212.123.1.45).
Escape character is '^]'.
220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 12:52:01 +0100
EHLO emsvr.com
250-webcommerce.be Hello [208.185.243.103] Pleased to meet you
250-ETRN
250-SIZE
250-AUTH LOGIN
250-AUTH=LOGIN
250 HELP
MAIL FROM: <>
250 <>... Sender ok
RCPT TO: <team@eigenbaaszijn.nl>
250 Recipient <team@eigenbaaszijn.nl> Ok
DATA
354 Enter Mail, end with "." on a line by itself
^]q

Connection closed.
[root@my root]#
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12503464
*INTERESTING*

I edited the "q" file, adding a suffix for a domain I own to the recipient email address in order to force sendmail to connect to a different machine.  On the different machine, I ran a proxy on port 25 which just echoed everything to the real SMTP destination.  Guess what?  My sendmail is not even iniating a connection out to the remote machine!! [I confirmed this by noticing that it fails immediately on all my "Deferred" emails - "immediately" is too fast - some of them take a long time to accept the connection]

[root@my etc]# sendmail -v -qIiA4Mpfao009544    

Running /var/spool/mqueue/iA4Mpfao009544 (sequence 1 of 1)
<team@eigenbaaszijn.nl.boatcode.com>... Connecting to eigenbaaszijn.nl.boatcode.com. via esmtp...
<team@eigenbaaszijn.nl.boatcode.com>... Connecting to copa.geek.net.au. via esmtp...
<team@eigenbaaszijn.nl.boatcode.com>... Deferred: Invalid argument

(on my proxy machine - no connection ever arrived)

[root@my etc]# telnet eigenbaaszijn.nl.boatcode.com 25    
Trying 203.217.18.13...
Connected to copa.geek.net.au (203.217.18.13).
Escape character is '^]'.
220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 13:09:35 +0100
QUIT
221 webcommerce.be closing connection
Connection closed by foreign host.
[root@my etc]#

(This test connection did work on my proxy of course) - here's the proxy's output:-

# perl SuperProx.pl -client 203.217.18.13:25 -server 212.123.1.45:25 -open -strip .boatcode.com
Connected to 203.217.18.13:25 for client connections thru 212.123.1.45:25
Connected to server 212.123.1.45:25
Accepted client 208.185.243.123:47625
220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 13:09:35 +0100\r\n

QUIT\r\n

221 webcommerce.be closing connection\r\n

server  closed connection!  
#

This is interesting too: it thinks it's tried all of hotmails' MX's (again - it didn't - since this came up instantly with no delay - hotmail just aint that fast...)

[root@my mqueue]# sendmail -v -qIiA4Fpfao008039

Running /var/spool/mqueue/iA4Fpfao008039 (sequence 1 of 1)
<jerryalaimo@msn.com>... Connecting to mx3.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Connecting to mx1.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Connecting to mx2.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Connecting to mx4.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Deferred: Invalid argument

Could it be tring to use SSL or some other port besides 25 maybe?
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12503535
***BINGO***

Here's the offending line:-

${daemon_flags}b h

Changed it to:-

${daemon_flags}

... and it went out without any problems.

Now - I just gotta track down where the heck it's getting that "b h" from :-)
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12503574
*** FIXED *** (I hope)

I found this in my sendmail.cf

O DaemonPortOptions=Port=smtp,Addr=208.185.243.123, Name=MTA, Modifiers=bh

After noticing the the "b" was the cause of the problem, and I found this in the doco:-

Flag 'b' = bind to interface through which mail has been received for outgoing message.

Now - since these are all bounces, I guess it was trying to reach the internet after binding to 127.0.0.1.
0
 
LVL 6

Expert Comment

by:anfi
ID: 12503601
:-)
Alternatively You may configure your submit.cf to relay locally submitted messages messages to public IP insted of 127.0.0.1
0
 
LVL 2

Author Comment

by:ChrisDrake
ID: 12503837
Would this be the submit.cf line I need to change for that?

D{MTAHost}[127.0.0.1]
0
 
LVL 6

Expert Comment

by:anfi
ID: 12504024
In submit.mc change:
   FEATURE(`msp', `[127.0.0.1]')dnl
to IP address of you public interface
   FEATURE(`msp', `[aaa.bbb.ccc.ddd]')dnl
and generate new submit.cf

It may create problems with delivering local mail when the interface is down.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

One-stop solution for Exchange Administrators to address all MS Exchange Server issues, which is known by the name of Stellar Exchange Toolkit.
Here in this article, you will get a step by step guidance on how to restore an Exchange database to a recovery database. Get a brief on Recovery Database and how it can be used to restore Exchange database in this section!
In this Micro Video tutorial you will learn the basics about Database Availability Groups and How to configure one using a live Exchange Server Environment. The video tutorial explains the basics of the Exchange server Database Availability grou…
The video tutorial explains the basics of the Exchange server Database Availability groups. The components of this video include: 1. Automatic Failover 2. Failover Clustering 3. Active Manager
Suggested Courses

650 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question