How to solve "Deferred: Invalid argument" problems in sendmail (bounces bouncing)

I have a constant large number of items in my sendmail queue, most of them containing "Deferred: Invalid argument" in the "q" file.

I know most of these should be deliverable, but it seems sendmail is somehow sending something stupid to the receiving server (if it's even trying that at all) or something else, because these sit here for 5 days then hit my "postmaster" address.

It seems the majority of these emails are bounces.

In other words - my mail server is having problems delivering some bounces, so they sit for 5 days and then bounce into my postmaster account.

I notice that they all have a strange line: H?P?Return-Path: <g>

In case you can't see it - there's a strange character before the "g" (in "vi" it shows up as <<81>g>). This may or may not be a problem? I've no idea where it came from.

Below is an example file (domain names have been changed to protect against spam, but otherwise it's unedited).  The RPF line is a valid email address (well, it was before I obscured it to protect against spam).  It happens to be my own address, on a different server running the same version of sendmail as the server having the problem sending it.

Any help, clues, hints, advice, or anything you can provide would be most welcome!!!


[root@my mqueue]# pwd
/var/spool/mqueue
[root@my mqueue]# grep -l "Deferred: Invalid argument" * |wc
     39      39     663

[root@my mqueue]# dir *fi9PMqwvt019196  
-rw-------    1 root     smmsp        1414 Oct 25 23:59 dfi9PMqwvt019196
-rw-------    1 root     smmsp         896 Oct 27 08:42 qfi9PMqwvt019196
[root@my mqueue]#
[root@my mqueue]# cat qfi9PMqwvt019196
V6
T1098748757
K1098866550
N33
P2970000
I104/2/9831242
MDeferred: Invalid argument
Frns
$_localhost
$r
$slocalhost
${daemon_flags}b h
${if_addr}127.0.0.1
SMAILER-DAEMON@maindomain.com
MDeferred: Invalid argument
Ccnd:500:500:cnd
rRFC822; postmaster@otherdomain.com
RPF:"cnd@recipient.net.au"
H?P?Return-Path: <g>
H??Received: from localhost (localhost)
        by otherdomain.com (8.12.10/8.12.10) id i9PMqwvt019196;
        Mon, 25 Oct 2004 23:59:17 GMT
H?D?Date: Mon, 25 Oct 2004 23:59:17 GMT
H?F?From: Mail Delivery Subsystem <MAILER-DAEMON@maindomain.com>
H?x?Full-Name: Mail Delivery Subsystem
H?M?Message-Id: <200410252359.i9PMqwvt019196@otherdomain.com>
H??To: postmaster
H??MIME-Version: 1.0
H??Content-Type: multipart/report; report-type=delivery-status;
        boundary="i9PMqwvt019196.1098748757/otherdomain.com"
H??Subject: Postmaster notify: see transcript for details
H??Auto-Submitted: auto-generated (postmaster-notification)
.
[root@my mqueue]#
[root@my mqueue]# cat dfi9PMqwvt019196
This is a MIME-encapsulated message

--i9PMqwvt019196.1098748757/otherdomain.com

The original message was received at Wed, 20 Oct 2004 23:13:02 GMT
from localhost
with id i9KMWfDp019382

   ----- The following addresses had permanent fatal errors -----
<jamas@xyzwebmail.net>

   ----- Transcript of session follows -----
<jamas@xyzwebmail.net>... Deferred: Invalid argument
Message could not be delivered for 5 days
Message will be deleted from queue

--i9PMqwvt019196.1098748757/otherdomain.com
Content-Type: message/delivery-status

Reporting-MTA: dns; otherdomain.com
Arrival-Date: Wed, 20 Oct 2004 23:13:02 GMT

Final-Recipient: RFC822; jamas@xyzwebmail.net
Action: failed
Status: 4.4.7
Remote-MTA: DNS; ns3.xyzwebmail.com
Last-Attempt-Date: Mon, 25 Oct 2004 23:59:17 GMT

--i9PMqwvt019196.1098748757/otherdomain.com
Content-Type: text/rfc822-headers

Return-Path: <MAILER-DAEMON@maindomain.com>
Received: from localhost (localhost)
        by otherdomain.com (8.12.10/8.12.10) id i9KMWfDp019382;
        Wed, 20 Oct 2004 23:13:02 GMT
Date: Wed, 20 Oct 2004 23:13:02 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON@maindomain.com>
Message-Id: <200410202313.i9KMWfDp019382@otherdomain.com>
To: <jamas@xyzwebmail.net>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="i9KMWfDp019382.1098313982/otherdomain.com"
Subject: Return receipt
Auto-Submitted: auto-generated (return-receipt)

--i9PMqwvt019196.1098748757/otherdomain.com--

[root@my mqueue]#
LVL 2
ChrisDrakeAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ChrisDrakeAuthor Commented:
My problem still remains, but I've investigated the "H?P?Return-Path: <g>" situation a bit more and it looks like that is a red herring: I've found some normal queue messages with the same thing that do go through OK (not "Invalid argument" one though).

I think the <<81>g> part is a sendmail macro, since my sendmail.cf file specifies "H?P?Return-Path: <$g>", and the queue file probably needs to replace the "$" so it know the expand that macro when the time comes to do the sending.

I also manually edited one of the queue files, replace the <<81>g> with a valid address, and tried "sendmail -q" (to reprocess my queue) and the queued message still remained there.

I remain stumped !
0
PsiCopCommented:
Yes, the "H?x?" and "<<81>g>" strings are part of sendmail's internal message format. Don't worry about it.

This draws my attention:

Reporting-MTA: dns; otherdomain.com
Arrival-Date: Wed, 20 Oct 2004 23:13:02 GMT

This suggests to me that your mailserver's traffic is being rejected by the "otherdomain.com" mailserver. It either doesn't like something your mailserver is saying, or it is misconfigured and is therefore rejecting you.

If you are able to send E-Mail to other places without any problems, I would tend to think the problem is on the "otherdomain.com" mailserver.
0
ChrisDrakeAuthor Commented:
Sorry to not explain the domains better:

maindomain.com is the main domain my server sends mails out from; here's what's in my sendmail.cf file:

# my name for error messages
DnMAILER-DAEMON@maindomain.com

I specifically set this because some broken mailers won't accept "MAIL FROM: <>" and I absolutely must get bounces back to all senders.

otherdomain.com is the same server; it's the "main name" that sendmail runs under; what gets added on the end of "root@" if you just type the "Mail" command in a shell.  Here's my "hosts" file, which I guess is where this comes from (not my real IP tho):-

127.0.0.1       localhost otherdomain.com localhost.localdomain localhost
208.285.243.123 otherdomain.com

I run a bunch of other virtual domains, and it's necessary that my server uses "otherdomain" so that customers don't get confused by headers and such.

I think you're correct (in a roundabout way) in suggesting the problem is on the "otherdomain.com" mailserver... that's my mailserver, and I reccon the problem is likley there: If only we could work out what it is!!

Is there a debugging mode where sendmail can run and output a detailed log of everything it's trying to do?  Maybe I can shut down sendmail, restart it in this mode, tell it to process my mailqueue, and see what the heck it's trying to do with all those queued things.  Maybe if we had some idea what, exactly, this "argument" is that's invalid we'd be some ways to working out what to change :-)

Hmm. Maybe I could search the source code to see what kinds of things produce that error?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

ChrisDrakeAuthor Commented:
Hmm. The mystery deepens - no "Invalid argument" in the binary nor source code.

[cnd@my sendmail-8.12.10]$ pwd
/usr/src/redhat/BUILD/sendmail-8.12.10
[cnd@my sendmail-8.12.10]$ grep -r  'Invalid argument' /usr/sbin/sendmail
[cnd@my sendmail-8.12.10]$ grep -r 'Invalid argument' .
./RELEASE_NOTES:                like "Cannot create qfXXXXXX: Invalid argument"
./libsmutil/lockfile.c: **  previous fcntl will fail with "Invalid argument" errors.
./sendmail/conf.c:      **  previous fcntl will fail with "Invalid argument" errors.
./libsmdb/smdb.c:       **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-4.ELsmp.i686/libsmutil/lockfile.c:   **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-4.ELsmp.i686/sendmail/conf.c:        **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-4.ELsmp.i686/libsmdb/smdb.c: **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-15.ELsmp.i686/sendmail/conf.c:       **  previous fcntl will fail with "Invalid argument" errors.
./obj.Linux.2.4.21-15.ELsmp.i686/libsmutil/lockfile.c:  **  previous fcntl will fail with "Invalid argument" errors.
[cnd@my sendmail-8.12.10]$
0
PsiCopCommented:
Hmmm.... I'm thinking it might have to do not with the format of the E-Mail, but with how sendmail is called when it is given the E-mail.

How is this E-Mail being submitted to sendmail?
0
anfiCommented:
Could you post the relevant entries from the log file?
0
ChrisDrakeAuthor Commented:
OK - Here's another one.  The bounce has been sitting in my mailq for some time now.  The original email was sent like this:-

/usr/sbin/sendmail -i -N delay,failure -R full -f "k***@***teams.com" -- "y***@scrr.com"

However, the scrr.com mailserver is not working, so the original sat for 5 days until sendmail gave up on it, and since then, the bounce has been stuck in my queue.

Here's the log file from the original.

Oct 22 03:46:57 my sendmail[5379]: i9M3kuf3005379: to=y***@scrr.com, ctladdr=k***@***teams.com (8/12), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=39575, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (i9M3kvDn005383 Message accepted for delivery)
Oct 22 03:52:57 my sendmail[5385]: i9M3kvDn005383: to=<y***@scrr.com>, delay=00:06:00, xdelay=00:06:00, mailer=esmtp, pri=39649, relay=scrr.com. [66.218.65.224], dsn=4.0.0, stat=Deferred: Connection timed out with scrr.com.
Oct 22 04:42:51 my sendmail[12228]: i9M3kvDn005383: to=<y***@scrr.com>, delay=00:55:54, xdelay=00:06:00, mailer=esmtp, pri=129649, relay=scrr.com. [66.218.65.226], dsn=4.0.0, stat=Deferred: Connection timed out with scrr.com.
[ lots of entries skipped ]

Then here's what appears to be the bounce that's now stuck in the queue

Oct 28 02:16:41 my sendmail[30551]: i9R3r5vu021108: to=<k***@***teams.com>, delay=20:59:17, xdelay=00:00:01, mailer=esmtp, pri=1844510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument
Oct 28 05:16:03 my sendmail[30813]: i9R3r5vu021108: to=<k***@***teams.com>, delay=23:58:39, xdelay=00:00:00, mailer=esmtp, pri=1934510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument

By coincidence, something unrelated was actually mailled directly to k***@***teams.com - so here's the log entries showing that this email address is receiving OK, followed by sendmail failing to deliver the bounce again:-

Oct 28 21:27:18 my sendmail[21800]: i9SLRGPH021800: to=k***@***teams.com, ctladdr=help@***mydomain.com (0/0), delay=00:00:02, xdelay=00:00:00, mailer=relay, pri=51754, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (i9SLRIvq021809 Message accepted for delivery)
Oct 28 21:27:24 my sendmail[21811]: i9SLRIvq021809: to=<k***@***teams.com>, delay=00:00:06, xdelay=00:00:06, mailer=esmtp, pri=51951, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=2.0.0, stat=Sent (4-0828838842 Message accepted for delivery)
Oct 28 21:46:25 my sendmail[13895]: i9R3r5vu021108: to=<k***@***teams.com>, delay=1+16:29:01, xdelay=00:00:00, mailer=esmtp, pri=3284510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument

I'm wondering if maybe it's some kind of obscure bug inside of the sendmail I'm running perhaps?

[root@my mqueue]# Oct 28 23:34:56 my sm-msp-queue[15030]: starting daemon (8.12.10):
0
anfiCommented:
Make sendmail deliver the stalled message in verbode mode. Execute:
   sendmail -v -qI_queueu_id_
e.g.
   sendmail -v -qIi9R3r5vu021108

One possible explanation: "***teams.com" server is misconfigure (or broken) and does not accept bounce messages (messages with envelope sender set to <>).
It is rare but happens. Such doamins are listed in DSN list at http://rfc-ignorant.org/
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ChrisDrakeAuthor Commented:
Hi anfi - great idea... it's given me a clue to try later (modifying my DNS so I can force sendmail to connect instead to a proxy so I can see what is being said by whom).

Here's what I just tried (verbatim - no alterations) - but I've told my sendmail never to use <> already because I need my DSN's to get back to the sender more than I care about my postmaster getting bounced DSN's :-)

[root@my root]# sendmail -v -qIiA4Mpfao009544

Running /var/spool/mqueue/iA4Mpfao009544 (sequence 1 of 1)
<team@eigenbaaszijn.nl>... Connecting to mail.eigenbaaszijn.nl. via esmtp...
<team@eigenbaaszijn.nl>... Deferred: Invalid argument
[root@my root]# telnet mail.eigenbaaszijn.nl 25
Trying 212.123.1.45...
Connected to hosted.by.hostbasket.com (212.123.1.45).
Escape character is '^]'.
220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 12:52:01 +0100
EHLO emsvr.com
250-webcommerce.be Hello [208.185.243.103] Pleased to meet you
250-ETRN
250-SIZE
250-AUTH LOGIN
250-AUTH=LOGIN
250 HELP
MAIL FROM: <>
250 <>... Sender ok
RCPT TO: <team@eigenbaaszijn.nl>
250 Recipient <team@eigenbaaszijn.nl> Ok
DATA
354 Enter Mail, end with "." on a line by itself
^]q

Connection closed.
[root@my root]#
0
ChrisDrakeAuthor Commented:
*INTERESTING*

I edited the "q" file, adding a suffix for a domain I own to the recipient email address in order to force sendmail to connect to a different machine.  On the different machine, I ran a proxy on port 25 which just echoed everything to the real SMTP destination.  Guess what?  My sendmail is not even iniating a connection out to the remote machine!! [I confirmed this by noticing that it fails immediately on all my "Deferred" emails - "immediately" is too fast - some of them take a long time to accept the connection]

[root@my etc]# sendmail -v -qIiA4Mpfao009544    

Running /var/spool/mqueue/iA4Mpfao009544 (sequence 1 of 1)
<team@eigenbaaszijn.nl.boatcode.com>... Connecting to eigenbaaszijn.nl.boatcode.com. via esmtp...
<team@eigenbaaszijn.nl.boatcode.com>... Connecting to copa.geek.net.au. via esmtp...
<team@eigenbaaszijn.nl.boatcode.com>... Deferred: Invalid argument

(on my proxy machine - no connection ever arrived)

[root@my etc]# telnet eigenbaaszijn.nl.boatcode.com 25    
Trying 203.217.18.13...
Connected to copa.geek.net.au (203.217.18.13).
Escape character is '^]'.
220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 13:09:35 +0100
QUIT
221 webcommerce.be closing connection
Connection closed by foreign host.
[root@my etc]#

(This test connection did work on my proxy of course) - here's the proxy's output:-

# perl SuperProx.pl -client 203.217.18.13:25 -server 212.123.1.45:25 -open -strip .boatcode.com
Connected to 203.217.18.13:25 for client connections thru 212.123.1.45:25
Connected to server 212.123.1.45:25
Accepted client 208.185.243.123:47625
220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 13:09:35 +0100\r\n

QUIT\r\n

221 webcommerce.be closing connection\r\n

server  closed connection!  
#

This is interesting too: it thinks it's tried all of hotmails' MX's (again - it didn't - since this came up instantly with no delay - hotmail just aint that fast...)

[root@my mqueue]# sendmail -v -qIiA4Fpfao008039

Running /var/spool/mqueue/iA4Fpfao008039 (sequence 1 of 1)
<jerryalaimo@msn.com>... Connecting to mx3.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Connecting to mx1.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Connecting to mx2.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Connecting to mx4.hotmail.com. via esmtp...
<jerryalaimo@msn.com>... Deferred: Invalid argument

Could it be tring to use SSL or some other port besides 25 maybe?
0
ChrisDrakeAuthor Commented:
***BINGO***

Here's the offending line:-

${daemon_flags}b h

Changed it to:-

${daemon_flags}

... and it went out without any problems.

Now - I just gotta track down where the heck it's getting that "b h" from :-)
0
ChrisDrakeAuthor Commented:
*** FIXED *** (I hope)

I found this in my sendmail.cf

O DaemonPortOptions=Port=smtp,Addr=208.185.243.123, Name=MTA, Modifiers=bh

After noticing the the "b" was the cause of the problem, and I found this in the doco:-

Flag 'b' = bind to interface through which mail has been received for outgoing message.

Now - since these are all bounces, I guess it was trying to reach the internet after binding to 127.0.0.1.
0
anfiCommented:
:-)
Alternatively You may configure your submit.cf to relay locally submitted messages messages to public IP insted of 127.0.0.1
0
ChrisDrakeAuthor Commented:
Would this be the submit.cf line I need to change for that?

D{MTAHost}[127.0.0.1]
0
anfiCommented:
In submit.mc change:
   FEATURE(`msp', `[127.0.0.1]')dnl
to IP address of you public interface
   FEATURE(`msp', `[aaa.bbb.ccc.ddd]')dnl
and generate new submit.cf

It may create problems with delivering local mail when the interface is down.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Email Servers

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.