Solved

How to solve "Deferred: Invalid argument" problems in sendmail (bounces bouncing)

Posted on 2004-10-27
4,765 Views
Last Modified: 2013-12-17
I have a constant large number of items in my sendmail queue, most of them containing "Deferred: Invalid argument" in the "q" file.

I know most of these should be deliverable, but it seems sendmail is somehow sending something stupid to the receiving server (if it's even trying that at all) or something else, because these sit here for 5 days then hit my "postmaster" address.

It seems the majority of these emails are bounces.

In other words - my mail server is having problems delivering some bounces, so they sit for 5 days and then bounce into my postmaster account.

I notice that they all have a strange line: H?P?Return-Path: <g>

In case you can't see it - there's a strange character before the "g" (in "vi" it shows up as <<81>g>). This may or may not be a problem? I've no idea where it came from.

Below is an example file (domain names have been changed to protect against spam, but otherwise it's unedited).  The RPF line is a valid email address (well, it was before I obscured it to protect against spam).  It happens to be my own address, on a different server running the same version of sendmail as the server having the problem sending it.

Any help, clues, hints, advice, or anything you can provide would be most welcome!!!


[root@my mqueue]# pwd
/var/spool/mqueue
[root@my mqueue]# grep -l "Deferred: Invalid argument" * |wc
     39      39     663

[root@my mqueue]# dir *fi9PMqwvt019196  
-rw-------    1 root     smmsp        1414 Oct 25 23:59 dfi9PMqwvt019196
-rw-------    1 root     smmsp         896 Oct 27 08:42 qfi9PMqwvt019196
[root@my mqueue]#
[root@my mqueue]# cat qfi9PMqwvt019196
V6
T1098748757
K1098866550
N33
P2970000
I104/2/9831242
MDeferred: Invalid argument
Frns
$_localhost
$r
$slocalhost
${daemon_flags}b h
${if_addr}127.0.0.1
SMAILER-DAEMON@maindomain.com
MDeferred: Invalid argument
Ccnd:500:500:cnd
rRFC822; postmaster@otherdomain.com
RPF:"cnd@recipient.net.au"
H?P?Return-Path: <g>
H??Received: from localhost (localhost)
        by otherdomain.com (8.12.10/8.12.10) id i9PMqwvt019196;
        Mon, 25 Oct 2004 23:59:17 GMT
H?D?Date: Mon, 25 Oct 2004 23:59:17 GMT
H?F?From: Mail Delivery Subsystem <MAILER-DAEMON@maindomain.com>
H?x?Full-Name: Mail Delivery Subsystem
H?M?Message-Id: <200410252359.i9PMqwvt019196@otherdomain.com>
H??To: postmaster
H??MIME-Version: 1.0
H??Content-Type: multipart/report; report-type=delivery-status;
        boundary="i9PMqwvt019196.1098748757/otherdomain.com"
H??Subject: Postmaster notify: see transcript for details
H??Auto-Submitted: auto-generated (postmaster-notification)
.
[root@my mqueue]#
[root@my mqueue]# cat dfi9PMqwvt019196
This is a MIME-encapsulated message

--i9PMqwvt019196.1098748757/otherdomain.com

The original message was received at Wed, 20 Oct 2004 23:13:02 GMT
from localhost
with id i9KMWfDp019382

   ----- The following addresses had permanent fatal errors -----
<jamas@xyzwebmail.net>

   ----- Transcript of session follows -----
<jamas@xyzwebmail.net>... Deferred: Invalid argument
Message could not be delivered for 5 days
Message will be deleted from queue

--i9PMqwvt019196.1098748757/otherdomain.com
Content-Type: message/delivery-status

Reporting-MTA: dns; otherdomain.com
Arrival-Date: Wed, 20 Oct 2004 23:13:02 GMT

Final-Recipient: RFC822; jamas@xyzwebmail.net
Action: failed
Status: 4.4.7
Remote-MTA: DNS; ns3.xyzwebmail.com
Last-Attempt-Date: Mon, 25 Oct 2004 23:59:17 GMT

--i9PMqwvt019196.1098748757/otherdomain.com
Content-Type: text/rfc822-headers

Return-Path: <MAILER-DAEMON@maindomain.com>
Received: from localhost (localhost)
        by otherdomain.com (8.12.10/8.12.10) id i9KMWfDp019382;
        Wed, 20 Oct 2004 23:13:02 GMT
Date: Wed, 20 Oct 2004 23:13:02 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON@maindomain.com>
Message-Id: <200410202313.i9KMWfDp019382@otherdomain.com>
To: <jamas@xyzwebmail.net>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="i9KMWfDp019382.1098313982/otherdomain.com"
Subject: Return receipt
Auto-Submitted: auto-generated (return-receipt)

--i9PMqwvt019196.1098748757/otherdomain.com--

[root@my mqueue]#
0
Question by:ChrisDrake
    15 Comments
     
    LVL 2

    Author Comment

    by:ChrisDrake
    My problem still remains, but I've investigated the "H?P?Return-Path: <g>" situation a bit more and it looks like that is a red herring: I've found some normal queue messages with the same thing that do go through OK (not "Invalid argument" one though).

    I think the <<81>g> part is a sendmail macro, since my sendmail.cf file specifies "H?P?Return-Path: <$g>", and the queue file probably needs to replace the "$" so it know the expand that macro when the time comes to do the sending.

    I also manually edited one of the queue files, replace the <<81>g> with a valid address, and tried "sendmail -q" (to reprocess my queue) and the queued message still remained there.

    I remain stumped !
    0
     
    LVL 34

    Expert Comment

    by:PsiCop
    Yes, the "H?x?" and "<<81>g>" strings are part of sendmail's internal message format. Don't worry about it.

    This draws my attention:

    Reporting-MTA: dns; otherdomain.com
    Arrival-Date: Wed, 20 Oct 2004 23:13:02 GMT

    This suggests to me that your mailserver's traffic is being rejected by the "otherdomain.com" mailserver. It either doesn't like something your mailserver is saying, or it is misconfigured and is therefore rejecting you.

    If you are able to send E-Mail to other places without any problems, I would tend to think the problem is on the "otherdomain.com" mailserver.
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    Sorry to not explain the domains better:

    maindomain.com is the main domain my server sends mails out from; here's what's in my sendmail.cf file:

    # my name for error messages
    DnMAILER-DAEMON@maindomain.com

    I specifically set this because some broken mailers won't accept "MAIL FROM: <>" and I absolutely must get bounces back to all senders.

    otherdomain.com is the same server; it's the "main name" that sendmail runs under; what gets added on the end of "root@" if you just type the "Mail" command in a shell.  Here's my "hosts" file, which I guess is where this comes from (not my real IP tho):-

    127.0.0.1       localhost otherdomain.com localhost.localdomain localhost
    208.285.243.123 otherdomain.com

    I run a bunch of other virtual domains, and it's necessary that my server uses "otherdomain" so that customers don't get confused by headers and such.

    I think you're correct (in a roundabout way) in suggesting the problem is on the "otherdomain.com" mailserver... that's my mailserver, and I reccon the problem is likley there: If only we could work out what it is!!

    Is there a debugging mode where sendmail can run and output a detailed log of everything it's trying to do?  Maybe I can shut down sendmail, restart it in this mode, tell it to process my mailqueue, and see what the heck it's trying to do with all those queued things.  Maybe if we had some idea what, exactly, this "argument" is that's invalid we'd be some ways to working out what to change :-)

    Hmm. Maybe I could search the source code to see what kinds of things produce that error?
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    Hmm. The mystery deepens - no "Invalid argument" in the binary nor source code.

    [cnd@my sendmail-8.12.10]$ pwd
    /usr/src/redhat/BUILD/sendmail-8.12.10
    [cnd@my sendmail-8.12.10]$ grep -r  'Invalid argument' /usr/sbin/sendmail
    [cnd@my sendmail-8.12.10]$ grep -r 'Invalid argument' .
    ./RELEASE_NOTES:                like "Cannot create qfXXXXXX: Invalid argument"
    ./libsmutil/lockfile.c: **  previous fcntl will fail with "Invalid argument" errors.
    ./sendmail/conf.c:      **  previous fcntl will fail with "Invalid argument" errors.
    ./libsmdb/smdb.c:       **  previous fcntl will fail with "Invalid argument" errors.
    ./obj.Linux.2.4.21-4.ELsmp.i686/libsmutil/lockfile.c:   **  previous fcntl will fail with "Invalid argument" errors.
    ./obj.Linux.2.4.21-4.ELsmp.i686/sendmail/conf.c:        **  previous fcntl will fail with "Invalid argument" errors.
    ./obj.Linux.2.4.21-4.ELsmp.i686/libsmdb/smdb.c: **  previous fcntl will fail with "Invalid argument" errors.
    ./obj.Linux.2.4.21-15.ELsmp.i686/sendmail/conf.c:       **  previous fcntl will fail with "Invalid argument" errors.
    ./obj.Linux.2.4.21-15.ELsmp.i686/libsmutil/lockfile.c:  **  previous fcntl will fail with "Invalid argument" errors.
    [cnd@my sendmail-8.12.10]$
    0
     
    LVL 34

    Expert Comment

    by:PsiCop
    Hmmm.... I'm thinking it might have to do not with the format of the E-Mail, but with how sendmail is called when it is given the E-mail.

    How is this E-Mail being submitted to sendmail?
    0
     
    LVL 6

    Expert Comment

    by:anfi
    Could you post the relevant entries from the log file?
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    OK - Here's another one.  The bounce has been sitting in my mailq for some time now.  The original email was sent like this:-

    /usr/sbin/sendmail -i -N delay,failure -R full -f "k***@***teams.com" -- "y***@scrr.com"

    However, the scrr.com mailserver is not working, so the original sat for 5 days until sendmail gave up on it, and since then, the bounce has been stuck in my queue.

    Here's the log file from the original.

    Oct 22 03:46:57 my sendmail[5379]: i9M3kuf3005379: to=y***@scrr.com, ctladdr=k***@***teams.com (8/12), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=39575, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (i9M3kvDn005383 Message accepted for delivery)
    Oct 22 03:52:57 my sendmail[5385]: i9M3kvDn005383: to=<y***@scrr.com>, delay=00:06:00, xdelay=00:06:00, mailer=esmtp, pri=39649, relay=scrr.com. [66.218.65.224], dsn=4.0.0, stat=Deferred: Connection timed out with scrr.com.
    Oct 22 04:42:51 my sendmail[12228]: i9M3kvDn005383: to=<y***@scrr.com>, delay=00:55:54, xdelay=00:06:00, mailer=esmtp, pri=129649, relay=scrr.com. [66.218.65.226], dsn=4.0.0, stat=Deferred: Connection timed out with scrr.com.
    [ lots of entries skipped ]

    Then here's what appears to be the bounce that's now stuck in the queue

    Oct 28 02:16:41 my sendmail[30551]: i9R3r5vu021108: to=<k***@***teams.com>, delay=20:59:17, xdelay=00:00:01, mailer=esmtp, pri=1844510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument
    Oct 28 05:16:03 my sendmail[30813]: i9R3r5vu021108: to=<k***@***teams.com>, delay=23:58:39, xdelay=00:00:00, mailer=esmtp, pri=1934510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument

    By coincidence, something unrelated was actually mailled directly to k***@***teams.com - so here's the log entries showing that this email address is receiving OK, followed by sendmail failing to deliver the bounce again:-

    Oct 28 21:27:18 my sendmail[21800]: i9SLRGPH021800: to=k***@***teams.com, ctladdr=help@***mydomain.com (0/0), delay=00:00:02, xdelay=00:00:00, mailer=relay, pri=51754, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (i9SLRIvq021809 Message accepted for delivery)
    Oct 28 21:27:24 my sendmail[21811]: i9SLRIvq021809: to=<k***@***teams.com>, delay=00:00:06, xdelay=00:00:06, mailer=esmtp, pri=51951, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=2.0.0, stat=Sent (4-0828838842 Message accepted for delivery)
    Oct 28 21:46:25 my sendmail[13895]: i9R3r5vu021108: to=<k***@***teams.com>, delay=1+16:29:01, xdelay=00:00:00, mailer=esmtp, pri=3284510, relay=mail-fwd.mx.g19.rapidsite.net. [199.239.254.18], dsn=4.0.0, stat=Deferred: Invalid argument

    I'm wondering if maybe it's some kind of obscure bug inside of the sendmail I'm running perhaps?

    [root@my mqueue]# Oct 28 23:34:56 my sm-msp-queue[15030]: starting daemon (8.12.10):
    0
     
    LVL 6

    Accepted Solution

    by:
    Make sendmail deliver the stalled message in verbode mode. Execute:
       sendmail -v -qI_queueu_id_
    e.g.
       sendmail -v -qIi9R3r5vu021108

    One possible explanation: "***teams.com" server is misconfigure (or broken) and does not accept bounce messages (messages with envelope sender set to <>).
    It is rare but happens. Such doamins are listed in DSN list at http://rfc-ignorant.org/
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    Hi anfi - great idea... it's given me a clue to try later (modifying my DNS so I can force sendmail to connect instead to a proxy so I can see what is being said by whom).

    Here's what I just tried (verbatim - no alterations) - but I've told my sendmail never to use <> already because I need my DSN's to get back to the sender more than I care about my postmaster getting bounced DSN's :-)

    [root@my root]# sendmail -v -qIiA4Mpfao009544

    Running /var/spool/mqueue/iA4Mpfao009544 (sequence 1 of 1)
    <team@eigenbaaszijn.nl>... Connecting to mail.eigenbaaszijn.nl. via esmtp...
    <team@eigenbaaszijn.nl>... Deferred: Invalid argument
    [root@my root]# telnet mail.eigenbaaszijn.nl 25
    Trying 212.123.1.45...
    Connected to hosted.by.hostbasket.com (212.123.1.45).
    Escape character is '^]'.
    220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 12:52:01 +0100
    EHLO emsvr.com
    250-webcommerce.be Hello [208.185.243.103] Pleased to meet you
    250-ETRN
    250-SIZE
    250-AUTH LOGIN
    250-AUTH=LOGIN
    250 HELP
    MAIL FROM: <>
    250 <>... Sender ok
    RCPT TO: <team@eigenbaaszijn.nl>
    250 Recipient <team@eigenbaaszijn.nl> Ok
    DATA
    354 Enter Mail, end with "." on a line by itself
    ^]q

    Connection closed.
    [root@my root]#
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    *INTERESTING*

    I edited the "q" file, adding a suffix for a domain I own to the recipient email address in order to force sendmail to connect to a different machine.  On the different machine, I ran a proxy on port 25 which just echoed everything to the real SMTP destination.  Guess what?  My sendmail is not even iniating a connection out to the remote machine!! [I confirmed this by noticing that it fails immediately on all my "Deferred" emails - "immediately" is too fast - some of them take a long time to accept the connection]

    [root@my etc]# sendmail -v -qIiA4Mpfao009544    

    Running /var/spool/mqueue/iA4Mpfao009544 (sequence 1 of 1)
    <team@eigenbaaszijn.nl.boatcode.com>... Connecting to eigenbaaszijn.nl.boatcode.com. via esmtp...
    <team@eigenbaaszijn.nl.boatcode.com>... Connecting to copa.geek.net.au. via esmtp...
    <team@eigenbaaszijn.nl.boatcode.com>... Deferred: Invalid argument

    (on my proxy machine - no connection ever arrived)

    [root@my etc]# telnet eigenbaaszijn.nl.boatcode.com 25    
    Trying 203.217.18.13...
    Connected to copa.geek.net.au (203.217.18.13).
    Escape character is '^]'.
    220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 13:09:35 +0100
    QUIT
    221 webcommerce.be closing connection
    Connection closed by foreign host.
    [root@my etc]#

    (This test connection did work on my proxy of course) - here's the proxy's output:-

    # perl SuperProx.pl -client 203.217.18.13:25 -server 212.123.1.45:25 -open -strip .boatcode.com
    Connected to 203.217.18.13:25 for client connections thru 212.123.1.45:25
    Connected to server 212.123.1.45:25
    Accepted client 208.185.243.123:47625
    220 webcommerce.be ESMTP MailMax 5.5 Fri, 05 Nov 2004 13:09:35 +0100\r\n

    QUIT\r\n

    221 webcommerce.be closing connection\r\n

    server  closed connection!  
    #

    This is interesting too: it thinks it's tried all of hotmails' MX's (again - it didn't - since this came up instantly with no delay - hotmail just aint that fast...)

    [root@my mqueue]# sendmail -v -qIiA4Fpfao008039

    Running /var/spool/mqueue/iA4Fpfao008039 (sequence 1 of 1)
    <jerryalaimo@msn.com>... Connecting to mx3.hotmail.com. via esmtp...
    <jerryalaimo@msn.com>... Connecting to mx1.hotmail.com. via esmtp...
    <jerryalaimo@msn.com>... Connecting to mx2.hotmail.com. via esmtp...
    <jerryalaimo@msn.com>... Connecting to mx4.hotmail.com. via esmtp...
    <jerryalaimo@msn.com>... Deferred: Invalid argument

    Could it be tring to use SSL or some other port besides 25 maybe?
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    ***BINGO***

    Here's the offending line:-

    ${daemon_flags}b h

    Changed it to:-

    ${daemon_flags}

    ... and it went out without any problems.

    Now - I just gotta track down where the heck it's getting that "b h" from :-)
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    *** FIXED *** (I hope)

    I found this in my sendmail.cf

    O DaemonPortOptions=Port=smtp,Addr=208.185.243.123, Name=MTA, Modifiers=bh

    After noticing the the "b" was the cause of the problem, and I found this in the doco:-

    Flag 'b' = bind to interface through which mail has been received for outgoing message.

    Now - since these are all bounces, I guess it was trying to reach the internet after binding to 127.0.0.1.
    0
     
    LVL 6

    Expert Comment

    by:anfi
    :-)
    Alternatively You may configure your submit.cf to relay locally submitted messages messages to public IP insted of 127.0.0.1
    0
     
    LVL 2

    Author Comment

    by:ChrisDrake
    Would this be the submit.cf line I need to change for that?

    D{MTAHost}[127.0.0.1]
    0
     
    LVL 6

    Expert Comment

    by:anfi
    In submit.mc change:
       FEATURE(`msp', `[127.0.0.1]')dnl
    to IP address of you public interface
       FEATURE(`msp', `[aaa.bbb.ccc.ddd]')dnl
    and generate new submit.cf

    It may create problems with delivering local mail when the interface is down.
    0

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone. Privacy Policy Terms of Use

    Featured Post

    Too many email signature updates to deal with?

    Are you constantly visiting users’ desks making changes to email signatures? Feel like it’s taking up all of your time? Wish you could manage all signatures from one central location, easily design them and deploy them quickly to users? Well, there is an easy way!

    MS outlook is a premier email client that enable you to send and receive the e-mails with various file formats of attachments such as document files, media file, and many others formats. There is some scenario occurs when a receiver of an e-mail mes…
    New-MailboxSearch Powershell Command and step by step approach to Search and Extract Emails form Exchange 2013 Journaling server.
    Familiarize people with the process of utilizing SQL Server stored procedures from within Microsoft Access. Microsoft Access is a very powerful client/server development tool. One of the SQL Server objects that you can interact with from within Micr…
    In this video we show how to create a Contact in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Contact ta…

    875 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    13 Experts available now in Live!

    Get 1:1 Help Now