i am not sure how to fix this problem

The bounce message excerpt just says that flanders relayed through mail, it doesn't say why it was rejected at the receiving end.

I'm going to guess that you don't have a valid PTR record for the IP you are sending from at mail.mydomain.com

Goto http://network-tools.com/ and put that IP in, and see what it says in the results for "Host name"

Or you can use mxtoolbox which is handy for checking other mail related things such SPF records and blacklists.

http://www.mxtoolbox.com/SuperTool.aspx?action=ptr%3a1.2.3.4

Put your IP in where 1.2.3.4 is

Papertrip:
the reverse lookup for the ip of mail.mydomain.com is the correct ip according to network-tools.
we are not listed on anyones blacklist according to mxtoolbox

according to mxtoolbox , mx:mydomain.com is correct.

this is the response i got from the ISP support

According to the Bounce errors you are seeing the Frontier Mail Server does not see the domain @flanders.mydomain.com as valid. I would double check the reply to email address and make sure it is correct. If it is correct, you will want to email the Bounce and Headers to the post master of @flanders.mydomain.com. They can then contact the Frontier Postmaster to resovle this issue

Ah, I see.  The problem here is the envelope-from (or MAIL FROM, Return-path, envelope-sender, etc).

Try this in sendmail

MASQUERADE_AS(`mydomain.com')
MASQUERADE_DOMAIN(`mydomain.com')
FEATURE(`masquerade_entire_domain')
FEATURE(`masquerade_envelope')

Select allOpen in new window

That should work as long as all mail from that machine is @mydomain.com.  I'm not able to test that right now but have done this years ago, so give it a shot and let me know.

You can also add -f to sendmail on the commandline
    -fname      Sets the name of the ``from'' person (i.e., the sender of the
                 mail).  -f can only be used by ``trusted'' users (normally
                 root, daemon, and network) or if the person you are trying to
                 become is the same as the person you are.

Select allOpen in new window

To be clear, I know for sure that -fname works, but only like 90% certain about the masquerading since I can't test it.

so this is what i did as suggested
uncomment - MASQUERADE_AS(`mydomain.com')
added - MASQUERADE_DOMAIN(`mydomain.com')
uncomment -FEATURE(`masquerade_entire_domain')
uncomment -FEATURE(`masquerade_envelope')

#make -C /etc/mail
#/etc/rc.d/init.d/sendmail restart


here is the header now that i sent:
i do not think i did it correctly
Delivered-To: myemail@gmail.com
Received: by 10.223.87.14 with SMTP id u14cs44896fal;
        Mon, 3 Oct 2011 14:42:53 -0700 (PDT)
Received: by 10.101.177.34 with SMTP id e34mr409773anp.73.1317678172514;
        Mon, 03 Oct 2011 14:42:52 -0700 (PDT)
Return-Path: <root@flanders.mydomain.com>
Received: from mail.mydomain.com (mail.mydomain.com. [xxx.xxx.190.170])
        by mx.google.com with ESMTP id j38si6362776ani.43.2011.10.03.14.42.52;
        Mon, 03 Oct 2011 14:42:52 -0700 (PDT)
Received-SPF: pass (google.com: domain of root@flanders.mydomain.com designates xxx.xxx.190.170 as permitted sender) client-ip=xxx.xxx.190.170;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of root@flanders.mydomain.com designates xxx.xxx.190.170 as permitted sender) smtp.mail=root@flanders.mydomain.com
Received: from flanders.mydomain.com [xxx.xxx.190.234] by mail.mydomain.com with ESMTP
  (SMTPD-11.5) id e7300000bdf73c43; Mon, 3 Oct 2011 16:42:40 -0500
Received: from flanders.mydomain.com (localhost.localdomain [127.0.0.1])
      by flanders.mydomain.com (8.13.8/8.13.8) with ESMTP id p93Lgp1Q013894
      for <myemail@gmail.com>; Mon, 3 Oct 2011 16:42:51 -0500
Received: (from root@localhost)
      by flanders.mydomain.com (8.13.8/8.13.8/Submit) id p93Lgpkd013893;
      Mon, 3 Oct 2011 16:42:51 -0500
Message-Id: <201110032142.p93Lgpkd013893@flanders.mydomain.com>
To: myemail@gmail.com
Subject: randy, October Newsletter
Date: Mon, 03 Oct 2011 16:42:51 -0500
From: "mydomain Newsletter" <noreply@mydomain.com>
X-Mailer: PHP-EMAIL,v1.1 (William Fowler)
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="-----91c2bc5a2725294704c4ba406763d00a"

Hmm yeah the problem is still there:

smtp.mail=root@flanders.mydomain.com

Select allOpen in new window

Try changing masquerade_domain, leave the others as-is.

MASQUERADE_DOMAIN(`flanders.mydomain.com')

Select allOpen in new window

yeah, no joy looks the same
smtp.mail=root@flanders.mydomain.com

this is from my mail log. not sure if it help
Oct  3 17:06:59 flanders sendmail[14899]: p93M6xmu014899: from=root, size=20606, class=0, nrcpts=1, msgid=<201110032206.p93M6xmu014899@flanders.mydomain.com>, relay=root@localhost
Oct  3 17:06:59 flanders sendmail[14900]: p93M6xHU014900: from=<root@flanders.mydomain.com>, size=20800, class=0, nrcpts=1, msgid=<201110032206.p93M6xmu014899@flanders.mydomain.com>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]
Oct  3 17:06:59 flanders sendmail[14899]: p93M6xmu014899: to=myemail@gmail.com, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=50606, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (p93M6xHU014900 Message accepted for delivery)
Oct  3 17:06:59 flanders sendmail[14902]: p93M6xHU014900: to=<myemail@gmail.com>, ctladdr=<root@flanders.mydomain.com> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=140800, relay=mail.mydomain.com. [xxx.xxx.190.170], dsn=2.0.0, stat=Sent (Message queued)

Ahh now this is something I didn't know about...

Remove this line

EXPOSED_USER(`root')

Select allOpen in new window

There are always users that need to be "exposed" -- that is, their internal site name should be displayed instead of the masquerade name. Root is an example (which has been "exposed" by default prior to 8.10). You can add users to this list using
EXPOSED_USER(`usernames')

http://www.sendmail.org/m4/masquerading.html should be able to help with any other questions, but I'm pretty sure we got it after this.