Messages Not Being Delivered

To whomever might help,
     Problem: I'm having many many emails that log as dsn=2.0.0, stat=Sent (Ok: queued as CA29A4FC7D) or dsn=2.0.0, stat=Sent ( <> Queued mail for delivery) for Hotmail emails, but they never get delivered.

     Relevant Facts:
     - Sendmail 8.12 on Linux. Java creates the messages and sends them to sendmail. The box has 2 new Xeon procs and 2 gigs of RAM.
     - A nightly email sends out and is delivered successfully to all legitimate addresses. Sends about 8,000 text only emails. I'm a little Linux ignorant, but I use "dir | wc -l" to get a file count in the queue (var/spool/mqueue) which reads about 220 at the height of the nightly run.
     - The problem run occurs once a week on weekends, and sends out approx. 80,000 emails. The messages are multipart, half text and half HTML, and each weigh about 50k. I took a count as above in the queue, and it peeked around 1,590.
    - AOL(once I got on their whitelist), Yahoo, Comcast and other fast ISPs get delivered to ok on the weekend.
    - I sort by filename, and sort the mail in Java before it's delivered to Sendmail by email address (NOT by host).
    - I mention fast ISPs, because I noticed that with at least on smaller domain, the addresses early in the alphabet go through, but the later ones don't. In contrast to that, my comcast address, although beginning with "r", which is after the smaller domains stop delivering, goes through fine.
   - Hotmail, as far as I can tell, does not go through even to my Hotmail address that begins with a "d". As stated above, hotmail shows as dsn=2.0.0, stat=Sent ( <> Queued mail for delivery), where AOL just says sent with no queuing.
   - On the nightly run, I see my Hotmail address as queued, but I receive the email almost immediately, every time.

   I believe it's a queue saturation problem, or maybe even messages getting dropped from the queue. I'm very familiar with both the and the files, having crawled through them more than a few times, so I can tell you what various settings are, if you need to know.

   Thank You.  This one's very important, as it's a final problem in the list of more than a few.

Who is Participating?

Improve company productivity with a Business Account.Sign Up

PsiCopConnect With a Mentor Commented:
Well, first, as a mail admin, I *despise* people who send duplicate parts, one text the other HTML. All you do is clutter my Internet link and litter my disk space with E-Mai that takes 2-2.5 times as much disk as it needs to. Don't do that. Let your recipients sign up for one or the other, but if they're smart enuf not to use an HTML-enabled mailreader, all you do is junk up what they see anyway.

Next, lotsa HTML is one of the things that SPAM filters look for. If what you have to say is really worth reading, you don't need HTML. A link or two is fine, but much more than that and SPAM filters tend to perk up. Same for multi-part messages that are just repeats (MIMEdefang/SpamAssassin has specific filters for that garbage).

Make sure your mailhost is properly registered in DNS, with a reverse-lookup record, and is listed as a Mail eXchanger.

Since you seem to be able to send out smaller amounts of E-mail fine, consider breaking the 80K honker into smaller jobs spread over several hours. You can pretty much guarantee that Hotmail throttles connections after X number of RCPT TO: requests during the SMTP conversation. And you can also guarantee that Hotmail's servers are configured to keep tabs on who sends them how much E-Mail, and regard lotsa E-Mail from one place in a short time as a spammer.

If Hotmail offers its users a white-list feature (I have no idea, never used Hotmail), encourage your subscribers to white-list you.

And quit sending text part/html part duplicate E-Mails.
To figure out what's happening here I need to see the all records in maillog for the queue ID of one of the messages to Hotmail. I'd expect there to be at least two records in the mail log associated with that message. Grepping for the local queue ID of the message will pull them all out.
To comment in addition to agreeing with PsiCop 100% (I'm also a mail admin =) one of the weights we have assigned heavily on spam scoring is HTML_TEXT_WITHUNBALANCEDTAG.  The ruleset for this is (roughly defined as follows): IF (twopart attachment TEXT+HTML) && UNBALANCED_HTML_TAGS then we assign a fairly hefty spam score to it right off the top.  We also keep a running history window of email received with certain attributes within the last Z minutes. If the number of messages tagged as spam from one of those entries reaches a certain threshold in the window interval, we temporarily block any further email matching that attribute. (Some of the attributes are sending MX, target domain, target domain+sending domain, etc). We also then add a decaying score for future mail from the domain/mail host so that we're more sensitive to potential spam for the next hour or so.

I know nothing about hotmail's anti-spam back end, but I can tell you that if you hit any of our mailservers with what you are describing you would probably get a handful of message through and then the rest would be blocked.  Consider configuring multiple outbound queues and running persistant background queue runners, throttling your outbound load and delivering fewer recipients per message.

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Oh yeah, I left one small detail out. Make sure your HTML has balanced tags. A hallmark of HTML spam is that the HTML is crap. =)
rosettatgAuthor Commented:
To all,
    I appreciate the general feedback and PsiCop's soapbox preaching; if only the world were that easy, that any one person would have control over such matters. I am but a developer brought in by company x, for the project of converting an existing email service that took over 14 hours to send out the 80,000 plus emails. I can make recommendations about changing the multipart to either text or HTML, but that's as far as I can go.

   Here's where I am at this point:
   1) I added some information to the header, such as errors-to, and wrapped the HTML in actaul <HTML><BODY></BODY></HTML> and other such tags, as it was going out as just a table structure. Having done that, Hotmail accepts the emails. Prior to this, I noticed that Hotmail had an issue regardless of the volume.
   2) I'm recommending that they change the emails to text only for now, for speed and for better clearance through firewalls. I will use some of your quotes to help make my case. Allowing people to opt for text or pure HTML is the best approach, but that will also require changes to their web site, which is out of the scope of this project.
   I read in the bat book that one good idea for running mail quickly through sendmail is to instantiate it with two different configurations, one that process the newly outgoing messages, only waiting a few seconds for each SMTP communication, and another configuration that process the queue, with more standard wait times. What do you think about that approach?
When I've set up mail systems for mass mailers I use an aged queueng system to speed up delivery and reduce the load on the MTA. The default delivery queue is processed with a Sendmail instance that has short timeouts on the order of 5 seconds. Each message in that queue is tried at least once and if it can't be immediately delivered the message is moved to a second queue. That queue is processed with normal timeouts and is run at 30 minute intervals. Each message in that queue is tried twice before moving to a third queue that's run at less frequent intervals, and so forth util the message is delivered or 4 days elapse.
I think the multiple-instantiation idea is worth a try. It will give you more granular control over delivery.
rosettatgAuthor Commented:
To all,
  I thank you for your participation. I've yet to solve this one, which is partially why I haven't tended to come back here, and the fact that I only run the software once a week, so testing is a long, long (did I mention long?) process.


No problem - if you'd let us know about those circumstances, I'm sure the Moderators would have let this Question remain open.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.