Solved

Messages Not Being Delivered

Posted on 2004-09-13
10
825 Views
Last Modified: 2013-12-17
To whomever might help,
   
     Problem: I'm having many many emails that log as dsn=2.0.0, stat=Sent (Ok: queued as CA29A4FC7D) or dsn=2.0.0, stat=Sent ( <21431207.1095024967457.JavaMail.root@mail.ceocast.com> Queued mail for delivery) for Hotmail emails, but they never get delivered.

     Relevant Facts:
     - Sendmail 8.12 on Linux. Java creates the messages and sends them to sendmail. The box has 2 new Xeon procs and 2 gigs of RAM.
     - A nightly email sends out and is delivered successfully to all legitimate addresses. Sends about 8,000 text only emails. I'm a little Linux ignorant, but I use "dir | wc -l" to get a file count in the queue (var/spool/mqueue) which reads about 220 at the height of the nightly run.
     - The problem run occurs once a week on weekends, and sends out approx. 80,000 emails. The messages are multipart, half text and half HTML, and each weigh about 50k. I took a count as above in the queue, and it peeked around 1,590.
    - AOL(once I got on their whitelist), Yahoo, Comcast and other fast ISPs get delivered to ok on the weekend.
    - I sort by filename, and sort the mail in Java before it's delivered to Sendmail by email address (NOT by host).
    - I mention fast ISPs, because I noticed that with at least on smaller domain, the addresses early in the alphabet go through, but the later ones don't. In contrast to that, my comcast address, although beginning with "r", which is after the smaller domains stop delivering, goes through fine.
   - Hotmail, as far as I can tell, does not go through even to my Hotmail address that begins with a "d". As stated above, hotmail shows as dsn=2.0.0, stat=Sent ( <21431207.1095024967457.JavaMail.root@mail.ceocast.com> Queued mail for delivery), where AOL just says sent with no queuing.
   - On the nightly run, I see my Hotmail address as queued, but I receive the email almost immediately, every time.

   I believe it's a queue saturation problem, or maybe even messages getting dropped from the queue. I'm very familiar with both the sendmail.mc and the sendmail.cf files, having crawled through them more than a few times, so I can tell you what various settings are, if you need to know.

   Thank You.  This one's very important, as it's a final problem in the list of more than a few.
   

0
Comment
Question by:rosettatg
  • 3
  • 2
  • 2
  • +1
10 Comments
 
LVL 40

Expert Comment

by:jlevie
ID: 12051369
To figure out what's happening here I need to see the all records in maillog for the queue ID of one of the messages to Hotmail. I'd expect there to be at least two records in the mail log associated with that message. Grepping for the local queue ID of the message will pull them all out.
0
 
LVL 34

Accepted Solution

by:
PsiCop earned 500 total points
ID: 12051382
Well, first, as a mail admin, I *despise* people who send duplicate parts, one text the other HTML. All you do is clutter my Internet link and litter my disk space with E-Mai that takes 2-2.5 times as much disk as it needs to. Don't do that. Let your recipients sign up for one or the other, but if they're smart enuf not to use an HTML-enabled mailreader, all you do is junk up what they see anyway.

Next, lotsa HTML is one of the things that SPAM filters look for. If what you have to say is really worth reading, you don't need HTML. A link or two is fine, but much more than that and SPAM filters tend to perk up. Same for multi-part messages that are just repeats (MIMEdefang/SpamAssassin has specific filters for that garbage).

Make sure your mailhost is properly registered in DNS, with a reverse-lookup record, and is listed as a Mail eXchanger.

Since you seem to be able to send out smaller amounts of E-mail fine, consider breaking the 80K honker into smaller jobs spread over several hours. You can pretty much guarantee that Hotmail throttles connections after X number of RCPT TO: requests during the SMTP conversation. And you can also guarantee that Hotmail's servers are configured to keep tabs on who sends them how much E-Mail, and regard lotsa E-Mail from one place in a short time as a spammer.

If Hotmail offers its users a white-list feature (I have no idea, never used Hotmail), encourage your subscribers to white-list you.

And quit sending text part/html part duplicate E-Mails.
0
 
LVL 5

Expert Comment

by:cgrey
ID: 12056280
To comment in addition to agreeing with PsiCop 100% (I'm also a mail admin =) one of the weights we have assigned heavily on spam scoring is HTML_TEXT_WITHUNBALANCEDTAG.  The ruleset for this is (roughly defined as follows): IF (twopart attachment TEXT+HTML) && UNBALANCED_HTML_TAGS then we assign a fairly hefty spam score to it right off the top.  We also keep a running history window of email received with certain attributes within the last Z minutes. If the number of messages tagged as spam from one of those entries reaches a certain threshold in the window interval, we temporarily block any further email matching that attribute. (Some of the attributes are sending MX, target domain, target domain+sending domain, etc). We also then add a decaying score for future mail from the domain/mail host so that we're more sensitive to potential spam for the next hour or so.

I know nothing about hotmail's anti-spam back end, but I can tell you that if you hit any of our mailservers with what you are describing you would probably get a handful of message through and then the rest would be blocked.  Consider configuring multiple outbound queues and running persistant background queue runners, throttling your outbound load and delivering fewer recipients per message.

0
 
LVL 5

Expert Comment

by:cgrey
ID: 12056310
Oh yeah, I left one small detail out. Make sure your HTML has balanced tags. A hallmark of HTML spam is that the HTML is crap. =)
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 

Author Comment

by:rosettatg
ID: 12076736
To all,
    I appreciate the general feedback and PsiCop's soapbox preaching; if only the world were that easy, that any one person would have control over such matters. I am but a developer brought in by company x, for the project of converting an existing email service that took over 14 hours to send out the 80,000 plus emails. I can make recommendations about changing the multipart to either text or HTML, but that's as far as I can go.

   Here's where I am at this point:
   1) I added some information to the header, such as errors-to, and wrapped the HTML in actaul <HTML><BODY></BODY></HTML> and other such tags, as it was going out as just a table structure. Having done that, Hotmail accepts the emails. Prior to this, I noticed that Hotmail had an issue regardless of the volume.
   2) I'm recommending that they change the emails to text only for now, for speed and for better clearance through firewalls. I will use some of your quotes to help make my case. Allowing people to opt for text or pure HTML is the best approach, but that will also require changes to their web site, which is out of the scope of this project.
   
   I read in the bat book that one good idea for running mail quickly through sendmail is to instantiate it with two different configurations, one that process the newly outgoing messages, only waiting a few seconds for each SMTP communication, and another configuration that process the queue, with more standard wait times. What do you think about that approach?
0
 
LVL 40

Expert Comment

by:jlevie
ID: 12077187
When I've set up mail systems for mass mailers I use an aged queueng system to speed up delivery and reduce the load on the MTA. The default delivery queue is processed with a Sendmail instance that has short timeouts on the order of 5 seconds. Each message in that queue is tried at least once and if it can't be immediately delivered the message is moved to a second queue. That queue is processed with normal timeouts and is run at 30 minute intervals. Each message in that queue is tried twice before moving to a third queue that's run at less frequent intervals, and so forth util the message is delivered or 4 days elapse.
0
 
LVL 34

Expert Comment

by:PsiCop
ID: 12077784
I think the multiple-instantiation idea is worth a try. It will give you more granular control over delivery.
0
 

Author Comment

by:rosettatg
ID: 12272722
To all,
  I thank you for your participation. I've yet to solve this one, which is partially why I haven't tended to come back here, and the fact that I only run the software once a week, so testing is a long, long (did I mention long?) process.

0
 
LVL 34

Expert Comment

by:PsiCop
ID: 12276596
rosettatq,

No problem - if you'd let us know about those circumstances, I'm sure the Moderators would have let this Question remain open.
0

Featured Post

Are your corporate email signatures appalling?

Is it scary how unprofessional your email signatures look? Do users create their own terrible designs and give themselves stupid job titles? You can make this a lot easier for yourself by choosing an email signature management solution from Exclaimer today.

Join & Write a Comment

The Microsoft Exchange server database may become damaged or corrupted due to many reasons, such as system failure. When this happens, it is usually urgent to restore the system to a stable and working state. There are different symptoms and causes …
Email signatures have numerous marketing benefits. Here are 8 top reasons to turn your email signature into a marketing channel.
The video tutorial explains the basics of the Exchange server Database Availability groups. The components of this video include: 1. Automatic Failover 2. Failover Clustering 3. Active Manager
The basic steps you have just learned will be implemented in this video. The basic steps are shown to configure an Exchange DAG in a live working Exchange Server Environment and manage the same (Exchange Server 2010 Software is used in a Windows Ser…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now