?
Solved

Messages Not Being Delivered

Posted on 2004-09-13
10
Medium Priority
?
838 Views
Last Modified: 2013-12-17
To whomever might help,
   
     Problem: I'm having many many emails that log as dsn=2.0.0, stat=Sent (Ok: queued as CA29A4FC7D) or dsn=2.0.0, stat=Sent ( <21431207.1095024967457.JavaMail.root@mail.ceocast.com> Queued mail for delivery) for Hotmail emails, but they never get delivered.

     Relevant Facts:
     - Sendmail 8.12 on Linux. Java creates the messages and sends them to sendmail. The box has 2 new Xeon procs and 2 gigs of RAM.
     - A nightly email sends out and is delivered successfully to all legitimate addresses. Sends about 8,000 text only emails. I'm a little Linux ignorant, but I use "dir | wc -l" to get a file count in the queue (var/spool/mqueue) which reads about 220 at the height of the nightly run.
     - The problem run occurs once a week on weekends, and sends out approx. 80,000 emails. The messages are multipart, half text and half HTML, and each weigh about 50k. I took a count as above in the queue, and it peeked around 1,590.
    - AOL(once I got on their whitelist), Yahoo, Comcast and other fast ISPs get delivered to ok on the weekend.
    - I sort by filename, and sort the mail in Java before it's delivered to Sendmail by email address (NOT by host).
    - I mention fast ISPs, because I noticed that with at least on smaller domain, the addresses early in the alphabet go through, but the later ones don't. In contrast to that, my comcast address, although beginning with "r", which is after the smaller domains stop delivering, goes through fine.
   - Hotmail, as far as I can tell, does not go through even to my Hotmail address that begins with a "d". As stated above, hotmail shows as dsn=2.0.0, stat=Sent ( <21431207.1095024967457.JavaMail.root@mail.ceocast.com> Queued mail for delivery), where AOL just says sent with no queuing.
   - On the nightly run, I see my Hotmail address as queued, but I receive the email almost immediately, every time.

   I believe it's a queue saturation problem, or maybe even messages getting dropped from the queue. I'm very familiar with both the sendmail.mc and the sendmail.cf files, having crawled through them more than a few times, so I can tell you what various settings are, if you need to know.

   Thank You.  This one's very important, as it's a final problem in the list of more than a few.
   

0
Comment
Question by:rosettatg
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
  • +1
10 Comments
 
LVL 40

Expert Comment

by:jlevie
ID: 12051369
To figure out what's happening here I need to see the all records in maillog for the queue ID of one of the messages to Hotmail. I'd expect there to be at least two records in the mail log associated with that message. Grepping for the local queue ID of the message will pull them all out.
0
 
LVL 34

Accepted Solution

by:
PsiCop earned 1500 total points
ID: 12051382
Well, first, as a mail admin, I *despise* people who send duplicate parts, one text the other HTML. All you do is clutter my Internet link and litter my disk space with E-Mai that takes 2-2.5 times as much disk as it needs to. Don't do that. Let your recipients sign up for one or the other, but if they're smart enuf not to use an HTML-enabled mailreader, all you do is junk up what they see anyway.

Next, lotsa HTML is one of the things that SPAM filters look for. If what you have to say is really worth reading, you don't need HTML. A link or two is fine, but much more than that and SPAM filters tend to perk up. Same for multi-part messages that are just repeats (MIMEdefang/SpamAssassin has specific filters for that garbage).

Make sure your mailhost is properly registered in DNS, with a reverse-lookup record, and is listed as a Mail eXchanger.

Since you seem to be able to send out smaller amounts of E-mail fine, consider breaking the 80K honker into smaller jobs spread over several hours. You can pretty much guarantee that Hotmail throttles connections after X number of RCPT TO: requests during the SMTP conversation. And you can also guarantee that Hotmail's servers are configured to keep tabs on who sends them how much E-Mail, and regard lotsa E-Mail from one place in a short time as a spammer.

If Hotmail offers its users a white-list feature (I have no idea, never used Hotmail), encourage your subscribers to white-list you.

And quit sending text part/html part duplicate E-Mails.
0
 
LVL 5

Expert Comment

by:cgrey
ID: 12056280
To comment in addition to agreeing with PsiCop 100% (I'm also a mail admin =) one of the weights we have assigned heavily on spam scoring is HTML_TEXT_WITHUNBALANCEDTAG.  The ruleset for this is (roughly defined as follows): IF (twopart attachment TEXT+HTML) && UNBALANCED_HTML_TAGS then we assign a fairly hefty spam score to it right off the top.  We also keep a running history window of email received with certain attributes within the last Z minutes. If the number of messages tagged as spam from one of those entries reaches a certain threshold in the window interval, we temporarily block any further email matching that attribute. (Some of the attributes are sending MX, target domain, target domain+sending domain, etc). We also then add a decaying score for future mail from the domain/mail host so that we're more sensitive to potential spam for the next hour or so.

I know nothing about hotmail's anti-spam back end, but I can tell you that if you hit any of our mailservers with what you are describing you would probably get a handful of message through and then the rest would be blocked.  Consider configuring multiple outbound queues and running persistant background queue runners, throttling your outbound load and delivering fewer recipients per message.

0
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

 
LVL 5

Expert Comment

by:cgrey
ID: 12056310
Oh yeah, I left one small detail out. Make sure your HTML has balanced tags. A hallmark of HTML spam is that the HTML is crap. =)
0
 

Author Comment

by:rosettatg
ID: 12076736
To all,
    I appreciate the general feedback and PsiCop's soapbox preaching; if only the world were that easy, that any one person would have control over such matters. I am but a developer brought in by company x, for the project of converting an existing email service that took over 14 hours to send out the 80,000 plus emails. I can make recommendations about changing the multipart to either text or HTML, but that's as far as I can go.

   Here's where I am at this point:
   1) I added some information to the header, such as errors-to, and wrapped the HTML in actaul <HTML><BODY></BODY></HTML> and other such tags, as it was going out as just a table structure. Having done that, Hotmail accepts the emails. Prior to this, I noticed that Hotmail had an issue regardless of the volume.
   2) I'm recommending that they change the emails to text only for now, for speed and for better clearance through firewalls. I will use some of your quotes to help make my case. Allowing people to opt for text or pure HTML is the best approach, but that will also require changes to their web site, which is out of the scope of this project.
   
   I read in the bat book that one good idea for running mail quickly through sendmail is to instantiate it with two different configurations, one that process the newly outgoing messages, only waiting a few seconds for each SMTP communication, and another configuration that process the queue, with more standard wait times. What do you think about that approach?
0
 
LVL 40

Expert Comment

by:jlevie
ID: 12077187
When I've set up mail systems for mass mailers I use an aged queueng system to speed up delivery and reduce the load on the MTA. The default delivery queue is processed with a Sendmail instance that has short timeouts on the order of 5 seconds. Each message in that queue is tried at least once and if it can't be immediately delivered the message is moved to a second queue. That queue is processed with normal timeouts and is run at 30 minute intervals. Each message in that queue is tried twice before moving to a third queue that's run at less frequent intervals, and so forth util the message is delivered or 4 days elapse.
0
 
LVL 34

Expert Comment

by:PsiCop
ID: 12077784
I think the multiple-instantiation idea is worth a try. It will give you more granular control over delivery.
0
 

Author Comment

by:rosettatg
ID: 12272722
To all,
  I thank you for your participation. I've yet to solve this one, which is partially why I haven't tended to come back here, and the fact that I only run the software once a week, so testing is a long, long (did I mention long?) process.

0
 
LVL 34

Expert Comment

by:PsiCop
ID: 12276596
rosettatq,

No problem - if you'd let us know about those circumstances, I'm sure the Moderators would have let this Question remain open.
0

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Email signatures have numerous marketing benefits. Here are 8 top reasons to turn your email signature into a marketing channel.
The main intent of this article is to make you aware of ‘Exchange fail to mount’ error, its effects, causes, and solution.
Familiarize people with the process of retrieving data from SQL Server using an Access pass-thru query. Microsoft Access is a very powerful client/server development tool. One of the ways that you can retrieve data from a SQL Server is by using a pa…
In this video we show how to create a User Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Mailb…
Suggested Courses

650 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question