Is your CI/CD pipeline a hodge-podge of randomly connected tools? You’ve likely got a tool to fix one problem & then a different tool to fix another, resulting in a cluster of tools with overlapping functionality. Learn how to optimize your pipeline with Gartner's recommendations
Course Of The Month
4 days, 21 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Messages Not Being Delivered
Posted on 2004-09-13
To whomever might help,
Problem: I'm having many many emails that log as dsn=2.0.0, stat=Sent (Ok: queued as CA29A4FC7D) or dsn=2.0.0, stat=Sent ( <21431207.1095024967457.Ja
Relevant Facts:
- Sendmail 8.12 on Linux. Java creates the messages and sends them to sendmail. The box has 2 new Xeon procs and 2 gigs of RAM.
- A nightly email sends out and is delivered successfully to all legitimate addresses. Sends about 8,000 text only emails. I'm a little Linux ignorant, but I use "dir | wc -l" to get a file count in the queue (var/spool/mqueue) which reads about 220 at the height of the nightly run.
- The problem run occurs once a week on weekends, and sends out approx. 80,000 emails. The messages are multipart, half text and half HTML, and each weigh about 50k. I took a count as above in the queue, and it peeked around 1,590.
- AOL(once I got on their whitelist), Yahoo, Comcast and other fast ISPs get delivered to ok on the weekend.
- I sort by filename, and sort the mail in Java before it's delivered to Sendmail by email address (NOT by host).
- I mention fast ISPs, because I noticed that with at least on smaller domain, the addresses early in the alphabet go through, but the later ones don't. In contrast to that, my comcast address, although beginning with "r", which is after the smaller domains stop delivering, goes through fine.
- Hotmail, as far as I can tell, does not go through even to my Hotmail address that begins with a "d". As stated above, hotmail shows as dsn=2.0.0, stat=Sent ( <21431207.1095024967457.Ja
- On the nightly run, I see my Hotmail address as queued, but I receive the email almost immediately, every time.
I believe it's a queue saturation problem, or maybe even messages getting dropped from the queue. I'm very familiar with both the sendmail.mc and the sendmail.cf files, having crawled through them more than a few times, so I can tell you what various settings are, if you need to know.
Thank You. This one's very important, as it's a final problem in the list of more than a few.
Problem: I'm having many many emails that log as dsn=2.0.0, stat=Sent (Ok: queued as CA29A4FC7D) or dsn=2.0.0, stat=Sent ( <21431207.1095024967457.Ja
Relevant Facts:
- Sendmail 8.12 on Linux. Java creates the messages and sends them to sendmail. The box has 2 new Xeon procs and 2 gigs of RAM.
- A nightly email sends out and is delivered successfully to all legitimate addresses. Sends about 8,000 text only emails. I'm a little Linux ignorant, but I use "dir | wc -l" to get a file count in the queue (var/spool/mqueue) which reads about 220 at the height of the nightly run.
- The problem run occurs once a week on weekends, and sends out approx. 80,000 emails. The messages are multipart, half text and half HTML, and each weigh about 50k. I took a count as above in the queue, and it peeked around 1,590.
- AOL(once I got on their whitelist), Yahoo, Comcast and other fast ISPs get delivered to ok on the weekend.
- I sort by filename, and sort the mail in Java before it's delivered to Sendmail by email address (NOT by host).
- I mention fast ISPs, because I noticed that with at least on smaller domain, the addresses early in the alphabet go through, but the later ones don't. In contrast to that, my comcast address, although beginning with "r", which is after the smaller domains stop delivering, goes through fine.
- Hotmail, as far as I can tell, does not go through even to my Hotmail address that begins with a "d". As stated above, hotmail shows as dsn=2.0.0, stat=Sent ( <21431207.1095024967457.Ja
- On the nightly run, I see my Hotmail address as queued, but I receive the email almost immediately, every time.
I believe it's a queue saturation problem, or maybe even messages getting dropped from the queue. I'm very familiar with both the sendmail.mc and the sendmail.cf files, having crawled through them more than a few times, so I can tell you what various settings are, if you need to know.
Thank You. This one's very important, as it's a final problem in the list of more than a few.
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
-
3
-
2
2- +1
10 Comments
To figure out what's happening here I need to see the all records in maillog for the queue ID of one of the messages to Hotmail. I'd expect there to be at least two records in the mail log associated with that message. Grepping for the local queue ID of the message will pull them all out.
Well, first, as a mail admin, I *despise* people who send duplicate parts, one text the other HTML. All you do is clutter my Internet link and litter my disk space with E-Mai that takes 2-2.5 times as much disk as it needs to. Don't do that. Let your recipients sign up for one or the other, but if they're smart enuf not to use an HTML-enabled mailreader, all you do is junk up what they see anyway.
Next, lotsa HTML is one of the things that SPAM filters look for. If what you have to say is really worth reading, you don't need HTML. A link or two is fine, but much more than that and SPAM filters tend to perk up. Same for multi-part messages that are just repeats (MIMEdefang/SpamAssassin has specific filters for that garbage).
Make sure your mailhost is properly registered in DNS, with a reverse-lookup record, and is listed as a Mail eXchanger.
Since you seem to be able to send out smaller amounts of E-mail fine, consider breaking the 80K honker into smaller jobs spread over several hours. You can pretty much guarantee that Hotmail throttles connections after X number of RCPT TO: requests during the SMTP conversation. And you can also guarantee that Hotmail's servers are configured to keep tabs on who sends them how much E-Mail, and regard lotsa E-Mail from one place in a short time as a spammer.
If Hotmail offers its users a white-list feature (I have no idea, never used Hotmail), encourage your subscribers to white-list you.
And quit sending text part/html part duplicate E-Mails.
Next, lotsa HTML is one of the things that SPAM filters look for. If what you have to say is really worth reading, you don't need HTML. A link or two is fine, but much more than that and SPAM filters tend to perk up. Same for multi-part messages that are just repeats (MIMEdefang/SpamAssassin has specific filters for that garbage).
Make sure your mailhost is properly registered in DNS, with a reverse-lookup record, and is listed as a Mail eXchanger.
Since you seem to be able to send out smaller amounts of E-mail fine, consider breaking the 80K honker into smaller jobs spread over several hours. You can pretty much guarantee that Hotmail throttles connections after X number of RCPT TO: requests during the SMTP conversation. And you can also guarantee that Hotmail's servers are configured to keep tabs on who sends them how much E-Mail, and regard lotsa E-Mail from one place in a short time as a spammer.
If Hotmail offers its users a white-list feature (I have no idea, never used Hotmail), encourage your subscribers to white-list you.
And quit sending text part/html part duplicate E-Mails.
To comment in addition to agreeing with PsiCop 100% (I'm also a mail admin =) one of the weights we have assigned heavily on spam scoring is HTML_TEXT_WITHUNBALANCEDTA
I know nothing about hotmail's anti-spam back end, but I can tell you that if you hit any of our mailservers with what you are describing you would probably get a handful of message through and then the rest would be blocked. Consider configuring multiple outbound queues and running persistant background queue runners, throttling your outbound load and delivering fewer recipients per message.
I know nothing about hotmail's anti-spam back end, but I can tell you that if you hit any of our mailservers with what you are describing you would probably get a handful of message through and then the rest would be blocked. Consider configuring multiple outbound queues and running persistant background queue runners, throttling your outbound load and delivering fewer recipients per message.
Oh yeah, I left one small detail out. Make sure your HTML has balanced tags. A hallmark of HTML spam is that the HTML is crap. =)
To all,
I appreciate the general feedback and PsiCop's soapbox preaching; if only the world were that easy, that any one person would have control over such matters. I am but a developer brought in by company x, for the project of converting an existing email service that took over 14 hours to send out the 80,000 plus emails. I can make recommendations about changing the multipart to either text or HTML, but that's as far as I can go.
Here's where I am at this point:
1) I added some information to the header, such as errors-to, and wrapped the HTML in actaul <HTML><BODY></BODY></HTML>
2) I'm recommending that they change the emails to text only for now, for speed and for better clearance through firewalls. I will use some of your quotes to help make my case. Allowing people to opt for text or pure HTML is the best approach, but that will also require changes to their web site, which is out of the scope of this project.
I read in the bat book that one good idea for running mail quickly through sendmail is to instantiate it with two different configurations, one that process the newly outgoing messages, only waiting a few seconds for each SMTP communication, and another configuration that process the queue, with more standard wait times. What do you think about that approach?
I appreciate the general feedback and PsiCop's soapbox preaching; if only the world were that easy, that any one person would have control over such matters. I am but a developer brought in by company x, for the project of converting an existing email service that took over 14 hours to send out the 80,000 plus emails. I can make recommendations about changing the multipart to either text or HTML, but that's as far as I can go.
Here's where I am at this point:
1) I added some information to the header, such as errors-to, and wrapped the HTML in actaul <HTML><BODY></BODY></HTML>
2) I'm recommending that they change the emails to text only for now, for speed and for better clearance through firewalls. I will use some of your quotes to help make my case. Allowing people to opt for text or pure HTML is the best approach, but that will also require changes to their web site, which is out of the scope of this project.
I read in the bat book that one good idea for running mail quickly through sendmail is to instantiate it with two different configurations, one that process the newly outgoing messages, only waiting a few seconds for each SMTP communication, and another configuration that process the queue, with more standard wait times. What do you think about that approach?
When I've set up mail systems for mass mailers I use an aged queueng system to speed up delivery and reduce the load on the MTA. The default delivery queue is processed with a Sendmail instance that has short timeouts on the order of 5 seconds. Each message in that queue is tried at least once and if it can't be immediately delivered the message is moved to a second queue. That queue is processed with normal timeouts and is run at 30 minute intervals. Each message in that queue is tried twice before moving to a third queue that's run at less frequent intervals, and so forth util the message is delivered or 4 days elapse.
I think the multiple-instantiation idea is worth a try. It will give you more granular control over delivery.
To all,
I thank you for your participation. I've yet to solve this one, which is partially why I haven't tended to come back here, and the fact that I only run the software once a week, so testing is a long, long (did I mention long?) process.
I thank you for your participation. I've yet to solve this one, which is partially why I haven't tended to come back here, and the fact that I only run the software once a week, so testing is a long, long (did I mention long?) process.
Featured Post
Synthetic Transaction Monitoring that you need for the day to day, which ensures your business website keeps running optimally, and that there is no downtime to impact your customer experience.
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
Microsoft has released various new features which are capable of handling various tasks. One of these tasks is ‘Migration from pop3 to Exchange Server’. Pop3 data stores various data along mailboxes like contacts, tasks, etc. So, it becomes the need…
As cyber crime continues to grow in both numbers and sophistication, a troubling trend of optimization has emerged over the last year.
In this video we show how to create a User Mailbox in Exchange 2013. We show this process by using the Exchange Admin Center.
Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center.
Navigate to the Recipients >> Mailb…
In this video we show how to create an Accepted Domain in Exchange 2013. We show this process by using the Exchange Admin Center.
Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center.
Navigate to the Mail Flow >> Ac…
Suggested Courses
Course of the Month4 days, 21 hours left to enroll
688 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.