[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.

Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

9.0

Limiting server load and speeding local delivery, when receiving very high volumes of email

Asked by cbns in SendMail Email Server

Tags: conflog_level, confqueue_la, deferred, delivery, high

I provide my users with email accounts specifically so that they can receive huge numbers of email from lists that they intentionally join.  Things have been running pretty well for 3 years, but lately I am seeing some very high spikes in cpu load that slow the server down, and of course causes sendmail to refuse connections until load is back under control.

This is junk mail, so suggestions can go light on protecting the arriving mail.  

I am running Sendmail on Redhat 7.3.  I provide OpenWebmail and run a cron job every hour that 'touches' each account in order to run email filters to move 'validation emails' out of the inbox.   This may be a load generator.  The filters also run whenever the member opens their account via OpenWebmail.  Each email account also has a cron job that deletes their inbox once every 3 hours.  These accounts accumulate and average of 10 meg of email in three hours with a high of 100meg in three hours.  

For 3 years Sendmail has run pretty well set in Queued Mode, attempting to deliver once an hour, allowing acceptance and delivery of mail when load is as high as 45 (high I know, but need to limit connection refusals, and normally we spike to no more than 15) I have included the entire M4 beneath my questions.  

Most M4 settings look as I set them myself 3 years or so ago, but I did have a consultant touch my system about 2 months ago, and am not sure if he tweaked this file.  Since I am a novice and set this up originally by trial and error, I need an experts eye to these settings.  The file seems smaller than I remember.  Also, I do not remember having both "mailer local" and "mailer procmail" entries before - and I thought there used to be a cancellation of safe mode in here.  The order of entries is different than I originally had them.

My primary concern is for Sendmail to never or very rarely refuse a connection.  My secondary concern is for the mail to be delivered locally as quickly as possible, since I suspect that an extremely full queue contributes to server load.  I realize that adding a second mx server to accept/hold mail when this server is busy will help, and I have one ready to add to the mix as soon as I am sure SendMail is setup as effectively as possible for this situation.

Specific Issues:
1.  YahooGroups bulk mail servers seem to have started delivering their mail in much larger batches this week from multiple servers at a time, and I think this is my primary problem with high load peaks.  Server load average is peaking up to 50 for as much as 10-15 minutes several times a day.  

Is there a way for me to 'throttle' email coming from their servers - either by limiting the number of connections they can make, or the number of emails they can drop before disconnecting?  This might require something done outside of Sendmail - ipTables perhaps?

2.  What Sendmail settings will minimize server load and speed up acceptance of email and local delivery of same?  

For example
a.  using Deferred versus, Queued, versus Interactive mode.  
b.  how to limite dns activity for inbound mail, etc...
c.  limiting 'protective' measures designed to preserve the mail if delivery fails, etc...

3.  Should I consider switching to PostFix for high volumen receipt and local delivery of email?  Assuming it will work with OpenWebmail.
4.  Should I be looking for a better, quicker way to filter the emails than with OpenWebmail, since this might be contributing to server load?

Thanks in advance for your advice.  Mike

=+++++++++++++++++++++++++++++++M4 below+++++++++++++++++++++++++++++
define(`confDEF_USER_ID',``8:12'')dnl
EXPOSED_USER(`root')dnl
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
define(`PROCMAIL_MAILER_PATH',`/usr/bin/procmail')dnl
FEATURE(local_procmail,`',`procmail -t -Y -a $h -d $u')dnl
define(`ALIAS_FILE', `/etc/aliases')dnl
define(`STATUS_FILE', `/etc/mail/statistics')dnl
define(`confUSERDB_SPEC', `/etc/mail/userdb.db')dnl
FEATURE(`mailertable',`hash -o /etc/mail/mailertable.db')dnl
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable.db')dnl
FEATURE(`genericstable')dnl
FEATURE(`domaintable')dnl
FEATURE(use_cw_file)dnl
FEATURE(use_ct_file)dnl
FEATURE(`access_db',`hash -o /etc/mail/access.db')dnl
FEATURE(`no_default_msa',`dnl')dnl
FEATURE(`blacklist_recipients')dnl
define(`confAUTO_REBUILD',`True')dnl
define(`confTO_CONNECT', `1m')dnl
dnl TRUST_AUTH_MECH(`DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl define(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confTO_QUEUEWARN',`7d')
define(`confTO_QUEUERETURN',`6d')
define(`confQUEUE_LA',45)
define(`confREFUSE_LA',45)
undefine(`confDELAY_LA')dnl
dnl This changes sendmail to only listen on the loopback device 127.0.0.1
dnl and not on any other network devices. Comment this out if you want
dnl to accept email over the network.
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')
dnl NOTE: binding both IPv4 and IPv6 daemon to the same port requires
dnl       a kernel patch
dnl DAEMON_OPTIONS(`port=smtp,Addr=::1, Name=MTA-v6, Family=inet6')
dnl We strongly recommend to comment this one out if you want to protect
dnl yourself from spam. However, the laptop and users on computers that do
dnl not have 24x7 DNS do need this.
define(`confMAX_DAEMON_CHILDREN',0)dnl
define(`confDOUBLE_BOUNCE_ADDRESS',`dblbounce')dnl
define(`confUSE_ERRORS_TO*',`True')dnl
define(`confSAFE_QUEUE*',`False')dnl
define(`confDF_BUFFER_SIZE',32768)dnl
define(`confMAX_QUEUE_RUN_SIZE',0)dnl
define(`confMIN_QUEUE_AGE',`1h')
define(`confLOG_LEVEL',4)dnl
define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn,restrictqrun')dnl
define(`confMAX_RCPTS_PER_MESSAGE',5)
MAILER(`local')dnl
MAILER(smtp)dnl
MAILER(procmail)dnl



 
Related Solutions
 
Loading Advertisement...
 
[+][-]03/03/05 12:06 PM, ID: 13452940Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]03/03/05 12:37 PM, ID: 13453263Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zone: SendMail Email Server
Tags: conflog_level, confqueue_la, deferred, delivery, high
Sign Up Now!
Solution Provided By: PsiCop
Participating Experts: 1
Solution Grade: A
 
[+][-]03/03/05 12:37 PM, ID: 13453269Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]03/03/05 12:43 PM, ID: 13453324Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]03/03/05 01:06 PM, ID: 13453562Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]03/03/05 01:42 PM, ID: 13453915Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]03/03/05 01:53 PM, ID: 13454015Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091111-EE-VQP-89