[Last Call] Learn how to a build a cloud-first strategyRegister Now


Limit amount of emails sent per hour

Posted on 2012-03-13
Medium Priority
Last Modified: 2013-11-13
We have an internal ESP that uses third party SMTP (Dynect) to send our emails out.  Once the scheduled sendjob is picked up by the cron job it builds an array of all the member emails that need to be sent to.  Once it has completed that, it creates the SMTP connection and begins to loop through the array sending one email at a time (we can't batch because we have dynamic fields that need to be populated for each email going out).

If the job needs to be paused or un-paused I initiate the creation of a flag file that is checked for each time through the loop:

                     if(file_get_contents($filename) == "STOP")

As you can see, there are two states of the flag file.  If the file contains "STOP", it initiates the deletion of the sendjob.  If the file is simply present but without "STOP" than it is a pause file and the script sleeps for 10 seconds before checking again.

Now my question is that I am now being asked to be able to set a limit of emails being sent per hour.  Would I use a flag file as well or is there another method I could use?  I don't want the jobs to stop completely, I just want them to pause if our hourly limit has been met and then start up again in the next hour until the threshold is reached again.  My only concern is having all the processes sleeping at once.  In one hour we may have 10 to 15 jobs sending at once and I want an overall limit set.
Question by:kageboy2003
  • 3
  • 2
LVL 111

Expert Comment

by:Ray Paseur
ID: 37719469
It seems to me that you might want to use some kind of data base to control this.  Example of the idea: When it is time to send, create tables with each hourly group of emails.  If you're sending at the rate of 2 per second and you wanted a limit of 7,200 per hour, you could make each of the tables contain 7,200 rows.  As one table is exhausted, move on to the next table.

Overall how many emails are you sending and at what rate now?  Are your scripts sleeping between outgoing emails?

Author Comment

ID: 37722507
Our normal application runs about 50 to 60k a day, about 4-5k an hour, but now we want to test throttling to prevent an onslaught with the major ISP's.  So I've been told that we will only be running 5-10k a day through the new application.  However, we want to throttle it at 1000 an hour.  I know how to limit an individual sendjob, but I need a way to keep a master count and limit ALL jobs per hour.  For instance, at 8:00 am tomorrow we will have 15 different sendjobs (to different accounts) starting at or around the same time (1 minute apart).  So lets say the first job has 800 emails.  Then when the second job starts,  I need to check the overall status of the first job and see if there is any left on the limit. If the limit has been reached, then all other jobs need to be delayed or paused until the hour is up and then the limit is reset.

The only time the script sleeps is if the admin initiates a "pause" at which time the flag file is created and it sleeps 10 seconds before checking for the pause file again.  It dosn't pause the whole system, just that specific sendjob.
LVL 111

Expert Comment

by:Ray Paseur
ID: 37722901
If I am doing the math correctly, it would appear that you have a problem.  Here is what I am seeing.  There are 86,400 seconds in a day.  If you send 5,000 per hour you have a sending capacity plan of 120,000 messages per day.  You are currently below the 50% load for that plan.

50K per day = 1.7 seconds per message.
60K per day = 1.4 seconds per message.
1,000 per hour = 24,000 per day.

So it would seem that somewhere between 26,000 and 36,000 messages are not going to get sent.  I don't make policy here, I just do the analysis.  But it looks like the new plan will leave you with an artificial limit that is set at less than 1/4 of the current capacity.
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

LVL 14

Accepted Solution

Scott Madeira earned 2000 total points
ID: 37724623
phplist is a mailing program that allows you to throttle emails.  You may want to download it and have a look at how they implement these capabilities.  It may give you some ideas.

From an architecture perspective you may want to look at implementing an email manager.  All of your jobs would use the email manager to coordinate sending emails.  That way you don't have 10 to 15 jobs running asynchronously and sending emails without regard to any of the other jobs.

We implemented something like that in one of our apps where the application would create a database record for each email it wants to send.  The email manager would then pull the data out of the table, send the email and move to the next one.

Another advantage of this approach is the ability to control emails to major ISPs.  Your email manager could track the number of emails sent to each ISP.  You could develop a scheduling algorithm that lets you know when you have the ability to send 200 more emails to yahoo addresses, for example.

Author Comment

ID: 37725460

We didn't originally go with PHP List because of the amount of customization that was required for the project.  So I'm trying to implement a solution that will require the least amount of change to the existing application.  Your suggestion regarding the control of email to ISP's is something we are looking at as well.  I'm already creating a "send table" for each job, but are you suggesting I create a master "send table" that stores all records no matter what account (sendjob) it is sending from?

My original thought was to have a master flag file or a counter table in the DB that kept track as each email went out.  This however would require a file check and db update for each email that goes out.  I'm also assuming I would need a cron script to run every hour to reset the master counter and if the limit is hit, all jobs starting within the same hour after would have to be paused.

We do direct mail marketing here and we have employees that schedule our email from all of our accounts each day.  We need to send each account as a seperate list because our Dynect accounts each have different sending emails. I give them the ability to schedule the sends at any given time of the day, so I can't control the schedule times.

Author Comment

ID: 37748072
I took a look at PHP List and Pommo, both with email throttling and batch processing.  Rather than me back-engineering their code, can somebody give me a summary of how that works?  I know they are sticking the emails in a mail queue, which is being picked up by the cron, but how does it pick up new emails that are added after the cron has already started?

Featured Post


Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article discusses how to implement server side field validation and display customized error messages to the client.
What do responsible coders do? They don't take detrimental shortcuts. They do take reasonable security precautions, create important automation, implement sufficient logging, fix things they break, and care about users.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Introduction to Processes
Suggested Courses
Course of the Month18 days, 12 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question