Solved

Limit amount of emails sent per hour

Posted on 2012-03-13
6
604 Views
Last Modified: 2013-11-13
We have an internal ESP that uses third party SMTP (Dynect) to send our emails out.  Once the scheduled sendjob is picked up by the cron job it builds an array of all the member emails that need to be sent to.  Once it has completed that, it creates the SMTP connection and begins to loop through the array sending one email at a time (we can't batch because we have dynamic fields that need to be populated for each email going out).

If the job needs to be paused or un-paused I initiate the creation of a flag file that is checked for each time through the loop:

                  while(file_exists($filename))
                  {
                     if(file_get_contents($filename) == "STOP")
                    {
                           $newsletter->cancel_send_cron($db,$send_id);
                        if(file_exists($filename))
                        unlink($filename);
                         die();
                    }
                     else
                   {
                     sleep(10);
                     }
                  }


As you can see, there are two states of the flag file.  If the file contains "STOP", it initiates the deletion of the sendjob.  If the file is simply present but without "STOP" than it is a pause file and the script sleeps for 10 seconds before checking again.

Now my question is that I am now being asked to be able to set a limit of emails being sent per hour.  Would I use a flag file as well or is there another method I could use?  I don't want the jobs to stop completely, I just want them to pause if our hourly limit has been met and then start up again in the next hour until the threshold is reached again.  My only concern is having all the processes sleeping at once.  In one hour we may have 10 to 15 jobs sending at once and I want an overall limit set.
0
Comment
Question by:kageboy2003
  • 3
  • 2
6 Comments
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 37719469
It seems to me that you might want to use some kind of data base to control this.  Example of the idea: When it is time to send, create tables with each hourly group of emails.  If you're sending at the rate of 2 per second and you wanted a limit of 7,200 per hour, you could make each of the tables contain 7,200 rows.  As one table is exhausted, move on to the next table.

Overall how many emails are you sending and at what rate now?  Are your scripts sleeping between outgoing emails?
0
 

Author Comment

by:kageboy2003
ID: 37722507
Our normal application runs about 50 to 60k a day, about 4-5k an hour, but now we want to test throttling to prevent an onslaught with the major ISP's.  So I've been told that we will only be running 5-10k a day through the new application.  However, we want to throttle it at 1000 an hour.  I know how to limit an individual sendjob, but I need a way to keep a master count and limit ALL jobs per hour.  For instance, at 8:00 am tomorrow we will have 15 different sendjobs (to different accounts) starting at or around the same time (1 minute apart).  So lets say the first job has 800 emails.  Then when the second job starts,  I need to check the overall status of the first job and see if there is any left on the limit. If the limit has been reached, then all other jobs need to be delayed or paused until the hour is up and then the limit is reset.

The only time the script sleeps is if the admin initiates a "pause" at which time the flag file is created and it sleeps 10 seconds before checking for the pause file again.  It dosn't pause the whole system, just that specific sendjob.
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 37722901
If I am doing the math correctly, it would appear that you have a problem.  Here is what I am seeing.  There are 86,400 seconds in a day.  If you send 5,000 per hour you have a sending capacity plan of 120,000 messages per day.  You are currently below the 50% load for that plan.

50K per day = 1.7 seconds per message.
60K per day = 1.4 seconds per message.
1,000 per hour = 24,000 per day.

So it would seem that somewhere between 26,000 and 36,000 messages are not going to get sent.  I don't make policy here, I just do the analysis.  But it looks like the new plan will leave you with an artificial limit that is set at less than 1/4 of the current capacity.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 14

Accepted Solution

by:
Scott Madeira earned 500 total points
ID: 37724623
phplist is a mailing program that allows you to throttle emails.  You may want to download it and have a look at how they implement these capabilities.  It may give you some ideas.

From an architecture perspective you may want to look at implementing an email manager.  All of your jobs would use the email manager to coordinate sending emails.  That way you don't have 10 to 15 jobs running asynchronously and sending emails without regard to any of the other jobs.

We implemented something like that in one of our apps where the application would create a database record for each email it wants to send.  The email manager would then pull the data out of the table, send the email and move to the next one.

Another advantage of this approach is the ability to control emails to major ISPs.  Your email manager could track the number of emails sent to each ISP.  You could develop a scheduling algorithm that lets you know when you have the ability to send 200 more emails to yahoo addresses, for example.
0
 

Author Comment

by:kageboy2003
ID: 37725460
smadeira,

We didn't originally go with PHP List because of the amount of customization that was required for the project.  So I'm trying to implement a solution that will require the least amount of change to the existing application.  Your suggestion regarding the control of email to ISP's is something we are looking at as well.  I'm already creating a "send table" for each job, but are you suggesting I create a master "send table" that stores all records no matter what account (sendjob) it is sending from?

My original thought was to have a master flag file or a counter table in the DB that kept track as each email went out.  This however would require a file check and db update for each email that goes out.  I'm also assuming I would need a cron script to run every hour to reset the master counter and if the limit is hit, all jobs starting within the same hour after would have to be paused.

We do direct mail marketing here and we have employees that schedule our email from all of our accounts each day.  We need to send each account as a seperate list because our Dynect accounts each have different sending emails. I give them the ability to schedule the sends at any given time of the day, so I can't control the schedule times.
0
 

Author Comment

by:kageboy2003
ID: 37748072
I took a look at PHP List and Pommo, both with email throttling and batch processing.  Rather than me back-engineering their code, can somebody give me a summary of how that works?  I know they are sticking the emails in a mail queue, which is being picked up by the cron, but how does it pick up new emails that are added after the cron has already started?
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

This is an explanation of a simple data model to help parse a JSON feed
Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now