Link to home
Start Free TrialLog in
Avatar of kageboy2003
kageboy2003

asked on

Limit amount of emails sent per hour

We have an internal ESP that uses third party SMTP (Dynect) to send our emails out.  Once the scheduled sendjob is picked up by the cron job it builds an array of all the member emails that need to be sent to.  Once it has completed that, it creates the SMTP connection and begins to loop through the array sending one email at a time (we can't batch because we have dynamic fields that need to be populated for each email going out).

If the job needs to be paused or un-paused I initiate the creation of a flag file that is checked for each time through the loop:

                  while(file_exists($filename))
                  {
                     if(file_get_contents($filename) == "STOP")
                    {
                           $newsletter->cancel_send_cron($db,$send_id);
                        if(file_exists($filename))
                        unlink($filename);
                         die();
                    }
                     else
                   {
                     sleep(10);
                     }
                  }


As you can see, there are two states of the flag file.  If the file contains "STOP", it initiates the deletion of the sendjob.  If the file is simply present but without "STOP" than it is a pause file and the script sleeps for 10 seconds before checking again.

Now my question is that I am now being asked to be able to set a limit of emails being sent per hour.  Would I use a flag file as well or is there another method I could use?  I don't want the jobs to stop completely, I just want them to pause if our hourly limit has been met and then start up again in the next hour until the threshold is reached again.  My only concern is having all the processes sleeping at once.  In one hour we may have 10 to 15 jobs sending at once and I want an overall limit set.
Avatar of Ray Paseur
Ray Paseur
Flag of United States of America image

It seems to me that you might want to use some kind of data base to control this.  Example of the idea: When it is time to send, create tables with each hourly group of emails.  If you're sending at the rate of 2 per second and you wanted a limit of 7,200 per hour, you could make each of the tables contain 7,200 rows.  As one table is exhausted, move on to the next table.

Overall how many emails are you sending and at what rate now?  Are your scripts sleeping between outgoing emails?
Avatar of kageboy2003
kageboy2003

ASKER

Our normal application runs about 50 to 60k a day, about 4-5k an hour, but now we want to test throttling to prevent an onslaught with the major ISP's.  So I've been told that we will only be running 5-10k a day through the new application.  However, we want to throttle it at 1000 an hour.  I know how to limit an individual sendjob, but I need a way to keep a master count and limit ALL jobs per hour.  For instance, at 8:00 am tomorrow we will have 15 different sendjobs (to different accounts) starting at or around the same time (1 minute apart).  So lets say the first job has 800 emails.  Then when the second job starts,  I need to check the overall status of the first job and see if there is any left on the limit. If the limit has been reached, then all other jobs need to be delayed or paused until the hour is up and then the limit is reset.

The only time the script sleeps is if the admin initiates a "pause" at which time the flag file is created and it sleeps 10 seconds before checking for the pause file again.  It dosn't pause the whole system, just that specific sendjob.
If I am doing the math correctly, it would appear that you have a problem.  Here is what I am seeing.  There are 86,400 seconds in a day.  If you send 5,000 per hour you have a sending capacity plan of 120,000 messages per day.  You are currently below the 50% load for that plan.

50K per day = 1.7 seconds per message.
60K per day = 1.4 seconds per message.
1,000 per hour = 24,000 per day.

So it would seem that somewhere between 26,000 and 36,000 messages are not going to get sent.  I don't make policy here, I just do the analysis.  But it looks like the new plan will leave you with an artificial limit that is set at less than 1/4 of the current capacity.
ASKER CERTIFIED SOLUTION
Avatar of Scott Madeira
Scott Madeira
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
smadeira,

We didn't originally go with PHP List because of the amount of customization that was required for the project.  So I'm trying to implement a solution that will require the least amount of change to the existing application.  Your suggestion regarding the control of email to ISP's is something we are looking at as well.  I'm already creating a "send table" for each job, but are you suggesting I create a master "send table" that stores all records no matter what account (sendjob) it is sending from?

My original thought was to have a master flag file or a counter table in the DB that kept track as each email went out.  This however would require a file check and db update for each email that goes out.  I'm also assuming I would need a cron script to run every hour to reset the master counter and if the limit is hit, all jobs starting within the same hour after would have to be paused.

We do direct mail marketing here and we have employees that schedule our email from all of our accounts each day.  We need to send each account as a seperate list because our Dynect accounts each have different sending emails. I give them the ability to schedule the sends at any given time of the day, so I can't control the schedule times.
I took a look at PHP List and Pommo, both with email throttling and batch processing.  Rather than me back-engineering their code, can somebody give me a summary of how that works?  I know they are sticking the emails in a mail queue, which is being picked up by the cron, but how does it pick up new emails that are added after the cron has already started?