Limit amount of emails sent per hour

We have an internal ESP that uses third party SMTP (Dynect) to send our emails out.  Once the scheduled sendjob is picked up by the cron job it builds an array of all the member emails that need to be sent to.  Once it has completed that, it creates the SMTP connection and begins to loop through the array sending one email at a time (we can't batch because we have dynamic fields that need to be populated for each email going out).

If the job needs to be paused or un-paused I initiate the creation of a flag file that is checked for each time through the loop:

                     if(file_get_contents($filename) == "STOP")

As you can see, there are two states of the flag file.  If the file contains "STOP", it initiates the deletion of the sendjob.  If the file is simply present but without "STOP" than it is a pause file and the script sleeps for 10 seconds before checking again.

Now my question is that I am now being asked to be able to set a limit of emails being sent per hour.  Would I use a flag file as well or is there another method I could use?  I don't want the jobs to stop completely, I just want them to pause if our hourly limit has been met and then start up again in the next hour until the threshold is reached again.  My only concern is having all the processes sleeping at once.  In one hour we may have 10 to 15 jobs sending at once and I want an overall limit set.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ray PaseurCommented:
It seems to me that you might want to use some kind of data base to control this.  Example of the idea: When it is time to send, create tables with each hourly group of emails.  If you're sending at the rate of 2 per second and you wanted a limit of 7,200 per hour, you could make each of the tables contain 7,200 rows.  As one table is exhausted, move on to the next table.

Overall how many emails are you sending and at what rate now?  Are your scripts sleeping between outgoing emails?
kageboy2003Author Commented:
Our normal application runs about 50 to 60k a day, about 4-5k an hour, but now we want to test throttling to prevent an onslaught with the major ISP's.  So I've been told that we will only be running 5-10k a day through the new application.  However, we want to throttle it at 1000 an hour.  I know how to limit an individual sendjob, but I need a way to keep a master count and limit ALL jobs per hour.  For instance, at 8:00 am tomorrow we will have 15 different sendjobs (to different accounts) starting at or around the same time (1 minute apart).  So lets say the first job has 800 emails.  Then when the second job starts,  I need to check the overall status of the first job and see if there is any left on the limit. If the limit has been reached, then all other jobs need to be delayed or paused until the hour is up and then the limit is reset.

The only time the script sleeps is if the admin initiates a "pause" at which time the flag file is created and it sleeps 10 seconds before checking for the pause file again.  It dosn't pause the whole system, just that specific sendjob.
Ray PaseurCommented:
If I am doing the math correctly, it would appear that you have a problem.  Here is what I am seeing.  There are 86,400 seconds in a day.  If you send 5,000 per hour you have a sending capacity plan of 120,000 messages per day.  You are currently below the 50% load for that plan.

50K per day = 1.7 seconds per message.
60K per day = 1.4 seconds per message.
1,000 per hour = 24,000 per day.

So it would seem that somewhere between 26,000 and 36,000 messages are not going to get sent.  I don't make policy here, I just do the analysis.  But it looks like the new plan will leave you with an artificial limit that is set at less than 1/4 of the current capacity.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Scott MadeiraCommented:
phplist is a mailing program that allows you to throttle emails.  You may want to download it and have a look at how they implement these capabilities.  It may give you some ideas.

From an architecture perspective you may want to look at implementing an email manager.  All of your jobs would use the email manager to coordinate sending emails.  That way you don't have 10 to 15 jobs running asynchronously and sending emails without regard to any of the other jobs.

We implemented something like that in one of our apps where the application would create a database record for each email it wants to send.  The email manager would then pull the data out of the table, send the email and move to the next one.

Another advantage of this approach is the ability to control emails to major ISPs.  Your email manager could track the number of emails sent to each ISP.  You could develop a scheduling algorithm that lets you know when you have the ability to send 200 more emails to yahoo addresses, for example.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
kageboy2003Author Commented:

We didn't originally go with PHP List because of the amount of customization that was required for the project.  So I'm trying to implement a solution that will require the least amount of change to the existing application.  Your suggestion regarding the control of email to ISP's is something we are looking at as well.  I'm already creating a "send table" for each job, but are you suggesting I create a master "send table" that stores all records no matter what account (sendjob) it is sending from?

My original thought was to have a master flag file or a counter table in the DB that kept track as each email went out.  This however would require a file check and db update for each email that goes out.  I'm also assuming I would need a cron script to run every hour to reset the master counter and if the limit is hit, all jobs starting within the same hour after would have to be paused.

We do direct mail marketing here and we have employees that schedule our email from all of our accounts each day.  We need to send each account as a seperate list because our Dynect accounts each have different sending emails. I give them the ability to schedule the sends at any given time of the day, so I can't control the schedule times.
kageboy2003Author Commented:
I took a look at PHP List and Pommo, both with email throttling and batch processing.  Rather than me back-engineering their code, can somebody give me a summary of how that works?  I know they are sticking the emails in a mail queue, which is being picked up by the cron, but how does it pick up new emails that are added after the cron has already started?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.