Faster, reliable, email handling with pear's Net_SMTP

I have a class that I am using to send authenticated SMTP emails. The best I am able to get out of it is about 1.5-3 messages per second. I need to get it to go faster. I know pear has a Mail_Queue package and it IS much faster but I can't use it right now. I am hoping to improve this class a bit to get better imrpovement. The class is first invoked where a connection is made and then I loop through emails 1 by 1 sending them. Here is the class:
<?php
	class SMTPMail {
		//Member Variables
		public $host, $username, $password, $from, $to, $subject, $html_body, $plaintext_body, $status, $attachment_name;
		private $smtp, $mail, $body, $headers;
		
		//Constructor
		public function __construct( $host, $username, $password, $from ) {
			//Import PEAR SMTP Library
			require_once("Mail.php");
			require_once("Mail/mime.php");
			
			//Set Member Variables from Parameters
			$this->host = $host;
			$this->username = $username;
			$this->password = $password;
			$this->from = $from;
			
			//Create SMTP Connection
			$this->smtp = Mail::factory('smtp',
				array ('host' => $this->host,
					'auth' => true,
					'username' => $this->username,
					'password' => $this->password)
				);
				return true;
		}
		
		//Send Mail
		public function sendMail( $to, $subject, $html_body, $plaintext_body = "", $attachment_name="" ) {
			//Set Member Variables from Parameters
			$this->to = $to;
			$this->subject = $subject;
			$this->html_body = $html_body;
			$this->plaintext_body = $plaintext_body;
			$this->attachment = $attachment_name;
			// var_dump($this->attachment); //for debugging
			//Build Headers
			$headers = array('From' => $this->from,
				'To' => $this->to,
				'Subject' => $this->subject
			);
		
			//Set MIME Types and Build Message
			$mime = new Mail_mime();
			$mime->setTXTBody( $this->plaintext_body );
			$mime->setHTMLBody( $this->html_body );
			$mime->addAttachment( $this->attachment,'application/octet-stream' );
			$this->body = $mime->get();
			$this->headers = $mime->headers($headers);
		
			//Send the Message
			$this->mail = $this->smtp->send($this->to, $this->headers, $this->body);

			//Check for Error Message, Return True/False if E-mail Sent or Not
			if (PEAR::isError($this->mail)) {
				$this->status = "FAIL:Error Code: ".$this->mail->getMessage();
				return false;
			}
			else {
				$this->status = "Sent";
				//$this->status = "OK:".$mail->getMessage();
				return true;
			}
		}
		
		//Accessor Methods
		public function getStatus() {
			//Return E-Mail Status Message
			return $this->status;
		}
		
		public function getMail() {
			//Return Mail Object Created by PEAR Mail/PEAR Mime
			return $this->mail();
		}
		
		public function getBody() {
			//Return Message Object Created by PEAR Mail/PEAR Mime
			return $this->body;
		}
		
		public function getHeaders() {
			//Return Message Headers Object Created by PEAR Mail/PEAR Mime
			return $this->headers;
		}
		public function showAttachment() {
			//Return Message Headers Object Created by PEAR Mail/PEAR Mime
			return $this->attachment;
		}
	}
?>

Open in new window


Here is the code that grabs it:

#!/usr/bin/php -q
<?php
	/**
	 * Blast Manager Customizable Messaging Tool
	 *(C) 2010 Gold Mobile
	 * Author: Michael Sole
	 * Read Through Message Queue and Send Messages Using Transactions
	 * Using process Daemon
	 */
	 
	// Allowed arguments & their defaults
	$runmode = array(
		'no-daemon' => false,
		'help' => false,
		'write-initd' => false
	);
	 
	// Scan command line attributes for allowed arguments
	foreach ($argv as $k=>$arg) {
		if (substr($arg, 0, 2) == '--' && isset($runmode[substr($arg, 2)])) {
			$runmode[substr($arg, 2)] = true;
		}
	}
	 
	// Help mode. Shows allowed argumentents and quit directly
	if ($runmode['help'] == true) {
		echo 'Usage: '.$argv[0].' [runmode]' . "\n";
		echo 'Available runmodes:' . "\n";
		foreach ($runmode as $runmod=>$val) {
			echo ' --'.$runmod . "\n";
		}
		die();
	}
	 
	// Make it possible to test in source directory
	// This is for PEAR developers only
	ini_set('include_path', ini_get('include_path').':..');
	 
	// Include Class
	require_once 'System/Daemon.php';
	require_once 'user.php';
	
	// Setup
	$options = array(
		'appName' => 'process_email_queue',
		'appDir' => dirname(__FILE__),
		'appDescription' => 'Process Email Queue',
		'authorName' => 'Gold Mobile',
		'authorEmail' => 'msole@gold-group.com',
		'sysMaxExecutionTime' => '0',
		'sysMaxInputTime' => '0',
		'sysMemoryLimit' => '1024M',
		'appRunAsGID' => $gid,
		'appRunAsUID' => $uid,
		'logLocation' => '../../log/process_email_queue.log',
		'appPidLocation' =>dirname(__FILE__).'/process_email_queue/process_email_queue.pid',
	);
	 
	System_Daemon::setOptions($options);
	 
	// This program can also be run in the forground with runmode --no-daemon
	if (!$runmode['no-daemon']) {
		// Spawn Daemon
		System_Daemon::start();
	}
	
	$engine_script = true;
	require_once('../config.php');
	require_once('config.php');
	require_once('../functions.php');
	$log = Logger::getLogger('myLogger');
	$log->info('Process Email Queue');
	$Service 				= new Service($server, $dbuser, $dbpass, $db,$app_log_path_file);
	include($base_app_path.'includes/simple_html_dom.php');
	require_once($base_app_path.'classes/SMTPMail.class.php');
	 
	// This variable gives your own code the ability to breakdown the daemon:
	$runningOkay = true;
	 
	// This variable keeps track of how many 'runs' or 'loops' your daemon has
	// done so far.
	$cnt = 1;
	 
	// While checks on 3 things in this case:
	// - That the Daemon Class hasn't reported it's dying
	// - That your own code has been running Okay
	while (!System_Daemon::isDying() && $runningOkay ) {
		// What mode are we in?
		$mode = '"'.(System_Daemon::isInBackground() ? '' : 'non-' ).'daemon" mode';
		$starttime = $Service->process_script_start();
			$header_html = '<html xmlns="http://www.w3.org/1999/xhtml">
			<head>
			<meta http-equiv="Content-Language" content="en-us">
			<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
			<title>Blastmanager</title>
			</head>
			<body>';
			//Get Messages from Message Queue
			$Service->runQuery("BEGIN");
			$sql = "SELECT m.mid as mid,
									m.id as id,
									m.title as title, 
									m.email as email, 
									m.message as message, 
									s.server as server, 
									s.username as username, 
									s.password as password, 
									c.shortcode as shortcode,
									c.attach_file,
									c.attachment,
									c.display_name,
									c.category_name
								FROM bm_message_queue m 
								LEFT JOIN bm_categories c ON m.cid=c.id 
								LEFT JOIN bm_servers s ON c.server=s.id 
								WHERE 
									c.type = 'email' 
									AND sent=0 
									ORDER BY m.id asc 
									LIMIT ".$message_rate." FOR UPDATE;";
			$updatesql = "UPDATE bm_message_queue SET sent=1 where type = 'email' and sent=0 order by id asc limit ".$message_rate.";";
			$get_queue = $Service->runQuery($sql);
			$mark_sent = $Service->runQuery($updatesql);
			if( $get_queue && $mark_sent ) {
				if( $Service->runQuery("COMMIT") ) {
					//While loop Executes for Each Item in Queue
					$prev_smtp_host='';
					$prev_smtp_user='';  
					$prev_smtp_pass=''; 
					$prev_smtp_from='';
					while( $row = mysql_fetch_array( $get_queue ) ) {
						$smtp_host = $row['server'];
						$smtp_user = $row['username'];
						$smtp_pass = $row['password'];
						if ($row['display_name'] == null || $row['display_name'] =='') {
							$display_name = $row['username'];
						} else {
							$display_name = $row['display_name'];
						}
						$smtp_from = '"'.$display_name.'" <'.$row['username'].'>';
						$attachment='';
						if ($row['attach_file']==1) {
							$attachment=$row['attachment'];
						}
						
						if ($smtp_host!=$prev_smtp_host && $smtp_user!=$prev_smtp_user && $smtp_pass!=$prev_smtp_pass && $smtp_from!=$prev_smtp_from ) {
							$email = new SMTPMail( $smtp_host, $smtp_user, $smtp_pass, $smtp_from );
						}
						
						//Set To E-Mail Address, Subject, and (HTML) E-Mail Body [Optionally Plaintext Replacement if HTML Disaabled]
						$to 				= $row['email'];
						$subject 	= $row['title'];			
						$mq_id 		= $row['id']; //  message_queue ID
						$mid 	    	= $row['mid'] > 0 ? $row['mid'] : 0; //  message_queue mid
						$body 		= $header_html;
						$body 		.= stripslashes($row['message']);
						
						//Add open tracking here
						$clickurl = $clickpath ."?m=".$mid."&mq_id=".$mq_id."&e=".($row['email']);
						$clickhtml = "<img height=\"1\" width=\"1\" src=\"".$clickurl."\" /></body>";
						if (preg_match("/\<\/body\>/", stripslashes($body))) {
							$body = str_replace("</body>",$clickhtml,stripslashes($body));
						} else {
							$body .= $clickhtml;
						}
						
						// Add Click Tracking here
						// Create DOM from URL or file
						$html = new simple_html_dom();  
						$html->load($body); 
						// Find all links 
						$anchors = $html->find('a');
						foreach ($anchors as $key=>$value) {
							$dest_url 		= stripslashes($value->href);
							$value->href 	= sprintf('%s&t=1&url=%s', $clickurl, $dest_url);
						}
						$body = $html->save(); 
						$html->clear();
						
						$email->sendMail( $to, $subject, $body, $body, $attachment );
						$status = $email->getStatus()."\n";
						$status_update = "UPDATE bm_message_queue SET status = '".$status."' where id=".$mq_id;
						$Service->runQuery($status_update);
						$logentry = "Email: ".$to.":Status: ".$status;
						$Service->log->info($logentry);
						$smtp_host=$prev_smtp_host;
						$smtp_user=$prev_smtp_user;  
						$smtp_pass=$prev_smtp_pass; 
						$smtp_from=$prev_smtp_from;
					}
				} else {
					$Service->runQuery("ROLLBACK");
					$log->error('[' . $dbuser . '] Transaction Problem in Process Email Queue Script:');
				}
			} else {
				$log->error('[' . $dbuser . '] Transaction Problem in Process Email Queue Script:');
			}
	
			$totaltime = $Service->process_script_time($starttime);
			$log->info("This email queue was processed in ".$totaltime." seconds"); 
		if (!$runningOkay) {
			System_Daemon::err('parseLog() produced an error, '.
				'so this will be my last run');
		}
	 
		// Relax the system by sleeping for a little bit
		// iterate also clears statcache
		System_Daemon::iterate(2);
	 
		$cnt++;
	}
	mysql_close();
	// Shut down the daemon nicely
	// This is ignored if the class is actually running in the foreground
	System_Daemon::stop();

?>

Open in new window


It also uses the pear packaged System_Daemon so it runs in the background.

Please help!
skioneAsked:
Who is Participating?
 
Ray PaseurCommented:
http://gold-group.com/ returns a 403 Forbidden

The variable $message_rate has an interesting name.  But it appears that it is undefined in the existing code.

I am curious about the code online 208.  What is that actually doing?

Most of the time when there are performance problems it's useful to isolate them with a timer.  This is a timer class that may be helpful. Try inserting timers around various control structures and seeing where the script is spending its time.
<?php // RAY_class_Stopwatch.php
error_reporting(E_ALL);


// DEMONSTRATE A SCRIPT TIMER FOR ALL OR PART OF A SCRIPT PHP 5+
// MAN PAGE http://php.net/manual/en/function.microtime.php


class StopWatch
{
    protected $a; // START TIME
    protected $s; // STATUS - IF RUNNING
    protected $z; // STOP TIME

    public function __construct()
    {
        $this->a = array();
        $this->s = array();
        $this->z = array();
    }

    // A METHOD TO REMOVE A TIMER
    public function reset($name='TIMER')
    {
        // RESET ALL TIMERS
        if ($name == 'TIMER')
        {
            $this->__construct();
        }
        else
        {
            unset($this->a[$name]);
            unset($this->s[$name]);
            unset($this->z[$name]);
        }
    }

    // A METHOD TO CAPTURE THE START TIME
    public function start($name='TIMER')
    {
        $this->a[$name] = microtime(TRUE);
        $this->z[$name] = $this->a[$name];
        $this->s[$name] = 'RUNNING';
    }

    // A METHOD TO CAPTURE THE END TIME
    public function stop($name='TIMER')
    {
        $ret = NULL;

        // STOP ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                // IF THIS TIMER IS STILL RUNNING, STOP IT
                if ($this->s[$name])
                {
                    $this->s[$name] = FALSE;
                    $this->z[$name] = microtime(TRUE);
                }
            }
        }

        // STOP ONLY ONE OF THE TIMERS
        else
        {
            if ($this->s[$name])
            {
                $this->s[$name] = FALSE;
                $this->z[$name] = microtime(TRUE);
            }
            else
            {
                $ret .= "ERROR: CALL TO STOP() METHOD FOR '$name' IS NOT RUNNING";
            }
        }

        // RETURN AN ERROR MESSAGE, IF ANY
        return $ret;
    }

    // A METHOD TO READ OUT THE TIMER(S)
    public function readout($name='TIMER', $dec=3, $m=1000, $eol=PHP_EOL)
    {
        $str = NULL;

        // GET READOUTS FOR ALL THE TIMERS
        if ($name == 'TIMER')
        {
            foreach ($this->a as $name => $start_time)
            {
                $str .= $name;

                // IF THIS TIMER IS STILL RUNNING UPDATE THE END TIME
                if ($this->s[$name])
                {
                    $this->z[$name] = microtime(TRUE);
                    $str .= " RUNNING ";
                }
                else
                {
                    $str .= " STOPPED ";
                }

                // RETURN A DISPLAY STRING
                $lapse_time = $this->z[$name] - $start_time;
                $lapse_msec = $lapse_time * $m;
                $lapse_echo = number_format($lapse_msec, $dec);
                $str .= " $lapse_echo";
                $str .= $eol;
            }
            return $str;
        }

        // GET A READOUT FOR ONLY ONE TIMER
        else
        {
            $str .= $name;

            // IF THIS TIME IS STILL RUNNING, UPDATE THE END TIME
            if ($this->s[$name])
            {
                $this->z[$name] = microtime(TRUE);
                $str .= " RUNNING ";
            }
            else
            {
                $str .= " STOPPED ";
            }


            // RETURN A DISPLAY STRING
            $lapse_time = $this->z[$name] - $this->a[$name];
            $lapse_msec = $lapse_time * $m;
            $lapse_echo = number_format($lapse_msec, $dec);
            $str .= " $lapse_echo";
            $str .= $eol;
            return $str;
        }
    }
}



// DEMONSTRATE THE USE -- INSTANTIATE THE STOPWATCH OBJECT
$sw  = new Stopwatch;

// SET A STOPWATCH NAME THAT REFLECTS THE PARTS OF THE SCRIPT WE WANT TO TIME
$g_timer = 'GOOGLE TIMER';

// START A TIMER TO GET ELAPSED TIME FOR A CALL TO GOOGLE
$sw->start($g_timer);

// PERFORM SOME ACTIVITY THAT YOU WANT TO TIME (READS GOOGLE WEB PAGE)
$page = 'http://google.com';
$html = file_get_contents($page);

// GET A READOUT OF THE TIMER WHILE IT IS STILL RUNNING
echo nl2br($sw->readout($g_timer));
echo "<br/>" . PHP_EOL;

// PERFORM SOME OTHER ACTIVITY (READS GOOGLE WEB PAGE AGAIN)
$page = 'http://google.com';
$html = file_get_contents($page);

// STOP THE TIMER AND GET A READOUT WITH SHORT DECIMALS
$x = $sw->stop($g_timer);
echo nl2br($sw->readout($g_timer, 1));
echo "<br/>" . PHP_EOL;




// START A SECOND TIMER
$y_timer = 'YAHOO TIMER';
$sw->start($y_timer);

// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME
$page = 'http://yahoo.com/';
$html = file_get_contents($page);

// REPORT THE STOPWATCHES CONTENT (ONE IS STOPPED AND ONE IS STILL RUNNING)
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";

// STOP ALL OF THE STOPWATCHES
$sw->stop();

// REPORT THE STOPWATCHES CONTENT AGAIN
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// TRY TO STOP A TIMER THAT IS NOT RUNNING
$x = $sw->stop($g_timer);
var_dump($x);
echo "<br/>" . PHP_EOL;
echo "<br/>" . PHP_EOL;



// START THIS TIMER OVER AGAIN
$sw->start($y_timer);

// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME
$page = 'http://weather.yahoo.com/';
$html = file_get_contents($page);

// REPORT THE STOPWATCHES CONTENT
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// REMOVE ONE OF THE STOPWATCHES
$sw->reset($g_timer);

// REPORT THE STOPWATCHES CONTENT
echo nl2br($sw->readout());
echo "<br/>" . PHP_EOL;

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";



// REMOVE ALL OF THE STOPWATCHES
$sw->reset();
echo "ALL STOPWATCHES HAVE BEEN REMOVED";

// REPORT THE STOPWATCHES CONTENT (SHOWS NOTHING)
echo nl2br($sw->readout());

// SHOW THE OBJECT
echo "<pre>";
var_dump($sw);
echo "</pre>";

Open in new window

0
 
skioneAuthor Commented:
The problem is I am looking for a 2-3x increase in perfomance which means I am measuring in thenths of a second. I had my own mechanism for measure time (although the suggestion is a good one) and here is the time for each step:

1Current process time from start is: 11.612640142441
2Current process time from start is: 11.625669002533
3Current process time from start is: 11.972162008286
4Current process time from start is: 11.973493099213


1 is the beginning of the run
2 is after all the text replacing occurs
3 is after the mail is being sent
4 is after I update the status of the email

So it appears it takes .3 seconds to send one email. sendMail() is wrapper for pear's send function and all it is doing is populating variables in the right place.

I am wondering if there are some switches that could be used to improve speed. I have control over all the servers as they are either pizza boxes or VPS machines I admin. So I could tweak postfix to be better but I think the "bottleneck" is in Net_SMTP somewhere
0
 
Ray PaseurCommented:
Scanning the second code snippet, I find three class instantiations.  See line 147.  Each time the constructor is called it will make a connection to the server.  I am not sure as I read the code what the exact flow of logic is around that, but I expect that could be one source of slowdown.  Another might be on line 169.  Just guessing, however.  The way I might attack this is to put the timer readouts before and after each statement that could be causing the delay.  We know the delay is occurring after point 2 and before point 3.  If that comes down to a call to (for example) to the sendmail() method called on line 180, then the timer instrumentation might be moved over to the SmtpMail class.

One thought -- the UPDATE query on line 182 has no LIMIT clause, so it will cause a table scan.

Where is $message_rate defined?
0
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
skioneAuthor Commented:
Message_rate is defined in the config. The script only processes 120 messages each cycle to control memory usage.

The constructor is only called if the host,user and pass change otherwise it uses it.

The update command has a where clause and that field is indexed so I doubt it is causing a table scan. (its an auto incremented unique primary key).

I tried phpmailer and was able to double the speed of emails, however you idea about baseline and measuring allowed me to better determine the effectiveness.

I was hoping someone knew of settings in Net_SMTP that would improve its performance. I am not sure why its much slower than another class.

So unless you have any other thoughts I'll just award you the points ;)
0
 
Ray PaseurCommented:
Well, I know that phpMailer is a popular solution.  And there may be a germ of an idea in this: http://www.perlmonks.org/?node_id=161077

Whenever there are performance problems I always suspect the I/O subsystem first.  You might consider putting a stethoscope on the server hard drive when this is going on.

And even if the UPDATE command has a WHERE clause on an indexed column, I am not sure that the DB engine is smart enough to know when it should stop looking for a match.  I always add a LIMIT to my update commands unless I really want to update all the rows that match the WHERE.
0
 
skioneAuthor Commented:
I'll add the limit, it can't hurt but a unique index pretty much tells us there can only be 1 match, since its unique. Its kind of how indexes work.

And in fact, after running a quick test there was no difference in query time. If the field was not a unique index you are right, it would do a table scan.

Again thanks for your help.
0
 
Ray PaseurCommented:
Thanks for the points.  Glad to know the LIMIT was not really needed after all.  Best of luck with it, ~Ray
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.