Link to home
Start Free TrialLog in
Avatar of Michael Sole
Michael Sole

asked on

Faster, reliable, email handling with pear's Net_SMTP

I have a class that I am using to send authenticated SMTP emails. The best I am able to get out of it is about 1.5-3 messages per second. I need to get it to go faster. I know pear has a Mail_Queue package and it IS much faster but I can't use it right now. I am hoping to improve this class a bit to get better imrpovement. The class is first invoked where a connection is made and then I loop through emails 1 by 1 sending them. Here is the class:
<?php
	class SMTPMail {
		//Member Variables
		public $host, $username, $password, $from, $to, $subject, $html_body, $plaintext_body, $status, $attachment_name;
		private $smtp, $mail, $body, $headers;
		
		//Constructor
		public function __construct( $host, $username, $password, $from ) {
			//Import PEAR SMTP Library
			require_once("Mail.php");
			require_once("Mail/mime.php");
			
			//Set Member Variables from Parameters
			$this->host = $host;
			$this->username = $username;
			$this->password = $password;
			$this->from = $from;
			
			//Create SMTP Connection
			$this->smtp = Mail::factory('smtp',
				array ('host' => $this->host,
					'auth' => true,
					'username' => $this->username,
					'password' => $this->password)
				);
				return true;
		}
		
		//Send Mail
		public function sendMail( $to, $subject, $html_body, $plaintext_body = "", $attachment_name="" ) {
			//Set Member Variables from Parameters
			$this->to = $to;
			$this->subject = $subject;
			$this->html_body = $html_body;
			$this->plaintext_body = $plaintext_body;
			$this->attachment = $attachment_name;
			// var_dump($this->attachment); //for debugging
			//Build Headers
			$headers = array('From' => $this->from,
				'To' => $this->to,
				'Subject' => $this->subject
			);
		
			//Set MIME Types and Build Message
			$mime = new Mail_mime();
			$mime->setTXTBody( $this->plaintext_body );
			$mime->setHTMLBody( $this->html_body );
			$mime->addAttachment( $this->attachment,'application/octet-stream' );
			$this->body = $mime->get();
			$this->headers = $mime->headers($headers);
		
			//Send the Message
			$this->mail = $this->smtp->send($this->to, $this->headers, $this->body);

			//Check for Error Message, Return True/False if E-mail Sent or Not
			if (PEAR::isError($this->mail)) {
				$this->status = "FAIL:Error Code: ".$this->mail->getMessage();
				return false;
			}
			else {
				$this->status = "Sent";
				//$this->status = "OK:".$mail->getMessage();
				return true;
			}
		}
		
		//Accessor Methods
		public function getStatus() {
			//Return E-Mail Status Message
			return $this->status;
		}
		
		public function getMail() {
			//Return Mail Object Created by PEAR Mail/PEAR Mime
			return $this->mail();
		}
		
		public function getBody() {
			//Return Message Object Created by PEAR Mail/PEAR Mime
			return $this->body;
		}
		
		public function getHeaders() {
			//Return Message Headers Object Created by PEAR Mail/PEAR Mime
			return $this->headers;
		}
		public function showAttachment() {
			//Return Message Headers Object Created by PEAR Mail/PEAR Mime
			return $this->attachment;
		}
	}
?>

Open in new window


Here is the code that grabs it:

#!/usr/bin/php -q
<?php
	/**
	 * Blast Manager Customizable Messaging Tool
	 *(C) 2010 Gold Mobile
	 * Author: Michael Sole
	 * Read Through Message Queue and Send Messages Using Transactions
	 * Using process Daemon
	 */
	 
	// Allowed arguments & their defaults
	$runmode = array(
		'no-daemon' => false,
		'help' => false,
		'write-initd' => false
	);
	 
	// Scan command line attributes for allowed arguments
	foreach ($argv as $k=>$arg) {
		if (substr($arg, 0, 2) == '--' && isset($runmode[substr($arg, 2)])) {
			$runmode[substr($arg, 2)] = true;
		}
	}
	 
	// Help mode. Shows allowed argumentents and quit directly
	if ($runmode['help'] == true) {
		echo 'Usage: '.$argv[0].' [runmode]' . "\n";
		echo 'Available runmodes:' . "\n";
		foreach ($runmode as $runmod=>$val) {
			echo ' --'.$runmod . "\n";
		}
		die();
	}
	 
	// Make it possible to test in source directory
	// This is for PEAR developers only
	ini_set('include_path', ini_get('include_path').':..');
	 
	// Include Class
	require_once 'System/Daemon.php';
	require_once 'user.php';
	
	// Setup
	$options = array(
		'appName' => 'process_email_queue',
		'appDir' => dirname(__FILE__),
		'appDescription' => 'Process Email Queue',
		'authorName' => 'Gold Mobile',
		'authorEmail' => 'msole@gold-group.com',
		'sysMaxExecutionTime' => '0',
		'sysMaxInputTime' => '0',
		'sysMemoryLimit' => '1024M',
		'appRunAsGID' => $gid,
		'appRunAsUID' => $uid,
		'logLocation' => '../../log/process_email_queue.log',
		'appPidLocation' =>dirname(__FILE__).'/process_email_queue/process_email_queue.pid',
	);
	 
	System_Daemon::setOptions($options);
	 
	// This program can also be run in the forground with runmode --no-daemon
	if (!$runmode['no-daemon']) {
		// Spawn Daemon
		System_Daemon::start();
	}
	
	$engine_script = true;
	require_once('../config.php');
	require_once('config.php');
	require_once('../functions.php');
	$log = Logger::getLogger('myLogger');
	$log->info('Process Email Queue');
	$Service 				= new Service($server, $dbuser, $dbpass, $db,$app_log_path_file);
	include($base_app_path.'includes/simple_html_dom.php');
	require_once($base_app_path.'classes/SMTPMail.class.php');
	 
	// This variable gives your own code the ability to breakdown the daemon:
	$runningOkay = true;
	 
	// This variable keeps track of how many 'runs' or 'loops' your daemon has
	// done so far.
	$cnt = 1;
	 
	// While checks on 3 things in this case:
	// - That the Daemon Class hasn't reported it's dying
	// - That your own code has been running Okay
	while (!System_Daemon::isDying() && $runningOkay ) {
		// What mode are we in?
		$mode = '"'.(System_Daemon::isInBackground() ? '' : 'non-' ).'daemon" mode';
		$starttime = $Service->process_script_start();
			$header_html = '<html xmlns="http://www.w3.org/1999/xhtml">
			<head>
			<meta http-equiv="Content-Language" content="en-us">
			<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
			<title>Blastmanager</title>
			</head>
			<body>';
			//Get Messages from Message Queue
			$Service->runQuery("BEGIN");
			$sql = "SELECT m.mid as mid,
									m.id as id,
									m.title as title, 
									m.email as email, 
									m.message as message, 
									s.server as server, 
									s.username as username, 
									s.password as password, 
									c.shortcode as shortcode,
									c.attach_file,
									c.attachment,
									c.display_name,
									c.category_name
								FROM bm_message_queue m 
								LEFT JOIN bm_categories c ON m.cid=c.id 
								LEFT JOIN bm_servers s ON c.server=s.id 
								WHERE 
									c.type = 'email' 
									AND sent=0 
									ORDER BY m.id asc 
									LIMIT ".$message_rate." FOR UPDATE;";
			$updatesql = "UPDATE bm_message_queue SET sent=1 where type = 'email' and sent=0 order by id asc limit ".$message_rate.";";
			$get_queue = $Service->runQuery($sql);
			$mark_sent = $Service->runQuery($updatesql);
			if( $get_queue && $mark_sent ) {
				if( $Service->runQuery("COMMIT") ) {
					//While loop Executes for Each Item in Queue
					$prev_smtp_host='';
					$prev_smtp_user='';  
					$prev_smtp_pass=''; 
					$prev_smtp_from='';
					while( $row = mysql_fetch_array( $get_queue ) ) {
						$smtp_host = $row['server'];
						$smtp_user = $row['username'];
						$smtp_pass = $row['password'];
						if ($row['display_name'] == null || $row['display_name'] =='') {
							$display_name = $row['username'];
						} else {
							$display_name = $row['display_name'];
						}
						$smtp_from = '"'.$display_name.'" <'.$row['username'].'>';
						$attachment='';
						if ($row['attach_file']==1) {
							$attachment=$row['attachment'];
						}
						
						if ($smtp_host!=$prev_smtp_host && $smtp_user!=$prev_smtp_user && $smtp_pass!=$prev_smtp_pass && $smtp_from!=$prev_smtp_from ) {
							$email = new SMTPMail( $smtp_host, $smtp_user, $smtp_pass, $smtp_from );
						}
						
						//Set To E-Mail Address, Subject, and (HTML) E-Mail Body [Optionally Plaintext Replacement if HTML Disaabled]
						$to 				= $row['email'];
						$subject 	= $row['title'];			
						$mq_id 		= $row['id']; //  message_queue ID
						$mid 	    	= $row['mid'] > 0 ? $row['mid'] : 0; //  message_queue mid
						$body 		= $header_html;
						$body 		.= stripslashes($row['message']);
						
						//Add open tracking here
						$clickurl = $clickpath ."?m=".$mid."&mq_id=".$mq_id."&e=".($row['email']);
						$clickhtml = "<img height=\"1\" width=\"1\" src=\"".$clickurl."\" /></body>";
						if (preg_match("/\<\/body\>/", stripslashes($body))) {
							$body = str_replace("</body>",$clickhtml,stripslashes($body));
						} else {
							$body .= $clickhtml;
						}
						
						// Add Click Tracking here
						// Create DOM from URL or file
						$html = new simple_html_dom();  
						$html->load($body); 
						// Find all links 
						$anchors = $html->find('a');
						foreach ($anchors as $key=>$value) {
							$dest_url 		= stripslashes($value->href);
							$value->href 	= sprintf('%s&t=1&url=%s', $clickurl, $dest_url);
						}
						$body = $html->save(); 
						$html->clear();
						
						$email->sendMail( $to, $subject, $body, $body, $attachment );
						$status = $email->getStatus()."\n";
						$status_update = "UPDATE bm_message_queue SET status = '".$status."' where id=".$mq_id;
						$Service->runQuery($status_update);
						$logentry = "Email: ".$to.":Status: ".$status;
						$Service->log->info($logentry);
						$smtp_host=$prev_smtp_host;
						$smtp_user=$prev_smtp_user;  
						$smtp_pass=$prev_smtp_pass; 
						$smtp_from=$prev_smtp_from;
					}
				} else {
					$Service->runQuery("ROLLBACK");
					$log->error('[' . $dbuser . '] Transaction Problem in Process Email Queue Script:');
				}
			} else {
				$log->error('[' . $dbuser . '] Transaction Problem in Process Email Queue Script:');
			}
	
			$totaltime = $Service->process_script_time($starttime);
			$log->info("This email queue was processed in ".$totaltime." seconds"); 
		if (!$runningOkay) {
			System_Daemon::err('parseLog() produced an error, '.
				'so this will be my last run');
		}
	 
		// Relax the system by sleeping for a little bit
		// iterate also clears statcache
		System_Daemon::iterate(2);
	 
		$cnt++;
	}
	mysql_close();
	// Shut down the daemon nicely
	// This is ignored if the class is actually running in the foreground
	System_Daemon::stop();

?>

Open in new window


It also uses the pear packaged System_Daemon so it runs in the background.

Please help!
ASKER CERTIFIED SOLUTION
Avatar of Ray Paseur
Ray Paseur
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Michael Sole
Michael Sole

ASKER

The problem is I am looking for a 2-3x increase in perfomance which means I am measuring in thenths of a second. I had my own mechanism for measure time (although the suggestion is a good one) and here is the time for each step:

1Current process time from start is: 11.612640142441
2Current process time from start is: 11.625669002533
3Current process time from start is: 11.972162008286
4Current process time from start is: 11.973493099213


1 is the beginning of the run
2 is after all the text replacing occurs
3 is after the mail is being sent
4 is after I update the status of the email

So it appears it takes .3 seconds to send one email. sendMail() is wrapper for pear's send function and all it is doing is populating variables in the right place.

I am wondering if there are some switches that could be used to improve speed. I have control over all the servers as they are either pizza boxes or VPS machines I admin. So I could tweak postfix to be better but I think the "bottleneck" is in Net_SMTP somewhere
Scanning the second code snippet, I find three class instantiations.  See line 147.  Each time the constructor is called it will make a connection to the server.  I am not sure as I read the code what the exact flow of logic is around that, but I expect that could be one source of slowdown.  Another might be on line 169.  Just guessing, however.  The way I might attack this is to put the timer readouts before and after each statement that could be causing the delay.  We know the delay is occurring after point 2 and before point 3.  If that comes down to a call to (for example) to the sendmail() method called on line 180, then the timer instrumentation might be moved over to the SmtpMail class.

One thought -- the UPDATE query on line 182 has no LIMIT clause, so it will cause a table scan.

Where is $message_rate defined?
Message_rate is defined in the config. The script only processes 120 messages each cycle to control memory usage.

The constructor is only called if the host,user and pass change otherwise it uses it.

The update command has a where clause and that field is indexed so I doubt it is causing a table scan. (its an auto incremented unique primary key).

I tried phpmailer and was able to double the speed of emails, however you idea about baseline and measuring allowed me to better determine the effectiveness.

I was hoping someone knew of settings in Net_SMTP that would improve its performance. I am not sure why its much slower than another class.

So unless you have any other thoughts I'll just award you the points ;)
Well, I know that phpMailer is a popular solution.  And there may be a germ of an idea in this: http://www.perlmonks.org/?node_id=161077

Whenever there are performance problems I always suspect the I/O subsystem first.  You might consider putting a stethoscope on the server hard drive when this is going on.

And even if the UPDATE command has a WHERE clause on an indexed column, I am not sure that the DB engine is smart enough to know when it should stop looking for a match.  I always add a LIMIT to my update commands unless I really want to update all the rows that match the WHERE.
I'll add the limit, it can't hurt but a unique index pretty much tells us there can only be 1 match, since its unique. Its kind of how indexes work.

And in fact, after running a quick test there was no difference in query time. If the field was not a unique index you are right, it would do a table scan.

Again thanks for your help.
Thanks for the points.  Glad to know the LIMIT was not really needed after all.  Best of luck with it, ~Ray