Solved

php mysql routine for creating report

Posted on 2009-04-15
7
273 Views
Last Modified: 2013-12-12
I have data in a system that I need to use to create some fairly complex reports. I tried to create the sql queries to get the data out in the way I want it, but they just take too long to run, so I finally created a script that runs in a chron job to generate a table with the data accumulated that I need for the tables. The problem is, this script is now failing, presumably timing out, and I don't know how to either break it up so that it can run without failure, nor can I figure out how to get it to let me know when it does fail. The data on the system changes constantly so I do have to be able to be able to run the script on the full data set, not just on newly added data.
<?php
 

///////////////////////////////////////////////////////////

//// DN: This code is run nightly via a cron job

//// to create the data needed for generating

//// direct contact reports for the Virginia data.

//// It takes too long to run to generate the data

//// on the fly, no matter how much I try to optimize

//// it!

//////////////////////////////////////////////////////////
 

require_once("fsnep_ini.php");

require_once("header.php");

require_once("functions.php");

ini_set("max_execution_time",3600);

ini_set("memory_limit","120M");

error_reporting(E_ALL);
 
 

$ethnic = Array();

$ethnic[0] = "American<BR>Indian/Alaska<br>Native";

$ethnic[1] = "Asian";

$ethnic[2] = "African<br>American";

$ethnic[3] = "Hispanic/<br>White";

$ethnic[4] = "N. Hawaiian/<br>Pac. Islander";

$ethnic[5] = "White";

$ethnic[6] = "Other/More<br> than one<br>race";
 
 
 
 

$old_id = 0;

// report filters

$where = 'sd.service_id = s.service_id';
 

$qry = "SELECT s.user_id, s.service_id, s.activity, s.instructor_id, s.location_id, s.county_id, s.audience, s.contact_type, s.num_sessions, s.series_cat, s.created, s.final, s.user_id, sd.* FROM session_data sd, service s WHERE ".$where." ORDER BY s.service_id DESC, sd.session_num";

//echo "SQL: $qry <br>";
 

//get rid of old report data

$sql2 = "delete from temp_direct_report where id>0";

$result2 = mysql_query($sql2) or die("SQL error: ".mysql_error());
 

$id = rand(100, 1000000);
 

$result = mysql_query($qry) or die("SQL error: ".mysql_error());

$numrows = @mysql_num_rows($result);
 

Echo "<br><br><b>Please wait, creating temporary table</b>...";
 

   // report body

   if ($numrows > 0)

   {

      while ($sd = mysql_fetch_assoc($result))

      {

		$session_data_id = $sd['session_data_id'];

		$lname = get_field('users','lname','user_id='.$sd['instructor_id']);

		$fname = get_field('users','fname','user_id='.$sd['instructor_id']);

		$cc_alert_1 = get_field('users','cc_alert_1','user_id='.$sd['instructor_id']);

		$cc_alert_2 = get_field('users','cc_alert_2','user_id='.$sd['instructor_id']);

		$educator = $lname.", ".$fname;

		$instructor_id = $sd['instructor_id'];
 

		echo $lname."...";
 

		$session_date = $sd['service_date'];

		$session_num = $sd['session_num'];

		$service_id = $sd['service_id'];
 

//White, African American, Hispanic/White, Other/More than one race, Asian, American Indian/Alaska Native, N. Hawaiian/Pac. Islander

			//need to pull these out of the participant_by_category table...

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='Hispanic/White' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());

			$m_white_hisp_u5 = 0;

			$m_white_hisp_5 = 0;

			$m_white_hisp_19 = 0;

			$m_white_hisp_65 = 0;

			$f_white_hisp_u5 = 0;

			$f_white_hisp_5 = 0;

			$f_white_hisp_19 = 0;

			$f_white_hisp_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_white_hisp_u5 = $row['m_u5'];

				$m_white_hisp_5  = $row['m_5'];

				$m_white_hisp_19 = $row['m_19'];

				$m_white_hisp_65 = $row['m_65'];

				$f_white_hisp_u5 = $row['f_u5'];

				$f_white_hisp_5  = $row['f_5'];

				$f_white_hisp_19 = $row['f_19'];

				$f_white_hisp_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_white_hisp_u5 = 0;

				$accum_m_white_hisp_5 = 0;

				$accum_m_white_hisp_19 = 0;

				$accum_m_white_hisp_65 = 0;

				$accum_f_white_hisp_u5 = 0;

				$accum_f_white_hisp_5 = 0;

				$accum_f_white_hisp_19 = 0;

				$accum_f_white_hisp_65 = 0;

			  }

			$accum_m_white_hisp_u5 += $row['m_u5'];

			$accum_m_white_hisp_5  += $row['m_5'];

			$accum_m_white_hisp_19 += $row['m_19'];

			$accum_m_white_hisp_65 += $row['m_65'];

			$accum_f_white_hisp_u5 += $row['f_u5'];

			$accum_f_white_hisp_5  += $row['f_5'];

			$accum_f_white_hisp_19 += $row['f_19'];

			$accum_f_white_hisp_65 += $row['f_65'];
 

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='White' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());

			$m_white_non_hisp_u5 = 0;

			$m_white_non_hisp_5 = 0;

			$m_white_non_hisp_19 = 0;

			$m_white_non_hisp_65 = 0;

			$f_white_non_hisp_u5 = 0;

			$f_white_non_hisp_5 = 0;

			$f_white_non_hisp_19 = 0;

			$f_white_non_hisp_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_white_non_hisp_u5 = $row['m_u5'];

				$m_white_non_hisp_5  = $row['m_5'];

				$m_white_non_hisp_19 = $row['m_19'];

				$m_white_non_hisp_65 = $row['m_65'];

				$f_white_non_hisp_u5 = $row['f_u5'];

				$f_white_non_hisp_5  = $row['f_5'];

				$f_white_non_hisp_19 = $row['f_19'];

				$f_white_non_hisp_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_white_non_hisp_u5 = 0;

				$accum_m_white_non_hisp_5 = 0;

				$accum_m_white_non_hisp_19 = 0;

				$accum_m_white_non_hisp_65 = 0;

				$accum_f_white_non_hisp_u5 = 0;

				$accum_f_white_non_hisp_5 = 0;

				$accum_f_white_non_hisp_19 = 0;

				$accum_f_white_non_hisp_65 = 0;

			  }

			$accum_m_white_non_hisp_u5 += $row['m_u5']; 

			$accum_m_white_non_hisp_5  += $row['m_5'];  

			$accum_m_white_non_hisp_19 += $row['m_19']; 

			$accum_m_white_non_hisp_65 += $row['m_65']; 

			$accum_f_white_non_hisp_u5 += $row['f_u5']; 

			$accum_f_white_non_hisp_5  += $row['f_5'];  

			$accum_f_white_non_hisp_19 += $row['f_19']; 

			$accum_f_white_non_hisp_65 += $row['f_65']; 
 

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='African American' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());

			$m_aa_u5 = 0;

			$m_aa_5 = 0;

			$m_aa_19 = 0;

			$m_aa_65 = 0;

			$f_aa_u5 = 0;

			$f_aa_5 = 0;

			$f_aa_19 = 0;

			$f_aa_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_aa_u5 = $row['m_u5'];

				$m_aa_5  = $row['m_5'];

				$m_aa_19 = $row['m_19'];

				$m_aa_65 = $row['m_65'];

				$f_aa_u5 = $row['f_u5'];

				$f_aa_5  = $row['f_5'];

				$f_aa_19 = $row['f_19'];

				$f_aa_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_aa_u5 = 0;

				$accum_m_aa_5 = 0;

				$accum_m_aa_19 = 0;

				$accum_m_aa_65 = 0;

				$accum_f_aa_u5 = 0;

				$accum_f_aa_5 = 0;

				$accum_f_aa_19 = 0;

				$accum_f_aa_65 = 0;

			  }

			$accum_m_aa_u5 += $row['m_u5']; 

			$accum_m_aa_5  += $row['m_5'];  

			$accum_m_aa_19 += $row['m_19']; 

			$accum_m_aa_65 += $row['m_65']; 

			$accum_f_aa_u5 += $row['f_u5']; 

			$accum_f_aa_5  += $row['f_5'];  

			$accum_f_aa_19 += $row['f_19']; 

			$accum_f_aa_65 += $row['f_65']; 

			

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='Asian' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());

			$m_asian_u5 = 0;

			$m_asian_5 = 0;

			$m_asian_19 = 0;

			$m_asian_65 = 0;

			$f_asian_u5 = 0;

			$f_asian_5 = 0;

			$f_asian_19 = 0;

			$f_asian_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_asian_u5 = $row['m_u5'];

				$m_asian_5  = $row['m_5'];

				$m_asian_19 = $row['m_19'];

				$m_asian_65 = $row['m_65'];

				$f_asian_u5 = $row['f_u5'];

				$f_asian_5  = $row['f_5'];

				$f_asian_19 = $row['f_19'];

				$f_asian_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_asian_u5 = 0;

				$accum_m_asian_5 = 0;

				$accum_m_asian_19 = 0;

				$accum_m_asian_65 = 0;

				$accum_f_asian_u5 = 0;

				$accum_f_asian_5 = 0;

				$accum_f_asian_19 = 0;

				$accum_f_asian_65 = 0;

			  }

			$accum_m_asian_u5 += $row['m_u5']; 

			$accum_m_asian_5  += $row['m_5'];  

			$accum_m_asian_19 += $row['m_19']; 

			$accum_m_asian_65 += $row['m_65']; 

			$accum_f_asian_u5 += $row['f_u5']; 

			$accum_f_asian_5  += $row['f_5'];  

			$accum_f_asian_19 += $row['f_19']; 

			$accum_f_asian_65 += $row['f_65']; 	
 

//White, African American, Hispanic/White, Other/More than one race, Asian, American Indian/Alaska Native, N. Hawaiian/Pac. Islander
 

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='Other/More than one race' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());

			$m_asian_u5 = 0;

			$m_other_u5 = 0;

			$m_other_5 = 0;

			$m_other_19 = 0;

			$m_other_65 = 0;

			$f_other_u5 = 0;

			$f_other_5 = 0;

			$f_other_19 = 0;

			$f_other_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_other_u5 = $row['m_u5'];

				$m_other_5  = $row['m_5'];

				$m_other_19 = $row['m_19'];

				$m_other_65 = $row['m_65'];

				$f_other_u5 = $row['f_u5'];

				$f_other_5  = $row['f_5'];

				$f_other_19 = $row['f_19'];

				$f_other_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_other_u5 = 0;

				$accum_m_other_5 = 0;

				$accum_m_other_19 = 0;

				$accum_m_other_65 = 0;

				$accum_f_other_u5 = 0;

				$accum_f_other_5 = 0;

				$accum_f_other_19 = 0;

				$accum_f_other_65 = 0;

			  }

			$accum_m_other_u5 += $row['m_u5']; 

			$accum_m_other_5  += $row['m_5'];  

			$accum_m_other_19 += $row['m_19']; 

			$accum_m_other_65 += $row['m_65']; 

			$accum_f_other_u5 += $row['f_u5']; 

			$accum_f_other_5  += $row['f_5'];  

			$accum_f_other_19 += $row['f_19']; 

			$accum_f_other_65 += $row['f_65']; 	

			
 

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='American Indian/Alaska Native' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());

			$m_na_u5 = 0;

			$m_na_5 = 0;

			$m_na_19 = 0;

			$m_na_65 = 0;

			$f_na_u5 = 0;

			$f_na_5 = 0;

			$f_na_19 = 0;

			$f_na_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_na_u5 = $row['m_u5'];

				$m_na_5  = $row['m_5'];

				$m_na_19 = $row['m_19'];

				$m_na_65 = $row['m_65'];

				$f_na_u5 = $row['f_u5'];

				$f_na_5  = $row['f_5'];

				$f_na_19 = $row['f_19'];

				$f_na_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_na_u5 = 0;

				$accum_m_na_5 = 0;

				$accum_m_na_19 = 0;

				$accum_m_na_65 = 0;

				$accum_f_na_u5 = 0;

				$accum_f_na_5 = 0;

				$accum_f_na_19 = 0;

				$accum_f_na_65 = 0;

			  }

			$accum_m_na_u5 += $row['m_u5']; 

			$accum_m_na_5  += $row['m_5'];  

			$accum_m_na_19 += $row['m_19']; 

			$accum_m_na_65 += $row['m_65']; 

			$accum_f_na_u5 += $row['f_u5']; 

			$accum_f_na_5  += $row['f_5'];  

			$accum_f_na_19 += $row['f_19']; 

			$accum_f_na_65 += $row['f_65']; 
 
 
 

			$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='N. Hawaiian/Pac. Islander' LIMIT 1";

			//echo "sql: $sql <br>";

			$sql = mysql_query($sql) or die (mysql_error());
 

			$m_hawaiian_u5 = 0;

			$m_hawaiian_5 = 0;

			$m_hawaiian_19 = 0;

			$m_hawaiian_65 = 0;

			$f_hawaiian_u5 = 0;

			$f_hawaiian_5 = 0;

			$f_hawaiian_19 = 0;

			$f_hawaiian_65 = 0;
 

			if(mysql_num_rows($sql) > 0)

			{

				$row = mysql_fetch_assoc($sql);

				$m_hawaiian_u5 = $row['m_u5'];

				$m_hawaiian_5  = $row['m_5'];

				$m_hawaiian_19 = $row['m_19'];

				$m_hawaiian_65 = $row['m_65'];

				$f_hawaiian_u5 = $row['f_u5'];

				$f_hawaiian_5  = $row['f_5'];

				$f_hawaiian_19 = $row['f_19'];

				$f_hawaiian_65 = $row['f_65'];

			}

			if ($old_id <> $service_id)

			  {

				//reset the accumulated 

				$accum_m_hawaiian_u5 = 0;

				$accum_m_hawaiian_5 = 0;

				$accum_m_hawaiian_19 = 0;

				$accum_m_hawaiian_65 = 0;

				$accum_f_hawaiian_u5 = 0;

				$accum_f_hawaiian_5 = 0;

				$accum_f_hawaiian_19 = 0;

				$accum_f_hawaiian_65 = 0;

			  }

			$accum_m_hawaiian_u5 += $row['m_u5']; 

			$accum_m_hawaiian_5  += $row['m_5'];  

			$accum_m_hawaiian_19 += $row['m_19']; 

			$accum_m_hawaiian_65 += $row['m_65']; 

			$accum_f_hawaiian_u5 += $row['f_u5']; 

			$accum_f_hawaiian_5  += $row['f_5'];  

			$accum_f_hawaiian_19 += $row['f_19']; 

			$accum_f_hawaiian_65 += $row['f_65']; 	
 

//now, write it into the temporary table used for the reports
 

				$sql3 = "insert into temp_direct_report (id, service_id, session_data_id, session_date, session_num, instructor_id, cc_alert_1, cc_alert_2, educator,m_white_non_hisp_u5, m_white_non_hisp_5,m_white_non_hisp_19, m_white_non_hisp_65,f_white_non_hisp_u5,f_white_non_hisp_5,f_white_non_hisp_19,f_white_non_hisp_65,accum_m_white_non_hisp_u5,accum_m_white_non_hisp_5,accum_m_white_non_hisp_19,accum_m_white_non_hisp_65,accum_f_white_non_hisp_u5,accum_f_white_non_hisp_5,accum_f_white_non_hisp_19,accum_f_white_non_hisp_65,	m_white_hisp_u5,m_white_hisp_5, m_white_hisp_19,m_white_hisp_65,f_white_hisp_u5,f_white_hisp_5,f_white_hisp_19,f_white_hisp_65,accum_m_white_hisp_u5,accum_m_white_hisp_5,accum_m_white_hisp_19,accum_m_white_hisp_65,accum_f_white_hisp_u5,accum_f_white_hisp_5,accum_f_white_hisp_19,accum_f_white_hisp_65,m_aa_u5,m_aa_5,m_aa_19,m_aa_65,f_aa_u5,f_aa_5,f_aa_19,f_aa_65,accum_m_aa_u5,accum_m_aa_5,accum_m_aa_19,accum_m_aa_65,accum_f_aa_u5,accum_f_aa_5,accum_f_aa_19,accum_f_aa_65,m_asian_u5,m_asian_5,m_asian_19,m_asian_65,f_asian_u5,f_asian_5,f_asian_19,f_asian_65,accum_m_asian_u5,accum_m_asian_5,accum_m_asian_19,accum_m_asian_65,accum_f_asian_u5,accum_f_asian_5,accum_f_asian_19,accum_f_asian_65,m_other_u5,m_other_5,m_other_19,m_other_65,f_other_u5,f_other_5,f_other_19,f_other_65,accum_m_other_u5,accum_m_other_5,accum_m_other_19,accum_m_other_65,accum_f_other_u5,accum_f_other_5,accum_f_other_19,accum_f_other_65,m_na_u5,m_na_5,m_na_19,m_na_65,f_na_u5,f_na_5,f_na_19,f_na_65,accum_m_na_u5,accum_m_na_5,accum_m_na_19,accum_m_na_65,accum_f_na_u5,accum_f_na_5,accum_f_na_19,accum_f_na_65,m_hawaiian_u5,m_hawaiian_5,m_hawaiian_19,m_hawaiian_65,f_hawaiian_u5,f_hawaiian_5,f_hawaiian_19,f_hawaiian_65,accum_m_hawaiian_u5,accum_m_hawaiian_5,accum_m_hawaiian_19,accum_m_hawaiian_65,accum_f_hawaiian_u5,accum_f_hawaiian_5,accum_f_hawaiian_19,accum_f_hawaiian_65) values (".$id.",".$service_id.",".$session_data_id.",'".$session_date."',".$session_num.",". $instructor_id.",".$cc_alert_1.",".$cc_alert_2.",'".addslashes($educator)."','".$m_white_non_hisp_u5. "','".$m_white_non_hisp_5."','".$m_white_non_hisp_19."','".$m_white_non_hisp_65."','".$f_white_non_hisp_u5."','".$f_white_non_hisp_5."','".$f_white_non_hisp_19."','".$f_white_non_hisp_65."','".$accum_m_white_non_hisp_u5."','".$accum_m_white_non_hisp_5."','".$accum_m_white_non_hisp_19."','".$accum_m_white_non_hisp_65."','".$accum_f_white_non_hisp_u5."','".$accum_f_white_non_hisp_5."','".$accum_f_white_non_hisp_19."','".$accum_f_white_non_hisp_65."','".$m_white_hisp_u5."','".$m_white_hisp_5."','".$m_white_hisp_19."','".$m_white_hisp_65."','".$f_white_hisp_u5."','".$f_white_hisp_5."','".$f_white_hisp_19."','".$f_white_hisp_65."','".$accum_m_white_hisp_u5."','".$accum_m_white_hisp_5."','".$accum_m_white_hisp_19."','".$accum_m_white_hisp_65."','".$accum_f_white_hisp_u5."','".$accum_f_white_hisp_5."','".$accum_f_white_hisp_19."','".$accum_f_white_hisp_65."','".$m_aa_u5."','".$m_aa_5."','".$m_aa_19."','".$m_aa_65."','".$f_aa_u5."','".$f_aa_5."','".$f_aa_19."','".$f_aa_65."','".$accum_m_aa_u5."','".$accum_m_aa_5."','".$accum_m_aa_19."','".$accum_m_aa_65."','".$accum_f_aa_u5."','".$accum_f_aa_5."','".$accum_f_aa_19."','".$accum_f_aa_65."','".$m_asian_u5."','".$m_asian_5."','".$m_asian_19."','".$m_asian_65."','".$f_asian_u5."','".$f_asian_5."','".$f_asian_19."','".$f_asian_65."','".$accum_m_asian_u5."','".$accum_m_asian_5."','".$accum_m_asian_19."','".$accum_m_asian_65."','".$accum_f_asian_u5."','".$accum_f_asian_5."','".$accum_f_asian_19."','".$accum_f_asian_65."','".$m_other_u5."','".$m_other_5."','".$m_other_19."','".$m_other_65."','".$f_other_u5."','".$f_other_5."','".$f_other_19."','".$f_other_65."','".$accum_m_other_u5."','".$accum_m_other_5."','".$accum_m_other_19."','".$accum_m_other_65."','".$accum_f_other_u5."','".$accum_f_other_5."','".$accum_f_other_19."','".$accum_f_other_65."','".$m_na_u5."','".$m_na_5."','".$m_na_19."','".$m_na_65."','".$f_na_u5."','".$f_na_5."','".$f_na_19."','".$f_na_65."','".$accum_m_na_u5."','".$accum_m_na_5."','".$accum_m_na_19."','".$accum_m_na_65."','".$accum_f_na_u5."','".$accum_f_na_5."','".$accum_f_na_19."','".$accum_f_na_65."','".$m_hawaiian_u5."','".$m_hawaiian_5."','".$m_hawaiian_19."','".$m_hawaiian_65."','".$f_hawaiian_u5."','".$f_hawaiian_5."','".$f_hawaiian_19."','".$f_hawaiian_65."','".$accum_m_hawaiian_u5."','".$accum_m_hawaiian_5."','".$accum_m_hawaiian_19."','".$accum_m_hawaiian_65."','".$accum_f_hawaiian_u5."','".$accum_f_hawaiian_5."','".$accum_f_hawaiian_19."','".$accum_f_hawaiian_65."')";

				

				//echo "SQL: $sql3 <br>";

				$result2 = mysql_query($sql3) or die (mysql_error());
 

				echo ".";
 

				if ($old_id <> $service_id) $old_id = $service_id;
 
 

	   }

  }
 

echo "<br><br>Table is complete. Run report tables now.<br>";
 
 
 

?>

Open in new window

0
Comment
Question by:dnatal
  • 4
  • 3
7 Comments
 
LVL 9

Accepted Solution

by:
Rurne earned 500 total points
ID: 24151002
Several things to note:

Are you using persistent connections?  That is, in one of your require_once files (lines 12-14), are you using mysql_connect() or mysql_pconnect()?  A persistent connection (pconnect) isn't released at termination of your script, so running this report a few times could leave a lot of open resources and eat up your memory.

Line 36: It seems like there's an implicit join going on here between session_data (sd) and service (s).  How are these tables supposed to be joined?  Can you provide the CREATE TABLE instructions for these?  You may be missing an index.


Lines 71, 116, 161, 206, 253, 300, 347:  The WHERE clauses look to text matching.  Do you have indexes on participant_by_category.ethnic?

Nothing seems overly complex in your queries, but I'm more likely to question your DB layout than the PHP code.  Full-text scanning without on a non-indexed field will absolutely cripple your query performance, and I'm inclined to think this may be the root of your issue.
0
 
LVL 2

Author Comment

by:dnatal
ID: 24151357
Yes, you're right, I do a mysql_connect but never explicitly disconnect it. However, I don't have other processes running at the same time. I think I will bite the bullet and rewrite the table so that the categories are not text but numeric--the guy I had that developed this for me did it this way for the flexibility, but it is just way too slow. I end up using databases a lot but I'm not really much of an expert--is it a lot more efficient to do a join that to do it how I did on line 36 (implicitly joining the tables?) Thanks for your help!
0
 
LVL 9

Expert Comment

by:Rurne
ID: 24151525
Not a problem.  If you're using mysql_connect() and not pconnect(), the resources are freed at termination of script; no need for you to explicitly call mysql_close().  The problem with mysql_pconnect() is that it keeps the connections open, even when your script ends... in some environments, this can lead to lower overhead, but it requires the programmer to know exactly how the web server is configured and generally isn't good for 95% of PHP programmers out there.


As for the meat of the issue... you don't need to rewrite the tables, but if you execute something like:


CREATE INDEX p_b_c_ethnic ON `participant_by_category` (`ethnic` (8));

You should be able to increase performance right off the bat.

As for line 36, using the implicit inner join works fairly decently, if you're working with InnoDB tables and have been able to set up foreign key constraints.  With MyISAM, it's a bit more of a crap shoot and sometimes you can get unexpected results.  Sometimes this can result in a cross-join, which ends up matching every row in table A to every row in table B using columns that have the same name (which is usually not what you want).

If you can provide the output from executing "SHOW CREATE TABLE session_data" and "SHOW CREATE TABLE service", I can help you structure an explicit join.  The advantage of that is that the server isn't left to guess how to match your tables, and the execution plan generally speeds up.  Also, if we can identify the fields used in the join and they are found to be lacking indexes, then we can create the needed indexes to also increase performance.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 2

Author Comment

by:dnatal
ID: 24158517
Thanks! Here are my tables:
CREATE TABLE `participant_by_category` (

  `id` int(11) NOT NULL auto_increment,

  `session_data_id` int(11) NOT NULL,

  `ethnic` varchar(64) collate latin1_general_ci NOT NULL,

  `m_u5` int(5) NOT NULL,

  `m_5` int(5) NOT NULL,

  `m_19` int(5) NOT NULL,

  `m_65` int(5) NOT NULL,

  `f_u5` int(5) NOT NULL,

  `f_5` int(5) NOT NULL,

  `f_19` int(5) NOT NULL,

  `f_65` int(5) NOT NULL,

  PRIMARY KEY  (`id`)

) ENGINE=MyISAM AUTO_INCREMENT=64527 DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci AUTO_INCREMENT=64527 ;
 
 

CREATE TABLE `session_data` (

  `session_data_id` int(11) NOT NULL auto_increment,

  `service_id` int(11) NOT NULL default '0',

  `session_num` int(11) NOT NULL default '1',

  `service_date` date default '0000-00-00',

  `project_setting` tinyint(1) NOT NULL default '0',

  `project_other` varchar(50) collate latin1_german2_ci default NULL,

  `curriculum_id` int(11) default '0',

  `curriculum_other` varchar(50) collate latin1_german2_ci default NULL,

  `evaluation_id` int(11) NOT NULL default '0',

  `focus_area` smallint(6) NOT NULL default '0',

  `topic_id` int(11) default NULL,

  `topic_id2` int(11) default '0',

  `recipe_id` int(11) default NULL,

  `recipe_other` varchar(255) collate latin1_german2_ci default NULL,

  `topic_other` varchar(255) collate latin1_german2_ci default NULL,

  `costs` float default NULL,

  `mileage` float NOT NULL,

  `session_length` int(11) NOT NULL default '0',

  `num_teacher_staff` smallint(5) unsigned default '0',

  `num_adults` smallint(5) unsigned default '0',

  `num_youth` smallint(5) unsigned default '0',

  `num_aa_hisp` smallint(5) unsigned default '0',

  `num_aa_non_hisp` smallint(5) unsigned default '0',

  `num_asian` smallint(5) unsigned default '0',

  `num_hawaiian` smallint(5) unsigned NOT NULL,

  `num_white_hisp` smallint(5) unsigned default '0',

  `num_white_non_hisp` smallint(5) unsigned default '0',

  `num_na` smallint(5) unsigned default '0',

  `num_other` smallint(5) unsigned default '0',

  `num_male` smallint(5) unsigned default '0',

  `num_female` smallint(5) unsigned default '0',

  `delivery_site` tinyint(2) default '0',

  `del_site_other` varchar(75) collate latin1_german2_ci default NULL,

  `del_site_new` tinyint(1) default '0',

  `fsp_eligible` int(11) default '0',

  `fsp_likely_eligible` int(11) default '0',

  `fsp_potent_eligible` int(11) default '0',

  `fsp_not_eligible` int(11) default '0',

  `pct_students_fsp_eligible` tinyint(3) default '0',

  `pct_students_free_slp` tinyint(3) default '0',

  `website_promo` tinyint(1) default '0',

  `num_brochures` int(11) default '0',

  `num_magnets` int(11) default '0',

  `num_calendars` int(11) default '0',

  `num_posters` int(11) default '0',

  `fsp_promo` tinyint(1) default '0',

  `audience_trainer` tinyint(1) default '0',

  `youth_under5` smallint(5) unsigned default '0',

  `youth_5` smallint(5) unsigned default '0',

  `youth_6` smallint(5) unsigned default '0',

  `youth_7` smallint(5) unsigned default '0',

  `youth_8` smallint(5) unsigned default '0',

  `youth_9` smallint(5) unsigned default '0',

  `youth_10` smallint(5) unsigned default '0',

  `youth_11` smallint(5) unsigned default '0',

  `youth_12` smallint(5) unsigned default '0',

  `youth_13` smallint(5) unsigned default '0',

  `youth_14` smallint(5) unsigned default '0',

  `youth_15` smallint(5) unsigned default '0',

  `youth_16` smallint(5) unsigned default '0',

  `youth_17` smallint(5) unsigned default '0',

  `youth_18` smallint(5) unsigned default '0',

  `final` varchar(10) collate latin1_german2_ci NOT NULL default '0',

  `accum_num_teacher_staff` smallint(5) NOT NULL,

  `accum_num_adults` smallint(5) NOT NULL,

  `accum_num_youth` smallint(5) NOT NULL,

  `accum_num_aa_hisp` smallint(5) NOT NULL,

  `accum_num_aa_non_hisp` smallint(5) NOT NULL,

  `accum_num_asian` smallint(5) NOT NULL,

  `accum_num_hawaiian` smallint(5) unsigned NOT NULL,

  `accum_num_white_hisp` smallint(5) NOT NULL,

  `accum_num_white_non_hisp` smallint(5) NOT NULL,

  `accum_num_na` smallint(5) NOT NULL,

  `accum_num_other` smallint(5) NOT NULL,

  `accum_num_male` smallint(5) NOT NULL,

  `accum_num_female` smallint(5) NOT NULL,

  `accum_fsp_eligible` smallint(5) NOT NULL,

  `accum_fsp_likely_eligible` smallint(5) NOT NULL,

  `accum_fsp_potent_eligible` smallint(5) NOT NULL,

  `accum_fsp_not_eligible` smallint(5) NOT NULL,

  `accum_youth_under5` smallint(5) NOT NULL,

  `accum_youth_5` smallint(5) NOT NULL,

  `accum_youth_6` smallint(5) NOT NULL,

  `accum_youth_7` smallint(5) NOT NULL,

  `accum_youth_8` smallint(5) NOT NULL,

  `accum_youth_9` smallint(5) NOT NULL,

  `accum_youth_10` smallint(5) NOT NULL,

  `accum_youth_11` smallint(5) NOT NULL,

  `accum_youth_12` smallint(5) NOT NULL,

  `accum_youth_13` smallint(5) NOT NULL,

  `accum_youth_14` smallint(5) NOT NULL,

  `accum_youth_15` smallint(5) NOT NULL,

  `accum_youth_16` smallint(5) NOT NULL,

  `accum_youth_17` smallint(5) NOT NULL,

  `accum_youth_18` smallint(5) NOT NULL,

  PRIMARY KEY  (`session_data_id`),

  KEY `service_id` (`service_id`)

) ENGINE=MyISAM AUTO_INCREMENT=17097 DEFAULT CHARSET=latin1 COLLATE=latin1_german2_ci COMMENT='This table stores all the data for the STS sessions' AUTO_INCREMENT=17097 ;
 

CREATE TABLE `service` (

  `service_id` int(11) NOT NULL auto_increment,

  `activity` varchar(100) collate latin1_german2_ci NOT NULL default '',

  `program` varchar(20) collate latin1_german2_ci NOT NULL,

  `instructor_id` int(11) NOT NULL default '0',

  `location_id` int(11) NOT NULL default '0',

  `county_id` int(11) NOT NULL default '0',

  `audience` tinyint(1) NOT NULL default '0',

  `contact_type` tinyint(1) NOT NULL default '0',

  `count_type` varchar(1) collate latin1_german2_ci NOT NULL,

  `direct_indirect` enum('direct','indirect') collate latin1_german2_ci NOT NULL default 'direct',

  `faith_community` varchar(1) collate latin1_german2_ci NOT NULL,

  `costs` float NOT NULL,

  `activity_descript` mediumtext collate latin1_german2_ci,

  `num_sessions` tinyint(4) default '1',

  `series_cat` tinyint(1) default '0',

  `created` date NOT NULL default '0000-00-00',

  `first_session_entered` date NOT NULL,

  `last_session_entered` date NOT NULL,

  `user_id` int(11) NOT NULL default '0',

  `final` varchar(10) collate latin1_german2_ci NOT NULL default '0',

  `audience_trainer` tinyint(4) NOT NULL,

  PRIMARY KEY  (`service_id`),

  KEY `location_id` (`location_id`),

  KEY `created` (`created`)

) ENGINE=MyISAM AUTO_INCREMENT=8072 DEFAULT CHARSET=latin1 COLLATE=latin1_german2_ci COMMENT='This table stores the master record for STS contact tracking' AUTO_INCREMENT=8072 ;

Open in new window

0
 
LVL 9

Expert Comment

by:Rurne
ID: 24158827
Yikes.  Those are some really big tables.  Outside of this particular issue, I'd recommend some normalization as relates to the session_data table, at the least.

Regardless, it looks like MySQL is (correctly) matching session_data.service_id to service.service_id... those two are indexed, so you're probably okay there.

Your bottleneck (other than dealing with overly large data sets) probably has a bit more to do with participant_by_category.   Both p_b_c.ethnic and p_b_c.session_data_id need to be indexed, since they are used WHERE clause parameters.  You're getting huge performance dings due to the fulltext scan of `ethnic`, and full-table scans by the query planner whenever you try to join p_b_c to session_data.


Without knowing any better, something in the above suggests you could refactor this all to use a join between session_data and participant_by_category, and use a GROUP BY p_b_c.ethnic to filter things down.  Then, you'd have only one query and could loop over the dataset instead of copy/pasting for every ethnicity you have to account for.  Cleans up the code and makes it easier to keep your place, plus it reduces the number of times you have to hit the DB.  Just a thought.
0
 
LVL 2

Author Comment

by:dnatal
ID: 24158953
What, exactly, does "CREATE INDEX p_b_c_ethnic ON `participant_by_category` (`ethnic` (8));" do?
0
 
LVL 9

Expert Comment

by:Rurne
ID: 24159320
It generates an index on the `ethnic` column (by default, a hash table) that will be used in joins and WHERE clauses to speed up performance.  It's a trade off between a (usually miniscule) bit of disk space to enhance performance.  If you search a CHAR, VARCHAR, or TEXT field without an index, it will attempt to do a full 1:1 match on every entry in the table, and full-text searching is several orders of magnitude slower than searching against an index.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Author Note: Since this E-E article was originally written, years ago, formal testing has come into common use in the world of PHP.  PHPUnit (http://en.wikipedia.org/wiki/PHPUnit) and similar technologies have enjoyed wide adoption, making it possib…
Nothing in an HTTP request can be trusted, including HTTP headers and form data.  A form token is a tool that can be used to guard against request forgeries (CSRF).  This article shows an improved approach to form tokens, making it more difficult to…
The viewer will learn how to count occurrences of each item in an array.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

28 Experts available now in Live!

Get 1:1 Help Now