dnatal
asked on
php mysql routine for creating report
I have data in a system that I need to use to create some fairly complex reports. I tried to create the sql queries to get the data out in the way I want it, but they just take too long to run, so I finally created a script that runs in a chron job to generate a table with the data accumulated that I need for the tables. The problem is, this script is now failing, presumably timing out, and I don't know how to either break it up so that it can run without failure, nor can I figure out how to get it to let me know when it does fail. The data on the system changes constantly so I do have to be able to be able to run the script on the full data set, not just on newly added data.
<?php
///////////////////////////////////////////////////////////
//// DN: This code is run nightly via a cron job
//// to create the data needed for generating
//// direct contact reports for the Virginia data.
//// It takes too long to run to generate the data
//// on the fly, no matter how much I try to optimize
//// it!
//////////////////////////////////////////////////////////
require_once("fsnep_ini.php");
require_once("header.php");
require_once("functions.php");
ini_set("max_execution_time",3600);
ini_set("memory_limit","120M");
error_reporting(E_ALL);
$ethnic = Array();
$ethnic[0] = "American<BR>Indian/Alaska<br>Native";
$ethnic[1] = "Asian";
$ethnic[2] = "African<br>American";
$ethnic[3] = "Hispanic/<br>White";
$ethnic[4] = "N. Hawaiian/<br>Pac. Islander";
$ethnic[5] = "White";
$ethnic[6] = "Other/More<br> than one<br>race";
$old_id = 0;
// report filters
$where = 'sd.service_id = s.service_id';
$qry = "SELECT s.user_id, s.service_id, s.activity, s.instructor_id, s.location_id, s.county_id, s.audience, s.contact_type, s.num_sessions, s.series_cat, s.created, s.final, s.user_id, sd.* FROM session_data sd, service s WHERE ".$where." ORDER BY s.service_id DESC, sd.session_num";
//echo "SQL: $qry <br>";
//get rid of old report data
$sql2 = "delete from temp_direct_report where id>0";
$result2 = mysql_query($sql2) or die("SQL error: ".mysql_error());
$id = rand(100, 1000000);
$result = mysql_query($qry) or die("SQL error: ".mysql_error());
$numrows = @mysql_num_rows($result);
Echo "<br><br><b>Please wait, creating temporary table</b>...";
// report body
if ($numrows > 0)
{
while ($sd = mysql_fetch_assoc($result))
{
$session_data_id = $sd['session_data_id'];
$lname = get_field('users','lname','user_id='.$sd['instructor_id']);
$fname = get_field('users','fname','user_id='.$sd['instructor_id']);
$cc_alert_1 = get_field('users','cc_alert_1','user_id='.$sd['instructor_id']);
$cc_alert_2 = get_field('users','cc_alert_2','user_id='.$sd['instructor_id']);
$educator = $lname.", ".$fname;
$instructor_id = $sd['instructor_id'];
echo $lname."...";
$session_date = $sd['service_date'];
$session_num = $sd['session_num'];
$service_id = $sd['service_id'];
//White, African American, Hispanic/White, Other/More than one race, Asian, American Indian/Alaska Native, N. Hawaiian/Pac. Islander
//need to pull these out of the participant_by_category table...
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='Hispanic/White' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_white_hisp_u5 = 0;
$m_white_hisp_5 = 0;
$m_white_hisp_19 = 0;
$m_white_hisp_65 = 0;
$f_white_hisp_u5 = 0;
$f_white_hisp_5 = 0;
$f_white_hisp_19 = 0;
$f_white_hisp_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_white_hisp_u5 = $row['m_u5'];
$m_white_hisp_5 = $row['m_5'];
$m_white_hisp_19 = $row['m_19'];
$m_white_hisp_65 = $row['m_65'];
$f_white_hisp_u5 = $row['f_u5'];
$f_white_hisp_5 = $row['f_5'];
$f_white_hisp_19 = $row['f_19'];
$f_white_hisp_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_white_hisp_u5 = 0;
$accum_m_white_hisp_5 = 0;
$accum_m_white_hisp_19 = 0;
$accum_m_white_hisp_65 = 0;
$accum_f_white_hisp_u5 = 0;
$accum_f_white_hisp_5 = 0;
$accum_f_white_hisp_19 = 0;
$accum_f_white_hisp_65 = 0;
}
$accum_m_white_hisp_u5 += $row['m_u5'];
$accum_m_white_hisp_5 += $row['m_5'];
$accum_m_white_hisp_19 += $row['m_19'];
$accum_m_white_hisp_65 += $row['m_65'];
$accum_f_white_hisp_u5 += $row['f_u5'];
$accum_f_white_hisp_5 += $row['f_5'];
$accum_f_white_hisp_19 += $row['f_19'];
$accum_f_white_hisp_65 += $row['f_65'];
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='White' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_white_non_hisp_u5 = 0;
$m_white_non_hisp_5 = 0;
$m_white_non_hisp_19 = 0;
$m_white_non_hisp_65 = 0;
$f_white_non_hisp_u5 = 0;
$f_white_non_hisp_5 = 0;
$f_white_non_hisp_19 = 0;
$f_white_non_hisp_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_white_non_hisp_u5 = $row['m_u5'];
$m_white_non_hisp_5 = $row['m_5'];
$m_white_non_hisp_19 = $row['m_19'];
$m_white_non_hisp_65 = $row['m_65'];
$f_white_non_hisp_u5 = $row['f_u5'];
$f_white_non_hisp_5 = $row['f_5'];
$f_white_non_hisp_19 = $row['f_19'];
$f_white_non_hisp_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_white_non_hisp_u5 = 0;
$accum_m_white_non_hisp_5 = 0;
$accum_m_white_non_hisp_19 = 0;
$accum_m_white_non_hisp_65 = 0;
$accum_f_white_non_hisp_u5 = 0;
$accum_f_white_non_hisp_5 = 0;
$accum_f_white_non_hisp_19 = 0;
$accum_f_white_non_hisp_65 = 0;
}
$accum_m_white_non_hisp_u5 += $row['m_u5'];
$accum_m_white_non_hisp_5 += $row['m_5'];
$accum_m_white_non_hisp_19 += $row['m_19'];
$accum_m_white_non_hisp_65 += $row['m_65'];
$accum_f_white_non_hisp_u5 += $row['f_u5'];
$accum_f_white_non_hisp_5 += $row['f_5'];
$accum_f_white_non_hisp_19 += $row['f_19'];
$accum_f_white_non_hisp_65 += $row['f_65'];
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='African American' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_aa_u5 = 0;
$m_aa_5 = 0;
$m_aa_19 = 0;
$m_aa_65 = 0;
$f_aa_u5 = 0;
$f_aa_5 = 0;
$f_aa_19 = 0;
$f_aa_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_aa_u5 = $row['m_u5'];
$m_aa_5 = $row['m_5'];
$m_aa_19 = $row['m_19'];
$m_aa_65 = $row['m_65'];
$f_aa_u5 = $row['f_u5'];
$f_aa_5 = $row['f_5'];
$f_aa_19 = $row['f_19'];
$f_aa_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_aa_u5 = 0;
$accum_m_aa_5 = 0;
$accum_m_aa_19 = 0;
$accum_m_aa_65 = 0;
$accum_f_aa_u5 = 0;
$accum_f_aa_5 = 0;
$accum_f_aa_19 = 0;
$accum_f_aa_65 = 0;
}
$accum_m_aa_u5 += $row['m_u5'];
$accum_m_aa_5 += $row['m_5'];
$accum_m_aa_19 += $row['m_19'];
$accum_m_aa_65 += $row['m_65'];
$accum_f_aa_u5 += $row['f_u5'];
$accum_f_aa_5 += $row['f_5'];
$accum_f_aa_19 += $row['f_19'];
$accum_f_aa_65 += $row['f_65'];
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='Asian' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_asian_u5 = 0;
$m_asian_5 = 0;
$m_asian_19 = 0;
$m_asian_65 = 0;
$f_asian_u5 = 0;
$f_asian_5 = 0;
$f_asian_19 = 0;
$f_asian_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_asian_u5 = $row['m_u5'];
$m_asian_5 = $row['m_5'];
$m_asian_19 = $row['m_19'];
$m_asian_65 = $row['m_65'];
$f_asian_u5 = $row['f_u5'];
$f_asian_5 = $row['f_5'];
$f_asian_19 = $row['f_19'];
$f_asian_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_asian_u5 = 0;
$accum_m_asian_5 = 0;
$accum_m_asian_19 = 0;
$accum_m_asian_65 = 0;
$accum_f_asian_u5 = 0;
$accum_f_asian_5 = 0;
$accum_f_asian_19 = 0;
$accum_f_asian_65 = 0;
}
$accum_m_asian_u5 += $row['m_u5'];
$accum_m_asian_5 += $row['m_5'];
$accum_m_asian_19 += $row['m_19'];
$accum_m_asian_65 += $row['m_65'];
$accum_f_asian_u5 += $row['f_u5'];
$accum_f_asian_5 += $row['f_5'];
$accum_f_asian_19 += $row['f_19'];
$accum_f_asian_65 += $row['f_65'];
//White, African American, Hispanic/White, Other/More than one race, Asian, American Indian/Alaska Native, N. Hawaiian/Pac. Islander
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='Other/More than one race' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_asian_u5 = 0;
$m_other_u5 = 0;
$m_other_5 = 0;
$m_other_19 = 0;
$m_other_65 = 0;
$f_other_u5 = 0;
$f_other_5 = 0;
$f_other_19 = 0;
$f_other_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_other_u5 = $row['m_u5'];
$m_other_5 = $row['m_5'];
$m_other_19 = $row['m_19'];
$m_other_65 = $row['m_65'];
$f_other_u5 = $row['f_u5'];
$f_other_5 = $row['f_5'];
$f_other_19 = $row['f_19'];
$f_other_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_other_u5 = 0;
$accum_m_other_5 = 0;
$accum_m_other_19 = 0;
$accum_m_other_65 = 0;
$accum_f_other_u5 = 0;
$accum_f_other_5 = 0;
$accum_f_other_19 = 0;
$accum_f_other_65 = 0;
}
$accum_m_other_u5 += $row['m_u5'];
$accum_m_other_5 += $row['m_5'];
$accum_m_other_19 += $row['m_19'];
$accum_m_other_65 += $row['m_65'];
$accum_f_other_u5 += $row['f_u5'];
$accum_f_other_5 += $row['f_5'];
$accum_f_other_19 += $row['f_19'];
$accum_f_other_65 += $row['f_65'];
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='American Indian/Alaska Native' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_na_u5 = 0;
$m_na_5 = 0;
$m_na_19 = 0;
$m_na_65 = 0;
$f_na_u5 = 0;
$f_na_5 = 0;
$f_na_19 = 0;
$f_na_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_na_u5 = $row['m_u5'];
$m_na_5 = $row['m_5'];
$m_na_19 = $row['m_19'];
$m_na_65 = $row['m_65'];
$f_na_u5 = $row['f_u5'];
$f_na_5 = $row['f_5'];
$f_na_19 = $row['f_19'];
$f_na_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_na_u5 = 0;
$accum_m_na_5 = 0;
$accum_m_na_19 = 0;
$accum_m_na_65 = 0;
$accum_f_na_u5 = 0;
$accum_f_na_5 = 0;
$accum_f_na_19 = 0;
$accum_f_na_65 = 0;
}
$accum_m_na_u5 += $row['m_u5'];
$accum_m_na_5 += $row['m_5'];
$accum_m_na_19 += $row['m_19'];
$accum_m_na_65 += $row['m_65'];
$accum_f_na_u5 += $row['f_u5'];
$accum_f_na_5 += $row['f_5'];
$accum_f_na_19 += $row['f_19'];
$accum_f_na_65 += $row['f_65'];
$sql = "SELECT * FROM participant_by_category WHERE session_data_id=$session_data_id AND ethnic='N. Hawaiian/Pac. Islander' LIMIT 1";
//echo "sql: $sql <br>";
$sql = mysql_query($sql) or die (mysql_error());
$m_hawaiian_u5 = 0;
$m_hawaiian_5 = 0;
$m_hawaiian_19 = 0;
$m_hawaiian_65 = 0;
$f_hawaiian_u5 = 0;
$f_hawaiian_5 = 0;
$f_hawaiian_19 = 0;
$f_hawaiian_65 = 0;
if(mysql_num_rows($sql) > 0)
{
$row = mysql_fetch_assoc($sql);
$m_hawaiian_u5 = $row['m_u5'];
$m_hawaiian_5 = $row['m_5'];
$m_hawaiian_19 = $row['m_19'];
$m_hawaiian_65 = $row['m_65'];
$f_hawaiian_u5 = $row['f_u5'];
$f_hawaiian_5 = $row['f_5'];
$f_hawaiian_19 = $row['f_19'];
$f_hawaiian_65 = $row['f_65'];
}
if ($old_id <> $service_id)
{
//reset the accumulated
$accum_m_hawaiian_u5 = 0;
$accum_m_hawaiian_5 = 0;
$accum_m_hawaiian_19 = 0;
$accum_m_hawaiian_65 = 0;
$accum_f_hawaiian_u5 = 0;
$accum_f_hawaiian_5 = 0;
$accum_f_hawaiian_19 = 0;
$accum_f_hawaiian_65 = 0;
}
$accum_m_hawaiian_u5 += $row['m_u5'];
$accum_m_hawaiian_5 += $row['m_5'];
$accum_m_hawaiian_19 += $row['m_19'];
$accum_m_hawaiian_65 += $row['m_65'];
$accum_f_hawaiian_u5 += $row['f_u5'];
$accum_f_hawaiian_5 += $row['f_5'];
$accum_f_hawaiian_19 += $row['f_19'];
$accum_f_hawaiian_65 += $row['f_65'];
//now, write it into the temporary table used for the reports
$sql3 = "insert into temp_direct_report (id, service_id, session_data_id, session_date, session_num, instructor_id, cc_alert_1, cc_alert_2, educator,m_white_non_hisp_u5, m_white_non_hisp_5,m_white_non_hisp_19, m_white_non_hisp_65,f_white_non_hisp_u5,f_white_non_hisp_5,f_white_non_hisp_19,f_white_non_hisp_65,accum_m_white_non_hisp_u5,accum_m_white_non_hisp_5,accum_m_white_non_hisp_19,accum_m_white_non_hisp_65,accum_f_white_non_hisp_u5,accum_f_white_non_hisp_5,accum_f_white_non_hisp_19,accum_f_white_non_hisp_65, m_white_hisp_u5,m_white_hisp_5, m_white_hisp_19,m_white_hisp_65,f_white_hisp_u5,f_white_hisp_5,f_white_hisp_19,f_white_hisp_65,accum_m_white_hisp_u5,accum_m_white_hisp_5,accum_m_white_hisp_19,accum_m_white_hisp_65,accum_f_white_hisp_u5,accum_f_white_hisp_5,accum_f_white_hisp_19,accum_f_white_hisp_65,m_aa_u5,m_aa_5,m_aa_19,m_aa_65,f_aa_u5,f_aa_5,f_aa_19,f_aa_65,accum_m_aa_u5,accum_m_aa_5,accum_m_aa_19,accum_m_aa_65,accum_f_aa_u5,accum_f_aa_5,accum_f_aa_19,accum_f_aa_65,m_asian_u5,m_asian_5,m_asian_19,m_asian_65,f_asian_u5,f_asian_5,f_asian_19,f_asian_65,accum_m_asian_u5,accum_m_asian_5,accum_m_asian_19,accum_m_asian_65,accum_f_asian_u5,accum_f_asian_5,accum_f_asian_19,accum_f_asian_65,m_other_u5,m_other_5,m_other_19,m_other_65,f_other_u5,f_other_5,f_other_19,f_other_65,accum_m_other_u5,accum_m_other_5,accum_m_other_19,accum_m_other_65,accum_f_other_u5,accum_f_other_5,accum_f_other_19,accum_f_other_65,m_na_u5,m_na_5,m_na_19,m_na_65,f_na_u5,f_na_5,f_na_19,f_na_65,accum_m_na_u5,accum_m_na_5,accum_m_na_19,accum_m_na_65,accum_f_na_u5,accum_f_na_5,accum_f_na_19,accum_f_na_65,m_hawaiian_u5,m_hawaiian_5,m_hawaiian_19,m_hawaiian_65,f_hawaiian_u5,f_hawaiian_5,f_hawaiian_19,f_hawaiian_65,accum_m_hawaiian_u5,accum_m_hawaiian_5,accum_m_hawaiian_19,accum_m_hawaiian_65,accum_f_hawaiian_u5,accum_f_hawaiian_5,accum_f_hawaiian_19,accum_f_hawaiian_65) values (".$id.",".$service_id.",".$session_data_id.",'".$session_date."',".$session_num.",". $instructor_id.",".$cc_alert_1.",".$cc_alert_2.",'".addslashes($educator)."','".$m_white_non_hisp_u5. "','".$m_white_non_hisp_5."','".$m_white_non_hisp_19."','".$m_white_non_hisp_65."','".$f_white_non_hisp_u5."','".$f_white_non_hisp_5."','".$f_white_non_hisp_19."','".$f_white_non_hisp_65."','".$accum_m_white_non_hisp_u5."','".$accum_m_white_non_hisp_5."','".$accum_m_white_non_hisp_19."','".$accum_m_white_non_hisp_65."','".$accum_f_white_non_hisp_u5."','".$accum_f_white_non_hisp_5."','".$accum_f_white_non_hisp_19."','".$accum_f_white_non_hisp_65."','".$m_white_hisp_u5."','".$m_white_hisp_5."','".$m_white_hisp_19."','".$m_white_hisp_65."','".$f_white_hisp_u5."','".$f_white_hisp_5."','".$f_white_hisp_19."','".$f_white_hisp_65."','".$accum_m_white_hisp_u5."','".$accum_m_white_hisp_5."','".$accum_m_white_hisp_19."','".$accum_m_white_hisp_65."','".$accum_f_white_hisp_u5."','".$accum_f_white_hisp_5."','".$accum_f_white_hisp_19."','".$accum_f_white_hisp_65."','".$m_aa_u5."','".$m_aa_5."','".$m_aa_19."','".$m_aa_65."','".$f_aa_u5."','".$f_aa_5."','".$f_aa_19."','".$f_aa_65."','".$accum_m_aa_u5."','".$accum_m_aa_5."','".$accum_m_aa_19."','".$accum_m_aa_65."','".$accum_f_aa_u5."','".$accum_f_aa_5."','".$accum_f_aa_19."','".$accum_f_aa_65."','".$m_asian_u5."','".$m_asian_5."','".$m_asian_19."','".$m_asian_65."','".$f_asian_u5."','".$f_asian_5."','".$f_asian_19."','".$f_asian_65."','".$accum_m_asian_u5."','".$accum_m_asian_5."','".$accum_m_asian_19."','".$accum_m_asian_65."','".$accum_f_asian_u5."','".$accum_f_asian_5."','".$accum_f_asian_19."','".$accum_f_asian_65."','".$m_other_u5."','".$m_other_5."','".$m_other_19."','".$m_other_65."','".$f_other_u5."','".$f_other_5."','".$f_other_19."','".$f_other_65."','".$accum_m_other_u5."','".$accum_m_other_5."','".$accum_m_other_19."','".$accum_m_other_65."','".$accum_f_other_u5."','".$accum_f_other_5."','".$accum_f_other_19."','".$accum_f_other_65."','".$m_na_u5."','".$m_na_5."','".$m_na_19."','".$m_na_65."','".$f_na_u5."','".$f_na_5."','".$f_na_19."','".$f_na_65."','".$accum_m_na_u5."','".$accum_m_na_5."','".$accum_m_na_19."','".$accum_m_na_65."','".$accum_f_na_u5."','".$accum_f_na_5."','".$accum_f_na_19."','".$accum_f_na_65."','".$m_hawaiian_u5."','".$m_hawaiian_5."','".$m_hawaiian_19."','".$m_hawaiian_65."','".$f_hawaiian_u5."','".$f_hawaiian_5."','".$f_hawaiian_19."','".$f_hawaiian_65."','".$accum_m_hawaiian_u5."','".$accum_m_hawaiian_5."','".$accum_m_hawaiian_19."','".$accum_m_hawaiian_65."','".$accum_f_hawaiian_u5."','".$accum_f_hawaiian_5."','".$accum_f_hawaiian_19."','".$accum_f_hawaiian_65."')";
//echo "SQL: $sql3 <br>";
$result2 = mysql_query($sql3) or die (mysql_error());
echo ".";
if ($old_id <> $service_id) $old_id = $service_id;
}
}
echo "<br><br>Table is complete. Run report tables now.<br>";
?>
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Not a problem. If you're using mysql_connect() and not pconnect(), the resources are freed at termination of script; no need for you to explicitly call mysql_close(). The problem with mysql_pconnect() is that it keeps the connections open, even when your script ends... in some environments, this can lead to lower overhead, but it requires the programmer to know exactly how the web server is configured and generally isn't good for 95% of PHP programmers out there.
As for the meat of the issue... you don't need to rewrite the tables, but if you execute something like:
CREATE INDEX p_b_c_ethnic ON `participant_by_category` (`ethnic` (8));
You should be able to increase performance right off the bat.
As for line 36, using the implicit inner join works fairly decently, if you're working with InnoDB tables and have been able to set up foreign key constraints. With MyISAM, it's a bit more of a crap shoot and sometimes you can get unexpected results. Sometimes this can result in a cross-join, which ends up matching every row in table A to every row in table B using columns that have the same name (which is usually not what you want).
If you can provide the output from executing "SHOW CREATE TABLE session_data" and "SHOW CREATE TABLE service", I can help you structure an explicit join. The advantage of that is that the server isn't left to guess how to match your tables, and the execution plan generally speeds up. Also, if we can identify the fields used in the join and they are found to be lacking indexes, then we can create the needed indexes to also increase performance.
As for the meat of the issue... you don't need to rewrite the tables, but if you execute something like:
CREATE INDEX p_b_c_ethnic ON `participant_by_category` (`ethnic` (8));
You should be able to increase performance right off the bat.
As for line 36, using the implicit inner join works fairly decently, if you're working with InnoDB tables and have been able to set up foreign key constraints. With MyISAM, it's a bit more of a crap shoot and sometimes you can get unexpected results. Sometimes this can result in a cross-join, which ends up matching every row in table A to every row in table B using columns that have the same name (which is usually not what you want).
If you can provide the output from executing "SHOW CREATE TABLE session_data" and "SHOW CREATE TABLE service", I can help you structure an explicit join. The advantage of that is that the server isn't left to guess how to match your tables, and the execution plan generally speeds up. Also, if we can identify the fields used in the join and they are found to be lacking indexes, then we can create the needed indexes to also increase performance.
ASKER
Thanks! Here are my tables:
CREATE TABLE `participant_by_category` (
`id` int(11) NOT NULL auto_increment,
`session_data_id` int(11) NOT NULL,
`ethnic` varchar(64) collate latin1_general_ci NOT NULL,
`m_u5` int(5) NOT NULL,
`m_5` int(5) NOT NULL,
`m_19` int(5) NOT NULL,
`m_65` int(5) NOT NULL,
`f_u5` int(5) NOT NULL,
`f_5` int(5) NOT NULL,
`f_19` int(5) NOT NULL,
`f_65` int(5) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=64527 DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci AUTO_INCREMENT=64527 ;
CREATE TABLE `session_data` (
`session_data_id` int(11) NOT NULL auto_increment,
`service_id` int(11) NOT NULL default '0',
`session_num` int(11) NOT NULL default '1',
`service_date` date default '0000-00-00',
`project_setting` tinyint(1) NOT NULL default '0',
`project_other` varchar(50) collate latin1_german2_ci default NULL,
`curriculum_id` int(11) default '0',
`curriculum_other` varchar(50) collate latin1_german2_ci default NULL,
`evaluation_id` int(11) NOT NULL default '0',
`focus_area` smallint(6) NOT NULL default '0',
`topic_id` int(11) default NULL,
`topic_id2` int(11) default '0',
`recipe_id` int(11) default NULL,
`recipe_other` varchar(255) collate latin1_german2_ci default NULL,
`topic_other` varchar(255) collate latin1_german2_ci default NULL,
`costs` float default NULL,
`mileage` float NOT NULL,
`session_length` int(11) NOT NULL default '0',
`num_teacher_staff` smallint(5) unsigned default '0',
`num_adults` smallint(5) unsigned default '0',
`num_youth` smallint(5) unsigned default '0',
`num_aa_hisp` smallint(5) unsigned default '0',
`num_aa_non_hisp` smallint(5) unsigned default '0',
`num_asian` smallint(5) unsigned default '0',
`num_hawaiian` smallint(5) unsigned NOT NULL,
`num_white_hisp` smallint(5) unsigned default '0',
`num_white_non_hisp` smallint(5) unsigned default '0',
`num_na` smallint(5) unsigned default '0',
`num_other` smallint(5) unsigned default '0',
`num_male` smallint(5) unsigned default '0',
`num_female` smallint(5) unsigned default '0',
`delivery_site` tinyint(2) default '0',
`del_site_other` varchar(75) collate latin1_german2_ci default NULL,
`del_site_new` tinyint(1) default '0',
`fsp_eligible` int(11) default '0',
`fsp_likely_eligible` int(11) default '0',
`fsp_potent_eligible` int(11) default '0',
`fsp_not_eligible` int(11) default '0',
`pct_students_fsp_eligible` tinyint(3) default '0',
`pct_students_free_slp` tinyint(3) default '0',
`website_promo` tinyint(1) default '0',
`num_brochures` int(11) default '0',
`num_magnets` int(11) default '0',
`num_calendars` int(11) default '0',
`num_posters` int(11) default '0',
`fsp_promo` tinyint(1) default '0',
`audience_trainer` tinyint(1) default '0',
`youth_under5` smallint(5) unsigned default '0',
`youth_5` smallint(5) unsigned default '0',
`youth_6` smallint(5) unsigned default '0',
`youth_7` smallint(5) unsigned default '0',
`youth_8` smallint(5) unsigned default '0',
`youth_9` smallint(5) unsigned default '0',
`youth_10` smallint(5) unsigned default '0',
`youth_11` smallint(5) unsigned default '0',
`youth_12` smallint(5) unsigned default '0',
`youth_13` smallint(5) unsigned default '0',
`youth_14` smallint(5) unsigned default '0',
`youth_15` smallint(5) unsigned default '0',
`youth_16` smallint(5) unsigned default '0',
`youth_17` smallint(5) unsigned default '0',
`youth_18` smallint(5) unsigned default '0',
`final` varchar(10) collate latin1_german2_ci NOT NULL default '0',
`accum_num_teacher_staff` smallint(5) NOT NULL,
`accum_num_adults` smallint(5) NOT NULL,
`accum_num_youth` smallint(5) NOT NULL,
`accum_num_aa_hisp` smallint(5) NOT NULL,
`accum_num_aa_non_hisp` smallint(5) NOT NULL,
`accum_num_asian` smallint(5) NOT NULL,
`accum_num_hawaiian` smallint(5) unsigned NOT NULL,
`accum_num_white_hisp` smallint(5) NOT NULL,
`accum_num_white_non_hisp` smallint(5) NOT NULL,
`accum_num_na` smallint(5) NOT NULL,
`accum_num_other` smallint(5) NOT NULL,
`accum_num_male` smallint(5) NOT NULL,
`accum_num_female` smallint(5) NOT NULL,
`accum_fsp_eligible` smallint(5) NOT NULL,
`accum_fsp_likely_eligible` smallint(5) NOT NULL,
`accum_fsp_potent_eligible` smallint(5) NOT NULL,
`accum_fsp_not_eligible` smallint(5) NOT NULL,
`accum_youth_under5` smallint(5) NOT NULL,
`accum_youth_5` smallint(5) NOT NULL,
`accum_youth_6` smallint(5) NOT NULL,
`accum_youth_7` smallint(5) NOT NULL,
`accum_youth_8` smallint(5) NOT NULL,
`accum_youth_9` smallint(5) NOT NULL,
`accum_youth_10` smallint(5) NOT NULL,
`accum_youth_11` smallint(5) NOT NULL,
`accum_youth_12` smallint(5) NOT NULL,
`accum_youth_13` smallint(5) NOT NULL,
`accum_youth_14` smallint(5) NOT NULL,
`accum_youth_15` smallint(5) NOT NULL,
`accum_youth_16` smallint(5) NOT NULL,
`accum_youth_17` smallint(5) NOT NULL,
`accum_youth_18` smallint(5) NOT NULL,
PRIMARY KEY (`session_data_id`),
KEY `service_id` (`service_id`)
) ENGINE=MyISAM AUTO_INCREMENT=17097 DEFAULT CHARSET=latin1 COLLATE=latin1_german2_ci COMMENT='This table stores all the data for the STS sessions' AUTO_INCREMENT=17097 ;
CREATE TABLE `service` (
`service_id` int(11) NOT NULL auto_increment,
`activity` varchar(100) collate latin1_german2_ci NOT NULL default '',
`program` varchar(20) collate latin1_german2_ci NOT NULL,
`instructor_id` int(11) NOT NULL default '0',
`location_id` int(11) NOT NULL default '0',
`county_id` int(11) NOT NULL default '0',
`audience` tinyint(1) NOT NULL default '0',
`contact_type` tinyint(1) NOT NULL default '0',
`count_type` varchar(1) collate latin1_german2_ci NOT NULL,
`direct_indirect` enum('direct','indirect') collate latin1_german2_ci NOT NULL default 'direct',
`faith_community` varchar(1) collate latin1_german2_ci NOT NULL,
`costs` float NOT NULL,
`activity_descript` mediumtext collate latin1_german2_ci,
`num_sessions` tinyint(4) default '1',
`series_cat` tinyint(1) default '0',
`created` date NOT NULL default '0000-00-00',
`first_session_entered` date NOT NULL,
`last_session_entered` date NOT NULL,
`user_id` int(11) NOT NULL default '0',
`final` varchar(10) collate latin1_german2_ci NOT NULL default '0',
`audience_trainer` tinyint(4) NOT NULL,
PRIMARY KEY (`service_id`),
KEY `location_id` (`location_id`),
KEY `created` (`created`)
) ENGINE=MyISAM AUTO_INCREMENT=8072 DEFAULT CHARSET=latin1 COLLATE=latin1_german2_ci COMMENT='This table stores the master record for STS contact tracking' AUTO_INCREMENT=8072 ;
Yikes. Those are some really big tables. Outside of this particular issue, I'd recommend some normalization as relates to the session_data table, at the least.
Regardless, it looks like MySQL is (correctly) matching session_data.service_id to service.service_id... those two are indexed, so you're probably okay there.
Your bottleneck (other than dealing with overly large data sets) probably has a bit more to do with participant_by_category. Both p_b_c.ethnic and p_b_c.session_data_id need to be indexed, since they are used WHERE clause parameters. You're getting huge performance dings due to the fulltext scan of `ethnic`, and full-table scans by the query planner whenever you try to join p_b_c to session_data.
Without knowing any better, something in the above suggests you could refactor this all to use a join between session_data and participant_by_category, and use a GROUP BY p_b_c.ethnic to filter things down. Then, you'd have only one query and could loop over the dataset instead of copy/pasting for every ethnicity you have to account for. Cleans up the code and makes it easier to keep your place, plus it reduces the number of times you have to hit the DB. Just a thought.
Regardless, it looks like MySQL is (correctly) matching session_data.service_id to service.service_id... those two are indexed, so you're probably okay there.
Your bottleneck (other than dealing with overly large data sets) probably has a bit more to do with participant_by_category. Both p_b_c.ethnic and p_b_c.session_data_id need to be indexed, since they are used WHERE clause parameters. You're getting huge performance dings due to the fulltext scan of `ethnic`, and full-table scans by the query planner whenever you try to join p_b_c to session_data.
Without knowing any better, something in the above suggests you could refactor this all to use a join between session_data and participant_by_category, and use a GROUP BY p_b_c.ethnic to filter things down. Then, you'd have only one query and could loop over the dataset instead of copy/pasting for every ethnicity you have to account for. Cleans up the code and makes it easier to keep your place, plus it reduces the number of times you have to hit the DB. Just a thought.
ASKER
What, exactly, does "CREATE INDEX p_b_c_ethnic ON `participant_by_category` (`ethnic` (8));" do?
It generates an index on the `ethnic` column (by default, a hash table) that will be used in joins and WHERE clauses to speed up performance. It's a trade off between a (usually miniscule) bit of disk space to enhance performance. If you search a CHAR, VARCHAR, or TEXT field without an index, it will attempt to do a full 1:1 match on every entry in the table, and full-text searching is several orders of magnitude slower than searching against an index.
ASKER