Link to home
Start Free TrialLog in
Avatar of mikelima
mikelima

asked on

PHP + Wordpress + Large Database = Hell

Hi,
I have a large wordpress blog with 120,000 posts. All posts are text. The posts table has 400 Mb.

I am trying to do a very single task: retrieve all permalinks and modified dates of all posts and save them to a text file.

The problem is that PHP is crap when dealing with with large datasets... not counting the one trillion bugs it has.

The problem with the code I have developed (see below) is that it works almost perfectly, until it stops somewhere after retrieving 100,000 posts. The problem? simple: memory exhausted, probably due to some PHP memory leak.

I am using PHP 5.2.0 and MySQL 5 and please do not suggest me to upgrade PHP or MySql, as I cannot do that. I need to come with a code that can do the task without exhausting memory.

Please check out the code and see if you have any idea!


<?php
//this is two lines are calling wordpress config and all the required files it will need to run
$wp_did_header = true;
require_once( '/home/user/mysite/wp-config.php');

//I have increased the script's memory up to this point with no success
ini_set ( "memory_limit", "700M");


$query = "select count(id) as number FROM wp_posts";
$result = mysql_query($query) or die(mysql_error());
$line = mysql_fetch_array($result);
$totalPosts = $line[0];
$counter = 0;
$output = "output.txt";
$fk = fopen($output, 'w') or die("can't open file");

while ($counter <= $totalPosts) {

      $start = $counter;
      
      //I have tried different values for delta, in order to retrieve posts in chunks... small success doing this, but not enough
      $delta = 5000;
      if ($start + $delta > $totalPosts)
            $delta = $totalPosts - $start;

      $query = "SELECT id,post_modified FROM wp_posts LIMIT $start,$delta";
      
      $result = mysql_query($query) or die(mysql_error());

      while ($line = mysql_fetch_row($result) ) {

            fwrite($fk, get_permalink($line[0]) . chr(10));
            fwrite($fk, mysql2date('Y-m-d\TH:i:s\Z', $line[1], false) . chr(10));
            $counter = $counter + 1;
      }
      Unset($line);
      Unset($query);
      Unset($result);
      //$counter = $counter + 10;
}
fclose($fk);
?>


 
ASKER CERTIFIED SOLUTION
Avatar of Roonaan
Roonaan
Flag of Netherlands image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of mikelima
mikelima

ASKER

Yes!!!!!!! mysql_free_result did the trick. But for some strange reason the script hangs at the end and do not finishes correctly. But for your solution to work I had to increase the variable max_input_time on php.ini from 60 to 360 as the script was being terminated by the interpreter after the timeout.

T H A N K S !!!!!!!!!!!
just to complete.... the script hangs at the end but does the job, i.e., generates the output file correctly.