Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Workaround to PHP bug is causing yet further issues

Posted on 2011-05-11
15
Medium Priority
?
486 Views
Last Modified: 2012-05-11
I had previously thought I found a workaround to a PHP bug having to do with a memory leak due to faulty garbage collection. The PHP bug is documented at http://bugs.php.net/48781 and my previous E-E question was at http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_26929909.html.

The workaround is shown from some text copied from http://bugs.php.net/48781, as follows:

 
<?php

while ($user = get_next_user()) {
    // do stuff with $user

    // php bug: have to unset before next iteration
    // to prevent memory leak caused by buggy gc
    unset($user);
}
?>

Open in new window


That is, unset the $user variable at the end of the loop.

My problem is coming in because in my script $user is a global variable, and unsetting global variables does not work because only the "local" value is reset. This confuses me, because I thought the point of a global variable was that there was only one variable and not any local version. But in any case, the PHP manual says that to avoid this problem, unset via the $GLOBALS variable, as follows:

To unset() a global variable inside of a function, then use the $GLOBALS array to do so:

<?php
function foo() 
{
    unset($GLOBALS['bar']);
}

$bar = "something";
foo();
?>

Open in new window


But this does not seem to work either. It seems that in the loop, each time through, the $user variable is creating another "hidden local" variable, and the script is thus not able to access the seemingly single instance of the global variable.

Here is the essence of what I am trying to do:

 
<?php

while ($user = mysqli_fetch_row($rs)) {
    // do stuff with $user including manipulation in functions

    // php bug: have to unset before next iteration
    // to prevent memory leak caused by buggy gc
    unset($user);
}
?>

Open in new window


The problem that I am having with the above code is that the loop shown has the right values for  $user (which is an array) within the loop, but in functions called, it does not. Instead it keeps the original value from the first time through the loop. This is something I don't understand as is seems to make no sense.

In any case, changing the usage to unsetting the $GLOBALS['user] does not work either. In this case, when the loop is repeated, nothing is assigned to $user. It is as though unsetting it makes it unable to be set again.

Specifically, here is another code snippet with a comment indicating what seems to be happening.

<?php

while ($user = mysqli_fetch_row($rs)) {
    // do stuff with $user including manipulation in functions

    // php bug: have to unset before next iteration
    // to prevent memory leak caused by buggy gc

    // at this point in the code on the second loop iteration, $user is still unset
    // which means that the assignment in the 'while' is not occuring as expected

    unset($GLOBALS['user']);
}
?>

Open in new window


If I remove the unset entirely, the code works properly. But then I get a memory leak and the program does not run to completion.

Any ideas?
0
Comment
Question by:jasimon9
  • 10
  • 4
15 Comments
 
LVL 4

Expert Comment

by:secondv
ID: 35743147
As far as unsetting a global variable, give this a try:

while ($user = mysqli_fetch_row($rs)) 
{
	// do stuff with $user including manipulation in functions

	// php bug: have to unset before next iteration
	// to prevent memory leak caused by buggy gc
	$GLOBALS['user'] = NULL;
	unset($GLOBALS['user']);
}

Open in new window

0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35746184
Wow, there are a lot of assumptions at work here!  Let's try to take this apart into the important pieces.

First, learn about variable scope.  This is a detailed page, but you need to understand 100% of it to use PHP correctly.
http://www.php.net/manual/en/language.variables.scope.php

Next learn about "references."
http://www.php.net/manual/en/language.references.php

Not sure if this is in play here, but you also want to understand this problem:
http://php.net/manual/en/security.globals.php

Some functions act upon the variable directly, but most functions make a copy of the variable and act upon the copy.  There is probably some combination of assumptions in your code that overlays the concepts of scope, reference and maybe register_globals in such a way that you are propagating values that you do not want to propagate.  Unsetting a variable inside the scope of a function only unsets the local copy.  And you probably do not want to be using the "global" declaration if you can avoid it.

If you're getting an array from the data base results set, there is no design pattern that would put this information into a global scope - it simply isn't done.  Instead you would have some kind of facade that enables getter and setter methods to retrieve the data from the row.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35746298
OK, I just reread the earlier question and the PHP.net bug report.  I am not convinced that the bug report is applicable to what you're doing here, and I cannot know for sure without seeing more of the code.  The bug report seems to refer to object creation, and the code I've looked at here seems to refer to fetching a row from a MySQL results set.  My guess is that those are different paths through PHP.

Can you please create a small, encapsulated test set - code, comments and data - that will enable us to see if this situation happens in other PHP installations?  I'll be glad to run a test on my servers and show the results.

And in unrelated matters, you should not ever have to program around a bug in PHP.  When you write code like that, you may be creating code that has dependencies on a particular version of PHP.  That's a characteristic that can never be eliminated, but is to be minimized.  We should fix PHP instead.

Thanks and regards, ~Ray
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:jasimon9
ID: 35746303
After posting this question, I decided to do some testing by getting rid of the global variable.
0
 

Author Comment

by:jasimon9
ID: 35746365
Also, I neglected to mention that this program has been in production since 2006, and has worked as intended.

A program that has been working for 5 years that suddenly fails due to a memory leak: what could cause that? Some possible answers:

 - the program was always flawed, but due to changes in the data over time the flaw changed from not having an effect to having an effect;

 - PHP has introduced a memory leak. Please refer to  http://bugs.php.net/48781 where the exact case is documented;

 - both of the above.

I am dealing with a workaround to handle the memory leak. The unsetting of the global variable to deal with the memory leak is part of recent efforts to deal with the issue. I had another issue with the memory leak which is well documented in the other question referenced above.

I will report back on the result of not using the global variable.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35746937
Here is one reason I suspect that the bug report at http://bugs.php.net/48781 is not applicable to your question here.  Search that bug report page for the string "mysql" - it is conspicuous by its absence.  If the bug had any relationship to mysql or mysqli you would expect to find that information in the bug report.

But instead of speculating, why not take advantage of this offer:

Can you please create a small, encapsulated test set - code, comments and data - that will enable us to see if this situation happens in other PHP installations?  I'll be glad to run a test on my servers and show the results.


Repeatable demonstrations are almost always better than "possible answers."  Thanks and regards, ~Ray
0
 

Author Comment

by:jasimon9
ID: 35747637
Before I respond, I do wish to thank you for previous help you have provide in other questions. In fact in the related question at  http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_26929909.html you were a contributor, and we almost got to the point of sending you test code and data for running on your system. However, the problem was worked around short of that step.

I do not believe mysql is related to the issue I am having.

If mysql was involved, then $user = mysqli_fetch_row($rs) and $user = array(1, 2, 3) would have to do something to cause $user in each case to be different types of objects. As far as I know, they are both just arrays. I do not see where the source of creating this array is relevant, unless there is a bug in the array created by the mysql call, which causes the memory leak. That would be interesting.

Do you know anything more in that regard?

Regarding the script - I cannot provide you with the whole script and database that it runs on. Small, encapsulated test code: please see the snippets in the original question.

Alternatively, the related question at  http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_26929909.html does provide similar cases of looping with the memory leak, and without any global variables involved. But that may be going to far afield.

Regarding "other PHP installations" - in addition to my production environment which is at PHP 5.3.5, I have a test environments running both 5.3.5 and 5.2.4. Putting the program into a test mode which removes the unsetting of the variable and displays memory_get_usage() on each iteration of the loop, I am finding the following interesting results:

1. The 5.2.4 instance shows the same memory used on each loop, and shows 10747904 used at the start and end of the test. That is, there is no memory leakage at all.

2. The 5.3.5 instance shows an increasing memory usage on each loop. The program does eventually crash if it is let run far enough through the database. On the first loop, memory starts at 36700160; the program crashes when memory exceeds the 134217728 limit.

I am also wondering why the starting memory is so much larger, but that could simply be explained by 5.3.5 using a lot more memory than 5.2.4.

So far I have just been testing. I have yet to go ahead with the removal of the global variable to allow the local variable to be unset. That is my next step.
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 35747935
Maybe it is just a matter of terminology, but neither of these statements instantiates an object; they return arrays:

$user = mysqli_fetch_row($rs) and $user = array(1, 2, 3)
http://php.net/manual/en/function.mysql-fetch-row.php
http://php.net/manual/en/function.array.php

The PHP bug report was about an object.  I think we may be wasting our time here.  Since we do not have any test data to work with, the rest of this dialog is going to have to be between you and yourself.  Hope you find the answer, ~Ray

0
 

Author Comment

by:jasimon9
ID: 35749524
I have changed the loop variable into a local variable. This has no effect on the memory usage.

But the unset also has no effect on memory usage either.

On our production server, the script does not complete because it runs out of memory. However, on a development server that has the identical script, the identical configuration, with the same OS, Apache, PHP and mysql, identical php.ini and almost identical my.cnf (mostly differences in buffer sizes), the program runs to completion. There is a memory leak on the development server as well, but it proceeds at about half the rate so that memory usage is well within limits, allowing the run to completion.

I am wondering if the mysql buffer sizes could be related.

Here are the setting with differences for the production server:

[mysqld]
port            = 3306
socket            = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 2048M
max_allowed_packet = 256M
table_open_cache = 512
sort_buffer_size = 16M
read_buffer_size = 16M
read_rnd_buffer_size = 32M
myisam_sort_buffer_size = 256M
thread_cache_size = 8
query_cache_size = 32M

Here are the settings for the development server:

[mysqld]
port            = 3306
socket            = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 2048M
max_allowed_packet = 1M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
0
 

Author Comment

by:jasimon9
ID: 35750319
I was inspired by the partial solution given to the related question at  http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_26929909.html -- namely, that the php memory leak bug introduced with 5.3 occurs in the mysqli connector but not in the mysql connector.

A preliminary test of switching to mysql_connect and related functions instead of mysqli_connect shows that there is no memory leak. I am thinking I will ask at the PHP bug site if they are interested in proof of this issue with 5.3.5.
0
 

Author Comment

by:jasimon9
ID: 35750704
No joy.

On our development server, reverting to the mysql connector instead of the enhanced mysqli connector eliminates 100% the memory leak.

But for some reason, on the nearly identically configured production server, the memory leak remains exactly as before.

This "impossible" result is really making me stretch to see what I am missing, because there must be something different that is producing the different results.

0
 

Author Comment

by:jasimon9
ID: 35751271
I still don't know why changing to mysql_connect on our development server eliminates all memory leaks, but the same change has no effect on the memory leaks on our production server, which continue as before. We are considering the following:

1. Upgrade to PHP 5.3.6, which could fix it, or at least enable us to report the bug.

2. Make the my.cnf identical on the off chance that that is having an unexpected effect.

In the meantime, I did some additional memory study on our production server. I have found the following odd results:

1. Immediately after executing the SQL statement, memory usage jumps 12MB. It does not change at all on our development server.

2. Putting memory displays into the code shows that memory usage is reported at exactly the same number though a number of processed rows. But then oddly, after some of the rows, memory usage jumps about 300,000 bytes. For example, this occurs after row 370, 447, 525 and 602. I did not check further. But in between these jumps, the reported usage stays exactly the same. There is nothing special happening on these rows where the reported usage jumps that is any different from any other rows.

I'd love to "send a small reproducible data set and code", and it might be worth seeing if I can create such.

0
 

Author Comment

by:jasimon9
ID: 35755788
I have developed a "small reproducible code" which demonstrates very clearly the problem. I am not yet ready to send a test data set because there are still some things I would like to try.

Here is the main loop of the code:


      // main loop
      $cnt = 0;
      echo 'mem=' . memory_get_usage(true) . "\n";
      while($row = mysql_fetch_row($rs))
      {
            if (++$cnt % 3000 == 0)
            {
                  echo '   id=' . $row[1] . '  mem=' . memory_get_usage(true) . "\n";
            }
      }


Here is the output from PHP 5.2.4 on Windows:

F:\Websites\RepHunter\current>php test-autoemail-memory.php
Test Autoemail Memory Leak
mem=262144
   id=43627  mem=262144
   id=40224  mem=262144
   id=37335  mem=262144
   id=34398  mem=262144
   id=31525  mem=262144
   id=28899  mem=262144
   id=26134  mem=262144
   id=21425  mem=262144
   id=16525  mem=262144
   id=13050  mem=262144
   id=9114  mem=262144
   id=3851  mem=262144
EOJ

Here is the output from PHP 5.3.5 on FreeBSD:

[jas1@www /var/www/rephunter/www/webroot]$ php ./test-autoemail-memory.php
Test Autoemail Memory Leak
mem=13631488
   id=43627  mem=23592960
   id=40224  mem=33292288
   id=37335  mem=43253760
   id=34398  mem=52953088
   id=31525  mem=62914560
   id=28899  mem=72613888
   id=26134  mem=82575360
   id=21425  mem=92274688
   id=16525  mem=102236160
   id=13050  mem=112197632
   id=9114  mem=121896960
   id=3851  mem=131858432
EOJ

Please note that the database used for both runs is the same. This very clearly shows we have a "smoking gun".

It also seems strange that the initial memory usage is so much larger.

To me this is pointing out a serious flaw in the mysql connector. Also, this time I have reverted to the mysql_connect command set rather than the mysqli_connect command set as the mysqli_ set seems to have the problem in more php versions, although I have not tested that with this "small reproducible code."

I did some playing around to either unset the $row variable, to set it to null, and to add the usage of gc_collect_cycles() along with these actions. None of these actions had any effect on the memory.

In the meantime, we have decided to upgrade to PHP 5.3.6, so that a bug report will be accepted. That will be rolled out into our production environment tonight, so I will very soon have some solid results to work from.
0
 

Accepted Solution

by:
jasimon9 earned 0 total points
ID: 35908603
As a followup, this issue has not yet been resolved, although the brute force solution of increasing memory from 128M to 256M for the one script allows it to run.

I believe there is a memory leak. I have opened a ticket at bugs.php.net. So as far as this question goes, I think I will accept this comment as the "solution."
0
 

Author Closing Comment

by:jasimon9
ID: 35937261
Workaround of allowing more memory allows script to run, but does not solve underlying memory leak. For that I have posted a ticket on bugs.php.net.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Build an array called $myWeek which will hold the array elements Today, Yesterday and then builds up the rest of the week by the name of the day going back 1 week.   (CODE) (CODE) Then you just need to pass your date to the function. If i…
Originally, this post was published on Monitis Blog, you can check it here . In business circles, we sometimes hear that today is the “age of the customer.” And so it is. Thanks to the enormous advances over the past few years in consumer techno…
The viewer will learn how to count occurrences of each item in an array.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Suggested Courses
Course of the Month20 days, 17 hours left to enroll

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question