Geoff Millikan
asked on
Windows 7 > PHP 5.3.6 > Cannot set memory limit
Tried setting the memory limit everywhere but no matter where I set it, it doesn't "take." What am I missing?
Here's the PHP code:
C:\Users\Public>php -f md553_to_csv.php -d memory_limit=1512927291
Fatal error: Out of memory (allocated 524288)
(tried to allocate 1512927290 bytes) in
C:\Users\Public\md553_to_csv.php on line 3
Here's the PHP code:
<?php
ini_set('memory_limit', '1512927291');
$lines = file('C:\Users\Public\CRMD3836.mask', FILE_IGNORE_NEW_LINES);
?>
try to edit the php.ini file, and you will probably need to run your text editor as Administrator.
ASKER
Further debugging shows the memory_limit is in fact getting changed - the issue has something to do with how big the file is (about 1.5GB) and the function, file(). The error seems to point to memory_limit bing the issue, but it's not.
So not sure what the issue is, this is very strange.
So not sure what the issue is, this is very strange.
A file of 1.5GB is kind of large for in-memory storage, even in this day and age. Can you give us a URL link to the file and a sense of what you want to get out of the file, please?
ASKER
Cannot give link but can tell you it's a big ASCII text file that we're parsing content out of, line-by-line.
In the mean time, we're using fopen(), fgets() and that works fine but still curious why file() doesn't work.
In the mean time, we're using fopen(), fgets() and that works fine but still curious why file() doesn't work.
OK, please create a big ASCII text file that looks like what you want to work on. Put it on a server where you can give the link. To understand why we need to see the test data, please see the article here about how professional programmers approach problems with test-driven development.
https://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/A_7830-A-Quick-Tour-of-Test-Driven-Development.html
Fopen() and fgets() read one line at a time, and this process uses relatively little memory. File() reads the entire file into an array, thus ensuring that the largest possible representation of the file is loaded all at once.
https://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/A_7830-A-Quick-Tour-of-Test-Driven-Development.html
Fopen() and fgets() read one line at a time, and this process uses relatively little memory. File() reads the entire file into an array, thus ensuring that the largest possible representation of the file is loaded all at once.
ASKER
This works: Run this at Windows command line:
Then run this:
And you get this error:
Fatal error: Out of memory (allocated 524288) (tried to allocate 2147483648 bytes) in Command line code on line 1
Merry Christmas!
php -r "while (filesize('C:\Users\Public\I_AM_A_2GB_FILE.txt')<=2147483648) {file_put_contents('C:\Users\Public\I_AM_A_2GB_FILE.txt',substr(str_shuffle(str_repeat('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 ',5000)),0,262144), FILE_APPEND);}"
Then run this:
C:\Users\Public>php -r "echo strlen(file_get_contents('C:\Users\Public\I_AM_A_2GB_FILE.txt'));" -d memory_limit=2500000000
And you get this error:
Fatal error: Out of memory (allocated 524288) (tried to allocate 2147483648 bytes) in Command line code on line 1
Merry Christmas!
ASKER
Actually, better to use this to make a 1GB file so we don't run into the 32bit integer issue described at link below:
http://php.net/manual/en/function.filesize.php
Note: Because PHP's integer type is signed and many platforms use 32bit integers, filesize() may return unexpected results for files which are larger than 2GB. For files between 2GB and 4GB in size this can usually be overcome by using sprintf("%u", filesize($file)).
C:\Users\Public>php -r "while (filesize('C:\Users\Public\I_AM_A_1GB_FILE.txt')<=1073741824) {file_put_contents('C:\Users\Public\I_AM_A_1GB_FILE.txt',substr(str_shuffle(str_repeat('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 ',5000)),0,262144), FILE_APPEND);}"
http://php.net/manual/en/function.filesize.php
Note: Because PHP's integer type is signed and many platforms use 32bit integers, filesize() may return unexpected results for files which are larger than 2GB. For files between 2GB and 4GB in size this can usually be overcome by using sprintf("%u", filesize($file)).
ASKER
Hold the phone, it's working now for 1GB file...
ASKER
file_get_contents() works for me on files up to about 1,182,531,584 bytes but if the files go over 1,606,156,288 bytes it fails consistently:
Fatal error: Out of memory (allocated 524288) (tried to allocate 1606164480 bytes) in Command line code on line 1
And just to confirm, my system has that much RAM available:
Fatal error: Out of memory (allocated 524288) (tried to allocate 1606164480 bytes) in Command line code on line 1
And just to confirm, my system has that much RAM available:
Where is the link to the test file we asked for?
PHP has limits on the amount of memory it uses. These may not be directly related to the amount of memory on the system -- it just depends on the PHP settings.
PHP has limits on the amount of memory it uses. These may not be directly related to the amount of memory on the system -- it just depends on the PHP settings.
ASKER
Ray_Paseur
Cut and paste the script below and you will have your test data:
Follow instructions above to reproduce. I don't believe this is a memory_limit ini file issue, much more complicated.
Where is the link to the test file we asked for?
Cut and paste the script below and you will have your test data:
C:\Users\Public>php -r "while (filesize('C:\Users\Public\I_AM_A_1GB_FILE.txt')<=1073741824) {file_put_contents('C:\Users\Public\I_AM_A_1GB_FILE.txt',substr(str_shuffle(str_repeat('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 ',5000)),0,262144), FILE_APPEND);}"
Follow instructions above to reproduce. I don't believe this is a memory_limit ini file issue, much more complicated.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
What version of PHP are you running? I'm running the Windows version of PHP 5.3.6 32bit on Windows 7 64bit.
Running this code:
Produces this result:
Running this code:
<?php
$file='C:\Users\Public\I_AM_A_1GB_FILE.txt';
error_reporting(E_ALL);
ini_set('memory_limit', -1);
echo filesize($file), "\r\n";
echo ini_get('memory_limit'),"\r\n";
$x = file($file);
$m = memory_get_peak_usage(TRUE);
var_dump($m);
?>
Produces this result:
c:\Users\Public>php -f "c:\Users\Public\memory_limit.php"
1548746752
-1
Fatal error: Out of memory (allocated 524288) (tried to allocate 1548754944 bytes) in C:\Users\Public\memory_limit.php on line 7
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ah ha! Yes, this works for me on Linux no problem. On Windows, it doesn't. Must be a Windows issue with how RAM is allocated.
ASKER
Bug in Windows version of PHP...