Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1220
  • Last Modified:

can not set php memory limit in .htaccess and ini_set()

Dear Experts,

I am using curl in php to grab one remote webpage site that is more than 36Meg byte
And I get the fatal error message which is "Out of memory(allocated 3407872( tried to
allocate 3147593 byte) in .../data/include/curl.php on line42" line42 is my $htm=curl_exec($url);
I have checked my php system 's max memory limit is 128M byte by php's phpinfo(). Why I will have such error ?

I go to google it about this at http://drupal.org/node/29268 
and I changed my .htaccess file with  php_value memory_limit   120M, I talked to my hosting domain support, they just replied it is right there is memory limit=128M byte.
So anyone have suggestion to solve the fatal error issue ?
I also use this in php at the top , ini_set("memory_limit", "120M");, but it still fail
I also use phpinfo() and host support confirm it is 128Meg in my php system.
Any variable use related to curl_exec() is just 2 so it should be less than 120Meg usage in my php ($htm,  $file_url)
And attached my .htaccess file and my php sytem configuration.
Is it one variable such as $file_url  can not store more than 32Meg byte ?


Please advise & Thanks
Duncan


my php script
=====================
<?php
$file_url=my_curl("wwww.example.com","" , 3000, TRUE);

function my_curl
( $url
, $get_array=array()
, $timeout=3
, $error_report=TRUE
)
{
    // PREPARE THE ARGUMENT STRING IF NEEDED
  //  $get_string = '';
  //  foreach ($get_array as $key => $val)
  //  {
  //      $get_string
 //       = $get_string
 //       . urlencode($key)
  //      . '='
  //      . urlencode($val)
  //      . '&';
  //  }
  //  $get_string = rtrim($get_string, '&');
    if (!empty($get_string)) $url .= '?' . $get_string;
  $curl = curl_init();
    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE BLANK
    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
    curl_setopt( $curl, CURLOPT_URL,            $url  );
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  );
    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );
    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );
    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );
    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );
  //  curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );
    // RUN THE CURL REQUEST AND GET THE RESULTS
    $htm = curl_exec($curl);

    // ON FAILURE HANDLE ERROR MESSAGE
      //global $track; // declaring global variable
      
    if ($htm === FALSE)
    {
        if ($error_report)
        {
            $err = curl_errno($curl);
            $inf = curl_getinfo($curl);
           // echo "FAILC: $url TIMEOUT=$timeout, CURL_ERRNOC=$err";
            var_dump($inf);
        }
            
      
        curl_close($curl);
            
        return FALSE;
    }

    // ON SUCCESS RETURN XML / HTML STRING
    curl_close($curl);
    return $htm;
}
?>


my .htaccess file
====================
# -FrontPage-


<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
php_value memory_limit 120M
AuthUserFile /home/aab/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/aab/public_html/_vti_pvt/service.grp
AuthName aaa.com
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti* Accept and Award Points Accept as Solution

Open in new window

my Php Configuration
===================

PHP Core

Directive => Local Value => Master Value
allow_call_time_pass_reference => On => On
allow_url_fopen => On => On
allow_url_include => Off => Off
always_populate_raw_post_data => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
asp_tags => Off => Off
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => /usr/lib/php/head.php => /usr/lib/php/head.php
browscap => no value => no value
default_charset => no value => no value
default_mimetype => text/html => text/html
define_syslog_variables => Off => Off
detect_unicode => On => On
disable_classes => no value => no value
disable_functions => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source
display_errors => STDOUT => STDOUT
display_startup_errors => Off => Off
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => Off => Off
error_append_string => no value => no value
error_log => error_log => error_log
error_prepend_string => no value => no value
error_reporting => 6135 => 6135
expose_php => On => On
extension_dir => /usr/local/lib/php/extensions/no-debug-non-zts-20060613 => /usr/local/lib/php/extensions/no-debug-non-zts-20060613
file_uploads => On => On
highlight.bg => <font style="color: #FFFFFF">#FFFFFF</font> => <font style="color: #FFFFFF">#FFFFFF</font>
highlight.comment => <font style="color: #FF8000">#FF8000</font> => <font style="color: #FF8000">#FF8000</font>
highlight.default => <font style="color: #0000BB">#0000BB</font> => <font style="color: #0000BB">#0000BB</font>
highlight.html => <font style="color: #000000">#000000</font> => <font style="color: #000000">#000000</font>
highlight.keyword => <font style="color: #007700">#007700</font> => <font style="color: #007700">#007700</font>
highlight.string => <font style="color: #DD0000">#DD0000</font> => <font style="color: #DD0000">#DD0000</font>
html_errors => Off => Off
ignore_repeated_errors => On => On
ignore_repeated_source => On => On
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .:/usr/lib/php:/usr/local/lib/php => .:/usr/lib/php:/usr/local/lib/php
log_errors => On => On
log_errors_max_len => 1024 => 1024
magic_quotes_gpc => On => On
magic_quotes_runtime => Off => Off
magic_quotes_sybase => Off => Off
mail.force_extra_parameters => no value => no value
max_execution_time => 0 => 0
max_file_uploads => 20 => 20
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
memory_limit => 128M => 128M
open_basedir => no value => no value
output_buffering => 0 => 0
output_handler => no value => no value
post_max_size => 128M => 128M
precision => 12 => 12
realpath_cache_size => 16K => 16K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
register_globals => Off => Off
register_long_arrays => On => On
report_memleaks => On => On
report_zend_debug => Off => Off
safe_mode => Off => Off
safe_mode_exec_dir => /usr/local/php/bin => /usr/local/php/bin
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sendmail_from => no value => no value
sendmail_path => /usr/sbin/sendmail -t -i => /usr/sbin/sendmail -t -i
serialize_precision => 100 => 100
short_open_tag => On => On
SMTP => localhost => localhost
smtp_port => 25 => 25
sql.safe_mode => Off => Off
track_errors => Off => Off
unserialize_callback_func => no value => no value
upload_max_filesize => 128M => 128M
upload_tmp_dir => /tmp => /tmp
user_dir => no value => no value
variables_order => EGPCS => EGPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
y2k_compliance => On => On
zend.ze1_compatibility_mode => Off => Off Accept and Award Points Accept as Solution

Open in new window

0
duncanb7
Asked:
duncanb7
  • 2
  • 2
2 Solutions
 
AmickCommented:
Set CURLOPT_RETURNTRANSFER False, and open a file-handle instead, passing it to CURLOPT_FILE. This lets curl_exec() saves the content to the file instead of holding it in memory.

http://us2.php.net/manual/en/function.curl-setopt.php
0
 
duncanb7Author Commented:
I thought that direction before and suppose what you said is correct
but finally I still need to open the saved file for data operation in memory so that
sitll have out of memory issue once I open it by fopen or file() in php.

Or Fopen and File won't open and put whole file data into memory ?

Please advise

Duncan
0
 
AmickCommented:
I can't advise you on how Fopen and File manage memory. I would try it and see the result.

If you want to pursue higher memory limits, you can locally set the memory limits in your script unless you've disallowed that on your system.
PHP_INI_ALL should be enabled to locally control memory_limit.

ini_set('memory_limit', '256M');
ini_set('post_max_size', '256M');
ini_set('upload_max_filesize', '256M');

Open in new window


Setting  memory limits to -1 in your script effectively removes them, and would be worth trying if transfer still fails at 256M.

you might use
echo memory_get_peak_usage() . "\n";

Open in new window

at exit to see how much memory you actually used.
0
 
duncanb7Author Commented:
Finally, I give up to
use memory instead of file
and it works fine in term speed
just a little bit slow
0

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now