Link to home
Start Free TrialLog in
Avatar of duncanb7
duncanb7

asked on

One Variable memory size in Linux system for curl and php

Dear Expert,

I run following curl php script to grab 36meg data from remote server site and save it
into one varaible like $htm and $file_url.  And the error report: Out of memory in my_curl function
at $htm=curl_exec($url)
Is it not allowed I put the data at such big size into one variable ?
If not, how can I put 32M  data into one variable, i know there is something related
to curl and memory reallocation. Any one has that code, please advise for me for a start
I have checked phpinfo() my  php max memory limit is 128Meg and also
add php_value memor_limit 120M in my .htaccess file



<?php
$file_url=my_curl("wwww.example.com","" , 3000, TRUE);

function my_curl
( $url
, $get_array=array()
, $timeout=3
, $error_report=TRUE
)
{
    // PREPARE THE ARGUMENT STRING IF NEEDED
  //  $get_string = '';
  //  foreach ($get_array as $key => $val)
  //  {
  //      $get_string
 //       = $get_string
 //       . urlencode($key)
  //      . '='
  //      . urlencode($val)
  //      . '&';
  //  }
  //  $get_string = rtrim($get_string, '&');
    if (!empty($get_string)) $url .= '?' . $get_string;
  $curl = curl_init();
    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE BLANK
    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
    curl_setopt( $curl, CURLOPT_URL,            $url  );
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  );
    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );
    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );
    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );
    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );
  //  curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );
    // RUN THE CURL REQUEST AND GET THE RESULTS
    $htm = curl_exec($curl);

    // ON FAILURE HANDLE ERROR MESSAGE
      //global $track; // declaring global variable
      
    if ($htm === FALSE)
    {
        if ($error_report)
        {
            $err = curl_errno($curl);
            $inf = curl_getinfo($curl);
           // echo "FAILC: $url TIMEOUT=$timeout, CURL_ERRNOC=$err";
            var_dump($inf);
        }
            
      
        curl_close($curl);
            
        return FALSE;
    }

    // ON SUCCESS RETURN XML / HTML STRING
    curl_close($curl);
    return $htm;
}
?>

Open in new window

configuration

PHP Core

Directive => Local Value => Master Value
allow_call_time_pass_reference => On => On
allow_url_fopen => On => On
allow_url_include => Off => Off
always_populate_raw_post_data => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
asp_tags => Off => Off
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => /usr/lib/php/head.php => /usr/lib/php/head.php
browscap => no value => no value
default_charset => no value => no value
default_mimetype => text/html => text/html
define_syslog_variables => Off => Off
detect_unicode => On => On
disable_classes => no value => no value
disable_functions => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source
display_errors => STDOUT => STDOUT
display_startup_errors => Off => Off
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => Off => Off
error_append_string => no value => no value
error_log => error_log => error_log
error_prepend_string => no value => no value
error_reporting => 6135 => 6135
expose_php => On => On
extension_dir => /usr/local/lib/php/extensions/no-debug-non-zts-20060613 => /usr/local/lib/php/extensions/no-debug-non-zts-20060613
file_uploads => On => On
highlight.bg => <font style="color: #FFFFFF">#FFFFFF</font> => <font style="color: #FFFFFF">#FFFFFF</font>
highlight.comment => <font style="color: #FF8000">#FF8000</font> => <font style="color: #FF8000">#FF8000</font>
highlight.default => <font style="color: #0000BB">#0000BB</font> => <font style="color: #0000BB">#0000BB</font>
highlight.html => <font style="color: #000000">#000000</font> => <font style="color: #000000">#000000</font>
highlight.keyword => <font style="color: #007700">#007700</font> => <font style="color: #007700">#007700</font>
highlight.string => <font style="color: #DD0000">#DD0000</font> => <font style="color: #DD0000">#DD0000</font>
html_errors => Off => Off
ignore_repeated_errors => On => On
ignore_repeated_source => On => On
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .:/usr/lib/php:/usr/local/lib/php => .:/usr/lib/php:/usr/local/lib/php
log_errors => On => On
log_errors_max_len => 1024 => 1024
magic_quotes_gpc => On => On
magic_quotes_runtime => Off => Off
magic_quotes_sybase => Off => Off
mail.force_extra_parameters => no value => no value
max_execution_time => 0 => 0
max_file_uploads => 20 => 20
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
memory_limit => 128M => 128M
open_basedir => no value => no value
output_buffering => 0 => 0
output_handler => no value => no value
post_max_size => 128M => 128M
precision => 12 => 12
realpath_cache_size => 16K => 16K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
register_globals => Off => Off
register_long_arrays => On => On
report_memleaks => On => On
report_zend_debug => Off => Off
safe_mode => Off => Off
safe_mode_exec_dir => /usr/local/php/bin => /usr/local/php/bin
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sendmail_from => no value => no value
sendmail_path => /usr/sbin/sendmail -t -i => /usr/sbin/sendmail -t -i
serialize_precision => 100 => 100
short_open_tag => On => On
SMTP => localhost => localhost
smtp_port => 25 => 25
sql.safe_mode => Off => Off
track_errors => Off => Off
unserialize_callback_func => no value => no value
upload_max_filesize => 128M => 128M
upload_tmp_dir => /tmp => /tmp
user_dir => no value => no value
variables_order => EGPCS => EGPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
y2k_compliance => On => On
zend.ze1_compatibility_mode => Off => Off

Open in new window

my .htaccess file 
# -FrontPage-


<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
php_value memory_limit 120M
AuthUserFile /home/aab/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/aab/public_html/_vti_pvt/service.grp
AuthName aaa.com
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of Zyloch
Zyloch
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of duncanb7
duncanb7

ASKER

I already curl it into file succesfully , now it is still error message about out of memory.
but when I open (using fopen or using file() ) i t and   store into $htm  about  2.5meg which is not allowed even I already set
max_limit=128.

Why ? please advise
I already curl it into file succesfully , now it has new  other error message about out of memory when I open (using fopen or using file() ) i t and   store into $csv  about  2.5meg which is not allowed even I already set
max_limit=128Meg.

Why ? please advise
For my memo:
We need to do
sprintf(%.3f, $var) that will
save or reduce variable size , and now it is  passed the out of memory
error

If the file size is too big, and then
php script do data calculation
for each data , and if the data is without truncation   like 123.123456789012333333333322222222222222222222222 which
will create the file size is bigger and bigger once we save the file after
calculation. So it might use up all memory