[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

One Variable memory size in Linux system for curl and php

Posted on 2011-05-12
5
Medium Priority
?
527 Views
Last Modified: 2012-05-11
Dear Expert,

I run following curl php script to grab 36meg data from remote server site and save it
into one varaible like $htm and $file_url.  And the error report: Out of memory in my_curl function
at $htm=curl_exec($url)
Is it not allowed I put the data at such big size into one variable ?
If not, how can I put 32M  data into one variable, i know there is something related
to curl and memory reallocation. Any one has that code, please advise for me for a start
I have checked phpinfo() my  php max memory limit is 128Meg and also
add php_value memor_limit 120M in my .htaccess file



<?php
$file_url=my_curl("wwww.example.com","" , 3000, TRUE);

function my_curl
( $url
, $get_array=array()
, $timeout=3
, $error_report=TRUE
)
{
    // PREPARE THE ARGUMENT STRING IF NEEDED
  //  $get_string = '';
  //  foreach ($get_array as $key => $val)
  //  {
  //      $get_string
 //       = $get_string
 //       . urlencode($key)
  //      . '='
  //      . urlencode($val)
  //      . '&';
  //  }
  //  $get_string = rtrim($get_string, '&');
    if (!empty($get_string)) $url .= '?' . $get_string;
  $curl = curl_init();
    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE BLANK
    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
    curl_setopt( $curl, CURLOPT_URL,            $url  );
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  );
    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );
    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );
    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );
    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );
  //  curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );
    // RUN THE CURL REQUEST AND GET THE RESULTS
    $htm = curl_exec($curl);

    // ON FAILURE HANDLE ERROR MESSAGE
      //global $track; // declaring global variable
      
    if ($htm === FALSE)
    {
        if ($error_report)
        {
            $err = curl_errno($curl);
            $inf = curl_getinfo($curl);
           // echo "FAILC: $url TIMEOUT=$timeout, CURL_ERRNOC=$err";
            var_dump($inf);
        }
            
      
        curl_close($curl);
            
        return FALSE;
    }

    // ON SUCCESS RETURN XML / HTML STRING
    curl_close($curl);
    return $htm;
}
?>

Open in new window

configuration

PHP Core

Directive => Local Value => Master Value
allow_call_time_pass_reference => On => On
allow_url_fopen => On => On
allow_url_include => Off => Off
always_populate_raw_post_data => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
asp_tags => Off => Off
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => /usr/lib/php/head.php => /usr/lib/php/head.php
browscap => no value => no value
default_charset => no value => no value
default_mimetype => text/html => text/html
define_syslog_variables => Off => Off
detect_unicode => On => On
disable_classes => no value => no value
disable_functions => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source
display_errors => STDOUT => STDOUT
display_startup_errors => Off => Off
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => Off => Off
error_append_string => no value => no value
error_log => error_log => error_log
error_prepend_string => no value => no value
error_reporting => 6135 => 6135
expose_php => On => On
extension_dir => /usr/local/lib/php/extensions/no-debug-non-zts-20060613 => /usr/local/lib/php/extensions/no-debug-non-zts-20060613
file_uploads => On => On
highlight.bg => <font style="color: #FFFFFF">#FFFFFF</font> => <font style="color: #FFFFFF">#FFFFFF</font>
highlight.comment => <font style="color: #FF8000">#FF8000</font> => <font style="color: #FF8000">#FF8000</font>
highlight.default => <font style="color: #0000BB">#0000BB</font> => <font style="color: #0000BB">#0000BB</font>
highlight.html => <font style="color: #000000">#000000</font> => <font style="color: #000000">#000000</font>
highlight.keyword => <font style="color: #007700">#007700</font> => <font style="color: #007700">#007700</font>
highlight.string => <font style="color: #DD0000">#DD0000</font> => <font style="color: #DD0000">#DD0000</font>
html_errors => Off => Off
ignore_repeated_errors => On => On
ignore_repeated_source => On => On
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .:/usr/lib/php:/usr/local/lib/php => .:/usr/lib/php:/usr/local/lib/php
log_errors => On => On
log_errors_max_len => 1024 => 1024
magic_quotes_gpc => On => On
magic_quotes_runtime => Off => Off
magic_quotes_sybase => Off => Off
mail.force_extra_parameters => no value => no value
max_execution_time => 0 => 0
max_file_uploads => 20 => 20
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
memory_limit => 128M => 128M
open_basedir => no value => no value
output_buffering => 0 => 0
output_handler => no value => no value
post_max_size => 128M => 128M
precision => 12 => 12
realpath_cache_size => 16K => 16K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
register_globals => Off => Off
register_long_arrays => On => On
report_memleaks => On => On
report_zend_debug => Off => Off
safe_mode => Off => Off
safe_mode_exec_dir => /usr/local/php/bin => /usr/local/php/bin
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sendmail_from => no value => no value
sendmail_path => /usr/sbin/sendmail -t -i => /usr/sbin/sendmail -t -i
serialize_precision => 100 => 100
short_open_tag => On => On
SMTP => localhost => localhost
smtp_port => 25 => 25
sql.safe_mode => Off => Off
track_errors => Off => Off
unserialize_callback_func => no value => no value
upload_max_filesize => 128M => 128M
upload_tmp_dir => /tmp => /tmp
user_dir => no value => no value
variables_order => EGPCS => EGPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
y2k_compliance => On => On
zend.ze1_compatibility_mode => Off => Off

Open in new window

my .htaccess file 
# -FrontPage-


<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
php_value memory_limit 120M
AuthUserFile /home/aab/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/aab/public_html/_vti_pvt/service.grp
AuthName aaa.com
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

Open in new window

0
Comment
Question by:duncanb7
  • 3
  • 2
5 Comments
 
LVL 36

Accepted Solution

by:
Zyloch earned 1500 total points
ID: 35759816
I agree that this is strange. One thing to make sure is that you have restarted your server after the changes to memory_limit in your php.ini file. If you do not own the server, make sure that you are allowed to raise your memory limit.

One way to bypass the need for cURL to store the entire file into a variable in memory is to have it write to a file. You can find out how to do that via this tutorial.
0
 
LVL 36

Assisted Solution

by:Zyloch
Zyloch earned 1500 total points
ID: 35759818
By the way, what is the exact error message you get? It may indicate how much memory you are trying to allocate out of how much is available.
0
 
LVL 13

Author Comment

by:duncanb7
ID: 35776160
I already curl it into file succesfully , now it is still error message about out of memory.
but when I open (using fopen or using file() ) i t and   store into $htm  about  2.5meg which is not allowed even I already set
max_limit=128.

Why ? please advise
0
 
LVL 13

Author Comment

by:duncanb7
ID: 35776166
I already curl it into file succesfully , now it has new  other error message about out of memory when I open (using fopen or using file() ) i t and   store into $csv  about  2.5meg which is not allowed even I already set
max_limit=128Meg.

Why ? please advise
0
 
LVL 13

Author Closing Comment

by:duncanb7
ID: 35777031
For my memo:
We need to do
sprintf(%.3f, $var) that will
save or reduce variable size , and now it is  passed the out of memory
error

If the file size is too big, and then
php script do data calculation
for each data , and if the data is without truncation   like 123.123456789012333333333322222222222222222222222 which
will create the file size is bigger and bigger once we save the file after
calculation. So it might use up all memory
0

Featured Post

NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is intended for those who are new to PHP error handling (https://www.experts-exchange.com/articles/11769/And-by-the-way-I-am-New-to-PHP.html).  It addresses one of the most common problems that plague beginning PHP develop…
3 proven steps to speed up Magento powered sites. The article focus is on optimizing time to first byte (TTFB), full page caching and configuring server for optimal performance.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month18 days, 4 hours left to enroll

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question