duncanb7
asked on
One Variable memory size in Linux system for curl and php
Dear Expert,
I run following curl php script to grab 36meg data from remote server site and save it
into one varaible like $htm and $file_url. And the error report: Out of memory in my_curl function
at $htm=curl_exec($url)
Is it not allowed I put the data at such big size into one variable ?
If not, how can I put 32M data into one variable, i know there is something related
to curl and memory reallocation. Any one has that code, please advise for me for a start
I have checked phpinfo() my php max memory limit is 128Meg and also
add php_value memor_limit 120M in my .htaccess file
I run following curl php script to grab 36meg data from remote server site and save it
into one varaible like $htm and $file_url. And the error report: Out of memory in my_curl function
at $htm=curl_exec($url)
Is it not allowed I put the data at such big size into one variable ?
If not, how can I put 32M data into one variable, i know there is something related
to curl and memory reallocation. Any one has that code, please advise for me for a start
I have checked phpinfo() my php max memory limit is 128Meg and also
add php_value memor_limit 120M in my .htaccess file
<?php
$file_url=my_curl("wwww.example.com","" , 3000, TRUE);
function my_curl
( $url
, $get_array=array()
, $timeout=3
, $error_report=TRUE
)
{
// PREPARE THE ARGUMENT STRING IF NEEDED
// $get_string = '';
// foreach ($get_array as $key => $val)
// {
// $get_string
// = $get_string
// . urlencode($key)
// . '='
// . urlencode($val)
// . '&';
// }
// $get_string = rtrim($get_string, '&');
if (!empty($get_string)) $url .= '?' . $get_string;
$curl = curl_init();
// HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
$header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: "; // BROWSERS USUALLY LEAVE BLANK
// SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6' );
curl_setopt( $curl, CURLOPT_HTTPHEADER, $header );
curl_setopt( $curl, CURLOPT_REFERER, 'http://www.google.com' );
curl_setopt( $curl, CURLOPT_ENCODING, 'gzip,deflate' );
curl_setopt( $curl, CURLOPT_AUTOREFERER, TRUE );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE );
// curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE );
curl_setopt( $curl, CURLOPT_TIMEOUT, $timeout );
// RUN THE CURL REQUEST AND GET THE RESULTS
$htm = curl_exec($curl);
// ON FAILURE HANDLE ERROR MESSAGE
//global $track; // declaring global variable
if ($htm === FALSE)
{
if ($error_report)
{
$err = curl_errno($curl);
$inf = curl_getinfo($curl);
// echo "FAILC: $url TIMEOUT=$timeout, CURL_ERRNOC=$err";
var_dump($inf);
}
curl_close($curl);
return FALSE;
}
// ON SUCCESS RETURN XML / HTML STRING
curl_close($curl);
return $htm;
}
?>
configuration
PHP Core
Directive => Local Value => Master Value
allow_call_time_pass_reference => On => On
allow_url_fopen => On => On
allow_url_include => Off => Off
always_populate_raw_post_data => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
asp_tags => Off => Off
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => /usr/lib/php/head.php => /usr/lib/php/head.php
browscap => no value => no value
default_charset => no value => no value
default_mimetype => text/html => text/html
define_syslog_variables => Off => Off
detect_unicode => On => On
disable_classes => no value => no value
disable_functions => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source => symlink,shell_exec,exec,proc_close,proc_open,popen,system,dl,passthru,escapeshellarg,escapeshellcmd,show_source
display_errors => STDOUT => STDOUT
display_startup_errors => Off => Off
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => Off => Off
error_append_string => no value => no value
error_log => error_log => error_log
error_prepend_string => no value => no value
error_reporting => 6135 => 6135
expose_php => On => On
extension_dir => /usr/local/lib/php/extensions/no-debug-non-zts-20060613 => /usr/local/lib/php/extensions/no-debug-non-zts-20060613
file_uploads => On => On
highlight.bg => <font style="color: #FFFFFF">#FFFFFF</font> => <font style="color: #FFFFFF">#FFFFFF</font>
highlight.comment => <font style="color: #FF8000">#FF8000</font> => <font style="color: #FF8000">#FF8000</font>
highlight.default => <font style="color: #0000BB">#0000BB</font> => <font style="color: #0000BB">#0000BB</font>
highlight.html => <font style="color: #000000">#000000</font> => <font style="color: #000000">#000000</font>
highlight.keyword => <font style="color: #007700">#007700</font> => <font style="color: #007700">#007700</font>
highlight.string => <font style="color: #DD0000">#DD0000</font> => <font style="color: #DD0000">#DD0000</font>
html_errors => Off => Off
ignore_repeated_errors => On => On
ignore_repeated_source => On => On
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .:/usr/lib/php:/usr/local/lib/php => .:/usr/lib/php:/usr/local/lib/php
log_errors => On => On
log_errors_max_len => 1024 => 1024
magic_quotes_gpc => On => On
magic_quotes_runtime => Off => Off
magic_quotes_sybase => Off => Off
mail.force_extra_parameters => no value => no value
max_execution_time => 0 => 0
max_file_uploads => 20 => 20
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
memory_limit => 128M => 128M
open_basedir => no value => no value
output_buffering => 0 => 0
output_handler => no value => no value
post_max_size => 128M => 128M
precision => 12 => 12
realpath_cache_size => 16K => 16K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
register_globals => Off => Off
register_long_arrays => On => On
report_memleaks => On => On
report_zend_debug => Off => Off
safe_mode => Off => Off
safe_mode_exec_dir => /usr/local/php/bin => /usr/local/php/bin
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sendmail_from => no value => no value
sendmail_path => /usr/sbin/sendmail -t -i => /usr/sbin/sendmail -t -i
serialize_precision => 100 => 100
short_open_tag => On => On
SMTP => localhost => localhost
smtp_port => 25 => 25
sql.safe_mode => Off => Off
track_errors => Off => Off
unserialize_callback_func => no value => no value
upload_max_filesize => 128M => 128M
upload_tmp_dir => /tmp => /tmp
user_dir => no value => no value
variables_order => EGPCS => EGPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
y2k_compliance => On => On
zend.ze1_compatibility_mode => Off => Off
my .htaccess file
# -FrontPage-
<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
php_value memory_limit 120M
AuthUserFile /home/aab/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/aab/public_html/_vti_pvt/service.grp
AuthName aaa.com
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I already curl it into file succesfully , now it has new other error message about out of memory when I open (using fopen or using file() ) i t and store into $csv about 2.5meg which is not allowed even I already set
max_limit=128Meg.
Why ? please advise
max_limit=128Meg.
Why ? please advise
ASKER
For my memo:
We need to do
sprintf(%.3f, $var) that will
save or reduce variable size , and now it is passed the out of memory
error
If the file size is too big, and then
php script do data calculation
for each data , and if the data is without truncation like 123.1234567890123333333333 2222222222 2222222222 222 which
will create the file size is bigger and bigger once we save the file after
calculation. So it might use up all memory
We need to do
sprintf(%.3f, $var) that will
save or reduce variable size , and now it is passed the out of memory
error
If the file size is too big, and then
php script do data calculation
for each data , and if the data is without truncation like 123.1234567890123333333333
will create the file size is bigger and bigger once we save the file after
calculation. So it might use up all memory
ASKER
but when I open (using fopen or using file() ) i t and store into $htm about 2.5meg which is not allowed even I already set
max_limit=128.
Why ? please advise