<div>
Looks like the function csv_output loops an array, and save as much lines as there are elements in the array. Next it opens the file again and loops all lines again, processes some stuff and saves the file again. 
 
This looks like a waste of resources. It should be done in a single loop. 
 
You could also consider using a for loop instead of foreach loop. This way you can do something like the attached code snippet. I build an array with 100 elements, and loop them in batches of 10. 
 
I do not have too much time to convert your current script, but i think if you run the example, it will be clear. I have the code snippet online for you to test on <a href="http://test.alwinroosen.be/23575412/cchrc.php" rel="ugc">http://test.alwinroosen.be/23575412/cchrc.php</a> 
 
It will allow you to buffer output of only a few lines at the time. The script will ofcourse work as long as it does now (maybe somewhat faster), but it won't consume he same amount of resources. I am sure it can be improved by other solutions, but that's for someone else :) 

<pre><code class="code-10 nolinks" id="code-121607">&lt;?php
 
for($i=0;$i&lt;100;$i++) {
 
	$cchrc_data[] = &quot;test&quot;.$i;
 
} // end for
 
$finished = false;
$batch = 0;
$step = 10;
 
/*
 * TODO: clear the destination csv file
 * TODO: add the headers to your destination csv file
 */
 
while(!$finished) {
 
	$start = $batch*$step;
	$stop = (($batch+1)*$step)-1;
	
	if($stop &gt; count($cchrc_data)-1) {
	
		$finished = true;
	
	}else{
	
		echo &quot;running batch &quot;.$batch.&quot; (&quot;.$start.&quot;-&quot;.$stop.&quot;)&lt;br /&gt;&quot;;
		
		/*
		 * TODO: open your destination csv file in append mode
		 */
		
		for($i=$start;$i&lt;=$stop;$i++) {
		
			/*
			 * TODO: process line, access it with $cchrc_data[$start+$i]
			 */
		
			echo $cchrc_data[$i].&quot;&lt;br /&gt;&quot;;
		
		} // end for
		
		/*
		 * TODO: close your destination csv file again
		 */
		
		$batch++;
	
	} // end if
 
} // end while
 
?&gt;
</code></pre>
</div>

<div>
Damn, i see i have an error in a comment block: 
 
/* 
&nbsp;* TODO: process line, access it with $cchrc_data[$start+$i] 
&nbsp;*/ 
 
should be 
 
/* 
&nbsp;* TODO: process line, access it with $cchrc_data[$i] 
&nbsp;*/
</div>

<div>
I comment the function as it stands now, the rest of the script is unchanged...let me know if I'm moving forward, or if any of this fails to make any sense, thanks... 

<pre><code class="code-10 nolinks" id="code-121971">...
function csv_output($cchrc_header, $cchrc_data) {
 
global $config;
$output = fopen('data.csv', 'w');
for($i=0;$i&lt;30;$i++) { //reduced $i to 30 to save time
 
 $cchrc_header[] = &quot;test&quot;.$i;
 
 } // end for
 
$finished = false;
$batch = 0;
$step = 10;
 
/*
 * TRYTHIS: clear the destination csv file (ok)
*/
$output = fopen('data.csv', 'w+');
/*
 * TRYTHIS: add the headers to your destination csv file (ok)
 */
fwrite($output, &quot;\&quot;Date\&quot;&quot;);
	if ($cchrc_data) foreach ($cchrc_data as $name =&gt; $data)
	{
		fwrite($output, &quot;,\&quot;$name\&quot;&quot;);
	}
	fwrite($output, &quot;\n&quot;);
 
 
while(!$finished) {
 
 $start = $batch*$step;
 $stop = (($batch+1)*$step)-1;
 
 if($stop &gt; count($cchrc_data)-1) {
 
 $finished = true;
 
 }else{
 
 echo &quot;running batch &quot;.$batch.&quot; (&quot;.$start.&quot;-&quot;.$stop.&quot;)&lt;br /&gt;&quot;;
 
 /*
 * TRYTHIS: open your destination csv file in append mode (ok)
 */
 $output=fopen('data.csv', 'a');
 
 for($i=$start;$i&lt;=$stop;$i++) {
 
 /*
 * TRYTHIS: process line, access it with $cchrc_data[$i] (???)
 */
 if ($cchrc_header) foreach ($cchrc_header as $date)
 {
 $newline = &quot;\&quot;$date\&quot;&quot;;
 if ($cchrc_data) foreach ($cchrc_data as $name =&gt; $data)
 {
 $full_line = (&quot;,\&quot;&quot;.$data[$date].&quot;\&quot;&quot;);
 }
 $cchrc_data[$i] = fwrite($full_line, $output, &quot;\n&quot;);
 } 
 $cchrc_data[$i]; 
 
 echo $cchrc_data[$i].&quot;&lt;br /&gt;&quot;;
/* prints &quot;running batch 0 - 212 (2120-2129)&lt;br /&gt;0 etc to stdout...so far, no pv_ files are created. */ 
 } // end for
 
 /*
 * TRYTHIS: close your destination csv file again
 */
 fclose($output);
 $batch++;
 
 } // end if
 
 } // end while
}
</code></pre>
</div>

<div>
You are making progress, but I think I might be able to save you some time. I will try to post the converted function later on today, and see if there are other functions that might need improvement.
</div>

<div>
Thanks :) &nbsp;As you may have inferred, I was able to replicate the iteration of $i in batches of 10, got the header printed to data.csv, and started appending values of the date-data array to data.csv, &nbsp;exiting due to time constraints. &nbsp;But the reformatted pv_ files were never created, and I don't understand the relationship between them and data.csv. &nbsp;I thought that hr1_archive.csv could be processed to produce a reformatted pv_hr1_archive.csv directly, but it looks like the data must be copied to data.csv first and then extracted from it. &nbsp;As you can see, I'm confused about that, and I'm also confused about why the process took so long: I use the data files instead of the archive files for testing, since they are so much smaller. &nbsp;The original script finished processing all 24 data files in less than a minute, but with yours I think I got stuck on a loop of some kind, because the batch numbers kept incrementing ad infinitum, way past the original limit of 100 for $i. &nbsp;
</div>

<div>
I see some errors in the script you wrote so it will surely not work as expected. It was too late yesterday for such scripting but i have time today. Could you post me an example csv to test with? 100 lines should be enough to get a feeling of what this script is supposed to do. I am sure looping in batches will reduce memory consumption of the script.
</div>

<div>
Please also post the contents of cchrc_convert.cfg since it is used in the script.
</div>

<div>
May I also suggest you never use /* */ to comment inside functions. It is very hard to comment an entire function and write a new version of it. If you only use // to comment inside functions, you will be able to very rapidly comment an entire function with /* function... */ 
 
Its not an error, but just good practice ;)
</div>

<div>
Sorry if I missed that window of opportunity! &nbsp;But I'll post what I have (forgot to do it this time since I had posted both files to earlier questions) in hopes we can still work this out. &nbsp;The default filter in the config file below omits the one-minute intervals in favor of the one-hour intervals (filters are alternately applied). 

<pre><code class="code-10 nolinks" id="code-122292"># This is the config file (convert.cfg)
# Header options #
 
# The header name references the first row of the full record. Records begin with
# a header name and end after the header count and content row.
HeaderName =Point Name:
 
# Number of rows associated with the header data. Content data follows immediatly.
HeaderCount =4
 
# Header filters #
 
# Header filters ignore records based on text. You may describe as many header
# filters as you wish here.
Filter =Trend Every:,1 Minutes
#Filter =Trend Every:,60 Minutes
 
# Column options #
 
# The key represented by column numbers in the content row. Seperating multiple
# columns by commas combines the text into one single key.
ColumnKey =0,1
 
# The value of the key represented by column number in the content row.
ColumnValue =2
 
# Number of columns for each content row.
ColumnCount =4 
</code></pre>
<a href="https://filedb.experts-exchange.com/incoming/2008/07_w30/42885/hr-data.txt" target="_blank" class="file-inline" title="data file where all headers are consolidated to row 1 (4 KB)"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M67.508 468.467c-58.005-58.013-58.016-151.92 0-209.943l225.011-225.04c44.643-44.645 117.279-44.645 161.92 0 44.743 44.749 44.753 117.186 0 161.944l-189.465 189.49c-31.41 31.413-82.518 31.412-113.926.001-31.479-31.482-31.49-82.453 0-113.944L311.51 110.491c4.687-4.687 12.286-4.687 16.972 0l16.967 16.971c4.685 4.686 4.685 12.283 0 16.969L184.983 304.917c-12.724 12.724-12.73 33.328 0 46.058 12.696 12.697 33.356 12.699 46.054-.001l189.465-189.489c25.987-25.989 25.994-68.06.001-94.056-25.931-25.934-68.119-25.932-94.049 0l-225.01 225.039c-39.249 39.252-39.258 102.795-.001 142.057 39.285 39.29 102.885 39.287 142.162-.028A739446.174 739446.174 0 0 1 439.497 238.49c4.686-4.687 12.282-4.684 16.969.004l16.967 16.971c4.685 4.686 4.689 12.279.004 16.965a755654.128 755654.128 0 0 0-195.881 195.996c-58.034 58.092-152.004 58.093-210.048.041z"/></svg>hr-data.txt</a>
</div>

data file where all headers are consolidated to row 1

hr-data.txt

<div>
Well, this looks more complicated then I expected, but no worry. I noticed several things that need improvement, and the original idea of batch processing will still help decreasing memory usage. 
 
The only problem is that I only have a PHP4 hosting available. Tuesday I am at work, and can move my test environment to a PHP5 server. I just started using this hosting to upload experts-exchange code, but did not realize it was still PHP4. Your script does not work in PHP4, that's for sure. 
 
I hope you can wait untill Tuesday, when I can test the changes I am already making to the script. I don't want to post the script before I tested it myself as it might confuse you even more if I left in some errors.
</div>

<div>
ok thanks :) &nbsp;The script picks up 40 header elements from the data files, so I could post the whole csv file if that would help.
</div>

<div>
Let me know if this is still do-able. &nbsp;If not, my fall-back position is to execute the script in turn from 24 different directories. &nbsp;However right now even that doesn't work... &gt;&nbsp;php myscript.php works from the command line in the directory where the archive file is located, but only there. &nbsp;If I run it from a central location and try &gt;&nbsp;php /path-to-myscript.php, it fails: no data.csv is created. &nbsp;So if this doesn't work, I'll need to pursue a new line of questioning.
</div>

<div>
I have been very sick, almost the entire week. But I am better now and catching up (sorry for the long waiting). I did already move my test-environment to a PHP5 server and will check out the script in a minute. I'll post later on with the results. 
 
The fact that no data.csv is created when using sub-directories could be a small issue in the script. I will look into this as well.
</div>

<div>
The alternative approach should be easy. If you want to make the directory where the original csv files are variable, you should addapt the script a bit. On line 7 you have this: 
 
$myDir='/home/my_user/my_data'; 
 
If you only plan to run the script ONLY from command line, you could pass an argument to the command like this: 
 
# php myscript.php &quot;/home/my_user/my_data&quot; 
 
and change your script (line 7) to this: 
 
$myDir=preg_replace('#/$#','',$argv[1]).&quot;/&quot;; 
 
This will load the path dynamically into the PHP script to use from there. It will scan the directory you pass to the command for csv files. This means your script can run from almost anywhere you would like, and it would search for csv files in the directory you passed as an argument to the command. 
 
It will however create the resulting csv in the directory where the script itself is located. 
 
To dump the resulting csv in the same folder as the original csv, change this on line 10: 
 
csv_output($cchrc_header, $cchrc_data, $myDir.'pv_' . $info['file']); 
 
This will make sure that the dumped csv file will be in the same directory as the source csv.
</div>

<div>
I noticed that there is another tweak you should do on line 59: 
 
$csv_file[]=array('file'=&gt;$file,'data'=&gt;file($myDir.$file));
</div>

<div>
This is brilliant, thanks!! &nbsp;I got diverted on another project yesterday and by the time I got back to this was so tired that I couldn't manage to get anything to work. &nbsp;Today I &nbsp;ran this on my home server much faster than it had taken me to process the data files before - but to process the archives, I need to return to the office, so I'm on my way now for a final test :)
</div>

<div>
The script works on the first five and on the last archive file, however all the files in between are too large, so the shell script that lists all the places for the php script to execute exits with Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 35 bytes) in /home/my_user/scripts/cchrc_26.php on line 64 
The largest archive I was able to process is 28410565 bytes and all of the archives that failed to convert range between 29321826 and 35484059 bytes each. &nbsp;As I mentioned, my system has 1 GB of RAM installed; php.ini has a memory limit of 256 MB. I think I can convince my boss to double the memory on this machine, but I would need some assurance that the script will work if we do. &nbsp;Let me know thanks. 
 
$ cat /proc/meminfo 
MemTotal: &nbsp; &nbsp; &nbsp;1034652 kB 
MemFree: &nbsp; &nbsp; &nbsp; &nbsp;389848 kB 

</div>

<div>
I think you don't have much choice. The limit in php.ini is causing the fatal error. You could increase this value to 312MB but don't go any further if you only have like 380MB available on your system. If you keep increasing beyond this point, your sytem will crash. 
 
The script is designed to only load the source file once, which will result in memory usage as big as the source file. It has to load the content of the file in a variable, which is stored in your memory. It also needs a bit of memory for other calculations, but that is minimal. 
 
This also means that if you let the script run on a directory containing several files, they will all be loaded into the memory before any processing will occur. 
 
You might be able to increase performance a little bit by cecking each php command used on php.net to see if there is a faster and better alternative. The original script contained a function ereg, which can be replaced by preg functions. These are perl functions that are faster and might consume less memory. But again, you cannot improve this much more so it consumes less memory. 
 
It also seems that your system consumes a whole lot of memory for a linux/unix system. I suppose this is a full-blown desktop environment. If so, try to disable most software at startup to create extra free memory and increase your limit in php.ini 
 
Good luck! 
 

</div>

<div>
We're working out the memory issues but I'm confident that with your latest advice and doubling the RAM, we'll be able to make this work. &nbsp;Working from 24 folders separates the files to minimize one of the memory problems, and the script already processes files over 28 MB in size. &nbsp;Thanks for all this!!
</div>

<div>
<div class="content wysiwyg-content">
I'm working with a PHP script that processes csv files files into a more efficient view - instead of displaying three columns of data in over 3000 lines, it consolidates the headers into one row, thereby reducing the number of lines of data by two thirds. &nbsp;The script was originally designed to process only one file at a time, but thanks to the (EE) insertion of a foreach statement (line 9), it can now process all the data files in a directory. 
 
Another (EE) change from the original script is that the timestamp column is now separated from the date column, which we wanted to try although we understand that there are inherent risks to this approach. 
 
So far I see no loss of data, so I'd like to automate the processing of the archive files, which I can currently process only one at a time (the archives are in the same format as the data files, but about 23 times larger). &nbsp;The csv_output function in the attached script creates a single data.csv which consumes far more memory than I have on my PC (1 GB of RAM, Linux OS, executing from the command line). &nbsp;This gets in the way when working with more than one file at a time, so the csv_output function needs a rewrite - which will also affect the date formatting function, but one thing at at time. &nbsp;Let me know what you can do, thanks. 
<a href="https://filedb.experts-exchange.com/incoming/2008/07_w29/42372/cchrc.txt" target="_blank" class="file-inline" title="PHP script to consolidate data headers and reduce the number of lines in a file (csv) (7 KB)"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M67.508 468.467c-58.005-58.013-58.016-151.92 0-209.943l225.011-225.04c44.643-44.645 117.279-44.645 161.92 0 44.743 44.749 44.753 117.186 0 161.944l-189.465 189.49c-31.41 31.413-82.518 31.412-113.926.001-31.479-31.482-31.49-82.453 0-113.944L311.51 110.491c4.687-4.687 12.286-4.687 16.972 0l16.967 16.971c4.685 4.686 4.685 12.283 0 16.969L184.983 304.917c-12.724 12.724-12.73 33.328 0 46.058 12.696 12.697 33.356 12.699 46.054-.001l189.465-189.489c25.987-25.989 25.994-68.06.001-94.056-25.931-25.934-68.119-25.932-94.049 0l-225.01 225.039c-39.249 39.252-39.258 102.795-.001 142.057 39.285 39.29 102.885 39.287 142.162-.028A739446.174 739446.174 0 0 1 439.497 238.49c4.686-4.687 12.282-4.684 16.969.004l16.967 16.971c4.685 4.686 4.689 12.279.004 16.965a755654.128 755654.128 0 0 0-195.881 195.996c-58.034 58.092-152.004 58.093-210.048.041z"/></svg>cchrc.txt</a>
</div>
</div>

PHP script to consolidate data headers and reduce the number of lines in a file (csv)

cchrc.txt

I'm working with a PHP script that processes csv files files into a more efficient view - instead of displaying three columns of data in over 3000 lines, it consolidates the headers into one row, thereby reducing the number of lines of data by two thirds.  The script was originally designed to process only one file at a time, but thanks to the (EE) insertion of a foreach statement (line 9), it can now process all the data files in a directory. 

Another (EE) change from the original script is that the timestamp column is now separated from the date column, which we wanted to try although we understand that there are inherent risks to this approach. 

So far I see no loss of data, so I'd like to automate the processing of the archive files, which I can currently process only one at a time (the archives are in the same format as the data files, but about 23 times larger).  The csv_output function in the attached script creates a single data.csv which consumes far more memory than I have on my PC (1 GB of RAM, Linux OS, executing from the command line).  This gets in the way when working with more than one file at a time, so the csv_output function needs a rewrite - which will also affect the date formatting function, but one thing at at time.  Let me know what you can do, thanks.

Modify PHP function to remove consolidated csv file output

PHP is a widely-used server-side scripting language especially suited for web development, powering tens of millions of sites from Facebook to personal WordPress blogs. PHP is often paired with the MySQL relational database, but includes support for most other mainstream databases. By utilizing different Server APIs, PHP can work on many different web servers as a server-side scripting language.