Avatar of sara_bellum
Flag for United States of America

asked on 

Modify PHP function to remove consolidated csv file output

I'm working with a PHP script that processes csv files files into a more efficient view - instead of displaying three columns of data in over 3000 lines, it consolidates the headers into one row, thereby reducing the number of lines of data by two thirds.  The script was originally designed to process only one file at a time, but thanks to the (EE) insertion of a foreach statement (line 9), it can now process all the data files in a directory.

Another (EE) change from the original script is that the timestamp column is now separated from the date column, which we wanted to try although we understand that there are inherent risks to this approach.

So far I see no loss of data, so I'd like to automate the processing of the archive files, which I can currently process only one at a time (the archives are in the same format as the data files, but about 23 times larger).  The csv_output function in the attached script creates a single data.csv which consumes far more memory than I have on my PC (1 GB of RAM, Linux OS, executing from the command line).  This gets in the way when working with more than one file at a time, so the csv_output function needs a rewrite - which will also affect the date formatting function, but one thing at at time.  Let me know what you can do, thanks.

Avatar of undefined
Last Comment

8/22/2022 - Mon