Merge csv and tsv files with PHP

I'm trying to merge all the csv or tsv files in a drectory (only one type at a time) into a "merge".csv or .tsv, plus insert the source file name as the first field in each record.

By inserting that field it seems to mess up the formatting of the csv or tsv data.  See sample files and outputs attached.

**Note: in the attached files all of the tsv files have been renamed to .jpg since expert-exchange won't allow uploading tsv files!?!?

<?php

    set_include_path('c:');

  $path='C:/Documents and Settings/Steve/My Documents/a_Holijoli/a_Projects/TCH-Group/SampleFiles/';
  $pathdir = opendir($path); 

    $outFile = "$path merge.csv";
    $fout = fopen($outFile, 'w');

    

    while (false !== ($file = readdir($pathdir))) 
    {
        $fullname="${path}${file}";

        
       if(stristr($file, '.csv') == TRUE) 
       {
           echo "...> $fullname<br/>";  
            
           $recArr = file("$fullname");
            
            foreach($recArr as $rec)
            {
                $newrec="$file.\t.$rec";
                fwrite($fout,$newrec);
                echo $newrec;
              
            }
        }    
    }
 
?>

Open in new window

samples.zip
stevelucyAsked:
Who is Participating?
 
karoldvlConnect With a Mentor Commented:
I don't know if Notepad++ macro recording could handle something like that. Probably not.

I'm certain that iconv with some batch script would do, but it's probably too much work.

SourceForge advises something of this sort:
http://sourceforge.net/projects/uni-transmuter/

or if you want it PHP based:
http://sourceforge.net/projects/batchconvert/
0
 
karoldvlCommented:
I fixed some minor issues here. See if that's what you wanted.
<?php
  set_include_path('c:');

  $path='C:/Documents and Settings/Steve/My Documents/a_Holijoli/a_Projects/TCH-Group/SampleFiles/';
  $pathdir = opendir($path); 

  $outFile = "${path}merge.csv";
  $fout = fopen($outFile, 'w');

  while (false !== ($file = readdir($pathdir))) 
  {
      $fullname="${path}${file}";

       
      if(stristr($file, '.csv') == TRUE && stristr($file, 'merge.csv') == FALSE) 
      {
         echo "...> $fullname<br/>";  
          
         $recArr = file("$fullname");
          
          foreach($recArr as $rec)
          {
              $newrec="$file\t$rec";
              fwrite($fout,$newrec);
              echo $newrec;
            
          }
      }
      
  }
 
?>

Open in new window

0
 
stevelucyAuthor Commented:
I stil get the same problem that the existing formatting of csv data gets lost .  Attached is a screenshot of excel loading the result, plus the actual file.

csvtest.jpg
merge.csv
0
Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
karoldvlCommented:
Try recreating your data files in uniform character encoding. Try using UTF-8 for all the files.
0
 
stevelucyAuthor Commented:
These files are all downloads from the Google Keyword tool or Yahoo Site Explorer.

It's funny because Google provides csv files but in notepad it looks like they're tab separated - not a comma to be found anywhere.

How would I convert all of them to UTF-8?

thanks for your help
0
 
karoldvlCommented:
You can use Notepad++:
http://notepad-plus.sourceforge.net/

Just open an appropriate file and use "Encoding->Convert to UTF-8 without BOM" and save.
0
 
stevelucyAuthor Commented:
Hey that did it!  But one more problem.  I have about 300 of these files - how could I do a batch conversion - any idea?

If not, I'll just go ahead and accept as solution - thanks!

0
 
stevelucyAuthor Commented:
thanks!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.