Merge CSV files in folders and sub folders

Do they have a header line in each, do they need checking to see if the headers are the same in each then too?

Also do you need to record the filename or directory name add part of the lines in the destination?

Steve

EG this stripped of mine just needs a /s added to the dir commands

http://scripts.dragon-it.co.uk/links/batch-combine-csv-files

something like this

 for /R %f in (*.txt) do type "%f" >> c:\yourLogFolder\output.txt

will append all *.txt files to output.txt starting at the folder you run it from and working through all subfolders

They have headers but no checking required

Then the script in my link will do that for you and strip off the headers apart from the first one.

(for /f "tokens=*" %%x in ('dir /b /a-d /on *.csv') do (

@echo off
cd /d "C:\my csv directory"
set first=true
setlocal enabledelayedexpansion
(for /f "tokens=*" %%x in ('dir /b /s /a-d /on *.csv') do (
if !first!==true (
  type "%%x"
  echo.
  set first=false
) ELSE (
  more +1 "%%x"
)
))> c:\destination\newfile.csv

Select allOpen in new window

or

@echo off
cd /d "C:\my csv directory"
set first=true
setlocal enabledelayedexpansion
(for /R %%x in (*.csv) do (
if !first!==true (
  type "%%x"
  echo.
  set first=false
) ELSE (
  more +1 "%%x"
)
))> c:\destination\newfile.csv

Select allOpen in new window

Save that as "Join-my-cv.cmd" or similar

Steve

Steve.. Do this cover sub-folders?

As all the CSV files are stored in folders within a directory

It does.  It works on all files reported by:

cd /d "c:\yourstartdir"
dir /b /s /a-d /on *.csv

or

cd /d "c:\yourstartdir"
for /R %% in (*.csv) do echo %x

It uses  "Type" command on the first file found to include the whole file then uses the MORE +1 command to skip the first line for all the other files.

Steve

Obviously to save confusing things a lot make sure your destination files is not pointing back into the source directory!

Sorry.. I can't get this to work..

if "date time user action" is your header ....

setlocal enabledelayedexpansion

for /R %f in (*.txt) do type "%f" >> c:\yourLogFolder\output.txt

findstr /V "date time user action" output.txt > output_2.txt

You shld have something very close to what you need in output_2.txt

Sorry my mistake.. I was being too generic with the criteria..

Folder - 'Audits'

Within the folder called 'Audits' there are sub-folders named relevant dates..

Within in the sub-folders there are various CSV files with different filename but the CSV I required is called

'template_822049ceaeaf4d8aad7cc9096cc32f14.csv'

These are the files I am looking for and I want to merge into one large CSV file..

In the header there are no dates or times just standard text headings..

for /R %f in (template_822049ceaeaf4d8aad7cc9096cc32f14.csv) do type "%f" >> c:\yourLogFolder\output.txt

I'm still warming up to Powershell, but when working with CSV files (and dates) it excels.

Your merge could be accomplished easily with this one line PS script:

Get-ChildItem -Path B:\EE\EE29163667\Audits -Recurse -Filter template_822049ceaeaf4d8aad7cc9096cc32f14.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv B:\EE\EE29163667\all-ps.csv -NoTypeInformation -Append

Select allOpen in new window

Which can of course be executed from a command line (or BAT script) as:

powershell -NoProfile -Command "Get-ChildItem -Path B:\EE\EE29163667\Audits -Recurse -Filter template_822049ceaeaf4d8aad7cc9096cc32f14.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv B:\EE\EE29163667\all-ps.csv -NoTypeInformation -Append"

Select allOpen in new window

---

»bp