Windows Batch file to count a set "N" number of files from a single source folder,merged multiple files together into a single output file

mreid3847
mreid3847 used Ask the Experts™
on
Windows Batch file to count a set "N" number of files from a single source folder,merged multiple files together into a single output file that contains <= N number of records, and move processed files to a save directory folder.

Request for a window bat file that can be run from command/CMD prompt.  Windows 7 environment.

Windows Batch file to copy a set "N" number of files from a single source folder to an output file, move processed records to a single save folder.  The single source directory may contain 1 up to a million files.  
Need to copy 200 files at a time to a newfile_N+ that should contain <= 200 files per newfile_N+ until all records have been read and processed.

Input file extension: *.hl7
Output file newfile_N+ extension: *.txt

Source directory - C:\temp\source
Destination directory - C:\temp\output\newfile_N+.txt
Destination directory - C:\temp\processed

For the purpose of demonstration I have attached (5) sample hl7 files.
Each newfile_N+.txt file should contain <= 2 records per file.

Prior to kicking off the batch file.  All (5) source files would be all be in the (C:\temp\source) directory.
Upon completion of the batch program running.  The expected output once the bat file has run would result in the following:

C:\temp\source
      dir: contains no files *.hl7 files
      
C:\temp\output
 dir: newfile.txt, newfile1.txt, newfile2.txt
      Note: Each of these files contains <=2 records each, max is 2 records per file
      ==> newfile.txt contains 2 records
      ==> newfile1.txt contains 2 records
      ==> newfile2.txt contains 1 record

C:\temp\processed
 dir: new 1.hl7, new 2.hl7, new 3.hl7, new 4.hl7, new 5.hl7
 Note: This processed directory should contain the records that were originally in the source directory

This is example is using a small # of input/output records.  In reality the source directory may hold 1 record or a million.
The ask is that the solution would accommodate and use variables where appropriate.
 
Note: The folder names may contain spaces, also the file names may contain spaces.
sample_source_input.zip
sample_destination_output.zip
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Just my opinion, but given the number of files you have, and the special characters that are contained in the data lines of the files, I wouldn't think a BAT script is going to work very well for this.  You would probably be better off looking at something like Powershell or VBscript.

Do the files from the source folder need to be processed in a certain order, perhaps in sorted name sequence?


»bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
You might take a good hard look at this utility, it looks promising and can handle multiple input files, as well as splitting into new files based on number of lines.  I'm just not sure if it can add lines from 2 different input files to the same output file, you'll need to experiment a bit...

GSplit - File Splitter - Split Any File - Split Text and Log Files



»bp

Commented:
Bill, I think it just needs a bit of a tweak to the script I did for him here, just haven't got time to do anything more with it at the moment for this Q:

"Windows Batch file to move a set "N" number of files from a single source folder to a destination folder. Then merged multiple files together as one file"
CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

Test your restores, not your backups...
Top Expert 2016
Commented:
Okay, give this a try, seems to work in a test here...

@ECHO OFF
SETLOCAL EnableDelayedExpansion

rem Set paths and files to work with, and number of files to merge into each output file
SET Source=C:\TEMP\source
SET Dest=C:\TEMP\output
SET Processed=C:\TEMP\processed
SET FileType="HL7"
SET Merge=2

rem Initialize input and output file counts
SET FileOut=0
SET FileIn=0

rem Sort input files by name, process each one
FOR /f "tokens=*" %%F IN ('dir /b /a-d /on "%Source%\*.%FileType%"') DO (

  rem Should we merge this input into current output file?
  SET /A FileIn+=1
  IF !FileIn! GTR %Merge% (
    rem Exceeded merge count, start a new output file number
    SET /A FileOut+=1
    SET FileIn=1
  )

  rem Copy this input file to current output file
  TYPE "%Source%\%%~F">>"%Dest%\NewFile!FileOut!%%~xF"

  rem Move this input file to processed folder
  MOVE "%Source%\%%~F" "%Processed%" > NUL
)

Open in new window


»bp

Author

Commented:
Thanks Bill and Steve for the review.  Will test the code shared out, thanks for the assistance.

Author

Commented:
Thanks Bill and Steve, appreciate the effort and providing a documented *.bat that has programmer notes. Appreciate the simplicity applied to it also. Spot on, completed the task requested.  Thank you both for your time, review and knowledge.
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Welcome, glad that was helpful.


»bp

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial