Windows Batch file to count a set "N" number of files from a single source folder,merged multiple files together into a single output file

Windows Batch file to count a set "N" number of files from a single source folder,merged multiple files together into a single output file that contains <= N number of records, and move processed files to a save directory folder.

Request for a window bat file that can be run from command/CMD prompt.  Windows 7 environment.

Windows Batch file to copy a set "N" number of files from a single source folder to an output file, move processed records to a single save folder.  The single source directory may contain 1 up to a million files.  
Need to copy 200 files at a time to a newfile_N+ that should contain <= 200 files per newfile_N+ until all records have been read and processed.

Input file extension: *.hl7
Output file newfile_N+ extension: *.txt

Source directory - C:\temp\source
Destination directory - C:\temp\output\newfile_N+.txt
Destination directory - C:\temp\processed

For the purpose of demonstration I have attached (5) sample hl7 files.
Each newfile_N+.txt file should contain <= 2 records per file.

Prior to kicking off the batch file.  All (5) source files would be all be in the (C:\temp\source) directory.
Upon completion of the batch program running.  The expected output once the bat file has run would result in the following:

      dir: contains no files *.hl7 files
 dir: newfile.txt, newfile1.txt, newfile2.txt
      Note: Each of these files contains <=2 records each, max is 2 records per file
      ==> newfile.txt contains 2 records
      ==> newfile1.txt contains 2 records
      ==> newfile2.txt contains 1 record

 dir: new 1.hl7, new 2.hl7, new 3.hl7, new 4.hl7, new 5.hl7
 Note: This processed directory should contain the records that were originally in the source directory

This is example is using a small # of input/output records.  In reality the source directory may hold 1 record or a million.
The ask is that the solution would accommodate and use variables where appropriate.
Note: The folder names may contain spaces, also the file names may contain spaces.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bill PrewIT / Software Engineering ConsultantCommented:
Just my opinion, but given the number of files you have, and the special characters that are contained in the data lines of the files, I wouldn't think a BAT script is going to work very well for this.  You would probably be better off looking at something like Powershell or VBscript.

Do the files from the source folder need to be processed in a certain order, perhaps in sorted name sequence?

Bill PrewIT / Software Engineering ConsultantCommented:
You might take a good hard look at this utility, it looks promising and can handle multiple input files, as well as splitting into new files based on number of lines.  I'm just not sure if it can add lines from 2 different input files to the same output file, you'll need to experiment a bit...

GSplit - File Splitter - Split Any File - Split Text and Log Files

Steve KnightIT ConsultancyCommented:
Bill, I think it just needs a bit of a tweak to the script I did for him here, just haven't got time to do anything more with it at the moment for this Q:

"Windows Batch file to move a set "N" number of files from a single source folder to a destination folder. Then merged multiple files together as one file"
Why Diversity in Tech Matters

Kesha Williams, certified professional and software developer, explores the imbalance of diversity in the world of technology -- especially when it comes to hiring women. She showcases ways she's making a difference through the Colors of STEM program.

Bill PrewIT / Software Engineering ConsultantCommented:
Okay, give this a try, seems to work in a test here...

SETLOCAL EnableDelayedExpansion

rem Set paths and files to work with, and number of files to merge into each output file
SET Source=C:\TEMP\source
SET Dest=C:\TEMP\output
SET Processed=C:\TEMP\processed
SET FileType="HL7"
SET Merge=2

rem Initialize input and output file counts
SET FileOut=0
SET FileIn=0

rem Sort input files by name, process each one
FOR /f "tokens=*" %%F IN ('dir /b /a-d /on "%Source%\*.%FileType%"') DO (

  rem Should we merge this input into current output file?
  SET /A FileIn+=1
  IF !FileIn! GTR %Merge% (
    rem Exceeded merge count, start a new output file number
    SET /A FileOut+=1
    SET FileIn=1

  rem Copy this input file to current output file
  TYPE "%Source%\%%~F">>"%Dest%\NewFile!FileOut!%%~xF"

  rem Move this input file to processed folder
  MOVE "%Source%\%%~F" "%Processed%" > NUL

Open in new window


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mreid3847Author Commented:
Thanks Bill and Steve for the review.  Will test the code shared out, thanks for the assistance.
mreid3847Author Commented:
Thanks Bill and Steve, appreciate the effort and providing a documented *.bat that has programmer notes. Appreciate the simplicity applied to it also. Spot on, completed the task requested.  Thank you both for your time, review and knowledge.
Bill PrewIT / Software Engineering ConsultantCommented:
Welcome, glad that was helpful.

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Batch

From novice to tech pro — start learning today.