Copy files from nested structure and rename by appending former directory name onto filename

I have thousands of files that are present in a series of directories like this:
SourceFolder
   FolderA
       1001
          FileUniqueNameA.txt
       1003
          FileUniqueNameB.txt
   FolderB
       2001
          FileUniqueNameC.txt
          FiletobeIgnored.pdf
       2004
          FileUniqueNameD.txt
          FiletobeIgnored.pdf

I'm looking to copy all of the txt files to a common directory like this:
CommonTargetFolder
    1001_FileUniqueNameA.txt
    1003_FileUniqueNameB.txt
    2001_FileUniqueNameC.txt
    2004_FileUniqueNameD.txt

Basic Points:
1) The directory name containing the files in question is unique, and we want to append that to the beginning of the filename as part of the copy...separated by an underscore.
2) We  can select by extension - we want only the txt files and nothing else.
3) The original directory and files needs to stay intact.
4) There are a lot of large files that will be ignored.  
5) There are no repetitions of the terminal directory names anywhere in the SourceFolder - they are truly unique.  They start at 01 and go to roughly 1000000.
6) There are no repetitions of the file names (txt) anywhere in the SourceFolder - they are also truly unique.
7) However, the ignored files do have a generic name.

For now it would be acceptable to just do this as a single batch job, however...
Down the road I will want to be able to keep things up to date.  Perhaps set a script to run once a day from 'Scheduled Tasks' or the like, and move newly written files from the source into the target, such that the target stays up to date as the source receives new files.  
Expecting ~20,000 text files.  Guaranteed to not bust 65,000 for a couple years.
Windows server 2003, 2008, or 2012.
Strong preference for a solution as a batch or an AutoIT script.

Thank you.
nosliwde99Asked:
Who is Participating?
 
Bill PrewConnect With a Mentor Commented:
Here's a small BAT that should do the job based on the structure you defined.  Save as a BAT, adjust the folder names, and then run as a test.

@echo off
setlocal

set BaseDir=c:\SourceFolder
set DestDir=c:\CommonTargetFolder

for /D %%A in ("%BaseDir%\*.*") do (
  for /D %%B in ("%%~A\*.*") do (
    for %%C in ("%%~B\*.txt") do (
      copy "%%~C" "%DestDir%\%%~nB_%%~nxC"
    )
  )
)

Open in new window

~bp
0
 
SurranoSystem EngineerCommented:
This is not a solution, just my opinion: It may be beneficial to install cygwin and use bash scripts for such repeated jobs of this complexity.
Alternatively, share the folder over samba and mount it in a Linux box to use bash script there.
If you can accept either of these, I can help you write a bash script.
0
 
Dan CraciunIT ConsultantCommented:
Here is a powershell solution:
Param (
[string]$inputPath = "\path\to\sourcefolder",
[string]$outputPath = "\path\to\targetfolder"
)

$fileList = Get-ChildItem -Path $inputPath -Recurse:1 -File -Include "*.txt"
foreach ($i in $fileList) {
    $newName = $i.FullName.split("\")
    $newName = $newName[($newName.count - 2)..($newName.count - 1)]  ## get only the name and last dir
    $newName = $newName -join("_")
    $newNameWithPath = $outputPath + "\" + $newName
    $test = Test-Path $newNameWithPath ## check if the file already exists
    if ( $test -like "False") {    
        Copy-Item $i -Destination $newNameWithPath
        }
}

Open in new window


The script will
- extract all txt files from your source folder,
- merge the file name with the name of the last containing folder, using "_" as the separator
- test if a file with that name already exists in your target folder
- copy the file if not already there.

HTH,
Dan
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
Steve KnightIT ConsultancyCommented:
Only thing I would add is maybe the use of  xcopy /d instead of copy then a second run would only copy the new files as requested, sound sensible Bill?


Steve
0
 
Bill PrewCommented:
Steve,

Yes, I sort of forgot about the rerun in the future part that was mentioned after the requirements. It wasn't clear what this meant though, so I'd want more info on that to design the best solution. More details would be needed, including things like:

- If a file was copied previously should it just be skipped?
- Or if a file was already copied do we need to copy if "newer"?
- Do any files in the dest directory that no longer exist in the source directory need to be removed?
- Etc...

~bp
0
 
nosliwde99Author Commented:
Sorry for the slow reply (hit with the flu).
Bill,  This is the kind of thing that I'm looking for.  Thank you.
Steve, Thank you for bringing up the part about re-running in the future.

To answer some of the questions:
-If a file was copied previously, it should in fact be skipped.
-Files will never be removed from either directory.
-If a previously existing file was re-added to the source folder, it would be appropriate to re-copy it to the destination, overwriting the prior copy by default.  (Fortunately in this particular case, in the source folder, the 'date modified' happens to be the date and time written to the source folder, so the newer copy would have a newer date.)

Would it be possible to get the most recent date modified from the destination directory, and plug that into the /d in xcopy?
Ed
0
 
Steve KnightConnect With a Mentor IT ConsultancyCommented:
Ed - xcopy with /d and no date will only copy if it is newer and not do anything otherwise, so something like this would do -- xcopy prompts whether a destination is a file or directory if it doesn't exist so could either do this in place of

copy "%%~C" "%DestDir%\%%~nB_%%~nxC"

Open in new window


echo f | xcopy /d "%%~C" "%DestDir%\%%~nB_%%~nxC" >NUL 2>&1

Open in new window


or

if exist "%DestDir%\%%~nB_%%~nxC" (
  xcopy /d "%%~C" "%DestDir%\%%~nB_%%~nxC"
) ELSE (
  copy "%%~C" "%DestDir%\%%~nB_%%~nxC"
)

Open in new window


i.e. if it doesn't exist then copy it, otherwise try xcopy /d to see if it has been updated

Steve
0
 
Bill PrewCommented:
With Steve's additional comments, are you all set now?

~bp
0
 
nosliwde99Author Commented:
Thank you both.   Have it set up and running nicely.  Appreciate your help.

FYI, as it is set up, it will copy roughly 1000 files per minute (from and to the same array), and will evaluate at roughly 1500 files per minute if no copy is involved.

Steve,
   I used the 'if exists' approach because the ' echo f | xcopy...' approach would occasionally and intermittently yield a "The process tried to write to a nonexistent pipe."  Most of the time it works fine.  When it fails it might return one or two errors out of a 500 file test batch.
0
 
nosliwde99Author Commented:
Also, for anyone else coming across this thread later, there is a third party solution (freeware) that could be viewed at http://www.bulkrenameutility.co.uk/Screenshots.php 

They also have a command line version that supposedly does most of the functionality of the GUI version.

I've tried neither.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.