Solved

Copy files from nested structure and rename by appending former directory name onto filename

Posted on 2014-01-05
Medium Priority
545 Views
I have thousands of files that are present in a series of directories like this:
SourceFolder
FolderA
1001
FileUniqueNameA.txt
1003
FileUniqueNameB.txt
FolderB
2001
FileUniqueNameC.txt
FiletobeIgnored.pdf
2004
FileUniqueNameD.txt
FiletobeIgnored.pdf

I'm looking to copy all of the txt files to a common directory like this:
CommonTargetFolder
1001_FileUniqueNameA.txt
1003_FileUniqueNameB.txt
2001_FileUniqueNameC.txt
2004_FileUniqueNameD.txt

Basic Points:
1) The directory name containing the files in question is unique, and we want to append that to the beginning of the filename as part of the copy...separated by an underscore.
2) We  can select by extension - we want only the txt files and nothing else.
3) The original directory and files needs to stay intact.
4) There are a lot of large files that will be ignored.
5) There are no repetitions of the terminal directory names anywhere in the SourceFolder - they are truly unique.  They start at 01 and go to roughly 1000000.
6) There are no repetitions of the file names (txt) anywhere in the SourceFolder - they are also truly unique.
7) However, the ignored files do have a generic name.

For now it would be acceptable to just do this as a single batch job, however...
Down the road I will want to be able to keep things up to date.  Perhaps set a script to run once a day from 'Scheduled Tasks' or the like, and move newly written files from the source into the target, such that the target stays up to date as the source receives new files.
Expecting ~20,000 text files.  Guaranteed to not bust 65,000 for a couple years.
Windows server 2003, 2008, or 2012.
Strong preference for a solution as a batch or an AutoIT script.

Thank you.
0
Question by:nosliwde99
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

• Help others & share knowledge
• Earn cash & points
• Learn & ask questions
• 3
• 3
• 2
• +2

LVL 8

Expert Comment

ID: 39758633
This is not a solution, just my opinion: It may be beneficial to install cygwin and use bash scripts for such repeated jobs of this complexity.
Alternatively, share the folder over samba and mount it in a Linux box to use bash script there.
If you can accept either of these, I can help you write a bash script.
0

LVL 35

Expert Comment

ID: 39758634
Here is a powershell solution:
Param (
[string]$inputPath = "\path\to\sourcefolder", [string]$outputPath = "\path\to\targetfolder"
)

$fileList = Get-ChildItem -Path$inputPath -Recurse:1 -File -Include "*.txt"
foreach ($i in$fileList) {
$newName =$i.FullName.split("\")
$newName =$newName[($newName.count - 2)..($newName.count - 1)]  ## get only the name and last dir
$newName =$newName -join("_")
$newNameWithPath =$outputPath + "\" + $newName$test = Test-Path $newNameWithPath ## check if the file already exists if ($test -like "False") {
Copy-Item $i -Destination$newNameWithPath
}
}


The script will
- extract all txt files from your source folder,
- merge the file name with the name of the last containing folder, using "_" as the separator
- test if a file with that name already exists in your target folder
- copy the file if not already there.

HTH,
Dan
0

LVL 56

Accepted Solution

Bill Prew earned 1600 total points
ID: 39759048
Here's a small BAT that should do the job based on the structure you defined.  Save as a BAT, adjust the folder names, and then run as a test.

@echo off
setlocal

set BaseDir=c:\SourceFolder
set DestDir=c:\CommonTargetFolder

for /D %%A in ("%BaseDir%\*.*") do (
for /D %%B in ("%%~A\*.*") do (
for %%C in ("%%~B\*.txt") do (
copy "%%~C" "%DestDir%\%%~nB_%%~nxC"
)
)
)

~bp
0

LVL 43

Expert Comment

ID: 39760565
Only thing I would add is maybe the use of  xcopy /d instead of copy then a second run would only copy the new files as requested, sound sensible Bill?

Steve
0

LVL 56

Expert Comment

ID: 39761176
Steve,

Yes, I sort of forgot about the rerun in the future part that was mentioned after the requirements. It wasn't clear what this meant though, so I'd want more info on that to design the best solution. More details would be needed, including things like:

- If a file was copied previously should it just be skipped?
- Or if a file was already copied do we need to copy if "newer"?
- Do any files in the dest directory that no longer exist in the source directory need to be removed?
- Etc...

~bp
0

Author Comment

ID: 39766958
Sorry for the slow reply (hit with the flu).
Bill,  This is the kind of thing that I'm looking for.  Thank you.
Steve, Thank you for bringing up the part about re-running in the future.

To answer some of the questions:
-If a file was copied previously, it should in fact be skipped.
-Files will never be removed from either directory.
-If a previously existing file was re-added to the source folder, it would be appropriate to re-copy it to the destination, overwriting the prior copy by default.  (Fortunately in this particular case, in the source folder, the 'date modified' happens to be the date and time written to the source folder, so the newer copy would have a newer date.)

Would it be possible to get the most recent date modified from the destination directory, and plug that into the /d in xcopy?
Ed
0

LVL 43

Assisted Solution

Steve Knight earned 400 total points
ID: 39767304
Ed - xcopy with /d and no date will only copy if it is newer and not do anything otherwise, so something like this would do -- xcopy prompts whether a destination is a file or directory if it doesn't exist so could either do this in place of

copy "%%~C" "%DestDir%\%%~nB_%%~nxC"


echo f | xcopy /d "%%~C" "%DestDir%\%%~nB_%%~nxC" >NUL 2>&1


or

if exist "%DestDir%\%%~nB_%%~nxC" (
xcopy /d "%%~C" "%DestDir%\%%~nB_%%~nxC"
) ELSE (
copy "%%~C" "%DestDir%\%%~nB_%%~nxC"
)


i.e. if it doesn't exist then copy it, otherwise try xcopy /d to see if it has been updated

Steve
0

LVL 56

Expert Comment

ID: 39774121
With Steve's additional comments, are you all set now?

~bp
0

Author Closing Comment

ID: 39775252
Thank you both.   Have it set up and running nicely.  Appreciate your help.

FYI, as it is set up, it will copy roughly 1000 files per minute (from and to the same array), and will evaluate at roughly 1500 files per minute if no copy is involved.

Steve,
I used the 'if exists' approach because the ' echo f | xcopy...' approach would occasionally and intermittently yield a "The process tried to write to a nonexistent pipe."  Most of the time it works fine.  When it fails it might return one or two errors out of a 500 file test batch.
0

Author Comment

ID: 39775258
Also, for anyone else coming across this thread later, there is a third party solution (freeware) that could be viewed at http://www.bulkrenameutility.co.uk/Screenshots.php

They also have a command line version that supposedly does most of the functionality of the GUI version.

I've tried neither.
0

Featured Post

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If like me you are one who spends a lot of time working and scripting with cmd.exe, sometimes it is handy to be able to quickly view a calendar for a given month and year. This script will quickly do just that!  Save the code posted below to a .bat …
This article was inspired by a question here at Experts Exchange (http://www.experts-exchange.com/Software/Photos_Graphics/Images_and_Photos/Q_28629170.html). The requirements stated in that question are (1) reduce the file size of a large number of…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…
Michael from AdRem Software outlines event notifications and Automatic Corrective Actions in network monitoring. Automatic Corrective Actions are scripts, which can automatically run upon discovery of a certain undesirable condition in your network.…
Suggested Courses
Course of the Month12 days, 2 hours left to enroll

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.