Solved

Copy files from nested structure and rename by appending former directory name onto filename

Posted on 2014-01-05
10
536 Views
Last Modified: 2014-01-12
I have thousands of files that are present in a series of directories like this:
SourceFolder
   FolderA
       1001
          FileUniqueNameA.txt
       1003
          FileUniqueNameB.txt
   FolderB
       2001
          FileUniqueNameC.txt
          FiletobeIgnored.pdf
       2004
          FileUniqueNameD.txt
          FiletobeIgnored.pdf

I'm looking to copy all of the txt files to a common directory like this:
CommonTargetFolder
    1001_FileUniqueNameA.txt
    1003_FileUniqueNameB.txt
    2001_FileUniqueNameC.txt
    2004_FileUniqueNameD.txt

Basic Points:
1) The directory name containing the files in question is unique, and we want to append that to the beginning of the filename as part of the copy...separated by an underscore.
2) We  can select by extension - we want only the txt files and nothing else.
3) The original directory and files needs to stay intact.
4) There are a lot of large files that will be ignored.  
5) There are no repetitions of the terminal directory names anywhere in the SourceFolder - they are truly unique.  They start at 01 and go to roughly 1000000.
6) There are no repetitions of the file names (txt) anywhere in the SourceFolder - they are also truly unique.
7) However, the ignored files do have a generic name.

For now it would be acceptable to just do this as a single batch job, however...
Down the road I will want to be able to keep things up to date.  Perhaps set a script to run once a day from 'Scheduled Tasks' or the like, and move newly written files from the source into the target, such that the target stays up to date as the source receives new files.  
Expecting ~20,000 text files.  Guaranteed to not bust 65,000 for a couple years.
Windows server 2003, 2008, or 2012.
Strong preference for a solution as a batch or an AutoIT script.

Thank you.
0
Comment
Question by:nosliwde99
  • 3
  • 3
  • 2
  • +2
10 Comments
 
LVL 8

Expert Comment

by:Surrano
ID: 39758633
This is not a solution, just my opinion: It may be beneficial to install cygwin and use bash scripts for such repeated jobs of this complexity.
Alternatively, share the folder over samba and mount it in a Linux box to use bash script there.
If you can accept either of these, I can help you write a bash script.
0
 
LVL 34

Expert Comment

by:Dan Craciun
ID: 39758634
Here is a powershell solution:
Param (
[string]$inputPath = "\path\to\sourcefolder",
[string]$outputPath = "\path\to\targetfolder"
)

$fileList = Get-ChildItem -Path $inputPath -Recurse:1 -File -Include "*.txt"
foreach ($i in $fileList) {
    $newName = $i.FullName.split("\")
    $newName = $newName[($newName.count - 2)..($newName.count - 1)]  ## get only the name and last dir
    $newName = $newName -join("_")
    $newNameWithPath = $outputPath + "\" + $newName
    $test = Test-Path $newNameWithPath ## check if the file already exists
    if ( $test -like "False") {    
        Copy-Item $i -Destination $newNameWithPath
        }
}

Open in new window


The script will
- extract all txt files from your source folder,
- merge the file name with the name of the last containing folder, using "_" as the separator
- test if a file with that name already exists in your target folder
- copy the file if not already there.

HTH,
Dan
0
 
LVL 51

Accepted Solution

by:
Bill Prew earned 400 total points
ID: 39759048
Here's a small BAT that should do the job based on the structure you defined.  Save as a BAT, adjust the folder names, and then run as a test.

@echo off
setlocal

set BaseDir=c:\SourceFolder
set DestDir=c:\CommonTargetFolder

for /D %%A in ("%BaseDir%\*.*") do (
  for /D %%B in ("%%~A\*.*") do (
    for %%C in ("%%~B\*.txt") do (
      copy "%%~C" "%DestDir%\%%~nB_%%~nxC"
    )
  )
)

Open in new window

~bp
0
 
LVL 43

Expert Comment

by:Steve Knight
ID: 39760565
Only thing I would add is maybe the use of  xcopy /d instead of copy then a second run would only copy the new files as requested, sound sensible Bill?


Steve
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 39761176
Steve,

Yes, I sort of forgot about the rerun in the future part that was mentioned after the requirements. It wasn't clear what this meant though, so I'd want more info on that to design the best solution. More details would be needed, including things like:

- If a file was copied previously should it just be skipped?
- Or if a file was already copied do we need to copy if "newer"?
- Do any files in the dest directory that no longer exist in the source directory need to be removed?
- Etc...

~bp
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 

Author Comment

by:nosliwde99
ID: 39766958
Sorry for the slow reply (hit with the flu).
Bill,  This is the kind of thing that I'm looking for.  Thank you.
Steve, Thank you for bringing up the part about re-running in the future.

To answer some of the questions:
-If a file was copied previously, it should in fact be skipped.
-Files will never be removed from either directory.
-If a previously existing file was re-added to the source folder, it would be appropriate to re-copy it to the destination, overwriting the prior copy by default.  (Fortunately in this particular case, in the source folder, the 'date modified' happens to be the date and time written to the source folder, so the newer copy would have a newer date.)

Would it be possible to get the most recent date modified from the destination directory, and plug that into the /d in xcopy?
Ed
0
 
LVL 43

Assisted Solution

by:Steve Knight
Steve Knight earned 100 total points
ID: 39767304
Ed - xcopy with /d and no date will only copy if it is newer and not do anything otherwise, so something like this would do -- xcopy prompts whether a destination is a file or directory if it doesn't exist so could either do this in place of

copy "%%~C" "%DestDir%\%%~nB_%%~nxC"

Open in new window


echo f | xcopy /d "%%~C" "%DestDir%\%%~nB_%%~nxC" >NUL 2>&1

Open in new window


or

if exist "%DestDir%\%%~nB_%%~nxC" (
  xcopy /d "%%~C" "%DestDir%\%%~nB_%%~nxC"
) ELSE (
  copy "%%~C" "%DestDir%\%%~nB_%%~nxC"
)

Open in new window


i.e. if it doesn't exist then copy it, otherwise try xcopy /d to see if it has been updated

Steve
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 39774121
With Steve's additional comments, are you all set now?

~bp
0
 

Author Closing Comment

by:nosliwde99
ID: 39775252
Thank you both.   Have it set up and running nicely.  Appreciate your help.

FYI, as it is set up, it will copy roughly 1000 files per minute (from and to the same array), and will evaluate at roughly 1500 files per minute if no copy is involved.

Steve,
   I used the 'if exists' approach because the ' echo f | xcopy...' approach would occasionally and intermittently yield a "The process tried to write to a nonexistent pipe."  Most of the time it works fine.  When it fails it might return one or two errors out of a 500 file test batch.
0
 

Author Comment

by:nosliwde99
ID: 39775258
Also, for anyone else coming across this thread later, there is a third party solution (freeware) that could be viewed at http://www.bulkrenameutility.co.uk/Screenshots.php

They also have a command line version that supposedly does most of the functionality of the GUI version.

I've tried neither.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

This script will sweep a range of IP addresses (class c only, 255.255.255.0) and report to a log the version of office installed. What it does: 1.)      Creates log file in the directory the script is run from (if it doesn't already exist) 2.)      Sweep…
This article is the result of a quest to better understand Task Scheduler 2.0 and all the newer objects available in vbscript in this version over  the limited options we had scripting in Task Scheduler 1.0.  As I started my journey of knowledge I f…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now