?
Solved

batch file to check for new files

Posted on 2009-02-13
26
Medium Priority
?
669 Views
Last Modified: 2013-11-25
I would like to write a batch file that
1) compares a list of file in a source directory to see if they are in the 'processed' directory
2) If not, the files are moved to the processing folder
3) each file is then renamed to Processing.txt one by one.  After naming it processing.txt, the file is returned to its original file name and moved to the processed folder.
4) Step 3 repeats for each file in the processing folder
0
Comment
Question by:lovingatx
  • 7
  • 3
  • 3
  • +5
24 Comments
 
LVL 28

Expert Comment

by:lesouef
ID: 23641108
using a cmd shell script I guess?
try that, done from memory, untested
---------
@echo off
for /f %%i in ('dir sourcefolder /b/s') do (set file==%%i & call :sub)
goto :eof

:sub
if not exist destination\%%file, move source\%%file destination\processing.txt
rem I guess you need to insert some action here...
move destination\processing.txt processed\%%file
goto :eof
0
 
LVL 17

Expert Comment

by:ramrom
ID: 23641128

Let me take a stab at understanding your specification. We start with:
 
source     processing    processed
foo.txt                  foo.txt
bar.txt
 
bar.txt is not in processed so we move it to processing giving:
 
source     processing    processed
foo.txt    bar.txt       foo.txt
 
rename bar.txt to processing.txt then back to bar.txt then move it to processed giving:
 
source     processing    processed
foo.txt                  foo.txt
                         bar.txt
 
That result can be achieved by just moving bar.txt from source to processes. 
What is the purpose of the intermediate steps?

Open in new window

0
 
LVL 17

Expert Comment

by:ramrom
ID: 23641142
Once we are clear about your objective - would you consider a Python script? That is what I can offer.
0
Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

 
LVL 28

Expert Comment

by:lesouef
ID: 23641196
I guess it is supposed to act as a hotfolder, that's why I said he should insert some action. or wait untill some other process completes
0
 
LVL 57

Expert Comment

by:giltjr
ID: 23641460
I would suggest something with a bit fewer moves and a bit less overhead.

You have two folders only: "new" and "processed".

Script looks for files in "new", if file(s) exist rename the file one, by one as you process them to "foo.txt.processing".  If processing is succesfull move the file to the "processed" folder with the ".processing" tag.  If the processing failed, rename the file to "foo.txt.failed" and move on to the next file.

This way:

If file is in processed you have processed it.
If file is in new, but is not tagged as processing you have done nothing with it.
If file is in new and tagged as processing you know you started processing it.
If file is inn new and tagged as failed, it failed processing.

You may actually want to tag the files with the data and time on the end.  That way if you get duplicate file names, they won't overwrite each other.


0
 
LVL 17

Expert Comment

by:ramrom
ID: 23642034
What OS? What process?
0
 
LVL 35

Expert Comment

by:ShineOn
ID: 23643283
lovingatx,

We can't help you unless you give us more information.

What has been requested so far include:

The OS you're planning to run this process on
The batch language you want to use
The goal for this batch process - what's it supposed to be for, beyond the process description you gave?

In addition:

Is there a reason for the "processing" folder with the "processing.txt" intermediate file name?  Is it supposed to be, er, processed by some other program before it gets renamed back to its original name and moved to the "processed" folder?

If not, then why the exercise?  

If so, then how is the batch routine supposed to know it's done "processing" so it can be renamed back to the original?

0
 
LVL 21

Expert Comment

by:AmazingTech
ID: 23645148
This uses XCOPY /d to find out which files are newer or does not exist in the Processing folder. With this list /l of files we move theses ones and rename them.
Set Source=C:\Source
Set Target=C:\Processing
 
for /f "Tokens=*" %%a in ('XCOPY /d /l /y "%Source%" "%Target%"') do (
    Move "%%a" "%Target%"
    Ren "%Target%\%%~nxa" "Processing.txt"
 
    ECHO Do your stuff here with the processing.txt"
    Pause
    Ren "%Target%\Processing.txt" "%%~nxa"
)

Open in new window

0
 
LVL 14

Expert Comment

by:ptakja
ID: 23645235
looks like homework to me....
0
 
LVL 57

Expert Comment

by:giltjr
ID: 23645459
The biggest problem with renaming every file to processing.txt while it is being processed is that you must keep track of what the real name of the file was.  By suffixing the name with ".processing" you don't need to remember the original name.

The other point I made about renaming the files with a date time stamp is if you don't do this, then you must assume that every single file will always have a unique name.  What happens if you get two files with the same name, but with different contents?
0
 
LVL 35

Expert Comment

by:ShineOn
ID: 23646968
ptakia, I don't think this is a homework question... based on Asker's history, they are in a production environment.  Their other Questions don't appear homework-like at all.

However, it sure would be nice to hear back from the Asker as to the purpose of this exercise rather than throwing random "solutions" at them.

It may be better to use something more robust than DOS-type command scripting.  If they have programming experience, VBS may be a better option. I have worked with this sort of process written into Microsoft Access VBA, using a tracking database to reduce file system I/O, which might be another option.  If they need it event-driven, they might benefit from a third-party tool like Schedule Wizard Automation Edition.  If they need to script a "macro" to process the file through another Windows-based program, they might benefit by using AutoIT or WinBatch.  

Won't know until they respond, which is why I'm not posting any code recommendations.  

0
 
LVL 16

Expert Comment

by:t0t0
ID: 23647334
lovingatx

(Solution at the bottom however,....) Please confirm the following logic in your question.

"1) compares a list of file in a source directory to see if they are in the 'processed' directory"
"2) If not, the files are moved to the processing folder"

The following code tests whether each file in source exists in processed then, each file that does not esixt in processed is moved to processing,
 
FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\"
   )
)


"3) each file is then renamed to Processing.txt one by one. After naming it processing.txt, the file is returned to its original file name and moved to the processed folder."

4) Step 3 repeats for each file in the processing folder

The following code renames each file in processing to Processing.txt. DOS only allows one file in the current folder to be named Processing.txt at any one time, so each file must be renamed back to it's original name before the next file is renamed Processing.txt. I assume you intend to 'process' each file named Processing.txt otherwise this would be a pointless procedure therefore, I have added a REM statement indicating where you need to add some 'process'.

FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "processing\"') DO (
   RENAME "processing\%%a" "processing\Processing.txt"
   :
   REM process Processing.txt
   :
   RENAME "processing\Processing.txt" "processing\%%a"
   MOVE "processing\%%a" "processed\"
)


Both fragments of code can be combined so that only one file is moved and processed at a time as in:

FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\"
      RENAME "processing\%%a" "processing\Processing.txt"
      :
      REM process Processing.txt
      :
      RENAME "processed\processing.txt" "processed\%%a"
      MOVE "processing\%%a" "processed\"
   )
)


Finally, the MOVE and RENAME process can be performed simultaneously as in:

FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\Processing.txt"
      :
      REM process Processing.txt
      :
      MOVE "processing\Processing.txt" "processed\%%a"
   )
)


Copy and paste code below into notepad and save it as you batch file.

@echo off
 
FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\Processing.txt"
      :
      REM process Processing.txt
      :
      MOVE "processing\Processing.txt" "processed\%%a"
   )
)

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 23648018
As for the rest of you, lovingatx makes his requirement clear and precise. See the following code:


FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\Processing.txt"
      ::
      ::
      ::
      MOVE "processing\Processing.txt" "processed\%%a"
   )
)
0
 
LVL 21

Assisted Solution

by:AmazingTech
AmazingTech earned 664 total points
ID: 23648045
Opps I totally missed the processed folder part.
Set Source=C:\Source
Set Processing=C:\Processing
Set Target=C:\Processed
 
for /f "Tokens=*" %%a in ('XCOPY /d /l /y "%Source%" "%Target%"') do (
    Move "%%a" "%Processing%"
    Ren "%Processing%\%%~nxa" "Processing.txt"
 
    ECHO Do your stuff here with the processing.txt"
    Pause
    Ren "%Processing%\Processing.txt" "%%~nxa"
    Move "%Processing%\%%~nxa" "%Target%"
)

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 23648209
Hello AmazingTech...

Apologies for my earlier rant.... I've been at my PC throughout the entire night in an attempt to get up to speed with Assembly on the AMD processor (something I haven't done for yonks) and after reading that nonesense posted by above I just flipped. Call it over-tiredness/irritability or what you like.

There's no need for garbage at 5 in the morning when you haven't slept all night.

By the way, none of that was directed at you and I should have made this clear earlier. In fact, of all the replies, yours was the only ensible one there. By the way, I liked the subtle use of the /D switch in your XCOPY.... Nice touch.
 
0
 
LVL 21

Expert Comment

by:AmazingTech
ID: 23648283
I don't take postings personally. Not to worry I actually did miss the point about processed folder totally.  I was coming back to change my code to do the move and rename with the 1 move command but you beat me to it. So instead I kept it with the 2 separate lines.
0
 
LVL 28

Assisted Solution

by:lesouef
lesouef earned 664 total points
ID: 23649008
Mr Tomasi,
 I was doing it from another OS, I typed a few things wrong, but lovingatx would probably have guessed that sourcefolder was a generic term to be replaced by the real path!
Last point: there is a good reason to use 'call :[label]' in scripts, sorry you did not discover that yet.
And here is my script, modified and tested (unlike on sundays).

@echo off
set sourcefolder=c:\tmp\source
set processingfolder=c:\tmp\processing
set processedfolder=c:\tmp\processed
 
for /f "tokens=*" %%i in ('dir "%sourcefolder%" /b/s') do (set file=%%~nxi & call :sub)
goto :eof
 
:sub
if not exist "%processingfolder%\%file%" move "%sourcefolder%\%file%" "%processingfolder%\processing.txt"
rem I guess you need to insert some action here...
if not exist "%processedfolder%\%file%" move "%processingfolder%\processing.txt" "%processedfolder%\%file%"
goto :eof

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 23649080
lovingatx

I have decided to add a little extra code to first test whether files do actually exist in the source folder

@echo off
SET f=0 & FOR %%a IN (*.*) DO SET /A f+=1
IF %f% EQU 0 (
   ECHO No files to process
   EXIT /B 1
)

FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\Processing.txt"
      ::
      ::
      ::
      MOVE "processing\Processing.txt" "processed\%%a"
   )
)
0
 
LVL 16

Expert Comment

by:t0t0
ID: 23649209
lovingatx

Here's the same code but without the "No files to process" messege.

NOTE: If there are nofiles to process, the batch file returns an errorcode 1 otherwise it returns an errorcode 0. You can then later examine the errorlevel from the command line or from within the CALLing batch file to determine whether any files were processed.


SET f=0 & FOR %%a IN (*.*) DO SET /A f+=1
IF %f% EQU 0 EXIT /B 1

FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\Processing.txt"
      ::
      ::
      MOVE "processing\Processing.txt" "processed\%%a"
   )
)
EXIT /B 0
0
 
LVL 16

Accepted Solution

by:
t0t0 earned 672 total points
ID: 23649509
lovingatx

The following code is so good I've decided to post it again.


SET f=0 & FOR %%a IN (*.*) DO SET /A f+=1
IF %f% EQU 0 EXIT /B 1

FOR /F "TOKENS=*" %%a IN ('DIR /B /A:-D "source\"') DO (
   IF NOT EXIST "processed\%%a" (
      MOVE "source\%%a" "processing\Processing.txt"
      ::
      ::
      MOVE "processing\Processing.txt" "processed\%%a"
   )
)
EXIT /B 0
0
 
LVL 35

Expert Comment

by:ShineOn
ID: 23656318
It's nice that all of you want to play 'battling batch scripts,' but the Asker still hasn't posted back any answers to any Expert questions.  In my opinion you're wasting your talents on someone that doesn't seem to care - at least not enough to post back - and as I said before, you might be going in the wrong direction, depending on what the purpose of this really is - which, contrary to toto's opinion, still is unclear.

lovingatx's other questions focus on database technologies, which is as far from DOS batch script as you can get.  It may actually be a fit, but it depends on what's really going on.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 23656514
hiya shineon

this would tie in nicely with his other questions then.... whatever he does between the two MOVEs must be database-related.

0
 

Author Comment

by:lovingatx
ID: 23687448
I apologize profusely for the delay in responding.  I had to have a solution the day of the question so I posted and then continued to work on the issue.   It was discourteous of me to forget to remove the question afterward.  You guys presented some great answers and I will split the points the best I can.

I was using an ETL tool (Cognos datamanager) and needed to process files from incremental file dump.  As you guys suspected, i wanted to pick up the files, check if they had been processed, if not process them, and then move them to the processed folder so that I would know they had been processed.  The ETL tool provides an interface where you can define a string and then execute the string as a windows shell command.  Below is the solution I ended up with.  Its 3 parts.  First part, copy any files that meet the criteria.  Next, determine if each file needs to processed.  If so, the etl tool processes the file.  THe file is then moved to the processed folder.  Next the ETL tool provides the looping functionality to process the next file.

Thanks again - I truly respect those who posts these intelligent replies and apologize for failing to maintain my thread.

Cheers
Get new files
STRCMD := 'copy \\Ntas34\KCPL_Shared\aqw*.txt  D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\';
System( $STRCMD );
 
 
Process files
$STRFILE := FileList('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\aqw*.txt');
$STRCHECKFILE := concat('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\',ExtractStr($STRFILE, 1));
$STRTARGETFILE := concat('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\PROCESSED\',ExtractStr($STRFILE, 1));
 
 
 
 
if Length($STRFILE)= 0 then
 begin
  $PROCEED := FALSE;
 end
 else
  begin
   $FILEEXISTS := FileCheck($STRTARGETFILE);
   if $FILEEXISTS then
    begin
     $STRCMD := concat('erase ', $STRCHECKFILE);
     System( $STRCMD );
     $PROCEED := FALSE;
     $STRFILE := FileList('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\aqw*.txt');
    end
   else
    begin
     $STRCMD := concat('COPY ', $STRCHECKFILE, ' D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\WeeklyContacts.txt' );
     System( $STRCMD );
     $PROCEED := TRUE;
    end
end
 
 
Move Processed File to Processed Folder
$STRFILE := FileList('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\aqw*.txt');
$STRCHECKFILE := concat('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\',ExtractStr($STRFILE, 1));
$STRTARGETFILE := concat('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\PROCESSED\',ExtractStr($STRFILE, 1));
 
$STRCMD := CONCAT('move ', $STRCHECKFILE, ' ', $STRTARGETFILE);
System( $STRCMD );
 
$STRCMD := 'erase   D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\WeeklyContacts.txt' ;
System( $STRCMD );
 
$STRFILE := FileList('D:\DataManagerImportFiles\GlobalDataFiles\GMO\WeeklyUpdate\aqw*.txt');

Open in new window

0
 

Author Closing Comment

by:lovingatx
ID: 31546763
Very creative - thank you!
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Unable to change the program that handles the scan event from a network attached Canon/Brother printer/scanner. This means you'll always have to choose which program handles this action, e.g. ControlCenter4 (in the case of a Brother).
When you discover the power of the R programming language, you are going to wonder how you ever lived without it! Learn why the language merits a place in your programming arsenal.
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Suggested Courses
Course of the Month16 days, 13 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question