Batch Script to run deletion of several folders nightly.

Well, it looks okay at the first glance.

Depending on what data processing logic is used, I would consider renaming "D:\CZM-HFA2i\EM\Data_Export" to "D:\CZM-HFA2i\EM\Data_Export-YYYYMMDD", where YYYYMMDD is yesterdays date and creating a new, empty export folder.
In this case you can access left artifacts as well as unprocessed files later on, if something weird happens (better safe than sorry).

In the case of deleting files, you should log the file names you have purged for logging reasons.

Ste5an I cant change the folder name as it is used by a streaming service and that Data export and processed folder has to stay the same fore the service to run. What is the line that should be added to create a log file showing the purge information you mention?

Ste5an I cant change the folder name as it is used by a streaming service and that Data export and processed folder has to stay the same fore the service to run.

This sounds like a contradiction. Is there a file locked, when the purge should run? If not, then renaming is the simpler strategy. Btw, renaming also implies that this export directory and the purge of it cover exactly one scenario.

You can log explicitly by using a for loop for enumerating the files to delete or you redirect the output of your delete command to a log file.

The simple logging case:

@Echo Off

SetLocal

Set LOG_FILE="D:\CZM-HFA2i\EM\Data_Export_Purge.log"
Set PATH="D:\CZM-HFA2i\EM\Data_Export\"
Set PATTERN="Processed *.*"

Echo --- %DATE% --- Purging "%PATTERN%" in "%PATH%".. >> "%LOG_FILE%
Del "%PATH%%PATTERN%" /S /F /Q >> "%LOG_FILE%
Echo --- %DATE% --- done. >> "%LOG_FILE%

EndLocal

Select allOpen in new window

Wait now ya got me confused.  Do I add this to my current batch script?

I think there is something wrong here:

Shouldn't this
\Processed *.*
be
\Processed\*.*
(notice the backslash)
if you run it like this it might try to also delete everything in current folder after deletenig everything in
Data_Export\Processed

Yeah I need to make sure Processed and Data_Export remain at all times.

Just tried to test this on my local machine made 4 temp folders called Temp Temp 2 Temp 3 and Temp 4 ran this script but it didnt work.

@Echo Off

SetLocal

Set LOG_FILE="C:\CZM-HFA2i\EM\Data_Export-Purge.log"

Call :PurgeDirectory "C:\Temp\*.*"
Call :PurgeDirectory "c:\Temp 2\*.*"
Call :PurgeDirectory "C:\Temp 3\*.*"
Call :PurgeDirectory "C:\Temp 4\*.*"
GoTo :Eof

EndLocal

:PurgeDirectory
Set PATH=%~1
Echo --- %DATE% %TIME%--- Purging "%PATH%".. >> "%LOG_FILE%
Del "%PATH%" /S /F /Q >> "%LOG_FILE%
Echo --- %DATE% %TIME% --- done. >> "%LOG_FILE%
Goto :Eof

First of all: Edit your posts and use the [}code][/code] tags (the document button with the </> symbol) and markup your code. This makes reading and copying code easier.

For testing you need as directories:

C:\Temp\Test
C:\Temp\Test\Data 1
C:\Temp\Test\Data 2
C:\Temp\Test\Data 3
C:\Temp\Test\Data 4

Select allOpen in new window

with

@Echo Off

SetLocal

Set LOG_FILE="C:\Temp\Test\Purge.log"

Call :PurgeDirectory "C:\Temp\Test\Data 1\Processed *.*"
Call :PurgeDirectory "C:\Temp\Test\Data 2\Processed *.*"
Call :PurgeDirectory "C:\Temp\Test\Data 3\Processed *.*"
Call :PurgeDirectory "C:\Temp\Test\Data 4\Processed *.*"
GoTo :Eof

EndLocal

:PurgeDirectory
Set PATH=%~1
Echo --- %DATE% %TIME%--- Purging "%PATH%".. >> "%LOG_FILE%
Del "%PATH%" /S /F /Q >> "%LOG_FILE%
Echo --- %DATE% %TIME% --- done. >> "%LOG_FILE%
Goto :Eof

Select allOpen in new window

ste5an I am testing this on my own machien not the environment to make sure it works.   I just made temps on my local c:\ so you're saying if I make this path and put data in there it will purge?

del will only remove files not directories
if you want to delete both use RD instead

Rd PATH /q/s

--- Thu 03/05/2020 10:35:28.82--- Purging "C:\Temp\Test\Data 1\Processed *.*"..
--- Thu 03/05/2020 10:35:28.82 --- done.
--- Thu 03/05/2020 10:35:28.83--- Purging "C:\Temp\Test\Data 2\Processed *.*"..
--- Thu 03/05/2020 10:35:28.84 --- done.
--- Thu 03/05/2020 10:35:28.85--- Purging "C:\Temp\Test\Data 3\Processed *.*"..
--- Thu 03/05/2020 10:35:28.85 --- done.
--- Thu 03/05/2020 10:35:28.86--- Purging "C:\Temp\Test\Data 4\Processed *.*"..
--- Thu 03/05/2020 10:35:28.86 --- done.

However the text files I put in those 4 directories are still there.   

I just noticed that I didnt put Processed.

I added the Processed folder but the files remain.. Strange maybe I do need those slashes.

Well, the files must be named accordingly. As Arana pointed out, you may have a typo, a missing backslash in your original post.

The given file pattern only deletes files named "Processed *.*", thus only files which start with the word processed followed by a space.

--- Thu 03/05/2020 10:35:28.82--- Purging "C:\Temp\Test\Data 1\Processed *.*"..
--- Thu 03/05/2020 10:35:28.82 --- done.
--- Thu 03/05/2020 10:35:28.83--- Purging "C:\Temp\Test\Data 2\Processed *.*"..
--- Thu 03/05/2020 10:35:28.84 --- done.
--- Thu 03/05/2020 10:35:28.85--- Purging "C:\Temp\Test\Data 3\Processed *.*"..
--- Thu 03/05/2020 10:35:28.85 --- done.
--- Thu 03/05/2020 10:35:28.86--- Purging "C:\Temp\Test\Data 4\Processed *.*"..
--- Thu 03/05/2020 10:35:28.86 --- done.
--- Thu 03/05/2020 10:37:44.96--- Purging "C:\Temp\Test\Data 1\Processed *.*"..
--- Thu 03/05/2020 10:37:44.97 --- done.
--- Thu 03/05/2020 10:37:44.98--- Purging "C:\Temp\Test\Data 2\Processed *.*"..
--- Thu 03/05/2020 10:37:44.99 --- done.
--- Thu 03/05/2020 10:37:45.00--- Purging "C:\Temp\Test\Data 3\Processed *.*"..
--- Thu 03/05/2020 10:37:45.00 --- done.
--- Thu 03/05/2020 10:37:45.01--- Purging "C:\Temp\Test\Data 4\Processed *.*"..
--- Thu 03/05/2020 10:37:45.01 --- done.
--- Thu 03/05/2020 10:38:38.72--- Purging "C:\Temp\Test\Data 1\Processed *.*"..
--- Thu 03/05/2020 10:38:38.73 --- done.
--- Thu 03/05/2020 10:38:38.74--- Purging "C:\Temp\Test\Data 2\Processed *.*"..
--- Thu 03/05/2020 10:38:38.75 --- done.
--- Thu 03/05/2020 10:38:38.76--- Purging "C:\Temp\Test\Data 3\Processed *.*"..
--- Thu 03/05/2020 10:38:38.76 --- done.
--- Thu 03/05/2020 10:38:38.77--- Purging "C:\Temp\Test\Data 4\Processed *.*"..
--- Thu 03/05/2020 10:38:38.78 --- done.

Files still there after adding the slashes... hmmm...

C:\Temp\Test\Data 1\Processed
C:\Temp\Test\Data 2\Processed
C:\Temp\Test\Data 3\Processed
C:\Temp\Test\Data 4\Processed

Those are the paths I have 2 temp txt files in them but they arent deleting.

change \Processed *.*"  to   \Processed\*.*"  for that script to work. using DEL (for files only)
and also change
Del "%PATH%" /S /F /Q >> "%LOG_FILE%
to
RD "%PATH%" /S  /Q >> "%LOG_FILE%
for it to delete files as well as directories.

Yup, in your original post, "Processed" is part of the file name pattern not a directory. Thus you need to add slashes as Arana pointed out:

Call :PurgeDirectory "C:\Temp\Test\Data 1\Processed\*.*"
Call :PurgeDirectory "C:\Temp\Test\Data 2\Processed\*.*"
Call :PurgeDirectory "C:\Temp\Test\Data 3\Processed\*.*"
Call :PurgeDirectory "C:\Temp\Test\Data 4\Processed\*.*"

Select allOpen in new window

Here is the edited script.  Again please remember I need Data_Export and Processed to remain and not delete, just the files within those folders.   I made the changes you made but the files still arent deleting.   I can copy and paste the files out so I know the folders are editable.   Here is the edited script.

@Echo Off

SetLocal

Set LOG_FILE="C:\Temp\Test\Data_Export-Purge.log"

Call :PurgeDirectory "C:\Temp\Test\Data 1\Processed\*.*" /s/f/q
Call :PurgeDirectory "C:\Temp\Test\Data 2\Processed\*.*" /s/f/q
Call :PurgeDirectory "C:\Temp\Test\Data 3\Processed\*.*" /s/f/q
Call :PurgeDirectory "C:\Temp\Test\Data 4\Processed\*.*" /s/f/q
GoTo :Eof

EndLocal

:PurgeDirectory
Set PATH=%~1
Echo --- %DATE% %TIME%--- Purging "%PATH%".. >> "%LOG_FILE%
RD "%PATH%" /S /F /Q >> "%LOG_FILE%
Echo --- %DATE% %TIME% --- done. >> "%LOG_FILE%
Goto :Eof

Sorry for the confusion.   Data_Export\Processed is in the real environment server 2008.   I am simply testing this script on my windows 10 pro workstation which i made my own temp directories if that clears things up.

@Andrew N. Kowtalo

I have used base version from ste5an

@ECHO OFF
REM BASE VERSION 0.01
REM BEGINS LOCALIZATION OF ENVIRONMENT CHANGES IN A BATCH FILE.
REM HELP SETLOCAL
SETLOCAL
SET DRIVENAME=D:
REM WE CANNOT REDIRECT PASS AND FAIL TO SAME FILE AT WINDOWS, SINCE SAME FILE WILL BE IN USE BY OTHER PROCESS(EITHER FAIL/PASS).
REM HENCE USING TWO LOG FILES
SET PASS_LOG_FILE="%DRIVENAME%\CZM-HFA2i\EM\Data_Export_Purge_PASS.log"
SET FAIL_LOG_FILE="%DRIVENAME%\CZM-HFA2i\EM\Data_Export_Purge_FAIL.log"
CALL :PURGEDIRECTORY "%DRIVENAME%\CZM-HFA2i\EM\Data_Export"
CALL :PURGEDIRECTORY "%DRIVENAME%\CZM-HFA2i\NM\Data_Export"
CALL :PURGEDIRECTORY "%DRIVENAME%\CZM-HFA2i\OP\Data_Export"
CALL :PURGEDIRECTORY "%DRIVENAME%\CZM-HFA2i\WM\Data_Export"
REM AFTER FINISHING CALL TO PURGEDIRECTORY EXIT CURRENT ROUTINE
GOTO :EOF
REM ENDS LOCALIZATION OF ENVIRONMENT CHANGES IN A BATCH FILE
REM HELP ENDLOCAL
ENDLOCAL
:PURGEDIRECTORY
SET CLEANPURGE=%~1
REM Search Processed*.* => Processed followed by with or without space...
REM Redirect error to NUL 2^>NUL
FOR /F "DELIMS=" %%A IN ('DIR /S /B %%CLEANPURGE%%\Processed*.* 2^>NUL') DO (
	IF EXIST "%%A" (
		ECHO DEL "%%A"
		ECHO DEL "%%A">> %PASS_LOG_FILE% 2>>%FAIL_LOG_FILE%
		DEL "%%A">> %PASS_LOG_FILE% 2>>%FAIL_LOG_FILE%
	)
)
REM EXIT CURRENT PURGEDIRECTORY SUBROUTINE USING EOF
GOTO :EOF

Select allOpen in new window

sample output:

D:\murugesandins> .\29174776.CMD
D:\murugesandins> .\29174776.CMD
DEL "D:\CZM-HFA2i\EM\Data_Export\Processed1.txt"
DEL "D:\CZM-HFA2i\EM\Data_Export\Processed2.txt"
DEL "D:\CZM-HFA2i\NM\Data_Export\Processed 1.txt"
DEL "D:\CZM-HFA2i\NM\Data_Export\Processed1.txt"
DEL "D:\CZM-HFA2i\OP\Data_Export\Processed 2.txt"
DEL "D:\CZM-HFA2i\OP\Data_Export\Processed1.txt"
DEL "D:\CZM-HFA2i\WM\Data_Export\Processed 3.txt"
DEL "D:\CZM-HFA2i\WM\Data_Export\Processed1.txt"
D:\murugesandins>TYPE D:\CZM-HFA2i\EM\Data_Export_Purge_PASS.log
DEL "D:\CZM-HFA2i\EM\Data_Export\Processed1.txt"
DEL "D:\CZM-HFA2i\EM\Data_Export\Processed2.txt"
DEL "D:\CZM-HFA2i\NM\Data_Export\Processed 1.txt"
DEL "D:\CZM-HFA2i\NM\Data_Export\Processed1.txt"
DEL "D:\CZM-HFA2i\OP\Data_Export\Processed 2.txt"
DEL "D:\CZM-HFA2i\OP\Data_Export\Processed1.txt"
DEL "D:\CZM-HFA2i\WM\Data_Export\Processed 3.txt"
DEL "D:\CZM-HFA2i\WM\Data_Export\Processed1.txt"
D:\murugesandins> .\29174776.CMD
D:\murugesandins> TYPE D:\CZM-HFA2i\EM\Data_Export_Purge_FAIL.log
D:\murugesandins>

Select allOpen in new window

if you will use RD you must not use /F

@Arana: When using RD is possible, then renaming the directory should be also possible...

murugesandins do you see the script I posted locally, I want to test this first with your changes before I implement them in the real environment.  We have not been given the go ahead yet to purge these folders.   This bought me enough time to test.

ok sure.
>> I want to test this first with your changes before I implement

Thank you and welcome for making "ASKER CERTIFIED SOLUTION" (from ste5an) and all related comments.

>> This bought me enough time to test.
interested in development, testing, deployment and bug fix too :)

@Ste5an /F is not a valid parameter for RD, (Remove Directory)  
Sorry I don't understand what you say about renaming.

I admit I am terrible at this LOL!  I have tested all these scripts locally on my machine and the files wont delete even though I can cut and paste them in and out.   Ugh!

To setup a test, run the following setup batch:

@Echo Off

SetLocal

Set BASE_PATH=C:\Temp\Test\
Set DATA1=Data 1\
Set DATA2=Data 2\
Set DATA3=Data 3\
Set DATA4=Data 4\

Call :CreateSampleData "%BASE_PATH%%DATA1%" 
Call :CreateSampleData "%BASE_PATH%%DATA2%" 
Call :CreateSampleData "%BASE_PATH%%DATA3%" 
Call :CreateSampleData "%BASE_PATH%%DATA4%" 

Dir "%BASE_PATH%" /S

EndLocal

:CreateSampleData
Set PATH=%~1
Set PROCESSED=%PATH%Processed\
Echo %PATH% %PROCESSED%
If Not Exist "%PATH%" ( MkDir "%PATH%" )
If Not Exist "%PROCESSED%" ( MkDir "%PROCESSED%" )
Echo --- %DATE% %TIME% -- >> "%PROCESSED%File1.txt"
Echo --- %DATE% %TIME% -- >> "%PROCESSED%File2.txt"
Echo --- %DATE% %TIME% -- >> "%PROCESSED%File3.txt"
Echo --- %DATE% %TIME% -- >> "%PROCESSED%File4.txt"
GoTo :Eof

Select allOpen in new window

Then you can test the purging with

@Echo Off

SetLocal

Set LOG_FILE="C:\Temp\ThePurge.log"

Call :PurgeDirectory "C:\Temp\Test\Data 1\Processed\*.*"
Call :PurgeDirectory "C:\Temp\Test\Data 2\Processed\*.*"
Call :PurgeDirectory "C:\Temp\Test\Data 3\Processed\*.*"
Call :PurgeDirectory "C:\Temp\Test\Data 4\Processed\*.*"
GoTo :Eof

EndLocal

:PurgeDirectory
Set PATH=%~1
Echo --- %DATE% %TIME%--- Purging "%PATH%".. >> "%LOG_FILE%
Del "%PATH%" /S /F /Q >> "%LOG_FILE%
Echo --- %DATE% %TIME% --- done. >> "%LOG_FILE%
Goto :Eof

Select allOpen in new window

ste5an That worked!! Hooray!! Now do I just change the paths to the real file location?  Hate to ask but could you provide the changes?

Ste5phan hooray!! I think it should work as far as folder creation just in case they delete something with the other script correct?