Deleting large qty of files - based on File name filters - what program to use?

Hi,
I have several years of files from our security system.  The names are in this type format, as an example:
Office.20150621_093154
Office.20150621_093034

name of room. year month date _ hour minutes seconds

What I'd like to do is find a way to go through and delete any file that has an 'hour' between, for instance, 01 and 06 (so any file between 1:00AM and 6:00AM).  To do this I'd need to be able to somehow filter based on the right 5th and 6th digits, or the two digits to the right of the "_".  Not sure the best way to do this.

There are over 500,000 of these files, so manually is not an option

Let me know if any suggestion
Mystical_IceAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

oBdACommented:
Do those file have a regular extension after the date, or is the date the extension?
Both of the batch scripts below are in test mode and will only display the files to be deleted. Remove the uppercase ECHO in front of the del command to run them for real.
This is for files with a regular extension (.mov in the example).
@echo off
setlocal enabledelayedexpansion
set Root=D:\Temp
set Mask=*.mov
set FirstHour=01
set LastHour=05
for /f "tokens=1-3* delims=_." %%a in ('dir /s /b /a:-d "%Root%\%Mask%"') do (
	<NUL set /p Dummy=Processing '%%a.%%b_%%c.%%d' ... 
	set FileTime=%%c
	set FileHour=!FileTime:~0,2!
	if !Filehour! geq %FirstHour% (
		if !Filehour! leq %LastHour% (
			echo deleted.
			ECHO del "%%a.%%b_%%c.%%d"
		) else (
			echo kept.
		)
	) else (
		echo kept.
	)
)

Open in new window


This one is for the date/time extension; it will take a bit longer, because the file names will be checked against the date/time format:
@echo off
setlocal enabledelayedexpansion
set Root=D:\Temp
set Mask=*.*
set FirstHour=01
set LastHour=05
for /f "tokens=1-3* delims=_." %%a in ('dir /s /b /a:-d "%Root%\%Mask%" ^| findstr.exe ".*\.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].*"') do (
	<NUL set /p Dummy=Processing '%%a.%%b_%%c' ... 
	set FileTime=%%c
	set FileHour=!FileTime:~0,2!
	if 1!Filehour! geq 1%FirstHour% (
		if 1!Filehour! leq 1%LastHour% (
			echo deleted.
			ECHO del "%%a.%%b_%%c"
		) else (
			echo kept.
		)
	) else (
		echo kept.
	)
)

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Mystical_IceAuthor Commented:
Files have a regular extension after the date.

One more thing - Is it possible to only delete where, for instance, there are more than 3 events for a particular hour and particular camera (Office, Entryway, Hallway, etc).

So for instance:
Office.20150621_093154
Office.20150621_093034
Office.20150621_092524
Office.20150621_092336
Office.20150621_092212
Office.20150621_092017
Office.20150621_071515

In the above example, because the "071515" is the only event in the hour, it's more than likely legitimate.  Whereas the others that are all in the '09' hour are probably a spider crawling over the lens triggering the file, so can be deleted.
oBdACommented:
Try this, in test mode again; note that there are two ECHOs that need to be removed to arm the script, in line 18 and line 28. Note that because of the need for sorting the results first, it might take a while for something to happen; test it on a smaller test folder first.
If it turns out to be too slow or just too much for batch with the 500000 files, would Powershell be an option? And are the files flat in one directory or spread out over several subfolders?
@echo off
setlocal enabledelayedexpansion
set Root=D:\Temp
set Mask=*.mov
set MaxEvents=3

set OldHash=
set /a FileCount = 0
for /f "tokens=1-3* delims=_." %%a in ('dir /s /b /a:-d "%Root%\%Mask%" ^| sort.exe') do (
	echo Processing '%%a.%%b_%%c' ...
	set FileTime=%%c
	set NewHash=%%a!FileTime:~0,2!
	if /i "!OldHash!"=="!NewHash!" (
		set /a FileCount += 1
	) else (
		if !FileCount! gtr %MaxEvents% (
			for /l %%i in (1, 1, !FileCount!) do (
				ECHO del "!DelFile[%%i]!"
			)
		)
		set OldHash=!NewHash!
		set /a FileCount = 1
	)
	set DelFile[!FileCount!]=%%a.%%b_%%c.%%d
)
if !FileCount! gtr %MaxEvents% (
	for /l %%i in (1, 1, !FileCount!) do (
		ECHO del "!DelFile[%%i]!"
	)
)

Open in new window

Mystical_IceAuthor Commented:
They are all in one big directory.  And powershell I don't know much about
Mystical_IceAuthor Commented:
What I ended up doing, and for anyone viewing this question in the future, was just using the built-in windows search capability.

Searching for:

"_03*", "_04*", "_05*", etc. would find anything that contained _03, _04, _05 and so forth.

Thanks all for your help!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Batch

From novice to tech pro — start learning today.