Script to split up a directory of 1.5 Million files into folders of 5000

We have a Windows 2003 server with an NTFS filesystem.  In it are log files from our webservers (all of them get dumped in one folder on this particular server)

There are 1,500,000 odd files in one folder (let's call it e:\logs\store1)

There are now too many files for an application we use to process / convert these log files to deal with.

I am after a script to go through the folder and split it up into sub folders of 5000.  That is, go through one by one and copy the first 5000 files to e:\logs\store1\1\ the next 5000 to e:\logs\store1\2\ etc.

I am unsure how to do this, hence my post.

I would appreciate any help anyone can give me.  Poor old windows explorer falls over when dealing with this many files in one spot!

Thanks in advance!
akeevillAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Richard QuadlingSenior Software DeveloperCommented:
If the files are over a LONG period of time, I would store them by Year/Month/Day.

e.g.

E:\logs\store1\archive\2008\01\30

sort of thing,

All the files are are created in e:\logs\store1.

On a regular basis you run the archiver which uses the last modification date to move the file to an applicable archive folder.

You can burn the archive folder to DVD and then delete what has been burnt.

You can restore multiple DVDs to the archive folder.

I have something similar (around 1000 faxes a day come into our electronic fax system - I process and archive them into a folder structure like the one above).

Here is a script which should help ...
0
Richard QuadlingSenior Software DeveloperCommented:

@ECHO OFF
IF "%1"=="" GOTO Start
IF "%~2"=="%0" GOTO Done
GOTO %1
 
:Start
ECHO Started on %DATE% at %TIME%
 
:Loop
FOR %%F IN ("%~dp0\*.log") DO CALL %0 Phase1 "%%~F" %%~tF
GOTO :EOF
 
:Phase1
FOR /F "tokens=1,2,3,4,5,6 delims=/: " %%G IN ("%3 %4") DO CALL %0 Phase2 "%~2" %%G %%H %%I %%J %%K
GOTO :EOF
 
:Phase2
SET YEAR=%3
IF %3 LEQ 2000 SET YEAR=20%3
IF NOT EXIST .\Archive\%YEAR%\%4\%5 MD .\Archive\%YEAR%\%4\%5
MOVE "%~2" ".\Archive\%YEAR%\%4\%5\%YEAR% %4 %5 %6 %7 - %~nx2"
IF EXIST ".\Archive\%YEAR%\%4\%5\%YEAR% %4 %5 %6 %7 - %~nx2" ECHO Moved .\Archive\%YEAR%\%4\%5\%YEAR% %4 %5 %6 %7 - %~nx2
IF NOT EXIST ".\Archive\%YEAR%\%4\%5\%YEAR% %4 %5 %6 %7 - %~nx2" ECHO ***** .\Archive\%YEAR%\%4\%5\%YEAR% %4 %5 %6 %7 - %~nx2

Open in new window

0
akeevillAuthor Commented:
Could you please explain the arguments in the script.

Maybe I was unclear - the files all have random names / dates.  There could be 100,000 files on one day, and 3 files on the other, so sorting by date really isn't what I was after.

I don't know if it's possible, but basically I was after something that would loop through and grab 5000 files and dump them in "Folder 1", and grab the next 5000 and dump them in "Folder 2" until there were no files left.

0
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Richard QuadlingSenior Software DeveloperCommented:
Consider this issue.

Say there are 12,000 files NOW.

We run the script for the first time.

It creates Folder_001 and puts 5,000 files in it.
It creates Folder_002 and puts 5,000 files in it.
If creates Folder_003 and puts 2,000 files in it.

The next day you get another 12,000 files.

What should happen?

Folders 4,5,6 with 5,000, 5,000 and 2,000?

Or

Folder 3 3,000, folder 4, 5000, folder 6 4,000 ?


I would STRONGLY recommend NOT trying to fill in 5,000 to a folder. I would leave the original 2,000 behind.

So, day 1 - folders 1 and 2 with 5,000 leaving 2,000 behind.
Day 2 folders 3 and 4

etc.



OOI. For the above program all that is needed is to run is to have the file in the same folder as the log files. Nothing else.

Try it first, create a folder with some log files in it. Put the batch file in the same folder and then run it.


Let me know what you think of the above plan and I'll talk you through a way.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
akeevillAuthor Commented:
Thanks for that - you made things a lot clearer.  I ended up writing a script manually :

e:\store1\dir /b /s > c:\files.txt
Then wrote a quick app that would make each line of files.txt look like this:

move /y e:\store1\file001.txt e:\newstore\1\
move /y e:\store1\file002.txt e:\newstore\1\
.
.
.
move /y e:\store1\file5000.txt e:\newstore\1\
move /y e:\store1\file5001.txt e:\newstore\2\

etc

Then simply ran that file - it took 4 hours to run, but did the job.  I have now changed the way the log files are stored, so they are archived more frequently.  Prevention is definitely better than cure.

I appreciate your help on this one.
0
akeevillAuthor Commented:
Thanks for your help - you made things a lot clearer.  I ended up writing a script manually :

1st got a list of the files in the dir:

e:\store1\dir /b /s > c:\files.txt

Then wrote a quick app that would make each line of files.txt look like this:

move /y e:\store1\file001.txt e:\newstore\1\
move /y e:\store1\file002.txt e:\newstore\1\
.
.
.
move /y e:\store1\file5000.txt e:\newstore\1\
move /y e:\store1\file5001.txt e:\newstore\2\

etc

Then simply ran that file - it took 4 hours to run - VERY hacked and manual, but did the job.  I have now changed the way the log files are stored, so they are archived more frequently.  Prevention is definitely better than cure.

I appreciate your help on this one.
0
Richard QuadlingSenior Software DeveloperCommented:
Glad you solved the problem.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows OS

From novice to tech pro — start learning today.