# How to move directory with more than 500,000 files?

I have a W2k3 server with more than 500,000 files in a directory. Cant seem to open the directory without crashing Windows Explorer. Much less copy to different directory without crashing. Any suggestions? Thanks.

Intel Xenon CPU 2.80 GHz
2.80 GHz, 1.00 GB
###### Who is Participating?

Commented:
>>I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash

It might be necessary to use DOS to COPY and DELETE one file at a time rather than attempt to run a command or application which 'logs' all the files before starting it's operation: COPY and XCOPY are two examples - they can take seemingly ages if you have a HUGE number of files giving you the impression the process has hung.

In my earlier comment, I assumed you would use COPY (or XCOPY) or MOVE inside a FOR-loop. You could try something like this:

@echo off
setlocal enabledelayedexpansion

set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (*.*) do (
set fd=%%~ta
set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

if !FileDate! lss %TwoYears% (
del "%%a"
) else (
move "%%a" "%Destination%"
)

)


It's fast because it does not rely on caching of filenames.

NOTES

DESTINATION FOLDER

Set your destination folder in line 4. Here it is set to c:\temp\destination\ so you will need to change this to point to your own destination folder.

CURRENT DATE

Line 6 simply sets the variable TwoYears to the current date minus 2 years. This is how it works:

The current date given by the system variable %DATE% is something like: 15/09/2011.

We can cleverly turn this into an ordered integer by rearranging the values for day-month-year to year-month-day ('15' '09' '2011' to '2011' '09' '15').

In this example, %date:~6,4% returns the year part of %DATE% starting at character-position 6 (0..6) comprising 4 digits ie, 2011

%date:~3,2% returns the month part of %DATE% starting at character-position 3 (0..3) comprising 2 digits ie, 09

%date:~0,2% returns the day part of %DATE% starting at character-position 0 comprising 2 digits ie, 15

The values used here may differ from yours depending how your system displays it's date with the following command:

echo %date%

Now that we know how to get the year, month and day we can string them togehter in year, month and day order with the following command:

set TwoYears=%date:~6,4%%date:~3,2%%date:~0,2%

So, where the current date is 15/09/2011, the variable TwoYears is set to '20110915' (leaving out the forward-slash delimiter characters)

By perfoming an arithmetic assignment using SET /A instead of just SET, we can perform arithmetic during the above assignment. In this case, we want to reduce the year value by 2. This is done by subtracting 20000 from 20110915 which gives us '20090915'.

This value will be used when testing the date of each file.

LOOP

Line 8 is the start of our loop. Here we loop for each file in the folder. The file's name is returned in the FOR-variable %%a. The loop ends at line 18.

COMPARING DATES

The FOR-loop's %%a variable returns the file's name whereas, %%~ta returns it's datestamp. This includes other information such as time. This is how it is shown on my system:

04/01/2011 01:17 AM

Again, this may differ on your system however, as we did before, we need to extract the year, month and day parts but before we do this, we have to first assign the whole thing to a variable - in this case, I've used the short variable name 'fd':

set fd=%%~ta

And now we can assemble the file's date in year-month-day order (as we did above) and assign it to a variable named 'FileDate'. Instead of using %fd%, we use !fd! becuase these values will need to be processed at run-time and therefore will require the line: SetLocal EnableDelayedExpansion as shown in line 2 of our code.

set FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

So, from the example datestamp shown above, the variable FileDate would be set to '20110104'.

With these two dates, the current system date 20090915 (year adjusted) and the file's date 20110104, we can perform a simple integer comparison to determine whether the file's date is older (less than) the current date two years ago.

MOVING FILES

Line 12 does all the work here:

if !FileDate! lss %TwoYears% (
del "%%a"
) else (
move "%%a" "%Destination%"
)

Here, we compare the file's date to the date two years ago and if the file's date is older (smaller) then the file (in "%%a" - note the double-quotes in case there are spaces in the file's name) is deleted otherwise, (if the file is newer than two years ago) it is moved to your destination folder (pointed to by "%Destination%" - again, note the use of double-quotes).

That's all there is to it!

FINAL CONSIDERATION

If you require a visual indication that your batch file is processing the files (rather than giving you the impression the batch file has hung), you can include the following line:

title %%a

in line 11 for example. But this only works if you run your batch file in a DOS window. In this case, you will see each file's name being processed in the window's (normally) blue titlebar.

0

Commented:
Have you tried a basic copy or move at a command prompt, like:

COPY "C:\FromDir\*.*" "D:\ToDir"

or

MOVE "C:\FromDir\*.*" "D:\ToDir"

You might also want to try XCOPY if you just want to copy, not move.

If those fail you could try ROBOCOPY as another option as well.

~bp
0

Commented:
If there are no requirements to the file move you could try the command prompt and use

Move c:\source\*.* c:\destination\

Or move the entire directory

Move c:\source c:\destination

That being said, I've never tried moving 500000 files from one directory and have no idea how the command prompt would react in this instance.
0

Commented:
I'm sure you've realized this by now, but never ever ever put anywhere near that many files into 1 directory.
0

IT ConsultancyCommented:
are they just some files to delete?

If so you can remove dir from command prompt.

Start cmd.exe

cd /d c:\yourdir\to\delete
rd . /s /q

it will show error but delete files and subdirs under current directory.... Careful!

If you need to retrieve the files into different directories say we could script moving them into directories with 10000 in each say, or how about copy all files to directory based on first letter wtc.

What are the files anyway - large number of random files, temp files, log files from something?

Steve
0

Commented:
The problem is that Windows will want to 'log' every file before displaying the folder's contents in Explorer... With that many files it will bring your system down on it's knees.

You should consider performing a CHKDSK operation ASAP to rule out and fix filesystem errors.

DOS is the way to go. As billpreww suggests, use either COPY (or XCOPY) with the VERIFY option enabled, or MOVE.

XCOPY has the ability to resume copying (with the '/C' option) if it encounters errors whereas COPY and MOVE will just frustratingly bomb-out.

To summerise, use COPY (or better still, XCOPY) (with the VERIFY option) to copy your files then, when you are satisfied all is well, delete the files from the original location.

BUT DON'T FORGET TO DO A CHKDSK FIRST !!

0

AWS Certified Solution Architect | L3 IT Specialist for CloudCommented:
0

Author Commented:
Thats nothing, theres a million+ files in another directory. XCOPY didnt do the job, but I will try it, since this was done by IT director. Not sure how he did it exactly, but Ill give it shot. I will also try Robocopy.

An application writes these small excel files for customer archiving. Theres a script to archive the directory, but it turns out, even the archive directory grew insanely. FYI: I didnt cause this. New here.

Also, turns out, we just need to delete files older than 2 years. You would think this would be easier, but its not since this is not one of DOS capabilities. I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash since previous scripts crashed when pointing to that directory.
0

Commented:
If you just need to delete them you might want to give DELAGE32 a shot, it's a great tool for removing old files.

http://www.horstmuc.de/win/delage.htm

http://www.horstmuc.de/wbat32.htm

~bp
0

Commented:
Oops!

I just realised if you drop the avove batch file into the folder containing your files, it will move ITSELF along with your other files. One way around this is to change the file:

for %%a in (*.*) do (

to something like:

for %%a in (*.xls) do (

if you know the extention names of the files to be moved. In this example, all the files are .XLS files.
0

Commented:
Try this modified batch file.

As stated above, you will need to edit the Source and Destination locations in lines 4 and 5.

@echo off
setlocal enabledelayedexpansion

set Source=c:\temp\Source
set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (%Source%\*.*) do (
set fd=%%~ta
set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

if !FileDate! lss %TwoYears% (
del "%%a"
) else (
move "%%a" "%Destination%"
)
)

0

Commented:
maybe you need someone else, who say "robocopy" ^^ here i am..

cmd: robocopy "c:\source dir" "c:\target dir" /MIR /SEC

/MIR = mirror (source -> target)
/SEC = copy security informations (ntfs permissions for example)
if you use " " (spaces) in foldernames .. put the hole path into "..."

at the end you will see the result of that job.
0

Commented:
dahesi

Yeah well, that doesn't address the the asker's question. How to MOVE files that are LESS than or EQUAL to two years old while at the same time DELETE files that are GREATER than two years old.

Oh, by the way, not all of us have Robocopy on our systems and besides, in all my years of experience, I have never had the need to use it.

(If it were possible to carve pyramids out of a single stone do you think they would have done so?)

0

Commented:
another parameter for all those robocopy haters out there:

/MAXAGE:n (n = days.. it wont copy files older than n)

cmd: robocopy ..... /MAXAGE:730

here we go
0

Commented:
dahesi

I won't say I 'hate' Robocopy... it's a powerful utility program and can save a lot of time when relied upon.

Personally, I haven't explored it's rich range of capabilities but from what bits I've seen I'm left with no doubt of it's use as a substitute to coding batch file routines.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.