Solved

How to move directory with more than 500,000 files?

Posted on 2011-09-12
15
613 Views
Last Modified: 2012-06-21

I have a W2k3 server with more than 500,000 files in a directory. Cant seem to open the directory without crashing Windows Explorer. Much less copy to different directory without crashing. Any suggestions? Thanks.

Intel Xenon CPU 2.80 GHz
2.80 GHz, 1.00 GB
0
Comment
Question by:fstinc
  • 6
  • 2
  • 2
  • +5
15 Comments
 
LVL 51

Expert Comment

by:Bill Prew
Comment Utility
Have you tried a basic copy or move at a command prompt, like:

COPY "C:\FromDir\*.*" "D:\ToDir"

or

MOVE "C:\FromDir\*.*" "D:\ToDir"

You might also want to try XCOPY if you just want to copy, not move.

If those fail you could try ROBOCOPY as another option as well.

~bp
0
 

Expert Comment

by:TwynFeyr
Comment Utility
If there are no requirements to the file move you could try the command prompt and use

Move c:\source\*.* c:\destination\

Or move the entire directory

Move c:\source c:\destination


That being said, I've never tried moving 500000 files from one directory and have no idea how the command prompt would react in this instance.
0
 
LVL 21

Expert Comment

by:Papertrip
Comment Utility
I'm sure you've realized this by now, but never ever ever put anywhere near that many files into 1 directory.
0
 
LVL 43

Expert Comment

by:Steve Knight
Comment Utility
are they just some files to delete?

If so you can remove dir from command prompt.

Start cmd.exe

cd /d c:\yourdir\to\delete
rd . /s /q

it will show error but delete files and subdirs under current directory.... Careful!

If you need to retrieve the files into different directories say we could script moving them into directories with 10000 in each say, or how about copy all files to directory based on first letter wtc.

What are the files anyway - large number of random files, temp files, log files from something?

Steve
0
 
LVL 11

Expert Comment

by:paultomasi
Comment Utility
The problem is that Windows will want to 'log' every file before displaying the folder's contents in Explorer... With that many files it will bring your system down on it's knees.

You should consider performing a CHKDSK operation ASAP to rule out and fix filesystem errors.

DOS is the way to go. As billpreww suggests, use either COPY (or XCOPY) with the VERIFY option enabled, or MOVE.

XCOPY has the ability to resume copying (with the '/C' option) if it encounters errors whereas COPY and MOVE will just frustratingly bomb-out.

To summerise, use COPY (or better still, XCOPY) (with the VERIFY option) to copy your files then, when you are satisfied all is well, delete the files from the original location.

BUT DON'T FORGET TO DO A CHKDSK FIRST !!

0
 
LVL 8

Expert Comment

by:Amitabh Singh
Comment Utility
0
 

Author Comment

by:fstinc
Comment Utility
Thats nothing, theres a million+ files in another directory. XCOPY didnt do the job, but I will try it, since this was done by IT director. Not sure how he did it exactly, but Ill give it shot. I will also try Robocopy.

An application writes these small excel files for customer archiving. Theres a script to archive the directory, but it turns out, even the archive directory grew insanely. FYI: I didnt cause this. New here.

Also, turns out, we just need to delete files older than 2 years. You would think this would be easier, but its not since this is not one of DOS capabilities. I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash since previous scripts crashed when pointing to that directory.
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 51

Expert Comment

by:Bill Prew
Comment Utility
If you just need to delete them you might want to give DELAGE32 a shot, it's a great tool for removing old files.

http://www.horstmuc.de/win/delage.htm

http://www.horstmuc.de/wbat32.htm

~bp
0
 
LVL 11

Accepted Solution

by:
paultomasi earned 250 total points
Comment Utility
>>I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash

Please post your script here...

It might be necessary to use DOS to COPY and DELETE one file at a time rather than attempt to run a command or application which 'logs' all the files before starting it's operation: COPY and XCOPY are two examples - they can take seemingly ages if you have a HUGE number of files giving you the impression the process has hung.

In my earlier comment, I assumed you would use COPY (or XCOPY) or MOVE inside a FOR-loop. You could try something like this:

@echo off
setlocal enabledelayedexpansion

set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (*.*) do (
   set fd=%%~ta
   set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

   if !FileDate! lss %TwoYears% (
      del "%%a"
   ) else (
      move "%%a" "%Destination%"
   )

)

Open in new window


It's fast because it does not rely on caching of filenames.



NOTES

DESTINATION FOLDER

Set your destination folder in line 4. Here it is set to c:\temp\destination\ so you will need to change this to point to your own destination folder.


CURRENT DATE

Line 6 simply sets the variable TwoYears to the current date minus 2 years. This is how it works:

The current date given by the system variable %DATE% is something like: 15/09/2011.

We can cleverly turn this into an ordered integer by rearranging the values for day-month-year to year-month-day ('15' '09' '2011' to '2011' '09' '15').

In this example, %date:~6,4% returns the year part of %DATE% starting at character-position 6 (0..6) comprising 4 digits ie, 2011

%date:~3,2% returns the month part of %DATE% starting at character-position 3 (0..3) comprising 2 digits ie, 09

%date:~0,2% returns the day part of %DATE% starting at character-position 0 comprising 2 digits ie, 15

The values used here may differ from yours depending how your system displays it's date with the following command:

   echo %date%

Now that we know how to get the year, month and day we can string them togehter in year, month and day order with the following command:

   set TwoYears=%date:~6,4%%date:~3,2%%date:~0,2%

So, where the current date is 15/09/2011, the variable TwoYears is set to '20110915' (leaving out the forward-slash delimiter characters)

By perfoming an arithmetic assignment using SET /A instead of just SET, we can perform arithmetic during the above assignment. In this case, we want to reduce the year value by 2. This is done by subtracting 20000 from 20110915 which gives us '20090915'.

This value will be used when testing the date of each file.


LOOP

Line 8 is the start of our loop. Here we loop for each file in the folder. The file's name is returned in the FOR-variable %%a. The loop ends at line 18.


COMPARING DATES

The FOR-loop's %%a variable returns the file's name whereas, %%~ta returns it's datestamp. This includes other information such as time. This is how it is shown on my system:

   04/01/2011 01:17 AM

Again, this may differ on your system however, as we did before, we need to extract the year, month and day parts but before we do this, we have to first assign the whole thing to a variable - in this case, I've used the short variable name 'fd':

   set fd=%%~ta

And now we can assemble the file's date in year-month-day order (as we did above) and assign it to a variable named 'FileDate'. Instead of using %fd%, we use !fd! becuase these values will need to be processed at run-time and therefore will require the line: SetLocal EnableDelayedExpansion as shown in line 2 of our code.
 
   set FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

So, from the example datestamp shown above, the variable FileDate would be set to '20110104'.

With these two dates, the current system date 20090915 (year adjusted) and the file's date 20110104, we can perform a simple integer comparison to determine whether the file's date is older (less than) the current date two years ago.


MOVING FILES

Line 12 does all the work here:

   if !FileDate! lss %TwoYears% (
      del "%%a"
   ) else (
      move "%%a" "%Destination%"
   )


Here, we compare the file's date to the date two years ago and if the file's date is older (smaller) then the file (in "%%a" - note the double-quotes in case there are spaces in the file's name) is deleted otherwise, (if the file is newer than two years ago) it is moved to your destination folder (pointed to by "%Destination%" - again, note the use of double-quotes).
 
That's all there is to it!



FINAL CONSIDERATION

If you require a visual indication that your batch file is processing the files (rather than giving you the impression the batch file has hung), you can include the following line:

  title %%a

in line 11 for example. But this only works if you run your batch file in a DOS window. In this case, you will see each file's name being processed in the window's (normally) blue titlebar.

0
 
LVL 11

Expert Comment

by:paultomasi
Comment Utility
Oops!

I just realised if you drop the avove batch file into the folder containing your files, it will move ITSELF along with your other files. One way around this is to change the file:

   for %%a in (*.*) do (

to something like:

   for %%a in (*.xls) do (

if you know the extention names of the files to be moved. In this example, all the files are .XLS files.
0
 
LVL 11

Expert Comment

by:paultomasi
Comment Utility
Try this modified batch file.

As stated above, you will need to edit the Source and Destination locations in lines 4 and 5.

Please read the notes in my earlier comment http:#36542046.


@echo off
setlocal enabledelayedexpansion

set Source=c:\temp\Source
set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (%Source%\*.*) do (
   set fd=%%~ta
   set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!
	
      if !FileDate! lss %TwoYears% (
         del "%%a"
      ) else (
         move "%%a" "%Destination%"
      )
)

Open in new window

0
 
LVL 3

Assisted Solution

by:dahesi
dahesi earned 250 total points
Comment Utility
maybe you need someone else, who say "robocopy" ^^ here i am..

cmd: robocopy "c:\source dir" "c:\target dir" /MIR /SEC

/MIR = mirror (source -> target)
/SEC = copy security informations (ntfs permissions for example)
if you use " " (spaces) in foldernames .. put the hole path into "..."

at the end you will see the result of that job.
0
 
LVL 11

Expert Comment

by:paultomasi
Comment Utility
dahesi

Yeah well, that doesn't address the the asker's question. How to MOVE files that are LESS than or EQUAL to two years old while at the same time DELETE files that are GREATER than two years old.

Oh, by the way, not all of us have Robocopy on our systems and besides, in all my years of experience, I have never had the need to use it.

(If it were possible to carve pyramids out of a single stone do you think they would have done so?)
 
0
 
LVL 3

Expert Comment

by:dahesi
Comment Utility
thx paultomasi for your info.. i didnt read the hole thread... but here we go:
another parameter for all those robocopy haters out there:

/MAXAGE:n (n = days.. it wont copy files older than n)

cmd: robocopy ..... /MAXAGE:730

here we go
0
 
LVL 11

Expert Comment

by:paultomasi
Comment Utility
dahesi

I won't say I 'hate' Robocopy... it's a powerful utility program and can save a lot of time when relied upon.

Personally, I haven't explored it's rich range of capabilities but from what bits I've seen I'm left with no doubt of it's use as a substitute to coding batch file routines.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This may not be a text book method to resolve VSS backup issues but it seemed to have worked on few of the Windows 2003 servers we had issues while performing a Volume Shadow Copy backup. If you have issues while performing a shadow copy backup usin…
Scenerio: You have a server running Server 2003 and have applied a retail pack of Terminal Server Licenses.  You want to change servers or your server has crashed and you need to reapply the Terminal Server Licenses. When you enter the 16-digit lic…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now