?
Solved

How to move directory with more than 500,000 files?

Posted on 2011-09-12
15
Medium Priority
?
637 Views
Last Modified: 2012-06-21

I have a W2k3 server with more than 500,000 files in a directory. Cant seem to open the directory without crashing Windows Explorer. Much less copy to different directory without crashing. Any suggestions? Thanks.

Intel Xenon CPU 2.80 GHz
2.80 GHz, 1.00 GB
0
Comment
Question by:fstinc
  • 6
  • 2
  • 2
  • +5
15 Comments
 
LVL 59

Expert Comment

by:Bill Prew
ID: 36526668
Have you tried a basic copy or move at a command prompt, like:

COPY "C:\FromDir\*.*" "D:\ToDir"

or

MOVE "C:\FromDir\*.*" "D:\ToDir"

You might also want to try XCOPY if you just want to copy, not move.

If those fail you could try ROBOCOPY as another option as well.

~bp
0
 

Expert Comment

by:TwynFeyr
ID: 36526676
If there are no requirements to the file move you could try the command prompt and use

Move c:\source\*.* c:\destination\

Or move the entire directory

Move c:\source c:\destination


That being said, I've never tried moving 500000 files from one directory and have no idea how the command prompt would react in this instance.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36526690
I'm sure you've realized this by now, but never ever ever put anywhere near that many files into 1 directory.
0
Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

 
LVL 43

Expert Comment

by:Steve Knight
ID: 36527450
are they just some files to delete?

If so you can remove dir from command prompt.

Start cmd.exe

cd /d c:\yourdir\to\delete
rd . /s /q

it will show error but delete files and subdirs under current directory.... Careful!

If you need to retrieve the files into different directories say we could script moving them into directories with 10000 in each say, or how about copy all files to directory based on first letter wtc.

What are the files anyway - large number of random files, temp files, log files from something?

Steve
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36528175
The problem is that Windows will want to 'log' every file before displaying the folder's contents in Explorer... With that many files it will bring your system down on it's knees.

You should consider performing a CHKDSK operation ASAP to rule out and fix filesystem errors.

DOS is the way to go. As billpreww suggests, use either COPY (or XCOPY) with the VERIFY option enabled, or MOVE.

XCOPY has the ability to resume copying (with the '/C' option) if it encounters errors whereas COPY and MOVE will just frustratingly bomb-out.

To summerise, use COPY (or better still, XCOPY) (with the VERIFY option) to copy your files then, when you are satisfied all is well, delete the files from the original location.

BUT DON'T FORGET TO DO A CHKDSK FIRST !!

0
 
LVL 8

Expert Comment

by:Amitabh Singh
ID: 36529560
0
 

Author Comment

by:fstinc
ID: 36540207
Thats nothing, theres a million+ files in another directory. XCOPY didnt do the job, but I will try it, since this was done by IT director. Not sure how he did it exactly, but Ill give it shot. I will also try Robocopy.

An application writes these small excel files for customer archiving. Theres a script to archive the directory, but it turns out, even the archive directory grew insanely. FYI: I didnt cause this. New here.

Also, turns out, we just need to delete files older than 2 years. You would think this would be easier, but its not since this is not one of DOS capabilities. I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash since previous scripts crashed when pointing to that directory.
0
 
LVL 59

Expert Comment

by:Bill Prew
ID: 36540246
If you just need to delete them you might want to give DELAGE32 a shot, it's a great tool for removing old files.

http://www.horstmuc.de/win/delage.htm

http://www.horstmuc.de/wbat32.htm

~bp
0
 
LVL 11

Accepted Solution

by:
paultomasi earned 1000 total points
ID: 36542046
>>I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash

Please post your script here...

It might be necessary to use DOS to COPY and DELETE one file at a time rather than attempt to run a command or application which 'logs' all the files before starting it's operation: COPY and XCOPY are two examples - they can take seemingly ages if you have a HUGE number of files giving you the impression the process has hung.

In my earlier comment, I assumed you would use COPY (or XCOPY) or MOVE inside a FOR-loop. You could try something like this:

@echo off
setlocal enabledelayedexpansion

set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (*.*) do (
   set fd=%%~ta
   set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

   if !FileDate! lss %TwoYears% (
      del "%%a"
   ) else (
      move "%%a" "%Destination%"
   )

)

Open in new window


It's fast because it does not rely on caching of filenames.



NOTES

DESTINATION FOLDER

Set your destination folder in line 4. Here it is set to c:\temp\destination\ so you will need to change this to point to your own destination folder.


CURRENT DATE

Line 6 simply sets the variable TwoYears to the current date minus 2 years. This is how it works:

The current date given by the system variable %DATE% is something like: 15/09/2011.

We can cleverly turn this into an ordered integer by rearranging the values for day-month-year to year-month-day ('15' '09' '2011' to '2011' '09' '15').

In this example, %date:~6,4% returns the year part of %DATE% starting at character-position 6 (0..6) comprising 4 digits ie, 2011

%date:~3,2% returns the month part of %DATE% starting at character-position 3 (0..3) comprising 2 digits ie, 09

%date:~0,2% returns the day part of %DATE% starting at character-position 0 comprising 2 digits ie, 15

The values used here may differ from yours depending how your system displays it's date with the following command:

   echo %date%

Now that we know how to get the year, month and day we can string them togehter in year, month and day order with the following command:

   set TwoYears=%date:~6,4%%date:~3,2%%date:~0,2%

So, where the current date is 15/09/2011, the variable TwoYears is set to '20110915' (leaving out the forward-slash delimiter characters)

By perfoming an arithmetic assignment using SET /A instead of just SET, we can perform arithmetic during the above assignment. In this case, we want to reduce the year value by 2. This is done by subtracting 20000 from 20110915 which gives us '20090915'.

This value will be used when testing the date of each file.


LOOP

Line 8 is the start of our loop. Here we loop for each file in the folder. The file's name is returned in the FOR-variable %%a. The loop ends at line 18.


COMPARING DATES

The FOR-loop's %%a variable returns the file's name whereas, %%~ta returns it's datestamp. This includes other information such as time. This is how it is shown on my system:

   04/01/2011 01:17 AM

Again, this may differ on your system however, as we did before, we need to extract the year, month and day parts but before we do this, we have to first assign the whole thing to a variable - in this case, I've used the short variable name 'fd':

   set fd=%%~ta

And now we can assemble the file's date in year-month-day order (as we did above) and assign it to a variable named 'FileDate'. Instead of using %fd%, we use !fd! becuase these values will need to be processed at run-time and therefore will require the line: SetLocal EnableDelayedExpansion as shown in line 2 of our code.
 
   set FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

So, from the example datestamp shown above, the variable FileDate would be set to '20110104'.

With these two dates, the current system date 20090915 (year adjusted) and the file's date 20110104, we can perform a simple integer comparison to determine whether the file's date is older (less than) the current date two years ago.


MOVING FILES

Line 12 does all the work here:

   if !FileDate! lss %TwoYears% (
      del "%%a"
   ) else (
      move "%%a" "%Destination%"
   )


Here, we compare the file's date to the date two years ago and if the file's date is older (smaller) then the file (in "%%a" - note the double-quotes in case there are spaces in the file's name) is deleted otherwise, (if the file is newer than two years ago) it is moved to your destination folder (pointed to by "%Destination%" - again, note the use of double-quotes).
 
That's all there is to it!



FINAL CONSIDERATION

If you require a visual indication that your batch file is processing the files (rather than giving you the impression the batch file has hung), you can include the following line:

  title %%a

in line 11 for example. But this only works if you run your batch file in a DOS window. In this case, you will see each file's name being processed in the window's (normally) blue titlebar.

0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36542090
Oops!

I just realised if you drop the avove batch file into the folder containing your files, it will move ITSELF along with your other files. One way around this is to change the file:

   for %%a in (*.*) do (

to something like:

   for %%a in (*.xls) do (

if you know the extention names of the files to be moved. In this example, all the files are .XLS files.
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36542161
Try this modified batch file.

As stated above, you will need to edit the Source and Destination locations in lines 4 and 5.

Please read the notes in my earlier comment http:#36542046.


@echo off
setlocal enabledelayedexpansion

set Source=c:\temp\Source
set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (%Source%\*.*) do (
   set fd=%%~ta
   set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!
	
      if !FileDate! lss %TwoYears% (
         del "%%a"
      ) else (
         move "%%a" "%Destination%"
      )
)

Open in new window

0
 
LVL 3

Assisted Solution

by:dahesi
dahesi earned 1000 total points
ID: 36542185
maybe you need someone else, who say "robocopy" ^^ here i am..

cmd: robocopy "c:\source dir" "c:\target dir" /MIR /SEC

/MIR = mirror (source -> target)
/SEC = copy security informations (ntfs permissions for example)
if you use " " (spaces) in foldernames .. put the hole path into "..."

at the end you will see the result of that job.
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36542412
dahesi

Yeah well, that doesn't address the the asker's question. How to MOVE files that are LESS than or EQUAL to two years old while at the same time DELETE files that are GREATER than two years old.

Oh, by the way, not all of us have Robocopy on our systems and besides, in all my years of experience, I have never had the need to use it.

(If it were possible to carve pyramids out of a single stone do you think they would have done so?)
 
0
 
LVL 3

Expert Comment

by:dahesi
ID: 36542488
thx paultomasi for your info.. i didnt read the hole thread... but here we go:
another parameter for all those robocopy haters out there:

/MAXAGE:n (n = days.. it wont copy files older than n)

cmd: robocopy ..... /MAXAGE:730

here we go
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36817635
dahesi

I won't say I 'hate' Robocopy... it's a powerful utility program and can save a lot of time when relied upon.

Personally, I haven't explored it's rich range of capabilities but from what bits I've seen I'm left with no doubt of it's use as a substitute to coding batch file routines.
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
While rebooting windows server 2003 server , it's showing "active directory rebuilding indices please wait" at startup. It took a little while for this process to complete and once we logged on not all the services were started so another reboot is …
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
Is your OST file inaccessible, Need to transfer OST file from one computer to another? Want to convert OST file to PST? If the answer to any of the above question is yes, then look no further. With the help of Stellar OST to PST Converter, you can e…

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question