Solved

How to move directory with more than 500,000 files?

Posted on 2011-09-12
15
623 Views
Last Modified: 2012-06-21

I have a W2k3 server with more than 500,000 files in a directory. Cant seem to open the directory without crashing Windows Explorer. Much less copy to different directory without crashing. Any suggestions? Thanks.

Intel Xenon CPU 2.80 GHz
2.80 GHz, 1.00 GB
0
Comment
Question by:fstinc
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 2
  • 2
  • +5
15 Comments
 
LVL 54

Expert Comment

by:Bill Prew
ID: 36526668
Have you tried a basic copy or move at a command prompt, like:

COPY "C:\FromDir\*.*" "D:\ToDir"

or

MOVE "C:\FromDir\*.*" "D:\ToDir"

You might also want to try XCOPY if you just want to copy, not move.

If those fail you could try ROBOCOPY as another option as well.

~bp
0
 

Expert Comment

by:TwynFeyr
ID: 36526676
If there are no requirements to the file move you could try the command prompt and use

Move c:\source\*.* c:\destination\

Or move the entire directory

Move c:\source c:\destination


That being said, I've never tried moving 500000 files from one directory and have no idea how the command prompt would react in this instance.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36526690
I'm sure you've realized this by now, but never ever ever put anywhere near that many files into 1 directory.
0
Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

 
LVL 43

Expert Comment

by:Steve Knight
ID: 36527450
are they just some files to delete?

If so you can remove dir from command prompt.

Start cmd.exe

cd /d c:\yourdir\to\delete
rd . /s /q

it will show error but delete files and subdirs under current directory.... Careful!

If you need to retrieve the files into different directories say we could script moving them into directories with 10000 in each say, or how about copy all files to directory based on first letter wtc.

What are the files anyway - large number of random files, temp files, log files from something?

Steve
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36528175
The problem is that Windows will want to 'log' every file before displaying the folder's contents in Explorer... With that many files it will bring your system down on it's knees.

You should consider performing a CHKDSK operation ASAP to rule out and fix filesystem errors.

DOS is the way to go. As billpreww suggests, use either COPY (or XCOPY) with the VERIFY option enabled, or MOVE.

XCOPY has the ability to resume copying (with the '/C' option) if it encounters errors whereas COPY and MOVE will just frustratingly bomb-out.

To summerise, use COPY (or better still, XCOPY) (with the VERIFY option) to copy your files then, when you are satisfied all is well, delete the files from the original location.

BUT DON'T FORGET TO DO A CHKDSK FIRST !!

0
 
LVL 8

Expert Comment

by:Amitabh Singh
ID: 36529560
0
 

Author Comment

by:fstinc
ID: 36540207
Thats nothing, theres a million+ files in another directory. XCOPY didnt do the job, but I will try it, since this was done by IT director. Not sure how he did it exactly, but Ill give it shot. I will also try Robocopy.

An application writes these small excel files for customer archiving. Theres a script to archive the directory, but it turns out, even the archive directory grew insanely. FYI: I didnt cause this. New here.

Also, turns out, we just need to delete files older than 2 years. You would think this would be easier, but its not since this is not one of DOS capabilities. I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash since previous scripts crashed when pointing to that directory.
0
 
LVL 54

Expert Comment

by:Bill Prew
ID: 36540246
If you just need to delete them you might want to give DELAGE32 a shot, it's a great tool for removing old files.

http://www.horstmuc.de/win/delage.htm

http://www.horstmuc.de/wbat32.htm

~bp
0
 
LVL 11

Accepted Solution

by:
paultomasi earned 250 total points
ID: 36542046
>>I wrote a script to delete files older than 2 years, but Im pretty confident that it will crash

Please post your script here...

It might be necessary to use DOS to COPY and DELETE one file at a time rather than attempt to run a command or application which 'logs' all the files before starting it's operation: COPY and XCOPY are two examples - they can take seemingly ages if you have a HUGE number of files giving you the impression the process has hung.

In my earlier comment, I assumed you would use COPY (or XCOPY) or MOVE inside a FOR-loop. You could try something like this:

@echo off
setlocal enabledelayedexpansion

set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (*.*) do (
   set fd=%%~ta
   set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

   if !FileDate! lss %TwoYears% (
      del "%%a"
   ) else (
      move "%%a" "%Destination%"
   )

)

Open in new window


It's fast because it does not rely on caching of filenames.



NOTES

DESTINATION FOLDER

Set your destination folder in line 4. Here it is set to c:\temp\destination\ so you will need to change this to point to your own destination folder.


CURRENT DATE

Line 6 simply sets the variable TwoYears to the current date minus 2 years. This is how it works:

The current date given by the system variable %DATE% is something like: 15/09/2011.

We can cleverly turn this into an ordered integer by rearranging the values for day-month-year to year-month-day ('15' '09' '2011' to '2011' '09' '15').

In this example, %date:~6,4% returns the year part of %DATE% starting at character-position 6 (0..6) comprising 4 digits ie, 2011

%date:~3,2% returns the month part of %DATE% starting at character-position 3 (0..3) comprising 2 digits ie, 09

%date:~0,2% returns the day part of %DATE% starting at character-position 0 comprising 2 digits ie, 15

The values used here may differ from yours depending how your system displays it's date with the following command:

   echo %date%

Now that we know how to get the year, month and day we can string them togehter in year, month and day order with the following command:

   set TwoYears=%date:~6,4%%date:~3,2%%date:~0,2%

So, where the current date is 15/09/2011, the variable TwoYears is set to '20110915' (leaving out the forward-slash delimiter characters)

By perfoming an arithmetic assignment using SET /A instead of just SET, we can perform arithmetic during the above assignment. In this case, we want to reduce the year value by 2. This is done by subtracting 20000 from 20110915 which gives us '20090915'.

This value will be used when testing the date of each file.


LOOP

Line 8 is the start of our loop. Here we loop for each file in the folder. The file's name is returned in the FOR-variable %%a. The loop ends at line 18.


COMPARING DATES

The FOR-loop's %%a variable returns the file's name whereas, %%~ta returns it's datestamp. This includes other information such as time. This is how it is shown on my system:

   04/01/2011 01:17 AM

Again, this may differ on your system however, as we did before, we need to extract the year, month and day parts but before we do this, we have to first assign the whole thing to a variable - in this case, I've used the short variable name 'fd':

   set fd=%%~ta

And now we can assemble the file's date in year-month-day order (as we did above) and assign it to a variable named 'FileDate'. Instead of using %fd%, we use !fd! becuase these values will need to be processed at run-time and therefore will require the line: SetLocal EnableDelayedExpansion as shown in line 2 of our code.
 
   set FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!

So, from the example datestamp shown above, the variable FileDate would be set to '20110104'.

With these two dates, the current system date 20090915 (year adjusted) and the file's date 20110104, we can perform a simple integer comparison to determine whether the file's date is older (less than) the current date two years ago.


MOVING FILES

Line 12 does all the work here:

   if !FileDate! lss %TwoYears% (
      del "%%a"
   ) else (
      move "%%a" "%Destination%"
   )


Here, we compare the file's date to the date two years ago and if the file's date is older (smaller) then the file (in "%%a" - note the double-quotes in case there are spaces in the file's name) is deleted otherwise, (if the file is newer than two years ago) it is moved to your destination folder (pointed to by "%Destination%" - again, note the use of double-quotes).
 
That's all there is to it!



FINAL CONSIDERATION

If you require a visual indication that your batch file is processing the files (rather than giving you the impression the batch file has hung), you can include the following line:

  title %%a

in line 11 for example. But this only works if you run your batch file in a DOS window. In this case, you will see each file's name being processed in the window's (normally) blue titlebar.

0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36542090
Oops!

I just realised if you drop the avove batch file into the folder containing your files, it will move ITSELF along with your other files. One way around this is to change the file:

   for %%a in (*.*) do (

to something like:

   for %%a in (*.xls) do (

if you know the extention names of the files to be moved. In this example, all the files are .XLS files.
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36542161
Try this modified batch file.

As stated above, you will need to edit the Source and Destination locations in lines 4 and 5.

Please read the notes in my earlier comment http:#36542046.


@echo off
setlocal enabledelayedexpansion

set Source=c:\temp\Source
set Destination=c:\temp\destination\

set /a TwoYears=%date:~6,4%%date:~3,2%%date:~0,2% - 20000

for %%a in (%Source%\*.*) do (
   set fd=%%~ta
   set /a FileDate=!fd:~6,4!!fd:~3,2!!fd:~0,2!
	
      if !FileDate! lss %TwoYears% (
         del "%%a"
      ) else (
         move "%%a" "%Destination%"
      )
)

Open in new window

0
 
LVL 3

Assisted Solution

by:dahesi
dahesi earned 250 total points
ID: 36542185
maybe you need someone else, who say "robocopy" ^^ here i am..

cmd: robocopy "c:\source dir" "c:\target dir" /MIR /SEC

/MIR = mirror (source -> target)
/SEC = copy security informations (ntfs permissions for example)
if you use " " (spaces) in foldernames .. put the hole path into "..."

at the end you will see the result of that job.
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36542412
dahesi

Yeah well, that doesn't address the the asker's question. How to MOVE files that are LESS than or EQUAL to two years old while at the same time DELETE files that are GREATER than two years old.

Oh, by the way, not all of us have Robocopy on our systems and besides, in all my years of experience, I have never had the need to use it.

(If it were possible to carve pyramids out of a single stone do you think they would have done so?)
 
0
 
LVL 3

Expert Comment

by:dahesi
ID: 36542488
thx paultomasi for your info.. i didnt read the hole thread... but here we go:
another parameter for all those robocopy haters out there:

/MAXAGE:n (n = days.. it wont copy files older than n)

cmd: robocopy ..... /MAXAGE:730

here we go
0
 
LVL 11

Expert Comment

by:paultomasi
ID: 36817635
dahesi

I won't say I 'hate' Robocopy... it's a powerful utility program and can save a lot of time when relied upon.

Personally, I haven't explored it's rich range of capabilities but from what bits I've seen I'm left with no doubt of it's use as a substitute to coding batch file routines.
0

Featured Post

Migrating Your Company's PCs

To keep pace with competitors, businesses must keep employees productive, and that means providing them with the latest technology. This document provides the tips and tricks you need to help you migrate an outdated PC fleet to new desktops, laptops, and tablets.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Run MS Access Module from Macro / Bat File 9 113
Windows 2003 SID Regeneration in ESXi 6.0 5 127
inplace upgrade from Windows 2003 R2 to 2012 8 141
Try / Exception 6 28
Recently, I had the need to build a standalone system to run a point-of-sale system. I’m running this on a low-voltage Atom processor, so I wanted a light-weight operating system, but still needed Windows. I chose to use Microsoft Windows Server 200…
Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial

740 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question