Link to home
Start Free TrialLog in
Avatar of sdruss
sdruss

asked on

Unix How to Bulk Remove Multiple Directories from FileSystem

I need to remove directory structure that has large amounts of data (e.g. over 3TB).  For database experts want to remove, no longer needed audit and archive log files.  Doing a "rm -rf" on a very large directory tree will fail.  

Will this method work better (using Solaris 10):
  $   cd /database/Archive/2013
  $   find . -delete -name "/database/Archive/2013"  -- read that this is faster???

Need suggestions, recommendations to delete large/huge amount of files.
ASKER CERTIFIED SOLUTION
Avatar of Joseph Gan
Joseph Gan
Flag of Australia image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
try this

cd /database/Archive/2013
find . -type f -exec  rm -f {} \;
rm -rf *
Further extension to it

If you only want to delete a certain type of file or run multiple threads handling each file type,
along with an option to keep files for last n days

find $DIR -name "*.ext" -type f -mtime +$TIME2KEEP -exec rm -f {} \;
Avatar of sdruss
sdruss

ASKER

ganjos:  I like the rsync command.  I did a small test today and it seemed to work okay.  

Will this work on a directory tree with 2+ TB of files?  Multiple directories under base directory.
Yes, it will. Thanks.
Avatar of sdruss

ASKER

For testing purposes I need to create huge number of files (e.g. 3,000 files) in several different directories.  So, I would need to populate the a directory structure similiar to below:

        /database/Archive/2013/January
                            ....                   /February
                            ....                   /March
                            ....                   /April
                           ....                   /May
                            ....                  /June
                            ....                  /July
                             ....                 /August
                            ....                  /September
                           ....                   /October
                            ....                  /November
                            ....                  /December

Currently the following unix "dd" command in a loop to create large number of files:

define i count =0
if [[ ${count} lt 1000 ]]
then
     dd if=/dev/zero of=test${count} count=10"
     let "count+=1"
fi

Any ideas -need to also poplulate sub-directories above to test deleting huge number of files.
I belive the following command can delete everything in one go:

rsync -a --delete empty_dir/ database/Archive/2013/

If you want to keep the sub-directories, then you need to do something like this:

rsync -a --delete empty_dir/ database/Archive/2013/January/

...
Avatar of sdruss

ASKER

rsync -a --delete empty_dir/ database/Archive/2013/January/  

This did seem to work for subordinate directories.  My directory tree above.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of sdruss

ASKER

Thanks.  This really did the job well!