sdruss
asked on
Unix How to Bulk Remove Multiple Directories from FileSystem
I need to remove directory structure that has large amounts of data (e.g. over 3TB). For database experts want to remove, no longer needed audit and archive log files. Doing a "rm -rf" on a very large directory tree will fail.
Will this method work better (using Solaris 10):
$ cd /database/Archive/2013
$ find . -delete -name "/database/Archive/2013" -- read that this is faster???
Need suggestions, recommendations to delete large/huge amount of files.
Will this method work better (using Solaris 10):
$ cd /database/Archive/2013
$ find . -delete -name "/database/Archive/2013" -- read that this is faster???
Need suggestions, recommendations to delete large/huge amount of files.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Further extension to it
If you only want to delete a certain type of file or run multiple threads handling each file type,
along with an option to keep files for last n days
find $DIR -name "*.ext" -type f -mtime +$TIME2KEEP -exec rm -f {} \;
If you only want to delete a certain type of file or run multiple threads handling each file type,
along with an option to keep files for last n days
find $DIR -name "*.ext" -type f -mtime +$TIME2KEEP -exec rm -f {} \;
ASKER
ganjos: I like the rsync command. I did a small test today and it seemed to work okay.
Will this work on a directory tree with 2+ TB of files? Multiple directories under base directory.
Will this work on a directory tree with 2+ TB of files? Multiple directories under base directory.
Yes, it will. Thanks.
ASKER
For testing purposes I need to create huge number of files (e.g. 3,000 files) in several different directories. So, I would need to populate the a directory structure similiar to below:
/database/Archive/2013/Jan uary
.... /February
.... /March
.... /April
.... /May
.... /June
.... /July
.... /August
.... /September
.... /October
.... /November
.... /December
Currently the following unix "dd" command in a loop to create large number of files:
define i count =0
if [[ ${count} lt 1000 ]]
then
dd if=/dev/zero of=test${count} count=10"
let "count+=1"
fi
Any ideas -need to also poplulate sub-directories above to test deleting huge number of files.
/database/Archive/2013/Jan
.... /February
.... /March
.... /April
.... /May
.... /June
.... /July
.... /August
.... /September
.... /October
.... /November
.... /December
Currently the following unix "dd" command in a loop to create large number of files:
define i count =0
if [[ ${count} lt 1000 ]]
then
dd if=/dev/zero of=test${count} count=10"
let "count+=1"
fi
Any ideas -need to also poplulate sub-directories above to test deleting huge number of files.
I belive the following command can delete everything in one go:
rsync -a --delete empty_dir/ database/Archive/2013/
If you want to keep the sub-directories, then you need to do something like this:
rsync -a --delete empty_dir/ database/Archive/2013/Janu ary/
...
rsync -a --delete empty_dir/ database/Archive/2013/
If you want to keep the sub-directories, then you need to do something like this:
rsync -a --delete empty_dir/ database/Archive/2013/Janu
...
ASKER
rsync -a --delete empty_dir/ database/Archive/2013/Janu ary/
This did seem to work for subordinate directories. My directory tree above.
This did seem to work for subordinate directories. My directory tree above.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks. This really did the job well!
cd /database/Archive/2013
find . -type f -exec rm -f {} \;
rm -rf *