sunhux
asked on
Efficient way to copy / backup millions of small files
Would like to know how people out there are handling this where
you have millions of small files (I've seen this in logfiles, Lotus Notes
email or Outlook files) :
In the past I dread copying or backing up (using Windows Explorer
or Windows or Unix OS copy commands) to copy directories with
million over small files.
Q1:
Does Veritas Netbackup or HP Dataprotector take very long to backup
such a directory to tape ?
Q2:
Would it be faster to use SAN or snapshot technology to just take a
quick snap of the entire SAN disk partition (where such small files
reside) or would it be more efficient to place these millions of small
files on a Solid State Disk (SSD) as SSD do not have seek/rotational
delays of conventional disks (SCSI or IDE) ?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Sjef Bosman
Small files? How small? Small enough to ZIP the lot in one single file?
ASKER
Most of it are static, but about 120,000 of the files get updated or new ones
get created daily. The small files' sizes range from 7kBytes to 90kBytes
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
What is the purpose of making the backup? To be able to revert to an earlier situation? Or to secure the current data? Could you maybe achieve the same using a RAID disk and weekly backups? Just asking...
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
> One, supported by most backup applications, is to perform an image backup, where you back
> up the sectors on the disk in the order they're laid out, without reference to the files or file system
Does DataProtector support the above method (image backup by sector)? I happen to have one
though I'm not the administrator
Hi Sun,
I'm not sure that you really want an image backup. It provides an outstanding restore point as restoring the image puts your system, including mass storage, exactly as it was at the time of the backup. But unless you need that kind of capability, or intend to clone the image to another machine it's overkill.
There are more efficient ways to take backups.
Kent
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
excellent insights