Link to home
Create AccountLog in
Avatar of sunhux
sunhux

asked on

Efficient way to copy / backup millions of small files


Would like to know how people out there are handling this where
you have millions of small files (I've seen this in logfiles, Lotus Notes
email or Outlook files) :

In the past I dread copying or backing up (using Windows Explorer
or Windows or Unix OS copy commands) to copy directories with
million over small files.

Q1:
Does Veritas Netbackup or HP Dataprotector take very long to backup
such a directory to tape ?  

Q2:
Would it be faster to use SAN or snapshot technology to just take a
quick snap of the entire SAN disk partition (where such small files
reside) or would it be more efficient to place these millions of small
files on a Solid State Disk (SSD) as SSD do not have seek/rotational
delays of conventional disks (SCSI or IDE) ?
ASKER CERTIFIED SOLUTION
Avatar of Kent Olsen
Kent Olsen
Flag of United States of America image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Small files? How small? Small enough to ZIP the lot in one single file?
Avatar of sunhux
sunhux

ASKER



Most of it are static, but about 120,000 of the files get updated or new ones
get created daily.  The small files' sizes range from 7kBytes to 90kBytes
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
What is the purpose of making the backup? To be able to revert to an earlier situation? Or to secure the current data? Could you maybe achieve the same using a RAID disk and weekly backups? Just asking...
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of sunhux

ASKER


> One, supported by most backup applications, is to perform an image backup, where you back
> up the sectors on the disk in the order they're laid out, without reference to the files or file system

Does DataProtector support the above method (image backup by sector)?  I happen to have one
though I'm not the administrator

Hi Sun,

I'm not sure that you really want an image backup.  It provides an outstanding restore point as restoring the image puts your system, including mass storage, exactly as it was at the time of the backup.  But unless you need that kind of capability, or intend to clone the image to another machine it's overkill.

There are more efficient ways to take backups.


Kent
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of sunhux

ASKER

excellent insights