• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 578
  • Last Modified:

Transfer data over NFS quickly

I need to transfer 460G over a Gigabit network as fast as possible.
We are dumping from a 1TB raid 10 SATA to a dedicated 1TB raid 5 NAS. Both have gigabit.

Being newer to linux, what is the best way to do this. It is a mail server and the server will be down.

My main questions is what commands to use. is a dump faster than a cp, etc.
0
chuckbeats
Asked:
chuckbeats
3 Solutions
 
ezatonCommented:
I use tar. There are several questions about it:
1. Do you have small amount of large files, or large amount of small files?
If the latest, copy will take a while.
Is the NAS Linux/Unix based? What device is it?
Do you need a backup archive (can be compressed) or an exact copy of the source (full directory tree)?

If you're into copying, I would use tar. Also, NFS mount should be done with the optimized settings - something like
vers=3,tcp,noint,rsize=32768,wsize=32768,actimeo=600
Tar command would look something like this (assuming your source is in /source and your NFS mount is in /target):
cd /source
tar cf - . | ( cd /target ; tar xf - )

If the target is *nix like, you can skip NFS and try for direct netcat (or nc). Something like that:
on the target:
cd /where/my/data/will/be
nc -l -p 1025 | tar xzf -

On the source:
cd /source
tar czf - . | nc NAS-server 1025

This will stress both system's CPU (since it will compress the data) however, it will decrease the amount of data to transfer on the network (as it will be compressed before sent, and extracted after being received). This is a bit risky, but it should be the fastest method available.
0
 
Duncan RoeSoftware DeveloperCommented:
FWIW, it's been my experience in the past that there's no gain from using tar compression on network transfers. Depends on CPU speed versus network speed of course, and whether you are transferring many small files or a few big ones,  but with a gigabit network I suspect you're going to need an really fast CPU to gain from compression. You might want to experiment while the mailer is running (you can't use the transferred data reliably but should get a handle on what is the fastest method).
You might want to look at rsync http://rsync.samba.org/ even if you only intend to do this transfer once, and certainly if you think you might do it again
0
 
CaseybeaCommented:
creating a tar file (and then stuffing it across the wire) is my favorite for one-time copies such as what you described.    Tar provides great checksumming capabilities, preserves file permissions and ownership, etc.    "simple and effective".

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now