[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Transfer data over NFS quickly

Posted on 2007-08-06
5
Medium Priority
?
555 Views
Last Modified: 2013-12-16
I need to transfer 460G over a Gigabit network as fast as possible.
We are dumping from a 1TB raid 10 SATA to a dedicated 1TB raid 5 NAS. Both have gigabit.

Being newer to linux, what is the best way to do this. It is a mail server and the server will be down.

My main questions is what commands to use. is a dump faster than a cp, etc.
0
Comment
Question by:chuckbeats
3 Comments
 
LVL 7

Accepted Solution

by:
ezaton earned 672 total points
ID: 19642386
I use tar. There are several questions about it:
1. Do you have small amount of large files, or large amount of small files?
If the latest, copy will take a while.
Is the NAS Linux/Unix based? What device is it?
Do you need a backup archive (can be compressed) or an exact copy of the source (full directory tree)?

If you're into copying, I would use tar. Also, NFS mount should be done with the optimized settings - something like
vers=3,tcp,noint,rsize=32768,wsize=32768,actimeo=600
Tar command would look something like this (assuming your source is in /source and your NFS mount is in /target):
cd /source
tar cf - . | ( cd /target ; tar xf - )

If the target is *nix like, you can skip NFS and try for direct netcat (or nc). Something like that:
on the target:
cd /where/my/data/will/be
nc -l -p 1025 | tar xzf -

On the source:
cd /source
tar czf - . | nc NAS-server 1025

This will stress both system's CPU (since it will compress the data) however, it will decrease the amount of data to transfer on the network (as it will be compressed before sent, and extracted after being received). This is a bit risky, but it should be the fastest method available.
0
 
LVL 35

Assisted Solution

by:Duncan Roe
Duncan Roe earned 664 total points
ID: 19644469
FWIW, it's been my experience in the past that there's no gain from using tar compression on network transfers. Depends on CPU speed versus network speed of course, and whether you are transferring many small files or a few big ones,  but with a gigabit network I suspect you're going to need an really fast CPU to gain from compression. You might want to experiment while the mailer is running (you can't use the transferred data reliably but should get a handle on what is the fastest method).
You might want to look at rsync http://rsync.samba.org/ even if you only intend to do this transfer once, and certainly if you think you might do it again
0
 
LVL 13

Assisted Solution

by:Caseybea
Caseybea earned 664 total points
ID: 19647929
creating a tar file (and then stuffing it across the wire) is my favorite for one-time copies such as what you described.    Tar provides great checksumming capabilities, preserves file permissions and ownership, etc.    "simple and effective".

0

Featured Post

Get quick recovery of individual SharePoint items

Free tool – Veeam Explorer for Microsoft SharePoint, enables fast, easy restores of SharePoint sites, documents, libraries and lists — all with no agents to manage and no additional licenses to buy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
Often times it's very very easy to extend a volume on a Linux instance in AWS, but impossible to shrink it. I wanted to contribute to the experts-exchange community a way of providing a procedure that works on an AWS instance. It can also be used on…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month19 days, 21 hours left to enroll

872 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question