Transfer data over NFS quickly

Posted on 2007-08-06
Last Modified: 2013-12-16
I need to transfer 460G over a Gigabit network as fast as possible.
We are dumping from a 1TB raid 10 SATA to a dedicated 1TB raid 5 NAS. Both have gigabit.

Being newer to linux, what is the best way to do this. It is a mail server and the server will be down.

My main questions is what commands to use. is a dump faster than a cp, etc.
Question by:chuckbeats
    LVL 7

    Accepted Solution

    I use tar. There are several questions about it:
    1. Do you have small amount of large files, or large amount of small files?
    If the latest, copy will take a while.
    Is the NAS Linux/Unix based? What device is it?
    Do you need a backup archive (can be compressed) or an exact copy of the source (full directory tree)?

    If you're into copying, I would use tar. Also, NFS mount should be done with the optimized settings - something like
    Tar command would look something like this (assuming your source is in /source and your NFS mount is in /target):
    cd /source
    tar cf - . | ( cd /target ; tar xf - )

    If the target is *nix like, you can skip NFS and try for direct netcat (or nc). Something like that:
    on the target:
    cd /where/my/data/will/be
    nc -l -p 1025 | tar xzf -

    On the source:
    cd /source
    tar czf - . | nc NAS-server 1025

    This will stress both system's CPU (since it will compress the data) however, it will decrease the amount of data to transfer on the network (as it will be compressed before sent, and extracted after being received). This is a bit risky, but it should be the fastest method available.
    LVL 34

    Assisted Solution

    by:Duncan Roe
    FWIW, it's been my experience in the past that there's no gain from using tar compression on network transfers. Depends on CPU speed versus network speed of course, and whether you are transferring many small files or a few big ones,  but with a gigabit network I suspect you're going to need an really fast CPU to gain from compression. You might want to experiment while the mailer is running (you can't use the transferred data reliably but should get a handle on what is the fastest method).
    You might want to look at rsync even if you only intend to do this transfer once, and certainly if you think you might do it again
    LVL 13

    Assisted Solution

    creating a tar file (and then stuffing it across the wire) is my favorite for one-time copies such as what you described.    Tar provides great checksumming capabilities, preserves file permissions and ownership, etc.    "simple and effective".


    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Why You Should Analyze Threat Actor TTPs

    After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

    Suggested Solutions

    The Samsung SSD 840 EVO and 840 EVO mSATA have a well-known problem with a drop in read performance. I first learned about this in an interesting thread here at Experts Exchange:…
    More or less everybody in the IT market understands the basics of Networking, however when we start talking about Storage Networks, things get a bit dizzier, and this is where I would like to help.
    Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    9 Experts available now in Live!

    Get 1:1 Help Now