Solved

tar slow on NFS

Posted on 2014-01-01
6
653 Views
Last Modified: 2014-01-21
We have many small files which stored on the NFS side. We want to tar the directory and then move these data to other NFS filesystem after two weeks. Now the problem is that the tar speed is very slow.One directory is 400M. It took 4 minutes to finish the tar command. We have many directories like this. It took a long time to tar all the files. We tested it on 3 servers and the time is almost same. If we move the data to other NFS filesystem, the tar time is reduced dramatically. But the data transferring time from filesystems also took a long time. What should I do ?
0
Comment
Question by:c11v11
  • 3
  • 2
6 Comments
 
LVL 11

Accepted Solution

by:
Technodweeb earned 500 total points
ID: 39749856
http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/

This will allow the tar command to run on the remote box as a local process and then copy the resulting file where you want it, instead of moving the 400MB worth of raw files across the network and then the resulting compressed version back again.

This could be easily scripted
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 39750921
Check you system and the NFS server network cards speeds and mode. It could be a network interface / card settings issue.
0
 

Author Comment

by:c11v11
ID: 39751114
we tested this tar command on different servers.  If my understanding is right, this should not work for my problem. We have two folders which are both mount at the NFS NAS. We need to tar one of the folders first and then move this tar file to another folder. The problem is that in that folder we have many small files, that took a long time to run the tar command. We really want to get a way to speed up the tar command.
0
Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

 
LVL 11

Expert Comment

by:Technodweeb
ID: 39751190
we tested this tar command on different servers.  If my understanding is right, this should not work for my problem.
Did you try this or did you assume it will not work?

Is the NAS box an actual computer with an operating system on it or one of those off-the-shelf drive sharing devices? If it is a computer, what operating system is it using? If not, then you are correct, it will probably not work and I should have asked my question in advance of the suggestion.
0
 

Author Comment

by:c11v11
ID: 39751260
We are using EMC NAS. I just assume it.  I noticed that I must have another server to run that command # tar zcvf - /wwwdata | ssh root@dumpserver.nixcraft.in "cat > /backup/wwwdata.tar.gz"  Can I use the same server as I am running this command ?   what is the secret behind this  if this is working?   I will try to test it.  Thank you.
0
 
LVL 11

Expert Comment

by:Technodweeb
ID: 39751272
The secret is that you must be able to ssh into the NAS device and execute the tar command via tunnel. I do not know the EMC product so you may just have to try it to know for certain.
0

Featured Post

New! My Passport Wireless Pro Wi-Fi Mobile Storage

Portable wireless storage to offload, edit, and stream anywhere.

High-capacity, wireless mobile storage designed to accompany professional photographers and videographers in the field to easily offload, edit and stream captured photos and high-definition videos.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Best way to virtualise a remote Linux server 2 52
Can't create RAID on a HP Z220 CMT, preparation for VMWare. 6 81
Hacked File Timestamps 4 51
Dell Server drives 9 38
Ever notice how you can't use a new drive in Windows without having Windows assigning a Disk Signature?  Ever have a signature collision problem (especially with Virtual Machines?)  This article is intended to help you understand what's going on and…
I previously wrote an article addressing the use of UBCD4WIN and SARDU. All are great, but I have always been an advocate of SARDU. Recently it was suggested that I go back and take a look at Easy2Boot in comparison.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now