[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

Does perl copy better than linux copy command?

Posted on 2014-01-01
9
Medium Priority
?
530 Views
Last Modified: 2014-01-21
Does anyone know if the Perl copy better than Linux copy command? I mean if  the copy time would be better than Linux copy.
0
Comment
Question by:c11v11
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
  • 2
  • +1
9 Comments
 
LVL 12

Expert Comment

by:Gregory Miller
ID: 39749858
what are you trying to do and what size and quantity of files are you trying to copy?

Have you tried rsync? Very fast...
0
 
LVL 62

Expert Comment

by:gheist
ID: 39750380
there is no copy command on linux....
Especially on Linux (not on old UNIX) |cp| command can take enormous number of arguments, so there is not much efficiency gained by using perl, or compiling C program to delete ton of files.
0
 

Author Comment

by:c11v11
ID: 39751123
we have hundreds thousands files of small files ( average 16K) to copy from one NFS directory to another NFS directory on one same server. Actually it is for archive purpose.  Does the rsync work for this situation ? Does rsync support multiple process to handle the file copy?
0
Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

 
LVL 12

Accepted Solution

by:
Gregory Miller earned 1500 total points
ID: 39751154
Have you considered scripting something on the remote box to tar.gz the files into one large file and then transfer it to your archive point? This would be faster by far as the large file gets many more efficiencies during the transfer plus you have compressed them for a smaller size. There is a lot of overhead when transferring 10000 1 byte files versus 1 10000 byte file that might compress to 8000 bytes.

To answer your question, yes, rsync will do what you want.
0
 
LVL 62

Expert Comment

by:gheist
ID: 39751709
To copy lots of files from nfs use actimeo=3600 mount option on Linux
0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 39753001
In my experience, tar without gzip is faster than with it (but you must have room for the big tar file at both ends) - the gzip takes longer than the time difference to transfer compressed / uncompressed.
So I would say tar without gzip on the systems where the files actually reside and will reside is your best shot. Don't even think of running tar anywhere else.
If you don't go the tar route, check man 5 nfs to understand what gheist's suggested actimeo=3600 mount option does: pretty neat for the copy but you don't want to leave it there.
0
 
LVL 62

Expert Comment

by:gheist
ID: 39753189
tar p keeps permissions (or cpio or pax) down to numeric uid
rsync would match usernames and try to get better permissions

no need for temporary files;)

(cd src ; tar cpf - .) | ssh dst '(cd dst ;tar xpf -)
0
 

Author Comment

by:c11v11
ID: 39753281
I don't have a dst server.  source directory and dest directory are both mount at NFS side.
0
 
LVL 62

Expert Comment

by:gheist
ID: 39754508
remove 'ssh' from command line....

best if source and destination are different mounts and you can set source with actimeo=3600 (or more) and each can use differnet thread for processing in client side.

Much more optimal would be having source mounted from destination and copied over that shortcut. (ask admin of the NFS if they can help...)
0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
In this post we will learn different types of Android Layout and some basics of an Android App.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses
Course of the Month14 days, 11 hours left to enroll

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question