Solved

ksh: /usr/bin/scp: arg list too long

Posted on 2009-05-20
47
2,989 Views
Last Modified: 2013-12-21
I am getting this error when I try to get files from another server. I have created the public/private key in my local machine and copied the key into a file called authorized_keys2 in the remote server.

Please let me know what wud be the problem.
0
Comment
Question by:Umavmishra
  • 14
  • 12
  • 11
  • +2
47 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24430683
Hi,
what is the full scp command you issue?
I think there is an aterisk (*) in it somewhere which gets expanded by the local shell where it shouldn't und thus must be escaped (using a backslash \)
wmp
 
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24430728
Hi,

Instead of listing files use find and exec switch such as:

find . -name "*.jpg" -exec scp {} user@host:{} \;

So that it will find all *.jpg files in he current directory will use scp to transfer it under the same path of the remote server.

Cheers,
K.
0
 

Author Comment

by:Umavmishra
ID: 24430733
yes i do have to get all file starting with XY. hence i have given XY* as /home/june/arc*/XY*
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24430740
use /home/june/arc\*/XY\*
or put the whole string in quotes.
wmp

 
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24430757
can you show the command used? Are you copying subdirectories? if not then consider using sftp with multi file upload
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24430765
use:

find "/home/june/arc*" -name "XY*" -exec scp {} user@host:{} \;

Cheers,
K.
0
 

Author Comment

by:Umavmishra
ID: 24430891
the command is as follows

scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:/usr/app/data/arc*/XY* /usr/app/dev/rel1/
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24430904
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:"/usr/app/data/arc*/XY*" /usr/app/dev/rel1/
0
 

Author Comment

by:Umavmishra
ID: 24430935
both arc\*/XY\* as well as "/usr/app/data/arc*/XY*"  din work :(
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 24430967
OK, my mistake. Try single quotes ( '  ...   ' ) !
0
 

Author Comment

by:Umavmishra
ID: 24431000
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:'/usr/app/data/arc*/XY*' /usr/app/dev/rel1/

even the above did not work :(:(:(

By any chance has it got to do anything with my permissions?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24431055
Same error?
No, permissions are not the problem.
Do you see a chance to log in to the remote server and try from there the 'find' thing suggested above?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431082
Hi,

Why don't you try the find version instead ?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431122
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:'/usr/app/data/arc*/XY*' /usr/app/dev/rel1/

This would not work becasue -i switch means to present the remote server an identitty file for remote login. So it won't accept anything with a wilcard character. Even if it did enclosing a paramer in single quotes means don't process it and send it a sa parameter to the program and it would not mean anyyting to scp without expansion.

So if you want to copy some files then the command would be:

find "/home/june/arc*" -name "XY*" -exec \
scp -i /home/jack/.ssh/id_rsa cust.sys.app.com {} user@host:{} \;


This way the command would copy all files but also authenticate user key so that it might not ask password.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431129
The syntax as you've written is not acceptable.

0
 

Author Comment

by:Umavmishra
ID: 24431131
find "/home/june/arc*" -name "XY*" -exec scp {} user@host:{} \;

in the above command "scp {} user@host:{} \" do i need to give the local rsa key path like below?


find "/home/june/arc*" -name "XY*" -exec scp {}  /home/jack/.ssh/id_rsa :{} \  
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431134
for scp to work propeorly it should have both source and destinatin must be present it does not assume.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431143

> in the above command "scp {} user@host:{} \" do i need to give the local rsa key path like below?
Yeah you would and the -i parameter does it.

find "/home/june/arc*" -name "XY*" -exec \
scp -i /home/jack/.ssh/id_rsa cust.sys.app.com {} user@host:{} \;
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24431144
try this

tar cf - /usr/app/data/arc*/XY* | ssh user@remotesys "( cd /path/to/remote ; tar xf - )"
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431171
Ooops sorry. I was thinking htat cust.sys.app.com was the part of the d I didn't notice the space betwnn them and id. The correct command should be like that:

find "/home/june/arc*" -name "XY*" -exec scp -i /home/jack/.ssh/id_rsa {} user@host:{} \;

Here's the similar output from my system:

# find . -name "vitra*" -exec scp -i /root/.ssh/identity {} root@moose:{} \;
vitra~                                                                                            100% 2157     2.1KB/s   00:00    
vitra_doms                                                                                        100% 1341     1.3KB/s   00:00    
vitra                                                                                             100% 2156     2.1KB/s   00:00    
vitra_dom_1                                                                                       100%  680     0.7KB/s   00:00    
vitra.1                                                                                           100% 3808     3.
0
 

Author Comment

by:Umavmishra
ID: 24431206
thanks KeremE

Can you please explain me the part " {} user@host:{} \;" if you dont mind?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431319
Of course I don't::

{} means the parameter find currently finds so it is eplaced by a filename.
and later as the remote filename.

The operation has been repeated for each file that find matches.

and \; is for stopping parameter expansion meaning the terminator for the -exec expression.

Cheers,
K.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24431334

Umavmishra,
please keep in mind that you're trying to copy the files from the remote server and not onto it!
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:Umavmishra
ID: 24431472
Yes Woolmilkporc is right! the above find command will put the files to the remote server
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24431514
To repeat my quote #24431055 -

Do you see a chance to log in to the remote server and try from there the 'find' thing suggested above?
 
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24431528
why don't you try the tar command:

 ssh user@remotesys "tar cf - /usr/app/data/arc*/XY*" | ( cd /path/to/dir ; tar xf - )
0
 

Author Comment

by:Umavmishra
ID: 24431569
i cannot access the remote server
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24431585
Yes, omarfarid,
the only missing part is the identity file -
ssh  -i /home/jack/.ssh/id_rsa cust.sys.app.com "tar cf - /usr/app/data/arc*/XY*" | ( cd /usr/app/dev/rel1 ; tar xf - )
and it will be perfect!
wmp
 
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431647
> ssh  -i /home/jack/.ssh/id_rsa cust.sys.app.com "tar cf - /usr/app/data/arc*/XY*" | ( cd /usr/app/dev/rel1 ; tar xf - )

This is not correct Please omit "cust.sys.app.com" it has no connection to id_rsa ! Read my comment : #24431171
      
0
 
LVL 61

Expert Comment

by:gheist
ID: 24431718
"smitty chgsys" allows you to change argument list size.
this is serious problem in AIX.
0
 

Author Comment

by:Umavmishra
ID: 24431728
Am extreemly sorry to say ..its not working for me...I appreciate all of you guys patience in helping me out.

I strongly feel that "*" is the culprit here
0
 
LVL 61

Expert Comment

by:gheist
ID: 24431754
Escape * with backslash = \*
shell adds all files as arguments, but scp will expand internally without problem.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24431764
The 'tar' version doesn't work either? What are the error messages?
gheist, ncargs would have to be be set at the remote server, where Umavmishra doesn't have access to (see #24431569)
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24431836
> Am extreemly sorry to say ..its not working for me...I appreciate all of you guys patience in helping me out.

Wil you post the outpuut from the commantd here?

> I strongly feel that "*" is the culprit here

it can not be the culprit here because it is properly placed in double quotes.
0
 
LVL 61

Expert Comment

by:gheist
ID: 24431848
Shell expands wildcard into very long list of files.
You can avoid that.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24431874
please look at the link below for using the tar and ssh together

http://www.experts-exchange.com/Networking/Linux_Networking/Q_24391829.html?cid=236#a24334624
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24432208
Hi,
You are AIX find will not accept the path with wildcard:


Use a command like that instead:

find /home/june -name "arc* -exec find {} -name "XY*" | xargs -i {} scp -i /home/jack/.ssh/id_rsa {} user@host:{} \;

Cheers,
K.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24432228
find /home/june -name "arc* -exec find {} -name "XY*" | xargs -i {} scp -I /home/jack/.ssh/id_rsa {} user@host:{} \;

The -i should be capital i for future compatibility "i" is obsolete now
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 24432244
find /home/june -name "arc* -exec find {} -name "XY*" | xargs -i {} scp -I /home/jack/.ssh/id_rsa {} user@host:{}

You won't need the trailing \; now sorry for the inconvenience :(

P.S. Tested on AIX 5.3 :)

0
 

Author Comment

by:Umavmishra
ID: 24432326
OK One last guess, I am able to get thru if I remove the XY*. scp is having problem to address two wild cards together!

So  if the dir is like an archive<systemdate>/XY<uniquerefnumb>, we do have a problem here

just "archive*" will work but "archive*/XY*" will not work
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24432360
Yes, because of the too many arguments in the second case.
I still think the 'tar' version should do the trick. Can't yo try it?
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24432389
why don't you do a loop

for dir in archive*
do
    scp $dir/XY* remotesystem:/path/to/$dir
done

this is just an example
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24432411
it seams the question author does not like tar :)
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24432433

omarfarid, it has to be a copy from the remote system, as you correctly did it in the 'tar' version.
Your last version copies to remote!
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 24432492
I know, and I was trying to say that you may do it the other way, login to remote system and run the loop from there. Any way , I still support the tar solution.
0
 

Author Comment

by:Umavmishra
ID: 24438266
All you great fellas,

I'm getting the files if I mention only arc*. The problem arises only when i give arc*/XY*. I got access to the remote server and cud check the count of XY* files. It's more than 200,000 files. Hence all the problem we faced in solving this yesterday!

I used "rsync" today. But no luck in that too. Guess have to find an alternative design approach instead.

I really cant thank you all enough for your time and so many solution options !

Uma
0
 

Author Comment

by:Umavmishra
ID: 24438287
forgot to mention that I used
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:'/usr/app/data/arc*' /usr/app/dev/rel1/'
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
Utilizing an array to gracefully append to a list of EmailAddresses
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now