ksh: /usr/bin/scp: arg list too long

I am getting this error when I try to get files from another server. I have created the public/private key in my local machine and copied the key into a file called authorized_keys2 in the remote server.

Please let me know what wud be the problem.
UmavmishraAsked:
Who is Participating?
 
woolmilkporcConnect With a Mentor Commented:
OK, my mistake. Try single quotes ( '  ...   ' ) !
0
 
woolmilkporcCommented:
Hi,
what is the full scp command you issue?
I think there is an aterisk (*) in it somewhere which gets expanded by the local shell where it shouldn't und thus must be escaped (using a backslash \)
wmp
 
0
 
Kerem ERSOYPresidentCommented:
Hi,

Instead of listing files use find and exec switch such as:

find . -name "*.jpg" -exec scp {} user@host:{} \;

So that it will find all *.jpg files in he current directory will use scp to transfer it under the same path of the remote server.

Cheers,
K.
0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

 
UmavmishraAuthor Commented:
yes i do have to get all file starting with XY. hence i have given XY* as /home/june/arc*/XY*
0
 
woolmilkporcCommented:
use /home/june/arc\*/XY\*
or put the whole string in quotes.
wmp

 
0
 
omarfaridCommented:
can you show the command used? Are you copying subdirectories? if not then consider using sftp with multi file upload
0
 
Kerem ERSOYPresidentCommented:
use:

find "/home/june/arc*" -name "XY*" -exec scp {} user@host:{} \;

Cheers,
K.
0
 
UmavmishraAuthor Commented:
the command is as follows

scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:/usr/app/data/arc*/XY* /usr/app/dev/rel1/
0
 
woolmilkporcCommented:
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:"/usr/app/data/arc*/XY*" /usr/app/dev/rel1/
0
 
UmavmishraAuthor Commented:
both arc\*/XY\* as well as "/usr/app/data/arc*/XY*"  din work :(
0
 
UmavmishraAuthor Commented:
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:'/usr/app/data/arc*/XY*' /usr/app/dev/rel1/

even the above did not work :(:(:(

By any chance has it got to do anything with my permissions?
0
 
woolmilkporcCommented:
Same error?
No, permissions are not the problem.
Do you see a chance to log in to the remote server and try from there the 'find' thing suggested above?
0
 
Kerem ERSOYPresidentCommented:
Hi,

Why don't you try the find version instead ?
0
 
Kerem ERSOYPresidentCommented:
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:'/usr/app/data/arc*/XY*' /usr/app/dev/rel1/

This would not work becasue -i switch means to present the remote server an identitty file for remote login. So it won't accept anything with a wilcard character. Even if it did enclosing a paramer in single quotes means don't process it and send it a sa parameter to the program and it would not mean anyyting to scp without expansion.

So if you want to copy some files then the command would be:

find "/home/june/arc*" -name "XY*" -exec \
scp -i /home/jack/.ssh/id_rsa cust.sys.app.com {} user@host:{} \;


This way the command would copy all files but also authenticate user key so that it might not ask password.
0
 
Kerem ERSOYPresidentCommented:
The syntax as you've written is not acceptable.

0
 
UmavmishraAuthor Commented:
find "/home/june/arc*" -name "XY*" -exec scp {} user@host:{} \;

in the above command "scp {} user@host:{} \" do i need to give the local rsa key path like below?


find "/home/june/arc*" -name "XY*" -exec scp {}  /home/jack/.ssh/id_rsa :{} \  
0
 
Kerem ERSOYPresidentCommented:
for scp to work propeorly it should have both source and destinatin must be present it does not assume.
0
 
Kerem ERSOYPresidentCommented:

> in the above command "scp {} user@host:{} \" do i need to give the local rsa key path like below?
Yeah you would and the -i parameter does it.

find "/home/june/arc*" -name "XY*" -exec \
scp -i /home/jack/.ssh/id_rsa cust.sys.app.com {} user@host:{} \;
0
 
omarfaridCommented:
try this

tar cf - /usr/app/data/arc*/XY* | ssh user@remotesys "( cd /path/to/remote ; tar xf - )"
0
 
Kerem ERSOYPresidentCommented:
Ooops sorry. I was thinking htat cust.sys.app.com was the part of the d I didn't notice the space betwnn them and id. The correct command should be like that:

find "/home/june/arc*" -name "XY*" -exec scp -i /home/jack/.ssh/id_rsa {} user@host:{} \;

Here's the similar output from my system:

# find . -name "vitra*" -exec scp -i /root/.ssh/identity {} root@moose:{} \;
vitra~                                                                                            100% 2157     2.1KB/s   00:00    
vitra_doms                                                                                        100% 1341     1.3KB/s   00:00    
vitra                                                                                             100% 2156     2.1KB/s   00:00    
vitra_dom_1                                                                                       100%  680     0.7KB/s   00:00    
vitra.1                                                                                           100% 3808     3.
0
 
UmavmishraAuthor Commented:
thanks KeremE

Can you please explain me the part " {} user@host:{} \;" if you dont mind?
0
 
Kerem ERSOYPresidentCommented:
Of course I don't::

{} means the parameter find currently finds so it is eplaced by a filename.
and later as the remote filename.

The operation has been repeated for each file that find matches.

and \; is for stopping parameter expansion meaning the terminator for the -exec expression.

Cheers,
K.
0
 
woolmilkporcCommented:

Umavmishra,
please keep in mind that you're trying to copy the files from the remote server and not onto it!
0
 
UmavmishraAuthor Commented:
Yes Woolmilkporc is right! the above find command will put the files to the remote server
0
 
woolmilkporcCommented:
To repeat my quote #24431055 -

Do you see a chance to log in to the remote server and try from there the 'find' thing suggested above?
 
0
 
omarfaridCommented:
why don't you try the tar command:

 ssh user@remotesys "tar cf - /usr/app/data/arc*/XY*" | ( cd /path/to/dir ; tar xf - )
0
 
UmavmishraAuthor Commented:
i cannot access the remote server
0
 
woolmilkporcCommented:
Yes, omarfarid,
the only missing part is the identity file -
ssh  -i /home/jack/.ssh/id_rsa cust.sys.app.com "tar cf - /usr/app/data/arc*/XY*" | ( cd /usr/app/dev/rel1 ; tar xf - )
and it will be perfect!
wmp
 
0
 
Kerem ERSOYPresidentCommented:
> ssh  -i /home/jack/.ssh/id_rsa cust.sys.app.com "tar cf - /usr/app/data/arc*/XY*" | ( cd /usr/app/dev/rel1 ; tar xf - )

This is not correct Please omit "cust.sys.app.com" it has no connection to id_rsa ! Read my comment : #24431171
      
0
 
gheistCommented:
"smitty chgsys" allows you to change argument list size.
this is serious problem in AIX.
0
 
UmavmishraAuthor Commented:
Am extreemly sorry to say ..its not working for me...I appreciate all of you guys patience in helping me out.

I strongly feel that "*" is the culprit here
0
 
gheistCommented:
Escape * with backslash = \*
shell adds all files as arguments, but scp will expand internally without problem.
0
 
woolmilkporcCommented:
The 'tar' version doesn't work either? What are the error messages?
gheist, ncargs would have to be be set at the remote server, where Umavmishra doesn't have access to (see #24431569)
0
 
Kerem ERSOYPresidentCommented:
> Am extreemly sorry to say ..its not working for me...I appreciate all of you guys patience in helping me out.

Wil you post the outpuut from the commantd here?

> I strongly feel that "*" is the culprit here

it can not be the culprit here because it is properly placed in double quotes.
0
 
gheistCommented:
Shell expands wildcard into very long list of files.
You can avoid that.
0
 
omarfaridCommented:
please look at the link below for using the tar and ssh together

http://www.experts-exchange.com/Networking/Linux_Networking/Q_24391829.html?cid=236#a24334624
0
 
Kerem ERSOYPresidentCommented:
Hi,
You are AIX find will not accept the path with wildcard:


Use a command like that instead:

find /home/june -name "arc* -exec find {} -name "XY*" | xargs -i {} scp -i /home/jack/.ssh/id_rsa {} user@host:{} \;

Cheers,
K.
0
 
Kerem ERSOYPresidentCommented:
find /home/june -name "arc* -exec find {} -name "XY*" | xargs -i {} scp -I /home/jack/.ssh/id_rsa {} user@host:{} \;

The -i should be capital i for future compatibility "i" is obsolete now
0
 
Kerem ERSOYPresidentCommented:
find /home/june -name "arc* -exec find {} -name "XY*" | xargs -i {} scp -I /home/jack/.ssh/id_rsa {} user@host:{}

You won't need the trailing \; now sorry for the inconvenience :(

P.S. Tested on AIX 5.3 :)

0
 
UmavmishraAuthor Commented:
OK One last guess, I am able to get thru if I remove the XY*. scp is having problem to address two wild cards together!

So  if the dir is like an archive<systemdate>/XY<uniquerefnumb>, we do have a problem here

just "archive*" will work but "archive*/XY*" will not work
0
 
woolmilkporcCommented:
Yes, because of the too many arguments in the second case.
I still think the 'tar' version should do the trick. Can't yo try it?
0
 
omarfaridCommented:
why don't you do a loop

for dir in archive*
do
    scp $dir/XY* remotesystem:/path/to/$dir
done

this is just an example
0
 
omarfaridCommented:
it seams the question author does not like tar :)
0
 
woolmilkporcCommented:

omarfarid, it has to be a copy from the remote system, as you correctly did it in the 'tar' version.
Your last version copies to remote!
0
 
omarfaridCommented:
I know, and I was trying to say that you may do it the other way, login to remote system and run the loop from there. Any way , I still support the tar solution.
0
 
UmavmishraAuthor Commented:
All you great fellas,

I'm getting the files if I mention only arc*. The problem arises only when i give arc*/XY*. I got access to the remote server and cud check the count of XY* files. It's more than 200,000 files. Hence all the problem we faced in solving this yesterday!

I used "rsync" today. But no luck in that too. Guess have to find an alternative design approach instead.

I really cant thank you all enough for your time and so many solution options !

Uma
0
 
UmavmishraAuthor Commented:
forgot to mention that I used
scp -r -i /home/jack/.ssh/id_rsa cust.sys.app.com:'/usr/app/data/arc*' /usr/app/dev/rel1/'
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.