Solved

rsync with expect

Posted on 2002-07-22
44
864 Views
Last Modified: 2010-04-21
I am using expect to anticipate a password prompt during an invocation of rsync over ssh. Unfortunately, expect prevents rsync from working properly in some way .. the rsync command has the desired effect when I run it directly, but when it is run by expect, it does nothing.

I am very new to expect, so please expect the mistake to be stupid.

Heres my expect script

#!/usr/bin/expect

set password [lindex $argv 0]


spawn rsync --recursive --rsh=ssh --size-only --archive --update --modify-window=900 --exclude='*.o' --exclude='*.ps' --exclude='*~' vil@libra1.rccp.tsukuba.ac.jp:/home/LATTICE/fellow/vil/gprogs/ /home/vil/gprogs

expect "vil@libra1.rccp.tsukuba.ac.jp's password:"
send "$password\r"
expect eof



spawn rsync --recursive --rsh=ssh --size-only --archive --modify-window=900 --exclude='*hosts*.static' --exclude='*~' vil@libra1.rccp.tsukuba.ac.jp:/home/LATTICE/fellow/vil/static/ /home/vil/static

expect "vil@libra1.rccp.tsukuba.ac.jp's password:"
send "$password\r"
expect eof


How do I need to modify this so that rsync commands work correctly under expect?
0
Comment
Question by:glebspy
  • 17
  • 14
  • 13
44 Comments
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7170402
first I suggest to use an rsa or dsa key with ssh , generate it with ssh-keygen and install the public key on the remote site's authorized_keys. The password for th ekey is empty.

About your expect script:
  did you try with a rule like:

    expect "assword:"
0
 
LVL 38

Accepted Solution

by:
yuzh earned 500 total points
ID: 7170848
You need to put the FULL PATH for rsync in your script,

eg,

spawn /usr/local/bin/rsync ........




0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7171271
damn, some answers are tooo simple
Probably I should prefix each suggestion with a bulk of "stupid question" (like: is PATH set propper?) to be proofen first ;-))
0
 
LVL 1

Author Comment

by:glebspy
ID: 7171475
Yeah!  sorry.

Can you explain to me why it failed, given that /usr/bin/rsync is the only executable rsync on the machine,
and that it didn't say "rsync not found" but rather simply
produced no results?

(There is a file
/etc/xinetd.d/rsync

but it doesnt have execute permissions for anyone.)
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7171496
probably your expect script runs in an environment where the PATH environment variable did not contaion /usr/bin, or where an executable rsync is found other than /usr/bin/rsync.

etc/xinetd.d/rsync is for the rsync server/deamon
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7171797
I have a look at the expect script, there isn't any syntax error, and rsync is a third party pachage. It is likely not
included in the default PATH.

BTW, I sometime use FULL PATH in script, and use FULL path for php, expect scripts, just for playing safe.

0
 
LVL 1

Author Comment

by:glebspy
ID: 7175709
ok sorry,

the answer didn't fix the problem after all. Basically the rsync process, or one of its subprocesses (it calls ssh to the remote machine and executes some command there) just seems to hang.expect just exits, or if I put
wait $spawn_id at the end, it just hangs.

Any thoughts?

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7175820
then it's time to reread my very first suggestion ...
0
 
LVL 1

Author Comment

by:glebspy
ID: 7176465
first I suggest to use an rsa or dsa key with ssh , generate it with ssh-keygen and install the public key on the remote site's authorized_keys. The password for th ekey is empty.

You mean, so that I don't need to type in my password? I tried this but it doesn't seem to work because my IP address is dynamically assigned by my ISP. It is different every time. Perhaps there is a fix?



About your expect script:
  did you try with a rule like:

    expect "assword:"

Sure, but like I say, it gets successfully gets past the password prompt, just seems to spontaneously shut down during execution after that.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7176681
authorized_keys does not depend on IP
Did you try?
0
 
LVL 1

Author Comment

by:glebspy
ID: 7176884
I thought I'd tried in the past, and it didnt work. I tried again, and indeed it doesn't work.

I'm not exactly sure why not, but one problem might be that I'm coming from a linux box, so the key looks like this:

ssh-dss .... rKOCf86RQKbWIL9Do4Uuec= vil@localhost.localdomain

this isn't much use to the remote machine I guess.

I tried with rsa key as well.

Any ideas?

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7178385
sounds like your usinf SSH Protocol Version 2
generate a RSA key, like:
   ssh-keygen -t rsa
0
 
LVL 1

Author Comment

by:glebspy
ID: 7178445
[vil@localhost vil]$ ssh -v
OpenSSH_3.0.2p1, SSH protocols 1.5/2.0, OpenSSL 0x0090602f
0
 
LVL 1

Author Comment

by:glebspy
ID: 7178465
I did what you said and it didnt work, sorry. Isn't the problem that theremote host doesnt know the proper hostname?
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7178489
please use

   ssh -v user@remote

and post the result
0
 
LVL 1

Author Comment

by:glebspy
ID: 7178522
[vil@localhost vil]$ ssh -v vil@libra1.rccp.tsukuba.ac.jp
OpenSSH_3.0.2p1, SSH protocols 1.5/2.0, OpenSSL 0x0090602f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Seeding random number generator
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: restore_uid
debug1: ssh_connect: getuid 500 geteuid 0 anon 1
debug1: Connecting to libra1.rccp.tsukuba.ac.jp [130.158.109.69] port 22.
debug1: temporarily_use_uid: 500/500 (e=0)
debug1: restore_uid
debug1: temporarily_use_uid: 500/500 (e=0)
debug1: restore_uid
debug1: Connection established.
debug1: read PEM private key done: type DSA
debug1: read PEM private key done: type RSA
debug1: identity file /home/vil/.ssh/identity type 0
debug1: identity file /home/vil/.ssh/id_rsa type 1
debug1: identity file /home/vil/.ssh/id_dsa type 2
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.1p1
debug1: match: OpenSSH_3.1p1 pat ^OpenSSH
debug1: Local version string SSH-1.5-OpenSSH_3.0.2p1
debug1: Waiting for server public key.
debug1: Received server public key (768 bits) and host key (1024 bits).
debug1: Host 'libra1.rccp.tsukuba.ac.jp' is known and matches the RSA1 host key.
debug1: Found key in /home/vil/.ssh/known_hosts:10
debug1: Encryption type: blowfish
debug1: Sent encrypted session key.
debug1: Installing crc compensation attack detector.
debug1: Received encrypted confirmation.
debug1: RSA authentication using agent refused.
debug1: Trying RSA authentication with key '/home/vil/.ssh/identity'
debug1: Server refused our key.
debug1: Doing challenge response authentication.
debug1: No challenge.
debug1: Doing password authentication.
vil@libra1.rccp.tsukuba.ac.jp's password:
debug1: Requesting pty.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Requesting authentication agent forwarding.
debug1: Requesting shell.
debug1: Entering interactive session.
Last login: Fri Jul 26 06:04:27 2002 from libra1
vil has logged on pts/0 from p6156-ipad02akatuka.
vil has logged on pts/1 from libra1.
vil has logged on pts/2 from rdtm07.
kawakatu has logged on pts/3 from rdtm04.
saigo has logged on pts/4 from taurus.
taisuke has logged on pts/5 from local.
vil has logged on pts/6 from p6156-ipad02akatuka.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7178532
> debug1: Trying RSA authentication with key '/home/vil/.ssh/identity'

The key in /home/vil/.ssh/identity does not match the public key your added to the remote site's autorized_keys.
0
 
LVL 1

Author Comment

by:glebspy
ID: 7178558
please tell me what to do..
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7178925
Hi  glebspy,

   Please ask EE support to help you reject the answer for
this question, and put it back to the active question are, so you can get help from more people.

   I'll try to figure out the problem when I get a chance

Regards
   yuzh
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7178947
Use FULL path for rsync , and replace "\r" with "\n"
send "$password\n"

0
 
LVL 38

Expert Comment

by:yuzh
ID: 7179010
If you want to try ahoffmann's suggestion, get rid of the ssh login password, you can try to do the followings:

1. Firstly, generate your public/private keys using ssh-keygen

% ssh-keygen -t rsa

You must use the -t option to specify that you are producing keys for SSHv2 using RSA. This will generate your id_rsa.pub and id_rsa.pub in the .ssh directory in your home directory. I strongly suggest using a passphrase.

2. Now copy the id_rsa.pub to the .ssh directory of the remote host you want to logon to as authorized_keys2 . You are basically telling the sshd daemon on the remote machine to encrypt the connection with this public key and that this key is authorized for version 2 of the ssh protocol. Try using something secure like scp for this copying.

scp ~foo/.ssh/id_rsa.pub foo@remote-server:~foo/.ssh/authorized_keys2

Your public key based authentication has been setup. You won't be asked your password on the remote machine.


0
 
LVL 1

Author Comment

by:glebspy
ID: 7179501
Yuzh I'll ask comm. suppt. to reopen the question. I'm grateful for you continuing help.

I changed \r to \n and the situation is unchanged. It gets through the password stage and starts up an rsync process, which however is screwed up and produces output which terminates in mid-line with no error message.

I did *exactly* what you said for ssh, and it still asks me for my password.

ssh -v vil@libra1.rccp.tsukuba.ac.jp (remote) produces[vil@localhost vil]$ ssh -v libra1.rccp.tsukuba.ac.jp
OpenSSH_3.0.2p1, SSH protocols 1.5/2.0, OpenSSL 0x0090602f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Seeding random number generator
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: restore_uid
debug1: ssh_connect: getuid 500 geteuid 0 anon 1
debug1: Connecting to libra1.rccp.tsukuba.ac.jp [130.158.109.69] port 22.
debug1: temporarily_use_uid: 500/500 (e=0)
debug1: restore_uid
debug1: temporarily_use_uid: 500/500 (e=0)
debug1: restore_uid
debug1: Connection established.
debug1: read PEM private key done: type DSA
debug1: read PEM private key done: type RSA
debug1: identity file /home/vil/.ssh/identity type 0
debug1: identity file /home/vil/.ssh/id_rsa type 1
debug1: identity file /home/vil/.ssh/id_dsa type 2
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.1p1
debug1: match: OpenSSH_3.1p1 pat ^OpenSSH
debug1: Local version string SSH-1.5-OpenSSH_3.0.2p1
debug1: Waiting for server public key.
debug1: Received server public key (768 bits) and host key (1024 bits).
debug1: Host 'libra1.rccp.tsukuba.ac.jp' is known and matches the RSA1 host key.
debug1: Found key in /home/vil/.ssh/known_hosts:10
debug1: Encryption type: blowfish
debug1: Sent encrypted session key.
debug1: Installing crc compensation attack detector.
debug1: Received encrypted confirmation.
debug1: RSA authentication using agent refused.
debug1: Trying RSA authentication with key '/home/vil/.ssh/identity'
debug1: Server refused our key.
debug1: Doing challenge response authentication.
debug1: No challenge.
debug1: Doing password authentication.
vil@libra1.rccp.tsukuba.ac.jp's password:
debug1: Requesting pty.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Requesting authentication agent forwarding.
debug1: Requesting shell.
debug1: Entering interactive session.
Last login: Fri Jul 26 16:51:07 2002 from p8253-ipad02akatuka.ibaraki.ocn.ne.jp

0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 51

Expert Comment

by:ahoffmann
ID: 7179503
thanks yuzh for giving a more detailed description about ssh-keygen.
Just with the scp to authorized_keys2 I do not fully aggree 'cause:
  1. it overwrites an existing file
  2. the remote server might use authorized_keys, even protocol version 2 is default

glebspy, as said in my last comment: the keys do not match.
Please check on the remote server in authorized_keys and authorized_keys2 if their are old key, probably remove them.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7179511
oops, you meanwhile posted another debug, good.

> debug1: Trying RSA authentication with key '/home/vil/.ssh/identity'
> debug1: Server refused our key.

still the same reason: keys do not match.

try following:

  ssh -1 -v ....

or

  ssh -oProtocol=1 -v ...
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7179562
Have a look at this page to see if it do anything good for you:

http://killyridols.net/rsyncssh.shtml
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7179577
I forgot to ask you to do a test.

Forget about the script, let get back to step one. Could
you please try to run rsync use ssh in command line?
of course you need to type in the password.

If this doesn't work, you might end up have to use rsh (not
secure !) to do the job.
0
 
LVL 1

Author Comment

by:glebspy
ID: 7179830
for ahoffman (yuzh see below)


[vil@localhost vil]$ ssh -1 -v libra1.rccp.tsukuba.ac.jp
OpenSSH_3.0.2p1, SSH protocols 1.5/2.0, OpenSSL 0x0090602f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Seeding random number generator
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: restore_uid
debug1: ssh_connect: getuid 500 geteuid 0 anon 1
debug1: Connecting to libra1.rccp.tsukuba.ac.jp [130.158.109.69] port 22.
debug1: temporarily_use_uid: 500/500 (e=0)
debug1: restore_uid
debug1: temporarily_use_uid: 500/500 (e=0)
debug1: restore_uid
debug1: Connection established.
debug1: read PEM private key done: type DSA
debug1: read PEM private key done: type RSA
debug1: identity file /home/vil/.ssh/identity type 0
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.1p1
debug1: match: OpenSSH_3.1p1 pat ^OpenSSH
debug1: Local version string SSH-1.5-OpenSSH_3.0.2p1
debug1: Waiting for server public key.
debug1: Received server public key (768 bits) and host key (1024 bits).
debug1: Host 'libra1.rccp.tsukuba.ac.jp' is known and matches the RSA1 host key.
debug1: Found key in /home/vil/.ssh/known_hosts:10
debug1: Encryption type: blowfish
debug1: Sent encrypted session key.
debug1: Installing crc compensation attack detector.
debug1: Received encrypted confirmation.
debug1: RSA authentication using agent refused.
debug1: Trying RSA authentication with key '/home/vil/.ssh/identity'
debug1: Server refused our key.
debug1: Doing challenge response authentication.
debug1: No challenge.
debug1: Doing password authentication.
vil@libra1.rccp.tsukuba.ac.jp's password:
debug1: Requesting pty.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Requesting authentication agent forwarding.
debug1: Requesting shell.
debug1: Entering interactive session.
Last login: Fri Jul 26 20:37:15 2002 from libra1


yuzh:
I tried running rsync using ssh from the command line, using exactly the same command that is called in my expect script. After I type in my password, it works perfectly.

Thanks very much for coming back to this question and your continued help, both of you. C.S. wouldnt let me reopen the question, so I will open another, if you both consent.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7180005
Why waste your points, just re-active the question, and get more input from other people.

We need to figure out how to make the 2 different version of ssh (v1 and v2) to work without password. It is our weekend now in Australia. I will try to figure something out on Monday if the question still outstanding, hopefully
someone can help you out by then.

Good luck and have a nice weekend.

Regards
   yuzh
0
 
LVL 1

Author Comment

by:glebspy
ID: 7180087
How do I reactivate the question..?

I'm in Japan so were almost on the same time-frame, cool hey.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7180165
Ask EE support to do it for you, deduct whatever points you gave to me for this question, and put it back to the question area. If you have problem to put it back, just create a link to the question.

Take it easy, and have a nice weekend !

=====
yuzh
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7180199
did you create new keys? did you install new keys on remote host? did you check for old/invalid keys?
It's a key problem, see the messages.
0
 
LVL 1

Author Comment

by:glebspy
ID: 7180215
did you create new keys? did you install new keys on r
remote host? did you check for old/invalid keys?

yes, yes and yes.

There were some old keys in the remote authorized_keys file, but they were for other machines, so I guess I did the right thing in not deleting them.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7186964
Hi glebspy,

   I did some testing yesterday. It look like very hard to
make 2 incompatible ssh to work together without password.

   Write a little expect script for the test:

#!/usr/local/bin/expect -f
set pwd [lindex $argv 0]

spawn /usr/local/bin/rsync --rsh=ssh --size-only --delete -az yuzh@poseidon:/home/yuzh/test1 /home/yuzh/test11

#spawn /usr/local/bin/rsync -e ssh --size-only --delete -az yuzh@poseidon:/home/yuzh/test1 /home/yuzh/test11

#expect "Password:"
expect "yuzh@poseidon's password:"
send "$pwd\n"
expect eof

  It looks like nothing wrong with the expect script, rsync can backup all my 117 small files (all < 1MB),
but when I test to do some large files (from 11MB - 57MB
a single file, rsync fail to handle it)

  It looks like we have to figure out what options to be used to run rsyn via ssh

  This is not an expect script problem any more !

  Note: there is no different to use:
        expect "Password:"
        or
        expect "yuzh@poseidon's password:"


   I hope that the above information can help.

Regards
   yuzh
 
0
 
LVL 1

Author Comment

by:glebspy
ID: 7186989
Yuzh, thanks very much for your continued help. If I understand you right, rsync just shuts down with large files, and this is related to

a) The versions of ssh on the two machines are slightly different

b) We try to circumvent ssh password prompt by using keys

c) We try to circumvent ssh password prompt using expect

some of the above.


I am way out of my depth in trying to figure out what may be done but I hope there is some solution. Which of a-c above do you think poses the most serious problem?
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7187297
> Note: there is no different to use:
WRONG.
At least case sensitive is different, to avoid this you need to use:
   expect -nocase ....

glebspy, should we continue in this thread, or in your new one?

BTW, one solution to your problem is as I already suggested: ssh with keys (I do this every day:)
0
 
LVL 1

Author Comment

by:glebspy
ID: 7187311
You can post in either thread.. I'll answer in the thread that you post. If we solve the problem together, then post in the new thread for the points.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7187346
I did another test,(without using expect script) it looks like the problem is using rsync via ssh, it hang on me when I try large files. So I believe that the problem is not the passord handling. I think it is the buffereing problem when use rsync via ssh.

0
 
LVL 38

Expert Comment

by:yuzh
ID: 7187351
Hi ahoffmann,

   Have you try to use key between differen version of ssh,
eg, ssh and openssh, ssh2 and openssh ?

   BTW, from my testing the expect script works fine, but have trouble to transfer large files.
0
 
LVL 1

Author Comment

by:glebspy
ID: 7187355
Ok yuzh but I still don't understand why rsync worked perfectly when I call it from the command line (entering my password) but the *exact same command* doesnt work when called from expect ..
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7187444
personally I'd like to continue discussion here, but no other experts may join 'cause they have to purchase this PAQ.

glebspy, I suggest that you post a short summary (suggestions + your tries & results) of this thread in the new question, then lets continue there.
0
 
LVL 1

Author Comment

by:glebspy
ID: 7187515
I agree to this in principle. Until I have a chance to do that, please continue posting in either thread as you please. When I have decided how to make an appropriate summary, I'll put a note here suggesting that we post exclusively to the new question.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 7187829
Sorry, I didn't think about this area could cost someone points.

glebspy, I totally argee with ahoffmann, please post a short summary to the new question, and we should move to
there.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7188085
yuzh, about your hanging ssh inside rsync: try to set the MTU of your interface to a lower value, for example 1400 (assuming that it is 1500 now)
0
 
LVL 1

Author Comment

by:glebspy
ID: 7189741
I made a summary of this thread at the new link. From now on, please post here:

http://www.experts-exchange.com/jsp/qManageQuestion.jsp?qid=20329801
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now