Avatar of Daniele Brunengo
Daniele Brunengo
Flag for Italy asked on

Rsync between two CENTOS servers: doesn't work without password

Hello, so I need to cron backups a few times a day from a local server to a remote server.
Both have Centos 6.0.

It's not the first time I did this, followed the usual procedure:
-created keys on the local server
-added the public key to authorized keys on remote server
-restarted sshd services
-allowed access with iptables
-checked all related folder permissions and ownership

Result: the remote server still asks for a password when I ssh from the local server.

I repeated the procedure 3 times to be sure.

I did a verbose ssh connection with
ssh root@SERVER -p PORT -vvv

Open in new window

.

Here are the results of the verbose connection.

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to SERVER [SERVER] port PORT.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /root/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 116/256
debug2: bits set: 521/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 1149
debug3: put_host_port: [SERVER]:PORT
debug3: put_host_port: [SERVER]:PORT
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host '[SERVER]:PORT' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: bits set: 492/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1165
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1213
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/identity ((nil))
debug2: key: /root/.ssh/id_rsa (0x7fc9b39b3a20)
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug3: Wrote 64 bytes for a total of 1277
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address SERVER.
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Offering public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1645
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

Open in new window


I hope somebody can help me with this.
SSH / Telnet SoftwareLinuxStorage Software

Avatar of undefined
Last Comment
Daniele Brunengo

8/22/2022 - Mon
Member_2_406981

Authorized keys on remote machine is set mode 600?
Daniele Brunengo

ASKER
Yes, authorized keys is 600 and its containing .ssh folder is 700, as should be.
Member_2_406981

how did you put the public key into the authorized_keys file? Manually or via ssh-copy-id?
maybe there are linebreaks in the public key inside the authorized file and the key is not on one single line?
Your help has saved me hundreds of hours of internet surfing.
fblack61
Daniele Brunengo

ASKER
I tried both manually and with ssh-copy-id. The key as it is now was created with ssh-copy-id and has no line breaks.
Member_2_406981

any hints in /var/log/... files on the remote end?
Daniele Brunengo

ASKER
The secure file just says password was accepted for ssh. The messages file doesn't report errors related to this.

Anyway, I think the server itself is not reading the authorized_keys file.

If I do this:
~/.ssh/authorized_keys

I get this:
Function return code: 126
bash: /root/.ssh/authorized_keys: Permission denied
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Member_2_406981

Aothorized keys is not an executable file thus you get an permission denied.

try
cat ~/.ssh/authorized_keys

as the account owner and see if it works.
Daniele Brunengo

ASKER
Yeah, you're right. And yes, it works.
Member_2_406981

is it a new machine? setup by you?

if not new or setup by others, review the /etc/sshd_config file if it is configured for public key authentication and its options.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Daniele Brunengo

ASKER
It's not new or setup by me. Anyway the sshd_config file has all default options.
Daniele Brunengo

ASKER
Specifically,
PubkeyAuthentication yes
RSAAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
Daniele Brunengo

ASKER
The ssh port is 22 on the server but I think they changed it to a non standard port using Nat on the hardware firewall. Could this be a problem? I mean, I know it's non standard because I'm connecting using that port, but it's set as 22 on the server.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
ASKER CERTIFIED SOLUTION
Member_2_406981

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Daniele Brunengo

ASKER
I tried that and it didn't work. So I tried clearing the authorized_keys file and it worked. Then I added the key from my local server and it worked again.

So I guess the problem was with the previous authorized_keys file. It starts like this:

ssh-dss

followed by a key I've been using to connect a Windows PC to the remote server and make a backup of the backup. This key has always worked, but it is somehow disrupting the other keys?
Member_2_406981

what if you put back the problematic key? Try to put back a fresh copy from the source machine, maybe this key was somehow corrupted.
Daniele Brunengo

ASKER
I'll try that. The key worked fine a few hours ago though, after all the initial tampering.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Daniele Brunengo

ASKER
The "backup of backup" machine doesn't have a fixed IP, don't know if that counts for something.
Member_2_406981

no IPs do not play a role, regarding key authentication.
Daniele Brunengo

ASKER
Tried adding it back using cygwin's console and ssh-copy-id, and I get:

WARNING: All keys were skipped because they already exist on the remote system.

How is that possible if I cleared the authorized-keys file?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Member_2_406981

wrong account/machine? to put the keys into?

if it says key already there you should be able to replace ssh-copy-id with ssh and login without password.
Member_2_406981

Or you have used one key piar on more than one machine and the other one was already added to the authorized_keys.
Daniele Brunengo

ASKER
Which I am.

I noticed something though. I have two ways of accessing the remote server. The firewall it's behind manages two WANs, so I can access with two different IPs.

The IP I've been using for the "backup of backup" is different from the IP I've been using for server to remote server backup (I hope I managed to explain this clearly).

This shouldn't change anything though, since the authorized-keys file is obviously the same.
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
Gerwin Jansen

So remote server has 2 IP addresses, can you try:

ssh root@ip_address_1 -p PORT
or
ssh root@ip_address_2 -p PORT

Do any of these 2 work? If so then I suggest you use the working IP address for your rsync.
Daniele Brunengo

ASKER
Right now everything is working, no matter what IP I use. And I know why. I randomly renamed the old, 'broken' authorized_keys file to authorized_keys2, only to discover that authorized_keys2 is a deprecated, but still working, way of listing DSA  and RSA keys. So that's what's happening. The remote server is using authorized_keys with the local server and authorized_keys2 with the Windows PC. I hope the fact I renamed the file randomly isn't lost on you. I find It hilarious.
Member_2_406981

hahahaha lol I also didnt thought of keys2 anymore. Always assuming RSA and DSA keys can be mixed. My last ssh DSA key I've seen many many years ago.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Daniele Brunengo

ASKER
The initial DSA key seems to be read fine, but the server won't read the other keys if they're in the same file. Anyway I'm closing this, all is mostly fine now.