Rsync between two CENTOS servers: doesn't work without password
Hello, so I need to cron backups a few times a day from a local server to a remote server.
Both have Centos 6.0.
It's not the first time I did this, followed the usual procedure:
-created keys on the local server
-added the public key to authorized keys on remote server
-restarted sshd services
-allowed access with iptables
-checked all related folder permissions and ownership
Result: the remote server still asks for a password when I ssh from the local server.
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013debug1: Reading configuration data /etc/ssh/ssh_configdebug1: Applying options for *debug2: ssh_connect: needpriv 0debug1: Connecting to SERVER [SERVER] port PORT.debug1: Connection established.debug1: permanently_set_uid: 0/0debug1: identity file /root/.ssh/identity type -1debug1: identity file /root/.ssh/identity-cert type -1debug3: Not a RSA1 key file /root/.ssh/id_rsa.debug2: key_type_from_name: unknown key type '-----BEGIN'debug3: key_read: missing keytypedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug3: key_read: missing whitespacedebug2: key_type_from_name: unknown key type '-----END'debug3: key_read: missing keytypedebug1: identity file /root/.ssh/id_rsa type 1debug1: identity file /root/.ssh/id_rsa-cert type -1debug1: identity file /root/.ssh/id_dsa type -1debug1: identity file /root/.ssh/id_dsa-cert type -1debug1: identity file /root/.ssh/id_ecdsa type -1debug1: identity file /root/.ssh/id_ecdsa-cert type -1debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3debug1: match: OpenSSH_5.3 pat OpenSSH*debug1: Enabling compatibility mode for protocol 2.0debug1: Local version string SSH-2.0-OpenSSH_5.3debug2: fd 3 setting O_NONBLOCKdebug1: SSH2_MSG_KEXINIT sentdebug3: Wrote 960 bytes for a total of 981debug1: SSH2_MSG_KEXINIT receiveddebug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dssdebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.sedebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.sedebug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96debug2: kex_parse_kexinit: none,zlib@openssh.com,zlibdebug2: kex_parse_kexinit: none,zlib@openssh.com,zlibdebug2: kex_parse_kexinit:debug2: kex_parse_kexinit:debug2: kex_parse_kexinit: first_kex_follows 0debug2: kex_parse_kexinit: reserved 0debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1debug2: kex_parse_kexinit: ssh-rsa,ssh-dssdebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.sedebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.sedebug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96debug2: kex_parse_kexinit: none,zlib@openssh.comdebug2: kex_parse_kexinit: none,zlib@openssh.comdebug2: kex_parse_kexinit:debug2: kex_parse_kexinit:debug2: kex_parse_kexinit: first_kex_follows 0debug2: kex_parse_kexinit: reserved 0debug2: mac_setup: found hmac-md5debug1: kex: server->client aes128-ctr hmac-md5 nonedebug2: mac_setup: found hmac-md5debug1: kex: client->server aes128-ctr hmac-md5 nonedebug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sentdebug1: expecting SSH2_MSG_KEX_DH_GEX_GROUPdebug3: Wrote 24 bytes for a total of 1005debug2: dh_gen_key: priv key bits set: 116/256debug2: bits set: 521/1024debug1: SSH2_MSG_KEX_DH_GEX_INIT sentdebug1: expecting SSH2_MSG_KEX_DH_GEX_REPLYdebug3: Wrote 144 bytes for a total of 1149debug3: put_host_port: [SERVER]:PORTdebug3: put_host_port: [SERVER]:PORTdebug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hostsdebug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hostsdebug3: check_host_in_hostfile: match line 1debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hostsdebug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hostsdebug3: check_host_in_hostfile: match line 1debug1: Host '[SERVER]:PORT' is known and matches the RSA host key.debug1: Found key in /root/.ssh/known_hosts:1debug2: bits set: 492/1024debug1: ssh_rsa_verify: signature correctdebug2: kex_derive_keysdebug2: set_newkeys: mode 1debug1: SSH2_MSG_NEWKEYS sentdebug1: expecting SSH2_MSG_NEWKEYSdebug3: Wrote 16 bytes for a total of 1165debug2: set_newkeys: mode 0debug1: SSH2_MSG_NEWKEYS receiveddebug1: SSH2_MSG_SERVICE_REQUEST sentdebug3: Wrote 48 bytes for a total of 1213debug2: service_accept: ssh-userauthdebug1: SSH2_MSG_SERVICE_ACCEPT receiveddebug2: key: /root/.ssh/identity ((nil))debug2: key: /root/.ssh/id_rsa (0x7fc9b39b3a20)debug2: key: /root/.ssh/id_dsa ((nil))debug2: key: /root/.ssh/id_ecdsa ((nil))debug3: Wrote 64 bytes for a total of 1277debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,passworddebug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,passworddebug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,passworddebug3: authmethod_lookup gssapi-keyexdebug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,passworddebug3: authmethod_is_enabled gssapi-keyexdebug1: Next authentication method: gssapi-keyexdebug1: No valid Key exchange contextdebug2: we did not send a packet, disable methoddebug3: authmethod_lookup gssapi-with-micdebug3: remaining preferred: publickey,keyboard-interactive,passworddebug3: authmethod_is_enabled gssapi-with-micdebug1: Next authentication method: gssapi-with-micdebug3: Trying to reverse map address SERVER.debug1: Unspecified GSS failure. Minor code may provide more informationCredentials cache file '/tmp/krb5cc_0' not founddebug1: Unspecified GSS failure. Minor code may provide more informationCredentials cache file '/tmp/krb5cc_0' not founddebug1: Unspecified GSS failure. Minor code may provide more informationdebug1: Unspecified GSS failure. Minor code may provide more informationCredentials cache file '/tmp/krb5cc_0' not founddebug2: we did not send a packet, disable methoddebug3: authmethod_lookup publickeydebug3: remaining preferred: keyboard-interactive,passworddebug3: authmethod_is_enabled publickeydebug1: Next authentication method: publickeydebug1: Trying private key: /root/.ssh/identitydebug3: no such identity: /root/.ssh/identitydebug1: Offering public key: /root/.ssh/id_rsadebug3: send_pubkey_testdebug2: we sent a publickey packet, wait for replydebug3: Wrote 368 bytes for a total of 1645debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,passworddebug1: Trying private key: /root/.ssh/id_dsadebug3: no such identity: /root/.ssh/id_dsadebug1: Trying private key: /root/.ssh/id_ecdsadebug3: no such identity: /root/.ssh/id_ecdsadebug2: we did not send a packet, disable methoddebug3: authmethod_lookup passworddebug3: remaining preferred: ,passworddebug3: authmethod_is_enabled passworddebug1: Next authentication method: password
Authorized keys on remote machine is set mode 600?
Daniele Brunengo
ASKER
Yes, authorized keys is 600 and its containing .ssh folder is 700, as should be.
Member_2_406981
how did you put the public key into the authorized_keys file? Manually or via ssh-copy-id?
maybe there are linebreaks in the public key inside the authorized file and the key is not on one single line?
The ssh port is 22 on the server but I think they changed it to a non standard port using Nat on the hardware firewall. Could this be a problem? I mean, I know it's non standard because I'm connecting using that port, but it's set as 22 on the server.
I tried that and it didn't work. So I tried clearing the authorized_keys file and it worked. Then I added the key from my local server and it worked again.
So I guess the problem was with the previous authorized_keys file. It starts like this:
ssh-dss
followed by a key I've been using to connect a Windows PC to the remote server and make a backup of the backup. This key has always worked, but it is somehow disrupting the other keys?
Member_2_406981
what if you put back the problematic key? Try to put back a fresh copy from the source machine, maybe this key was somehow corrupted.
Daniele Brunengo
ASKER
I'll try that. The key worked fine a few hours ago though, after all the initial tampering.
if it says key already there you should be able to replace ssh-copy-id with ssh and login without password.
Member_2_406981
Or you have used one key piar on more than one machine and the other one was already added to the authorized_keys.
Daniele Brunengo
ASKER
Which I am.
I noticed something though. I have two ways of accessing the remote server. The firewall it's behind manages two WANs, so I can access with two different IPs.
The IP I've been using for the "backup of backup" is different from the IP I've been using for server to remote server backup (I hope I managed to explain this clearly).
This shouldn't change anything though, since the authorized-keys file is obviously the same.
ssh root@ip_address_1 -p PORT or
ssh root@ip_address_2 -p PORT
Do any of these 2 work? If so then I suggest you use the working IP address for your rsync.
Daniele Brunengo
ASKER
Right now everything is working, no matter what IP I use. And I know why. I randomly renamed the old, 'broken' authorized_keys file to authorized_keys2, only to discover that authorized_keys2 is a deprecated, but still working, way of listing DSA and RSA keys. So that's what's happening. The remote server is using authorized_keys with the local server and authorized_keys2 with the Windows PC. I hope the fact I renamed the file randomly isn't lost on you. I find It hilarious.
Member_2_406981
hahahaha lol I also didnt thought of keys2 anymore. Always assuming RSA and DSA keys can be mixed. My last ssh DSA key I've seen many many years ago.
The initial DSA key seems to be read fine, but the server won't read the other keys if they're in the same file. Anyway I'm closing this, all is mostly fine now.