asked on Rsync between two CENTOS servers: doesn't work without password
Hello, so I need to cron backups a few times a day from a local server to a remote server.
Both have Centos 6.0.
It's not the first time I did this, followed the usual procedure:
-created keys on the local server
-added the public key to authorized keys on remote server
-restarted sshd services
-allowed access with iptables
-checked all related folder permissions and ownership
Result: the remote server still asks for a password when I ssh from the local server.
I repeated the procedure 3 times to be sure.
I did a verbose ssh connection with
Here are the results of the verbose connection.
I hope somebody can help me with this.
Both have Centos 6.0.
It's not the first time I did this, followed the usual procedure:
-created keys on the local server
-added the public key to authorized keys on remote server
-restarted sshd services
-allowed access with iptables
-checked all related folder permissions and ownership
Result: the remote server still asks for a password when I ssh from the local server.
I repeated the procedure 3 times to be sure.
I did a verbose ssh connection with
ssh root@SERVER -p PORT -vvv
Here are the results of the verbose connection.
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to SERVER [SERVER] port PORT.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /root/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 116/256
debug2: bits set: 521/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 1149
debug3: put_host_port: [SERVER]:PORT
debug3: put_host_port: [SERVER]:PORT
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host '[SERVER]:PORT' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: bits set: 492/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1165
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1213
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/identity ((nil))
debug2: key: /root/.ssh/id_rsa (0x7fc9b39b3a20)
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug3: Wrote 64 bytes for a total of 1277
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address SERVER.
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found
debug1: Unspecified GSS failure. Minor code may provide more information
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Offering public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1645
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
I hope somebody can help me with this.
SSH / Telnet SoftwareLinuxStorage Software
Last Comment
Daniele Brunengo
Authorized keys on remote machine is set mode 600?
ASKER
Yes, authorized keys is 600 and its containing .ssh folder is 700, as should be.
how did you put the public key into the authorized_keys file? Manually or via ssh-copy-id?
maybe there are linebreaks in the public key inside the authorized file and the key is not on one single line?
maybe there are linebreaks in the public key inside the authorized file and the key is not on one single line?
ASKER
I tried both manually and with ssh-copy-id. The key as it is now was created with ssh-copy-id and has no line breaks.
any hints in /var/log/... files on the remote end?
ASKER
The secure file just says password was accepted for ssh. The messages file doesn't report errors related to this.
Anyway, I think the server itself is not reading the authorized_keys file.
If I do this:
~/.ssh/authorized_keys
I get this:
Function return code: 126
bash: /root/.ssh/authorized_keys
Anyway, I think the server itself is not reading the authorized_keys file.
If I do this:
~/.ssh/authorized_keys
I get this:
Function return code: 126
bash: /root/.ssh/authorized_keys
Aothorized keys is not an executable file thus you get an permission denied.
try
cat ~/.ssh/authorized_keys
as the account owner and see if it works.
try
cat ~/.ssh/authorized_keys
as the account owner and see if it works.
ASKER
Yeah, you're right. And yes, it works.
is it a new machine? setup by you?
if not new or setup by others, review the /etc/sshd_config file if it is configured for public key authentication and its options.
if not new or setup by others, review the /etc/sshd_config file if it is configured for public key authentication and its options.
ASKER
It's not new or setup by me. Anyway the sshd_config file has all default options.
ASKER
Specifically,
PubkeyAuthentication yes
RSAAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PubkeyAuthentication yes
RSAAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
ASKER
The ssh port is 22 on the server but I think they changed it to a non standard port using Nat on the hardware firewall. Could this be a problem? I mean, I know it's non standard because I'm connecting using that port, but it's set as 22 on the server.
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
I tried that and it didn't work. So I tried clearing the authorized_keys file and it worked. Then I added the key from my local server and it worked again.
So I guess the problem was with the previous authorized_keys file. It starts like this:
ssh-dss
followed by a key I've been using to connect a Windows PC to the remote server and make a backup of the backup. This key has always worked, but it is somehow disrupting the other keys?
So I guess the problem was with the previous authorized_keys file. It starts like this:
ssh-dss
followed by a key I've been using to connect a Windows PC to the remote server and make a backup of the backup. This key has always worked, but it is somehow disrupting the other keys?
what if you put back the problematic key? Try to put back a fresh copy from the source machine, maybe this key was somehow corrupted.
ASKER
I'll try that. The key worked fine a few hours ago though, after all the initial tampering.
ASKER
The "backup of backup" machine doesn't have a fixed IP, don't know if that counts for something.
no IPs do not play a role, regarding key authentication.
ASKER
Tried adding it back using cygwin's console and ssh-copy-id, and I get:
WARNING: All keys were skipped because they already exist on the remote system.
How is that possible if I cleared the authorized-keys file?
WARNING: All keys were skipped because they already exist on the remote system.
How is that possible if I cleared the authorized-keys file?
wrong account/machine? to put the keys into?
if it says key already there you should be able to replace ssh-copy-id with ssh and login without password.
if it says key already there you should be able to replace ssh-copy-id with ssh and login without password.
Or you have used one key piar on more than one machine and the other one was already added to the authorized_keys.
ASKER
Which I am.
I noticed something though. I have two ways of accessing the remote server. The firewall it's behind manages two WANs, so I can access with two different IPs.
The IP I've been using for the "backup of backup" is different from the IP I've been using for server to remote server backup (I hope I managed to explain this clearly).
This shouldn't change anything though, since the authorized-keys file is obviously the same.
I noticed something though. I have two ways of accessing the remote server. The firewall it's behind manages two WANs, so I can access with two different IPs.
The IP I've been using for the "backup of backup" is different from the IP I've been using for server to remote server backup (I hope I managed to explain this clearly).
This shouldn't change anything though, since the authorized-keys file is obviously the same.
So remote server has 2 IP addresses, can you try:
ssh root@ip_address_1 -p PORT
or
ssh root@ip_address_2 -p PORT
Do any of these 2 work? If so then I suggest you use the working IP address for your rsync.
ssh root@ip_address_1 -p PORT
or
ssh root@ip_address_2 -p PORT
Do any of these 2 work? If so then I suggest you use the working IP address for your rsync.
ASKER
Right now everything is working, no matter what IP I use. And I know why. I randomly renamed the old, 'broken' authorized_keys file to authorized_keys2, only to discover that authorized_keys2 is a deprecated, but still working, way of listing DSA and RSA keys. So that's what's happening. The remote server is using authorized_keys with the local server and authorized_keys2 with the Windows PC. I hope the fact I renamed the file randomly isn't lost on you. I find It hilarious.
hahahaha lol I also didnt thought of keys2 anymore. Always assuming RSA and DSA keys can be mixed. My last ssh DSA key I've seen many many years ago.
ASKER
The initial DSA key seems to be read fine, but the server won't read the other keys if they're in the same file. Anyway I'm closing this, all is mostly fine now.