Rsync between two CENTOS servers: doesn't work without password

Hello, so I need to cron backups a few times a day from a local server to a remote server.
Both have Centos 6.0.

It's not the first time I did this, followed the usual procedure:
-created keys on the local server
-added the public key to authorized keys on remote server
-restarted sshd services
-allowed access with iptables
-checked all related folder permissions and ownership

Result: the remote server still asks for a password when I ssh from the local server.

I repeated the procedure 3 times to be sure.

I did a verbose ssh connection with
ssh root@SERVER -p PORT -vvv

Open in new window

.

Here are the results of the verbose connection.

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to SERVER [SERVER] port PORT.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /root/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 116/256
debug2: bits set: 521/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 1149
debug3: put_host_port: [SERVER]:PORT
debug3: put_host_port: [SERVER]:PORT
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host [SERVER]:PORT filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host '[SERVER]:PORT' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: bits set: 492/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1165
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1213
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/identity ((nil))
debug2: key: /root/.ssh/id_rsa (0x7fc9b39b3a20)
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug3: Wrote 64 bytes for a total of 1277
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address SERVER.
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_0' not found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Offering public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1645
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

Open in new window


I hope somebody can help me with this.
Daniele BrunengoIT Consultant, Web DesignerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

andreasSystem AdminCommented:
Authorized keys on remote machine is set mode 600?
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
Yes, authorized keys is 600 and its containing .ssh folder is 700, as should be.
andreasSystem AdminCommented:
how did you put the public key into the authorized_keys file? Manually or via ssh-copy-id?
maybe there are linebreaks in the public key inside the authorized file and the key is not on one single line?
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
I tried both manually and with ssh-copy-id. The key as it is now was created with ssh-copy-id and has no line breaks.
andreasSystem AdminCommented:
any hints in /var/log/... files on the remote end?
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
The secure file just says password was accepted for ssh. The messages file doesn't report errors related to this.

Anyway, I think the server itself is not reading the authorized_keys file.

If I do this:
~/.ssh/authorized_keys

I get this:
Function return code: 126
bash: /root/.ssh/authorized_keys: Permission denied
andreasSystem AdminCommented:
Aothorized keys is not an executable file thus you get an permission denied.

try
cat ~/.ssh/authorized_keys

as the account owner and see if it works.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
Yeah, you're right. And yes, it works.
andreasSystem AdminCommented:
is it a new machine? setup by you?

if not new or setup by others, review the /etc/sshd_config file if it is configured for public key authentication and its options.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
It's not new or setup by me. Anyway the sshd_config file has all default options.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
Specifically,
PubkeyAuthentication yes
RSAAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
The ssh port is 22 on the server but I think they changed it to a non standard port using Nat on the hardware firewall. Could this be a problem? I mean, I know it's non standard because I'm connecting using that port, but it's set as 22 on the server.
andreasSystem AdminCommented:
config looks ok. Port-forwards should no be a problem as long as its really a true forward.

make a local key pair for the user itself on the server and add it into own authorized_keys file
then you should be able to do

ssh account@localhost

without password. IF this works then there is some problem with either the key from the other system or with the forward.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
I tried that and it didn't work. So I tried clearing the authorized_keys file and it worked. Then I added the key from my local server and it worked again.

So I guess the problem was with the previous authorized_keys file. It starts like this:

ssh-dss

followed by a key I've been using to connect a Windows PC to the remote server and make a backup of the backup. This key has always worked, but it is somehow disrupting the other keys?
andreasSystem AdminCommented:
what if you put back the problematic key? Try to put back a fresh copy from the source machine, maybe this key was somehow corrupted.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
I'll try that. The key worked fine a few hours ago though, after all the initial tampering.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
The "backup of backup" machine doesn't have a fixed IP, don't know if that counts for something.
andreasSystem AdminCommented:
no IPs do not play a role, regarding key authentication.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
Tried adding it back using cygwin's console and ssh-copy-id, and I get:

WARNING: All keys were skipped because they already exist on the remote system.

How is that possible if I cleared the authorized-keys file?
andreasSystem AdminCommented:
wrong account/machine? to put the keys into?

if it says key already there you should be able to replace ssh-copy-id with ssh and login without password.
andreasSystem AdminCommented:
Or you have used one key piar on more than one machine and the other one was already added to the authorized_keys.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
Which I am.

I noticed something though. I have two ways of accessing the remote server. The firewall it's behind manages two WANs, so I can access with two different IPs.

The IP I've been using for the "backup of backup" is different from the IP I've been using for server to remote server backup (I hope I managed to explain this clearly).

This shouldn't change anything though, since the authorized-keys file is obviously the same.
Gerwin Jansen, EE MVETopic Advisor Commented:
So remote server has 2 IP addresses, can you try:

ssh root@ip_address_1 -p PORT
or
ssh root@ip_address_2 -p PORT

Do any of these 2 work? If so then I suggest you use the working IP address for your rsync.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
Right now everything is working, no matter what IP I use. And I know why. I randomly renamed the old, 'broken' authorized_keys file to authorized_keys2, only to discover that authorized_keys2 is a deprecated, but still working, way of listing DSA  and RSA keys. So that's what's happening. The remote server is using authorized_keys with the local server and authorized_keys2 with the Windows PC. I hope the fact I renamed the file randomly isn't lost on you. I find It hilarious.
andreasSystem AdminCommented:
hahahaha lol I also didnt thought of keys2 anymore. Always assuming RSA and DSA keys can be mixed. My last ssh DSA key I've seen many many years ago.
Daniele BrunengoIT Consultant, Web DesignerAuthor Commented:
The initial DSA key seems to be read fine, but the server won't read the other keys if they're in the same file. Anyway I'm closing this, all is mostly fine now.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
SSH / Telnet Software

From novice to tech pro — start learning today.