can't use public/private keys to get in through SSH

Which keys are you using, DSA or RSA?

What are the permissions of your .ssh folders?

What are the permissions of your key files?

ls -ld .ssh
ls -l .ssh

RSA keys

drwx------. 2 root root 4096 Mar 14 23:31 /root/.ssh
-rw-------. 1 root root 4472 Jan 28 06:44 authorized_keys

I suppose that you already have:
~/.ssh/id_rsa key on the Source Machine <-- PRIVATE KEY
And
~/.ssh/authorized_keys contains the counter part public key.


Just a quick test:  I am doing this because after many such troubleshootings, I sometimes found this silly thing.

Try dsa keys

ssh-keygen -t dsa

Copy the public key to the ~/.ssh/authorized_keys and see if that would make it work.

I can do that, but because these keys are just copied from one box to another (and they're all working on the other boxes) I don't believe the keys are actually the problem here.

This system is working on 45 other servers and all the files were copied directly to this one. I even rebuilt the server yesterday to make sure I didn't botch something with the install....

Can you log on at all if you enter your password?

If not, you may want to check your TCP wrappers.

Your IP address or hostname should be in
/etc/hosts.allow

Make sure there is no firewall (IP tables entry) blocking it

iptables -F  (on both hosts).

service iptables stop

What kind of Linux is it?

Do the other 45 systems have the same version as the one that does not ?
It looks as though the attempt is for a DSA key match
the DSA and RSA .pub belong in authorized_keys2 for SSH v2.
The other issue to check is whether this server is configured to allow Root logins.
/etc/ssh/sshd_config
PermitRootLogin yes

http://ubuntuforums.org/showthread.php?t=1314425
Check /var/log/messages to make sure selinux is not the one blocking the login.

we have a script running where you are putting in root, but it's actually logging in as you.

This system is RHEL6 and the other ones are a mixture of RHEL4, 5, 6.

The setup is identical on all boxes and I have even re-written the sshd_conf file in case there was some problem with the copy that messed it up. However, always the same outcome.

Did you try checking

iptables -F
service iptables stop

vi /etc/hosts.allow
sshd : <IP-ADDR>

Overwriting a configuration where you have different version is a rather poor approach.
Can you check the server's /var/log/messages /var/log/secure /var/log/audit/audit.log for information that might explain the issue.
try ssh -1

Try a more verbose log for ssh using the -vvv
See which protocol it is connecting with. make sure that the "user" that is presented actually has the id_dsa id_rsa and identity keys in their .ssh directory.

Look at the authorized_keys to make sure there are no extraneous characters there
cat -v authorized_keys
Were these authorized keys copied/scped?  

He already said, it is access denied.  I don't think you need any more information before you would check what is blocking the authorization.  Am I write Arnold?

iptables is stopped and you can tell that there is no problem with connecting because you go in through putting and it will say "login as:". So it's making the connection but as soon as you put in the username it's trashes it.

farzani, the checks you suggest are not an issue given the ssh connection is establish which knocks out the iptables issue.
The ssh daemon goes through the checks which knocks out the hosts.deny/hosts.allow rules if existed the sshd daemon would have dropped the connection without the added processing reflected in the verbose log.

The configuration you copied over the one included might be incomplete/incompatible.
run in sequence:
grep openssh /var/log/rpm
locate the -server
rpm --verify openssh-server
this will tell you which files are out of whack.
Do you have another RHEL 6 from which you can copy the configuration?

Is the only type of access to this system is via console?

Also, it's not overwriting with a different version. I just said we had 4,5, and 6 boxes. But I was taking the configuration from the 6 box and putting it on another 6 box. Same version, same everything.

The keys were copied in the same way that they always are from another working system. I have checked on the spaces before and there was nothing. I also tried removing the call to the .pl file we have that is being run from the authorized keys file and just using the key, but that was no go.

I have checked the /var/log/messages and secure and the only things I could find were:

Mar 15 09:57:22 server sshd[14828]: Server listening on server.domain.com port 22.
Mar 15 09:57:24 server sshd[14828]: Received signal 15; terminating.
Mar 15 09:57:25 server sshd[14846]: Server listening on server.domain.com port 22.
Mar 15 09:57:35 server sshd[14849]: Received disconnect from client_machine: 14: No supported authentication methods available

type=USER_END msg=audit(1300203001.939:49316): user pid=14935 uid=0 auid=0 ses=103 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

from another server with -vvv
==============================
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa
debug1: Offering public key: /root/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

yeah, I copied the files from another (working) RHEL6 box and then made sure the permissions were identical. The only thing I changed was the listening address in the sshd_conf file.

I can also access the system through the VM console...

rpm --verify openssh-server
S.5....T.  c /etc/ssh/sshd_config

and there is no rpm directory on that box....

sshd_config is the only one who has changed from the install.

What is issuing a restart?
Mar 15 09:57:24 server sshd[14828]: Received signal 15; terminating.
Do you have a monitor of the system that is incorrectly configure and keeps resetting SSHD?

no, that would have been me after I made a change attempting to fix this and then restarted the sshd.

I actually deleted the sshd_config file and then copied the text right from the file on the working RHEL6 box too and that was the same thing.

Just totally stumped here about how I can have an identical configuration on another identical OS and it's not working on one of them.

Could you double check the authorized_keys files in the root user's shell?
try ssh -2 -vvv
Did you also copy the server keys from one system to the other?
Try this, delete the keys in /etc/ssh/ssh_host*
The start script will regenerate new ones on start. /etc/init.d/sshd

there is a way using ssh -2 -4
-2 sets SSH protocol version2
-4 tells it to use IPv4 connection method.

Could you check the state of the system when the connection is being made since this is a VM, might it lack the resources to setup an SSH session?

the VM that this is on is even more powerful than the other one that is working so I don't 'think' that would be an issue....

same results with the -2

I copied the server keys from the working RHEL6 box and then I also made a new file and took them out of the working one and put them into the new server.

Deleted the keys and then regenerated them but still nothing....although it was nice to see a difference now when it warned me about the key being different. :)

Server keys are unique to the server and should not be copied.
Presumably you deleted this server's entry from your known_hosts files.

Could you make sure that your id_rsa.pub and your id_dsa.pub are are added into authorized_keys2 and see whether that makes a difference. Also comment out the GSS* options or disable them GSS<parameter> no
And see if that makes a difference.
The ssh connection should fall through and prompt for the password, but in your case given you are getting a deny, you may have the setup that prevents root login.

Create another user on the system and copy the authorized_keys* into its home dir/.ssh and see if ssh -vvv <newuser>@remoteserver -p <port> has issues as well.

sorry, when I said server keys I was actually meaning the user keys from the authorized_keys file...just to say that there was no way that the keys were wrong because they had been taken from a working box.

are you able while logged into the VM console, ssh -vvvv root@localhost
Are you able to get a session?
You may have other issues that is causing difficulties.

same deal where it says:

permission denied (publickey, .....)

check your /etc/nsswitch.conf passwd file?
check your /etc/pam.d/system-auth*
run system-config-authentication you may have omitted/locked out remote access

[root@server home]# system-config-authentication
Traceback (most recent call last):
  File "/usr/sbin/system-config-authentication", line 27, in <module>
    import msgarea
  File "/usr/share/authconfig/msgarea.py", line 19, in <module>
    import gtk, gobject
  File "/usr/lib/python2.6/site-packages/gtk-2.0/gtk/__init__.py", line 64, in <module>
    _init()
  File "/usr/lib/python2.6/site-packages/gtk-2.0/gtk/__init__.py", line 52, in _init
    _gtk.init_check()
RuntimeError: could not open display

oh man, it was SELinux.

Once I got that sorted out and rebooted it connected right away.

Man, kudos for sticking with this. :)

Thanks.