NoMachine NX client connection fails

/var/log/messages:

Feb 27 09:41:27 ODT002 NXSERVER-3.4.0-12[10427]: User 'atle' logged in from '192.168.10.2'. 'NXLogin::set'
Feb 27 09:41:28 ODT002 NXSERVER-3.4.0-12[10427]: Selected node host:localhost with port:22 'main::selectNode'
Feb 27 09:41:28 ODT002 NXSERVER-3.4.0-12[10427]: Current selected node: localhost is in status: running  'main::selectNode'
Feb 27 09:41:28 ODT002 NXSERVER-3.4.0-12[10427]: Selected session type: unix-gnome allowed in the profile of user: atle 'NXShell::Static'
Feb 27 09:41:31 ODT002 NXSERVER-3.4.0-12[10427]: Session 'B9C5B975EF57F0C40FC95F6EF85609F7' started by user 'atle'. 'NXShell::handler_session_start'
Feb 27 09:41:31 ODT002 NXSERVER-3.4.0-12[10427]: User 'atle' from '192.168.10.2' logged out. 'NXLogin::reset'
Feb 27 09:41:31 ODT002 NXNODE-3.4.0-11[10622]: Using port '1014' on node 'ODT002' for session 'unix-gnome'. Logger::log nxnode 6238
Feb 27 09:41:31 ODT002 NXNODE-3.4.0-11[10622]: Using host from available host list: '192.168.10.101'. Logger::log nxnode 6239
Feb 27 09:41:32 ODT002 NXSERVER-3.4.0-12[10427]: ERROR: nxssh process exit with exit status: 255 and flag connected set to: [0] 'NXShell::handler_bye'
Feb 27 09:41:32 ODT002 NXSERVER-3.4.0-12[10427]: ERROR: Cannot establish ssh tunnel between nxserver and nxnode 'NXShell::handler_bye'
Feb 27 09:41:32 ODT002 NXSERVER-3.4.0-12[10427]: ERROR: Please check permissions of user's home directory on the 'NXShell::handler_bye'
Feb 27 09:41:32 ODT002 NXSERVER-3.4.0-12[10427]: ERROR: node host and file name for authorized keys set in the NX 'NXShell::handler_bye'
Feb 27 09:41:32 ODT002 NXSERVER-3.4.0-12[10427]: ERROR: node and SSHD configurations. 'NXShell::handler_bye'
Feb 27 09:42:31 ODT002 NXNODE-3.4.0-11[10622]: ERROR: run command: process: 10855 finished with: 1 Logger::log nxnode 3902
Feb 27 09:42:32 ODT002 NXNODE-3.4.0-11[10862]: ERROR: Error when monitoring session: Unable to open display 'nx/nx,options=/home/atle/.nx/C-ODT002-1014-B9C5B975EF57F0C4
0FC95F6EF85609F7/options:1014' 'NXSessionMonitor::__setSessionStatus'
Feb 27 09:42:32 ODT002 NXNODE-3.4.0-11[10862]: Directory '/home/atle/.nx/C-ODT002-1014-B9C5B975EF57F0C40FC95F6EF85609F7' renamed into '/home/atle/.nx/F-C-ODT002-1014-B9
C5B975EF57F0C40FC95F6EF85609F7' for further investigation Logger::log nxnode 6427
Feb 27 09:42:33 ODT002 NXNODE-3.4.0-11[10622]: Session 'unix-gnome' on port '1014' failed. Logger::log nxnode 6508
Feb 27 09:42:39 ODT002 NXSERVER-3.4.0-12[10857]: ERROR: NXNodeExec: Cannot kill nxssh process: No such process 'NXNodeExec::exec'
Feb 27 09:42:39 ODT002 NXSERVER-3.4.0-12[10857]: User 'atle' from '192.168.10.2' logged out. 'NXLogin::reset'

Reinstalled the whole thing, I get just about the same from /var/log/messages:

Feb 27 11:11:59 ODT002 NXSERVER-3.4.0-12[5400]: User 'atle' logged in from '192.168.10.2'. 'NXLogin::set'
Feb 27 11:12:02 ODT002 NXSERVER-3.4.0-12[5400]: Selected node host:localhost with port:22 'main::selectNode'
Feb 27 11:12:02 ODT002 NXSERVER-3.4.0-12[5400]: Current selected node: localhost is in status: running  'main::selectNode'
Feb 27 11:12:02 ODT002 NXSERVER-3.4.0-12[5400]: Selected session type: unix-gnome allowed in the profile of user: atle 'NXShell::Static'
Feb 27 11:12:04 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: nxssh process exited with '255' 'NXNodeExec::exec'
Feb 27 11:12:07 ODT002 NXSERVER-3.4.0-12[5400]: Session 'A76A3031E2F940A161A7702925DA55A8' started by user 'atle'. 'NXShell::handler_session_start'
Feb 27 11:12:07 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: run command: no child process with pid 5412 Logger::log nxserver 3127
Feb 27 11:12:07 ODT002 NXSERVER-3.4.0-12[5400]: User 'atle' from '192.168.10.2' logged out. 'NXLogin::reset'
Feb 27 11:12:07 ODT002 NXNODE-3.4.0-11[5419]: Using port '1003' on node 'ODT002' for session 'unix-gnome'. Logger::log nxnode 6238
Feb 27 11:12:07 ODT002 NXNODE-3.4.0-11[5419]: Using host from available host list: '192.168.10.101'. Logger::log nxnode 6239
Feb 27 11:12:08 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: nxssh process exit with exit status: 255 and flag connected set to: [0] 'NXShell::handler_bye'
Feb 27 11:12:08 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: Cannot establish ssh tunnel between nxserver and nxnode 'NXShell::handler_bye'
Feb 27 11:12:08 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: Please check permissions of user's home directory on the 'NXShell::handler_bye'
Feb 27 11:12:08 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: node host and file name for authorized keys set in the NX 'NXShell::handler_bye'
Feb 27 11:12:08 ODT002 NXSERVER-3.4.0-12[5400]: ERROR: node and SSHD configurations. 'NXShell::handler_bye'

is the directory protection on ~nx/.ssh directory correct?

it needs to be chmod 700 ~nx/.ssh
nx must be the owner of all files in ~nx ( ls -laR ~nx )

I have tried to give full access to anyone (chmod 777) to the whole home area of the user, without results. I have also tried what you now mention, but then I get the error that "/bin/mkdir: cannot create directory `/home/atle/.nx': Permission denied."

ssh WILL deny access to improperly secured files. So chmod 700 for .ssh WILL work anything else Won't...

Oh... ~nx is not .nx ~nx is the login directory of the nx account you connect with on the remote system.  .nx is your local config settings storage.

Looks like the home of the user nx is /usr/NX/home/nx, owned by nx and has the access rights 700.

ls -al /usr/NX/home/nx/.ssh
drwx------ 2 nx root    4096 2010-02-28 11:38 .
drwx------ 3 nx root    4096 2010-02-28 11:52 ..
-rw-r--r-- 2 nx root     668 2010-02-02 16:49 authorized_keys2
-rw-r--r-- 2 nx root     668 2010-02-02 16:49 default.id_dsa.pub
-rw-r--r-- 1 nx nogroup  782 2010-02-27 10:55 known_hosts
-rw-r--r-- 1 nx root     668 2010-02-02 16:49 restore.id_dsa.pub

All looks OK. I guess that the path /usr/NX/home/nx is set some where. I will just on the offchance set the same setup for /home/nx

Didn't seem to work.

the logon directory is specified in /etc/passwd for use nx, or equivalent account info in ldap.
The known hosts is still nogroup.

After this is the error message still the same?, as that indicated bad access to .ssh config for user nx.

Looks pretty much the same to me:
Feb 28 17:24:56 ODT002 NXSERVER-3.4.0-12[3840]: User 'atle' logged in from '192.168.10.2'. 'NXLogin::set'
Feb 28 17:24:57 ODT002 NXSERVER-3.4.0-12[3840]: Selected node host:localhost with port:22 'main::selectNode'
Feb 28 17:24:57 ODT002 NXSERVER-3.4.0-12[3840]: Current selected node: localhost is in status: running  'main::selectNode'
Feb 28 17:24:57 ODT002 NXSERVER-3.4.0-12[3840]: Selected session type: unix-gnome allowed in the profile of user: atle 'NXShell::Static'
Feb 28 17:25:00 ODT002 NXSERVER-3.4.0-12[3840]: Session 'EC5DEC7EF149509F38EAE2FD96D93F7B' started by user 'atle'. 'NXShell::handler_session_start'
Feb 28 17:25:00 ODT002 NXSERVER-3.4.0-12[3840]: User 'atle' from '192.168.10.2' logged out. 'NXLogin::reset'
Feb 28 17:25:00 ODT002 NXNODE-3.4.0-11[3856]: Using port '1012' on node 'ODT002' for session 'unix-gnome'. Logger::log nxnode 6238
Feb 28 17:25:00 ODT002 NXNODE-3.4.0-11[3856]: Using host from available host list: '192.168.10.101'. Logger::log nxnode 6239
Feb 28 17:25:01 ODT002 NXSERVER-3.4.0-12[3840]: ERROR: nxssh process exit with exit status: 255 and flag connected set to: [0] 'NXShell::handler_bye'
Feb 28 17:25:01 ODT002 NXSERVER-3.4.0-12[3840]: ERROR: Cannot establish ssh tunnel between nxserver and nxnode 'NXShell::handler_bye'
Feb 28 17:25:01 ODT002 NXSERVER-3.4.0-12[3840]: ERROR: Please check permissions of user's home directory on the 'NXShell::handler_bye'
Feb 28 17:25:01 ODT002 NXSERVER-3.4.0-12[3840]: ERROR: node host and file name for authorized keys set in the NX 'NXShell::handler_bye'
Feb 28 17:25:01 ODT002 NXSERVER-3.4.0-12[3840]: ERROR: node and SSHD configurations. 'NXShell::handler_bye'
Feb 28 17:26:00 ODT002 NXNODE-3.4.0-11[3856]: ERROR: run command: process: 3882 finished with: 1 Logger::log nxnode 3902
Feb 28 17:26:00 ODT002 NXNODE-3.4.0-11[3889]: ERROR: Error when monitoring session: Unable to open display 'nx/nx,options=/home/atle/.nx/C-ODT002-1012-EC5DEC7EF149509F38EAE2FD96D93F7B/options:1012' 'NXSessionMonitor::__setSessionStatus'
Feb 28 17:26:01 ODT002 NXNODE-3.4.0-11[3889]: Directory '/home/atle/.nx/C-ODT002-1012-EC5DEC7EF149509F38EAE2FD96D93F7B' renamed into '/home/atle/.nx/F-C-ODT002-1012-EC5DEC7EF149509F38EAE2FD96D93F7B' for further investigation Logger::log nxnode 6427
Feb 28 17:26:02 ODT002 NXNODE-3.4.0-11[3856]: Session 'unix-gnome' on port '1012' failed. Logger::log nxnode 6508

which howto did you follow?

please five a look at
http://wiki.debian.org/freenx

While I see the problem with the permissions, I think you can just follow a howto and have the result faster

I don't think freeNX is exactly the same as NoMachine NX, but I can try it.
I used NoMachine's guides: http://www.nomachine.com/installation.php
basically it is just:
dpkg -i <client package>
dpkg -i <node package>
dpkg -i <server package>

the authorized_keys[2] files in (~/.ssh & ~nx/.ssh) are both mode 600
(-rw-------)?

Is the name localhost resolvable?
ping localhost #(on the server)?

If you restrict the users that are allowed to logon in sh_config you need to allow the nx user to logon.

Has the sshd daemon be restarted after changing the /etc/sshd_config file. (or after install of nx?)

localhost does not resolve. I will update DNS with 127.0.0.1. There are no access restrictions in /etc/ssh/sshd_config other then that root is not allowed to log on via ssh. But I will specifically mark nx user as allowed. The SSH service has been restarted after NoMachine NX was installed. I will be back with the results as soon as I can do the changes mentioned here.

Yes! It worked! Thanks!