Cluster node won't come online

hi all,
I have a two node Windows Server 2012 cluster and cant bring both nodes online, only one node at the time can be online.
Errors in system log when trying to start cluster service are eventid 1573(Node 'host01' failed to form a cluster. This was because the witness was not accessible) following with eventid 1570(Node 'host01' failed to establish a communication session while joining the cluster. This was due to an authorization failure)

part of the cluster log of currently offline host


000007c0.00000f3c::2015/04/18-09:30:14.667 INFO  [NODE] Node 1: New join with n2: stage: 'Authorize Initial Connection' status HrError(0x8009030c) reason: '[SV] Authorization failed'
000007c0.00000f3c::2015/04/18-09:30:14.667 DBG   [CHANNEL 10.12.12.143:~3343~] Close().
000007c0.00000f3c::2015/04/18-09:30:14.667 WARN  cxl::ConnectWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
000007c0.00000f3c::2015/04/18-09:30:14.667 DBG   [HM] Connection attempt to host02 failed with error ERROR_DISK_REPAIR_DISABLED(780): Failed to post-process connection to remote endpoint 10.12.12.143:~3343~.
000007c0.00000d94::2015/04/18-09:30:17.652 DBG   [JPM] Node 1: contacts size for node host02 is 4, current index 1
000007c0.00000d94::2015/04/18-09:30:17.652 DBG   [JPM] Node 1: Trying to connect to node host02 (IP: 12.0.0.2:~0~)
000007c0.00000d94::2015/04/18-09:30:17.652 DBG   [HM] Trying to connect to host02 at 12.0.0.2:~3343~
000007c0.00000d94::2015/04/18-09:30:17.653 INFO  [CONNECT] 12.0.0.2:~3343~: Established connection to remote endpoint 12.0.0.2:~3343~.
000007c0.00000d94::2015/04/18-09:30:17.653 INFO  [SV] New real route: local (12.0.0.1:~56724~) to remote host02 (12.0.0.2:~3343~).
000007c0.00000d94::2015/04/18-09:30:17.653 INFO  [SV] Got a new outgoing stream to host02 at 12.0.0.2:~3343~
000007c0.00000d94::2015/04/18-09:30:17.653 DBG   [SM] Joiner: Initialized with SPN = host02, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
000007c0.00000c60::2015/04/18-09:30:17.653 DBG   [SM] Handling auth handshake posted by thread id 3476
000007c0.00000c60::2015/04/18-09:30:17.654 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1535, service principal name HOST/host02, auth type MSG_AUTH_PACKAGE::KerberosAuth
000007c0.00000c60::2015/04/18-09:30:17.654 DBG   [SM] Joiner: Sending SSPI blob of size 1535 to Sponsor
000007c0.00000c60::2015/04/18-09:30:17.655 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 155, auth type: MSG_AUTH_PACKAGE::CurrentAuth
000007c0.00000c60::2015/04/18-09:30:17.655 DBG   [SM] Joiner: ISC returned status = 0 output Blob size 0, service principal name HOST/host02, auth type MSG_AUTH_PACKAGE::KerberosAuth
000007c0.00000c60::2015/04/18-09:30:17.655 DBG   [SM] Joiner: SSPI blob - nothing to send to Sponsor
000007c0.00000c60::2015/04/18-09:30:17.655 DBG   [SM] Joiner: Sending synchronize packet to Sponsor
000007c0.00000c60::2015/04/18-09:30:17.655 DBG   [SM] Joiner: Received packet from the Sponsor during synchronization: size 0, auth type: MSG_AUTH_PACKAGE::Synchronize
000007c0.00000d94::2015/04/18-09:30:17.669 WARN  [ClRtl] Cannot open crypto container (error 2148073494). Giving up.
000007c0.00000d94::2015/04/18-09:30:17.669 INFO  [NODE] Node 1: New join with n2: stage: 'Authorize Initial Connection' status HrError(0x8009030c) reason: '[SV] Authorization failed'
000007c0.00000d94::2015/04/18-09:30:17.669 DBG   [CHANNEL 12.0.0.2:~3343~] Close().
000007c0.00000d94::2015/04/18-09:30:17.669 WARN  cxl::ConnectWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
000007c0.00000d94::2015/04/18-09:30:17.669 DBG   [HM] Connection attempt to host02 failed with error ERROR_DISK_REPAIR_DISABLED(780): Failed to post-process connection to remote endpoint 12.0.0.2:~3343~.
000007c0.00000f3c::2015/04/18-09:30:20.653 DBG   [JPM] Node 1: contacts size for node host02 is 4, current index 2
000007c0.00000f3c::2015/04/18-09:30:20.653 DBG   [JPM] Node 1: Trying to connect to node host02 (IP: 11.0.0.22:~0~)
000007c0.00000f3c::2015/04/18-09:30:20.653 DBG   [HM] Trying to connect to host02 at 11.0.0.22:~3343~
000007c0.00000f3c::2015/04/18-09:30:20.653 INFO  [CONNECT] 11.0.0.22:~3343~: Established connection to remote endpoint 11.0.0.22:~3343~.
000007c0.00000f3c::2015/04/18-09:30:20.653 INFO  [SV] New real route: local (11.0.0.21:~56725~) to remote host02 (11.0.0.22:~3343~).
000007c0.00000f3c::2015/04/18-09:30:20.653 INFO  [SV] Got a new outgoing stream to host02 at 11.0.0.22:~3343~
000007c0.00000f3c::2015/04/18-09:30:20.654 DBG   [SM] Joiner: Initialized with SPN = host02, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
000007c0.00000c60::2015/04/18-09:30:20.654 DBG   [SM] Handling auth handshake posted by thread id 3900
000007c0.00000c60::2015/04/18-09:30:20.654 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1535, service principal name HOST/host02, auth type MSG_AUTH_PACKAGE::KerberosAuth
000007c0.00000c60::2015/04/18-09:30:20.655 DBG   [SM] Joiner: Sending SSPI blob of size 1535 to Sponsor
000007c0.00000c60::2015/04/18-09:30:20.656 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 155, auth type: MSG_AUTH_PACKAGE::CurrentAuth
000007c0.00000c60::2015/04/18-09:30:20.656 DBG   [SM] Joiner: ISC returned status = 0 output Blob size 0, service principal name HOST/host02, auth type MSG_AUTH_PACKAGE::KerberosAuth
000007c0.00000c60::2015/04/18-09:30:20.656 DBG   [SM] Joiner: SSPI blob - nothing to send to Sponsor
000007c0.00000c60::2015/04/18-09:30:20.656 DBG   [SM] Joiner: Sending synchronize packet to Sponsor
000007c0.00000c60::2015/04/18-09:30:20.656 DBG   [SM] Joiner: Received packet from the Sponsor during synchronization: size 0, auth type: MSG_AUTH_PACKAGE::Synchronize
000007c0.00000f3c::2015/04/18-09:30:20.670 WARN  [ClRtl] Cannot open crypto container (error 2148073494). Giving up.
000007c0.00000f3c::2015/04/18-09:30:20.670 INFO  [NODE] Node 1: New join with n2: stage: 'Authorize Initial Connection' status HrError(0x8009030c) reason: '[SV] Authorization failed'
000007c0.00000f3c::2015/04/18-09:30:20.670 DBG   [CHANNEL 11.0.0.22:~3343~] Close().
000007c0.00000f3c::2015/04/18-09:30:20.670 WARN  cxl::ConnectWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
000007c0.00000f3c::2015/04/18-09:30:20.670 DBG   [HM] Connection attempt to host02 failed with error ERROR_DISK_REPAIR_DISABLED(780): Failed to post-process connection to remote endpoint 11.0.0.22:~3343~.
000007c0.00000d94::2015/04/18-09:30:23.656 DBG   [JPM] Node 1: contacts size for node host02 is 4, current index 3
000007c0.00000d94::2015/04/18-09:30:23.656 DBG   [JPM] Node 1: Trying to connect to node host02 (IP: 11.1.0.22:~0~)
000007c0.00000d94::2015/04/18-09:30:23.656 DBG   [HM] Trying to connect to host02 at 11.1.0.22:~3343~
000007c0.00000d94::2015/04/18-09:30:23.656 INFO  [CONNECT] 11.1.0.22:~3343~: Established connection to remote endpoint 11.1.0.22:~3343~.
000007c0.00000d94::2015/04/18-09:30:23.657 INFO  [SV] New real route: local (11.1.0.21:~56726~) to remote host02 (11.1.0.22:~3343~).
000007c0.00000d94::2015/04/18-09:30:23.657 INFO  [SV] Got a new outgoing stream to host02 at 11.1.0.22:~3343~
000007c0.00000d94::2015/04/18-09:30:23.657 DBG   [SM] Joiner: Initialized with SPN = host02, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
000007c0.00000c60::2015/04/18-09:30:23.657 DBG   [SM] Handling auth handshake posted by thread id 3476
000007c0.00000c60::2015/04/18-09:30:23.658 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1535, service principal name HOST/host02, auth type MSG_AUTH_PACKAGE::KerberosAuth
000007c0.00000c60::2015/04/18-09:30:23.658 DBG   [SM] Joiner: Sending SSPI blob of size 1535 to Sponsor
000007c0.00000c60::2015/04/18-09:30:23.664 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 155, auth type: MSG_AUTH_PACKAGE::CurrentAuth
000007c0.00000c60::2015/04/18-09:30:23.664 DBG   [SM] Joiner: ISC returned status = 0 output Blob size 0, service principal name HOST/host02, auth type MSG_AUTH_PACKAGE::KerberosAuth
000007c0.00000c60::2015/04/18-09:30:23.664 DBG   [SM] Joiner: SSPI blob - nothing to send to Sponsor
000007c0.00000c60::2015/04/18-09:30:23.664 DBG   [SM] Joiner: Sending synchronize packet to Sponsor
000007c0.00000c60::2015/04/18-09:30:23.664 DBG   [SM] Joiner: Received packet from the Sponsor during synchronization: size 0, auth type: MSG_AUTH_PACKAGE::Synchronize
000007c0.00000d94::2015/04/18-09:30:23.678 WARN  [ClRtl] Cannot open crypto container (error 2148073494). Giving up.
000007c0.00000d94::2015/04/18-09:30:23.678 INFO  [NODE] Node 1: New join with n2: stage: 'Authorize Initial Connection' status HrError(0x8009030c) reason: '[SV] Authorization failed'
000007c0.00000d94::2015/04/18-09:30:23.678 DBG   [CHANNEL 11.1.0.22:~3343~] Close().
000007c0.00000d94::2015/04/18-09:30:23.678 WARN  cxl::ConnectWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
000007c0.00000d94::2015/04/18-09:30:23.678 DBG   [HM] Connection attempt to host02 failed with error ERROR_DISK_REPAIR_DISABLED(780): Failed to post-process connection to remote endpoint 11.1.0.22:~3343~.
000007c0.00000cb8::2015/04/18-09:30:29.068 INFO  [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 10.12.12.143:~57180~.
000007c0.00000f3c::2015/04/18-09:30:29.069 INFO  [SV] New real route: local (10.12.12.142:~3343~) to remote  (10.12.12.143:~57180~).
000007c0.00000f3c::2015/04/18-09:30:29.069 INFO  [SV] Got a new incoming stream from 10.12.12.143:~57180~
000007c0.00000f3c::2015/04/18-09:30:29.069 DBG   [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName =  host01
000007c0.00001724::2015/04/18-09:30:29.069 DBG   [SM] Handling auth handshake posted by thread id 3900
000007c0.00001724::2015/04/18-09:30:29.069 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1535, auth type: MSG_AUTH_PACKAGE::CurrentAuth
000007c0.00001724::2015/04/18-09:30:29.070 DBG   [SM] Sponsor: SSPI ASC returned status = 0
000007c0.00001724::2015/04/18-09:30:29.071 DBG   [SM] Sponsor: Sending SSPI blob of size 155 to Joiner
000007c0.00001724::2015/04/18-09:30:29.071 DBG   [SM] Sponsor: Sending synchronize packet to Joiner
000007c0.00001724::2015/04/18-09:30:29.071 DBG   [SM] Sponsor: Received packet from the Joiner during synchronization: size 0, auth type: MSG_AUTH_PACKAGE::Synchronize
000007c0.00001724::2015/04/18-09:30:29.071 DBG   [SM] Sponsor: Authentication handshake final Status 0
000007c0.00000f3c::2015/04/18-09:30:29.085 WARN  [ClRtl] Cannot open crypto container (error 2148073494). Giving up.
000007c0.00000f3c::2015/04/18-09:30:29.085 DBG   [CHANNEL 10.12.12.143:~57180~] Close().
000007c0.00000f3c::2015/04/18-09:30:29.085 WARN  mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
000007c0.00000cb8::2015/04/18-09:30:32.070 INFO  [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 12.0.0.2:~57181~.
000007c0.00000f3c::2015/04/18-09:30:32.070 INFO  [SV] New real route: local (12.0.0.1:~3343~) to remote  (12.0.0.2:~57181~).
000007c0.00000f3c::2015/04/18-09:30:32.070 INFO  [SV] Got a new incoming stream from 12.0.0.2:~57181~
000007c0.00000f3c::2015/04/18-09:30:32.070 DBG   [SM] SrvCtxt initialized with MaxTokenSize = 48000, RequiredCtxAttrib = 0, HandShakeTimeout = 40000, ServerName =  host01
000007c0.00001724::2015/04/18-09:30:32.070 DBG   [SM] Handling auth handshake posted by thread id 3900
000007c0.00001724::2015/04/18-09:30:32.071 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1535, auth type: MSG_AUTH_PACKAGE::CurrentAuth
000007c0.00001724::2015/04/18-09:30:32.071 DBG   [SM] Sponsor: SSPI ASC returned status = 0
000007c0.00001724::2015/04/18-09:30:32.071 DBG   [SM] Sponsor: Sending SSPI blob of size 155 to Joiner
000007c0.00001724::2015/04/18-09:30:32.071 DBG   [SM] Sponsor: Sending synchronize packet to Joiner
000007c0.00001724::2015/04/18-09:30:32.072 DBG   [SM] Sponsor: Received packet from the Joiner during synchronization: size 0, auth type: MSG_AUTH_PACKAGE::Synchronize
000007c0.00001724::2015/04/18-09:30:32.072 DBG   [SM] Sponsor: Authentication handshake final Status 0
000007c0.00000f3c::2015/04/18-09:30:32.086 WARN  [ClRtl] Cannot open crypto container (error 2148073494). Giving up.
000007c0.00000f3c::2015/04/18-09:30:32.086 DBG   [CHANNEL 12.0.0.2:~57181~] Close().
000007c0.00000f3c::2015/04/18-09:30:32.086 WARN  mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'
oompaloompaAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Zephyr ICTCloud ArchitectCommented:
Might be a security issues on the machinekeys, check this to see if it can help your situation.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
oompaloompaAuthor Commented:
already seen that and local admins have all rights on RSA folder, that was previous issue that i had with this cluster when i couldn't do live migration
0
Zephyr ICTCloud ArchitectCommented:
Hmmm, still think it's a security issue though ... Seeing these errors:

000007c0.00000f3c::2015/04/18-09:30:32.086 WARN  [ClRtl] Cannot open crypto container (error 2148073494). Giving up.
000007c0.00000f3c::2015/04/18-09:30:32.086 DBG   [CHANNEL 12.0.0.2:~57181~] Close().
000007c0.00000f3c::2015/04/18-09:30:32.086 WARN  mscs::ListenerWorker::operator (): HrError(0x8009030c)' because of '[SV] Authentication or Authorization Failed'

Did you also restart after changing the security settings? No Anti-Virus that could be blocking things?
0
oompaloompaAuthor Commented:
damn, sorry, something keeps rewriting permissions, maybe some hotfix, i rechecked and repaired and node is online, thanks
0
Zephyr ICTCloud ArchitectCommented:
No worries ... This happens ;)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server OS

From novice to tech pro — start learning today.