Link to home
Start Free TrialLog in
Avatar of Thlware
Thlware

asked on

Cluster Solaris10

Hi
I have performed a reboot on the machine that was running over a year now. It is now failing to come up(it is a V880) running Solaris 10.
Below is the error i get and the machine just goes back to OK prompt.

"Warning: Illegal configuration for Basic Failover mode
Info: Paths not selected for BasicFailover to 60060160DB601800EE8123BBCF6CDC11 a
re alive .
May 25 12:37:39 svc.startd[8]: svc:/system/cluster/scslm:default: Method "/usr/c
luster/lib/svc/method/svc_cl_slm" failed with exit status 96.
May 25 12:37:39 svc.startd[8]: system/cluster/scslm:default misconfigured: trans
itioned to maintenance (see 'svcs -xv' for details)
Info: Not all paths selected for BasicFailover to 60060160A1601800DE949F59C759DF
11 are dead.
Info: 60060160A160180072D46A4B3263DF11 is alive.
Info: 60060160A16018005EF9817E3163DF11 is alive.
Info: Paths not selected for BasicFailover to 60060160A160180072D46A4B3263DF11 a
re alive .
Info: Paths not selected for BasicFailover to 60060160A16018005EF9817E3163DF11 a
re alive .
Info: Not all paths selected for BasicFailover to 60060160A1601800C4FA31703263DF
11 are dead.
Info: Not all paths selected for BasicFailover to 60060160A16018009AEDA5793263DF
11 are dead.
Info: Not all paths selected for BasicFailover to 60060160A1601800D2F14B833263DF
11 are dead.
Info: 60060160A160180086435C5E3263DF11 is alive.
Info: Paths not selected for BasicFailover to 60060160A160180086435C5E3263DF11 a
re alive .
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c10t5006016130226F24d1s2"
to verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c10t5006016930226F24d1s2"
to verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c9t5006016030226F24d1s2" t
o verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c9t5006016830226F24d1s2" t
o verfiy device ID - No such device or address.
May 25 12:37:42 svc.startd[8]: system/sysevent:default failed repeatedly: transi
tioned to maintenance (see 'svcs -xv' for details)
May 25 12:37:42 svc.startd[8]: failed to abandon contract 54: Permission denied
/usr/cluster/lib/svc/method/bootcluster: /etc/cluster/chkinfr.err: cannot create

cat: cannot open /etc/cluster/chkinfr.err
UNRECOVERABLE ERROR: /etc/cluster/ccr/infrastructure file is corrupted
Please reboot in noncluster mode(boot -x) and Repair
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
May 25 12:37:43 Cluster.CCR: /usr/cluster/bin/scgdevs: _cladm failed. Pleaseensu
re the node is in cluster mode."

I then issue a boot command as the system suggests "boot -x" this is what i get ...

Rebooting with command: boot -x
Boot device: /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w2100002037eb6c45,0:a  File an
d args: -x
WARNING: rpcmod:svc_default_stksize is set more than once in /etc/system. "set r
pcmod:svc_default_stksize = 0x6000" applied as the current setting.

sorry, variable 'noexec_user_Stack' is not defined in the 'kernel'
SunOS Release 5.10 Version Generic_125100-10 64-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
SUNW,pci-gem0: Using Gigabit SERDES Interface
SUNW,pci-gem0: Auto-Negotiated 1000 Mbps Full-Duplex Link Up
Failed to configure IPv4 interface(s): ge0
Hostname: stpdb1
Info: Not all paths selected for BasicFailover to 60060160DB6018004CFA8EF92631DC
11 are dead.
Info: 60060160DB601800EE8123BBCF6CDC11 is alive.
Warning: Illegal configuration for Basic Failover mode
Info: Paths not selected for BasicFailover to 60060160DB601800EE8123BBCF6CDC11 a
re alive .
The / file system (/dev/md/rdsk/d0) is being checked.
May 25 12:21:08 svc.startd[7]: svc:/system/cluster/scslm:default: Method "/usr/c
luster/lib/svc/method/svc_cl_slm" failed with exit status 96.
May 25 12:21:08 svc.startd[7]: system/cluster/scslm:default misconfigured: trans
itioned to maintenance (see 'svcs -xv' for details)
Info: Not all paths selected for BasicFailover to 60060160A1601800DE949F59C759DF
11 are dead.
Info: 60060160A160180072D46A4B3263DF11 is alive.
Info: 60060160A16018005EF9817E3163DF11 is alive.
Info: Paths not selected for BasicFailover to 60060160A160180072D46A4B3263DF11 a
re alive .
Info: Paths not selected for BasicFailover to 60060160A16018005EF9817E3163DF11 a
re alive .
Info: Not all paths selected for BasicFailover to 60060160A1601800C4FA31703263DF
11 are dead.
Info: Not all paths selected for BasicFailover to 60060160A16018009AEDA5793263DF
11 are dead.
Info: Not all paths selected for BasicFailover to 60060160A1601800D2F14B833263DF
11 are dead.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c10t5006016130226F24d1s2"
to verfiy device ID - No such device or address.
Info: 60060160A160180086435C5E3263DF11 is alive.
Info: Paths not selected for BasicFailover to 60060160A160180086435C5E3263DF11 a
re alive .
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c10t5006016930226F24d1s2"
to verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c9t5006016030226F24d1s2" t
o verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c9t5006016830226F24d1s2" t
o verfiy device ID - No such device or address.
/dev/md/rdsk/d110 is clean
May 25 12:21:10 svc.startd[7]: system/sysevent:default failed repeatedly: transi
tioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:10 svc.startd[7]: failed to abandon contract 60: Permission denied
cron aborted: cannot create fifo queue
cron aborted: cannot create fifo queue
cron aborted: cannot create fifo queue
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:12 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:12 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:12 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
May 25 12:21:14 svc.startd[7]: system/cron:default failed repeatedly: transition
ed to maintenance (see 'svcs -xv' for details)
May 25 12:21:14 svc.startd[7]: failed to abandon contract 96: Permission denied
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:14 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:14 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
May 25 12:21:14 svc.startd[7]: system/sac:default failed repeatedly: transitione
d to maintenance (see 'svcs -xv' for details)
May 25 12:21:18 svc.startd[7]: system/filesystem/volfs:default failed repeatedly
: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:18 svc.startd[7]: failed to abandon contract 117: Permission denied

syslogd: /var/adm/messages: Read-only file system
syslogd: /var/adm/messages: Read-only file system
May 25 12:21:19 stpdb1 svc.startd[7]: application/management/seaport:default fai
led: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:20 stpdb1 sendmail[778]: My unqualified host name (stpdb1) unknown;
 sleeping for retry
May 25 12:21:20 stpdb1 sendmail[779]: My unqualified host name (stpdb1) unknown;
 sleeping for retry
May 25 12:21:21 stpdb1 svc.startd[7]: application/print/ipp-listener:default fai
led: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:22 stpdb1 svc.startd[7]: application/print/server:default failed re
peatedly: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:26 stpdb1 ufs: NOTICE: /: unexpected free inode 129698, run fsck(1M
) -o f
May 25 12:21:26 stpdb1 xntpd[966]: can't open /var/ntp/ntp.drift: I/O error
May 25 12:21:37 stpdb1 svc.startd[7]: application/graphical-login/cde-login:defa
ult failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:43 stpdb1 svc.startd[7]: system/webconsole:console failed fatally:
transitioned to maintenance (see 'svcs -xv' for details)

May 25 12:22:20 stpdb1 sendmail[779]: unable to qualify my own domain name (stpd
b1) -- using short name
May 25 12:22:20 stpdb1 sendmail[778]: unable to qualify my own domain name (stpd
b1) -- using short name
May 25 12:22:20 stpdb1 sendmail[1346]: unable to write pid to /var/spool/clientm
queue/sm-client.pid: Read-only file system

And i typed in the "boot -x" and this is what i get

Rebooting with command: boot -x
Boot device: /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w2100002037eb6c45,0:a  File an
d args: -x
WARNING: rpcmod:svc_default_stksize is set more than once in /etc/system. "set r
pcmod:svc_default_stksize = 0x6000" applied as the current setting.

sorry, variable 'noexec_user_Stack' is not defined in the 'kernel'
SunOS Release 5.10 Version Generic_125100-10 64-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
SUNW,pci-gem0: Using Gigabit SERDES Interface
SUNW,pci-gem0: Auto-Negotiated 1000 Mbps Full-Duplex Link Up
Failed to configure IPv4 interface(s): ge0
Hostname: server1
Info: Not all paths selected for BasicFailover to 60060160DB6018004CFA8EF92631DC
11 are dead.
Info: 60060160DB601800EE8123BBCF6CDC11 is alive.
Warning: Illegal configuration for Basic Failover mode
Info: Paths not selected for BasicFailover to 60060160DB601800EE8123BBCF6CDC11 a
re alive .
The / file system (/dev/md/rdsk/d0) is being checked.
May 25 12:21:08 svc.startd[7]: svc:/system/cluster/scslm:default: Method "/usr/c
luster/lib/svc/method/svc_cl_slm" failed with exit status 96.
May 25 12:21:08 svc.startd[7]: system/cluster/scslm:default misconfigured: trans
itioned to maintenance (see 'svcs -xv' for details)
Info: Not all paths selected for BasicFailover to 60060160A1601800DE949F59C759DF
11 are dead.
Info: 60060160A160180072D46A4B3263DF11 is alive.
Info: 60060160A16018005EF9817E3163DF11 is alive.
Info: Paths not selected for BasicFailover to 60060160A160180072D46A4B3263DF11 a
re alive .
Info: Paths not selected for BasicFailover to 60060160A16018005EF9817E3163DF11 a
re alive .
Info: Not all paths selected for BasicFailover to 60060160A1601800C4FA31703263DF
11 are dead.
Info: Not all paths selected for BasicFailover to 60060160A16018009AEDA5793263DF
11 are dead.
Info: Not all paths selected for BasicFailover to 60060160A1601800D2F14B833263DF
11 are dead.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c10t5006016130226F24d1s2"
to verfiy device ID - No such device or address.
Info: 60060160A160180086435C5E3263DF11 is alive.
Info: Paths not selected for BasicFailover to 60060160A160180086435C5E3263DF11 a
re alive .
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c10t5006016930226F24d1s2"
to verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c9t5006016030226F24d1s2" t
o verfiy device ID - No such device or address.
/usr/cluster/bin/scdidadm:  Could not open "/dev/rdsk/c9t5006016830226F24d1s2" t
o verfiy device ID - No such device or address.
/dev/md/rdsk/d110 is clean
May 25 12:21:10 svc.startd[7]: system/sysevent:default failed repeatedly: transi
tioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:10 svc.startd[7]: failed to abandon contract 60: Permission denied
cron aborted: cannot create fifo queue
cron aborted: cannot create fifo queue
cron aborted: cannot create fifo queue
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:12 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:12 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:12 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
cron aborted: cannot create fifo queue
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:13 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
May 25 12:21:14 svc.startd[7]: system/cron:default failed repeatedly: transition
ed to maintenance (see 'svcs -xv' for details)
May 25 12:21:14 svc.startd[7]: failed to abandon contract 96: Permission denied
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:14 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
SAC: could not open logfile /var/saf/_log: Read-only file system
May 25 12:21:14 svc.startd[7]: instance svc:/system/sac:default exited with stat
us 1
May 25 12:21:14 svc.startd[7]: system/sac:default failed repeatedly: transitione
d to maintenance (see 'svcs -xv' for details)
May 25 12:21:18 svc.startd[7]: system/filesystem/volfs:default failed repeatedly
: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:18 svc.startd[7]: failed to abandon contract 117: Permission denied

syslogd: /var/adm/messages: Read-only file system
syslogd: /var/adm/messages: Read-only file system
May 25 12:21:19 server1 svc.startd[7]: application/management/seaport:default fai
led: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:20 server1 sendmail[778]: My unqualified host name (server1) unknown;
 sleeping for retry
May 25 12:21:20 server1 sendmail[779]: My unqualified host name (server1) unknown;
 sleeping for retry
May 25 12:21:21 server1 svc.startd[7]: application/print/ipp-listener:default fai
led: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:22 server1 svc.startd[7]: application/print/server:default failed re
peatedly: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:26 server1 ufs: NOTICE: /: unexpected free inode 129698, run fsck(1M
) -o f
May 25 12:21:26 server1 xntpd[966]: can't open /var/ntp/ntp.drift: I/O error
May 25 12:21:37 server1 svc.startd[7]: application/graphical-login/cde-login:defa
ult failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details)
May 25 12:21:43 server1 svc.startd[7]: system/webconsole:console failed fatally:
transitioned to maintenance (see 'svcs -xv' for details)

May 25 12:22:20 server1 sendmail[779]: unable to qualify my own domain name (server1
) -- using short name
May 25 12:22:20 server1 sendmail[778]: unable to qualify my own domain name (server1
) -- using short name
May 25 12:22:20 server1 sendmail[1346]: unable to write pid to /var/spool/clientm
queue/sm-client.pid: Read-only file system

Any advice to get this machine up will be apreciated
ASKER CERTIFIED SOLUTION
Avatar of Joseph Gan
Joseph Gan
Flag of Australia image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Thlware
Thlware

ASKER

Good document. hhmm

I managed to log to the system.  The problem i am experiencing is still give me the above output when i execute the "svcadm milestone all" is there a way i can exclude this service when starting all my service? maybe remove it from my conf files.. Because i do not need it for now.

chz
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Thlware

ASKER

Project on Hold