jayatallen
asked on
how to find when rc script started tomcat
Hi Folks,
I need to find when rc script started the tomcat. Currently my prod solaris 10 server is scheduled to reboot weekly.Every Sat. at 8:00 AM. And when its comes up after it reboot,we have script setup in /etc/rc2.d to start the tomcat.
It has been working fine.But yesterday, tomcat didnt come up after reboot and later it came up automatically after 4 hours.
So far what i have done:
checked the server was actually rebooted at sheduled time and found it did reboot at sheduled time.
bash-3.00$ last reboot | head
reboot   system boot          Sat Mar 26 08:05
reboot   system down          Sat Mar 26 08:00
reboot   system boot          Sat Mar 19 08:05
reboot   system down          Sat Mar 19 08:00
reboot   system boot          Sat Mar 12 08:05
reboot   system down          Sat Mar 12 08:00
reboot   system boot          Sat Mar  5 08:05
reboot   system down          Sat Mar  5 08:00
reboot   system boot          Sat Feb 26 08:05
reboot   system down          Sat Feb 26 08:00
Then i checked catalina out file to see when generally it starts after reboot:
grep -i 'server startup' catalina.out | tail
catalina.out:2010-12-04 08:14:26,176 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2010-12-11 08:14:29,202 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2010-12-18 08:14:24,104 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2010-12-25 08:14:24,515 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-01 08:14:26,042 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-08 08:14:24,199 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-15 08:14:41,326 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-22 08:14:29,239 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-29 08:14:24,634 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-04 18:37:32,357 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-05 08:14:26,014 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-12 08:14:25,305 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-19 08:13:27,119 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-26 08:13:28,873 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-05 08:13:28,152 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-12 08:13:30,109 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-19 08:13:28,806 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-26 12:30:47,702 (Catalina.java:560) INFO Â org.apache.catalina.startu
So, i can see it was started at catalina.out:2011-03-26 12:30:47.
Lastly i checked rc.2d folder and if has below script which is responsible for stop/start of tomcat.
-rwxr--r--  1 root   other    1064 Aug  6  2010 K99stopLAIOPRODEnv
bash-3.00$ pwd
/etc/rc2.d
bash-3.00$
1)How can i find if the solaris10 tried to start the tomcat after rebooting?
2)when it didnt up after reboot, how come tomcat started after 4 hours automatically? --Is rc script tries to restart the application after a failure? i mean it rc script and it didnt bring up the application,would i try to bring it up again itself?
The main thing i need to bebug why it didnt when machine came up?
and why it started at  12:30:47 itself?
Please help/suugest or guide me  to any resource . i'm familiar with basics of  unix.
Thank you
Joe
I need to find when rc script started the tomcat. Currently my prod solaris 10 server is scheduled to reboot weekly.Every Sat. at 8:00 AM. And when its comes up after it reboot,we have script setup in /etc/rc2.d to start the tomcat.
It has been working fine.But yesterday, tomcat didnt come up after reboot and later it came up automatically after 4 hours.
So far what i have done:
checked the server was actually rebooted at sheduled time and found it did reboot at sheduled time.
bash-3.00$ last reboot | head
reboot   system boot          Sat Mar 26 08:05
reboot   system down          Sat Mar 26 08:00
reboot   system boot          Sat Mar 19 08:05
reboot   system down          Sat Mar 19 08:00
reboot   system boot          Sat Mar 12 08:05
reboot   system down          Sat Mar 12 08:00
reboot   system boot          Sat Mar  5 08:05
reboot   system down          Sat Mar  5 08:00
reboot   system boot          Sat Feb 26 08:05
reboot   system down          Sat Feb 26 08:00
Then i checked catalina out file to see when generally it starts after reboot:
grep -i 'server startup' catalina.out | tail
catalina.out:2010-12-04 08:14:26,176 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2010-12-11 08:14:29,202 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2010-12-18 08:14:24,104 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2010-12-25 08:14:24,515 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-01 08:14:26,042 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-08 08:14:24,199 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-15 08:14:41,326 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-22 08:14:29,239 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-01-29 08:14:24,634 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-04 18:37:32,357 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-05 08:14:26,014 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-12 08:14:25,305 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-19 08:13:27,119 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-02-26 08:13:28,873 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-05 08:13:28,152 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-12 08:13:30,109 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-19 08:13:28,806 (Catalina.java:560) INFO Â org.apache.catalina.startu
catalina.out:2011-03-26 12:30:47,702 (Catalina.java:560) INFO Â org.apache.catalina.startu
So, i can see it was started at catalina.out:2011-03-26 12:30:47.
Lastly i checked rc.2d folder and if has below script which is responsible for stop/start of tomcat.
-rwxr--r--  1 root   other    1064 Aug  6  2010 K99stopLAIOPRODEnv
bash-3.00$ pwd
/etc/rc2.d
bash-3.00$
1)How can i find if the solaris10 tried to start the tomcat after rebooting?
2)when it didnt up after reboot, how come tomcat started after 4 hours automatically? --Is rc script tries to restart the application after a failure? i mean it rc script and it didnt bring up the application,would i try to bring it up again itself?
The main thing i need to bebug why it didnt when machine came up?
and why it started at  12:30:47 itself?
Please help/suugest or guide me  to any resource . i'm familiar with basics of  unix.
Thank you
Joe
SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
ASKER
i see it now. A similar script in /etc/rc3.d
Is it responsible to startup ?
bash-3.00$ pwd
/etc/rc3.d
bash-3.00$ ls -l S99startLionnPRODEnv
-rwxr--r--  1 root   root     1064 Feb 19  2009 S99startLionnPRODEnv
bash-3.00$
bash-3.00$ diff S99startLionnPRODEnv /etc/rc2.d/K99stopLionnPRO
bash-3.00$ pwd
/etc/rc3.d
So for now, can i say that there must be some issues with startup rc script?
How can i debug/find the issue?
Please suggest.
Is it responsible to startup ?
bash-3.00$ pwd
/etc/rc3.d
bash-3.00$ ls -l S99startLionnPRODEnv
-rwxr--r--  1 root   root     1064 Feb 19  2009 S99startLionnPRODEnv
bash-3.00$
bash-3.00$ diff S99startLionnPRODEnv /etc/rc2.d/K99stopLionnPRO
bash-3.00$ pwd
/etc/rc3.d
So for now, can i say that there must be some issues with startup rc script?
How can i debug/find the issue?
Please suggest.
You can run "strace" command to debug the issue by
# stace -o /tmp/output.txt /etc/rc3.d/S99startLionnPR
# view /tmp/output.txt
# stace -o /tmp/output.txt /etc/rc3.d/S99startLionnPR
# view /tmp/output.txt
ASKER
Thank you ganjos.
I ran it but says ERROR. do only root can run the given command?
bash-3.00$ strace  -o /tmp/output.txt /etc/rc3.d/S99startLionnrP
ERROR: unable to open /dev/log
bash-3.00$ ls -l log
lrwxrwxrwx  1 root   other     27 Feb 17  2009 log -> ../devices/pseudo/log@0:lo
bash-3.00$ more log
log: Permission denied
bash-3.00$ ls -l  ../devices/pseudo/log@0:lo
crw-r-----  1 root   sys    21,  5 Feb 17  2009 ../devices/pseudo/log@0:lo
bash-3.00$ pwd
/dev
Is there any other way to debug?
Does rc script try to run an application after successive failure until it succeeded?
I ran it but says ERROR. do only root can run the given command?
bash-3.00$ strace  -o /tmp/output.txt /etc/rc3.d/S99startLionnrP
ERROR: unable to open /dev/log
bash-3.00$ ls -l log
lrwxrwxrwx  1 root   other     27 Feb 17  2009 log -> ../devices/pseudo/log@0:lo
bash-3.00$ more log
log: Permission denied
bash-3.00$ ls -l  ../devices/pseudo/log@0:lo
crw-r-----  1 root   sys    21,  5 Feb 17  2009 ../devices/pseudo/log@0:lo
bash-3.00$ pwd
/dev
Is there any other way to debug?
Does rc script try to run an application after successive failure until it succeeded?
Usually owner of the script can run debug, but because some script access other paths which may need root permision, in this case better to run as root.
The rc script will run when system starts or run by root, won't be wait until it succeeded. It looks to me Tomcat is depanded on other processes such as Appach and Java. The depandant will cause Tomcat fail to start as well.
The rc script will run when system starts or run by root, won't be wait until it succeeded. It looks to me Tomcat is depanded on other processes such as Appach and Java. The depandant will cause Tomcat fail to start as well.
ASKER
Hi ganjos,
actually we are using maven to start all the required components together.Basically only one rc script starts apache,siteminder and tomcat.Apache and siteminder agent came up.
For simplicity,if i consider it failed for some reason...
Is there way how tomcat got started itself later?
Please help
actually we are using maven to start all the required components together.Basically only one rc script starts apache,siteminder and tomcat.Apache and siteminder agent came up.
For simplicity,if i consider it failed for some reason...
Is there way how tomcat got started itself later?
Please help
Check your start scripts:
su - compass -c "/apps/compass/startStopJa
     /apps/compass/startStopLio
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
One of them must be Tomcat. If not, that means Tomcat start manually later.
su - compass -c "/apps/compass/startStopJa
     /apps/compass/startStopLio
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
One of them must be Tomcat. If not, that means Tomcat start manually later.
ASKER
Hi ganjos,
i know /apps/compass/startStopLio
And the major issue is its Prod env. so i cant start them again.
I was asked to get the evidence why the script failed ? and how it came up after 4 hours?
is there any way to find something in log files?
i know /apps/compass/startStopLio
And the major issue is its Prod env. so i cant start them again.
I was asked to get the evidence why the script failed ? and how it came up after 4 hours?
is there any way to find something in log files?
Do the sub scripts write out to logs? you will have to check each script and see how/where the log is written.
SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
Solaris 10 uses the svcadm mechanism to manage startup process with the exception of some legacy where the /etc/rc*.d scripts come into place.
It seems that your tomcat/startup process sets the delay before start based on the log.
It seems that someone thought the dealy will be in milliseconds versus seconds.
Solaris uses the truss command that is comparable to the strace.
Your issue is that a delay is included in the start of tomcat after reboot. Â Often if a delay is needed, one should set the startup value higher.
look in /etc/init.d to see whether tomcat script is there and then see whether you see a reference to 28800.
It seems that your tomcat/startup process sets the delay before start based on the log.
It seems that someone thought the dealy will be in milliseconds versus seconds.
Solaris uses the truss command that is comparable to the strace.
Your issue is that a delay is included in the start of tomcat after reboot. Â Often if a delay is needed, one should set the startup value higher.
look in /etc/init.d to see whether tomcat script is there and then see whether you see a reference to 28800.
SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
ASKER
Hi Ganjos...
syslog.conf looks very cryptic to me :(
bash-3.00$ grep -v \# syslog.conf
*.err;kern.notice;auth.not
*.err;kern.debug;daemon.no
*.alert;kern.err;daemon.er
*.alert                     root
*.emerg                     *
mail.debug            ifdef(`LOGHOST', /var/log/syslog, @loghost)
ifdef(`LOGHOST', ,
user.err                     /dev/sysmsg
user.err                     /var/adm/messages
user.alert                    `root, operator'
user.emerg                    *
)
daemon.info                   /var/adm/messages
user.info                    /var/adm/messages
auth.info                    /var/adm/messages
arnold-- i checked reboot time of machine and start up server in catalina.out and pasted above..
its been running fine for almost an year. I think i need to make sure whether it was started manually by someone in my team..
In case,,how can i add delay to startup of script to a higher value?
Tomunique - i have checked to startup rc script. it resides in rc3.d. and its same as kill script.
bash-3.00$ pwd
/etc/rc3.d
bash-3.00$ ls -l S99startLionnPRODEnv
-rwxr--r--  1 root   root     1064 Feb 19  2009 S99startLionnPRODEnv
bash-3.00$
bash-3.00$ diff S99startLionnPRODEnv /etc/rc2.d/K99stopLionnPRO
bash-3.00$ pwd
/etc/rc3.d
and we use maven which starts apache ,siteminder and tomcat at once taking care of dependency.
syslog.conf looks very cryptic to me :(
bash-3.00$ grep -v \# syslog.conf
*.err;kern.notice;auth.not
*.err;kern.debug;daemon.no
*.alert;kern.err;daemon.er
*.alert                     root
*.emerg                     *
mail.debug            ifdef(`LOGHOST', /var/log/syslog, @loghost)
ifdef(`LOGHOST', ,
user.err                     /dev/sysmsg
user.err                     /var/adm/messages
user.alert                    `root, operator'
user.emerg                    *
)
daemon.info                   /var/adm/messages
user.info                    /var/adm/messages
auth.info                    /var/adm/messages
arnold-- i checked reboot time of machine and start up server in catalina.out and pasted above..
its been running fine for almost an year. I think i need to make sure whether it was started manually by someone in my team..
In case,,how can i add delay to startup of script to a higher value?
Tomunique - i have checked to startup rc script. it resides in rc3.d. and its same as kill script.
bash-3.00$ pwd
/etc/rc3.d
bash-3.00$ ls -l S99startLionnPRODEnv
-rwxr--r--  1 root   root     1064 Feb 19  2009 S99startLionnPRODEnv
bash-3.00$
bash-3.00$ diff S99startLionnPRODEnv /etc/rc2.d/K99stopLionnPRO
bash-3.00$ pwd
/etc/rc3.d
and we use maven which starts apache ,siteminder and tomcat at once taking care of dependency.
ASKER
it was started manually by one of my team member after 4 hours:(
SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
ASKER
Thank you jgiordano. i tried your suggestion.
bash-3.00$ svcs
STATE Â Â Â Â Â STIME Â Â FMRI
legacy_run   Mar_26  lrc:/etc/rcS_d/S29wrsmcfg
legacy_run   Mar_26  lrc:/etc/rc2_d/S01nddconfi
legacy_run   Mar_26  lrc:/etc/rc2_d/S20sysetup
legacy_run   Mar_26  lrc:/etc/rc2_d/S42ncakmod
legacy_run   Mar_26  lrc:/etc/rc2_d/S45vxpbx_ex
legacy_run   Mar_26  lrc:/etc/rc2_d/S47pppd
legacy_run   Mar_26  lrc:/etc/rc2_d/S50vxvail
legacy_run   Mar_26  lrc:/etc/rc2_d/S64preisisd
legacy_run   Mar_26  lrc:/etc/rc2_d/S70vxatd
legacy_run   Mar_26  lrc:/etc/rc2_d/S73cachefs_
legacy_run   Mar_26  lrc:/etc/rc2_d/S73isisd
legacy_run   Mar_26  lrc:/etc/rc2_d/S750vxpal_g
legacy_run   Mar_26  lrc:/etc/rc2_d/S75vxpal_St
legacy_run   Mar_26  lrc:/etc/rc2_d/S75vxsmfd
legacy_run   Mar_26  lrc:/etc/rc2_d/S760vxpal_a
legacy_run   Mar_26  lrc:/etc/rc2_d/S79opsware-
legacy_run   Mar_26  lrc:/etc/rc2_d/S89PRESERVE
legacy_run   Mar_26  lrc:/etc/rc2_d/S91afbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91gfbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91ifbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91jfbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91kfbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91zuluinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S94ncalogd
legacy_run   Mar_26  lrc:/etc/rc2_d/S94vxnm-vxn
legacy_run   Mar_26  lrc:/etc/rc2_d/S96vradmind
legacy_run   Mar_26  lrc:/etc/rc2_d/S96vxrsyncd
legacy_run   Mar_26  lrc:/etc/rc2_d/S98dealloca
legacy_run   Mar_26  lrc:/etc/rc2_d/S99bokcfg
legacy_run   Mar_26  lrc:/etc/rc2_d/S99boksm
legacy_run   Mar_26  lrc:/etc/rc2_d/S99eccmad
legacy_run   Mar_26  lrc:/etc/rc2_d/S99sneep
legacy_run   Mar_26  lrc:/etc/rc2_d/S99xboksm-a
legacy_run   Mar_26  lrc:/etc/rc3_d/S69esmrc
legacy_run   Mar_26  lrc:/etc/rc3_d/S75vxsmfd
legacy_run   Mar_26  lrc:/etc/rc3_d/S78bgsu
legacy_run   Mar_26  lrc:/etc/rc3_d/S80mipagent
legacy_run   Mar_26  lrc:/etc/rc3_d/S89jpmc-net
legacy_run   Mar_26  lrc:/etc/rc3_d/S95X11R6
legacy_run   Mar_26  lrc:/etc/rc3_d/S95dbora
legacy_run   Mar_26  lrc:/etc/rc3_d/S98control-
legacy_run   Mar_26  lrc:/etc/rc3_d/S99Tivoli_l
legacy_run   Mar_26  lrc:/etc/rc3_d/S99aqueduct
legacy_run   Mar_26  lrc:/etc/rc3_d/S99gcstartu
legacy_run   Mar_26  lrc:/etc/rc3_d/S99infra
legacy_run   Mar_26  lrc:/etc/rc3_d/S99startTig
legacy_run   Mar_26  lrc:/etc/rc3_d/S99vxtf_chk
online     Mar_26  svc:/system/svc/restarter:
online     Mar_26  svc:/network/pfil:default
online     Mar_26  svc:/network/tnctl:default
online     Mar_26  svc:/network/loopback:defa
online     Mar_26  svc:/system/installupdates
online     Mar_26  svc:/network/physical:defa
online     Mar_26  svc:/system/identity:node
online     Mar_26  svc:/system/metainit:defau
online     Mar_26  svc:/system/vxvm/vxvm-sysb
online     Mar_26  svc:/system/filesystem/roo
online     Mar_26  svc:/system/scheduler:defa
online     Mar_26  svc:/system/vxvm/vxvm-star
online     Mar_26  svc:/system/boot-archive:d
online     Mar_26  svc:/system/filesystem/usr
online     Mar_26  svc:/system/keymap:default
online     Mar_26  svc:/system/device/local:d
online     Mar_26  svc:/system/filesystem/min
online     Mar_26  svc:/system/resource-mgmt:
online     Mar_26  svc:/system/rmtmpfiles:def
online     Mar_26  svc:/system/name-service-c
online     Mar_26  svc:/system/identity:domai
online     Mar_26  svc:/site/JPMCsvc_prop_mod
online     Mar_26  svc:/system/coreadm:defaul
online     Mar_26  svc:/system/sysevent:defau
online     Mar_26  svc:/system/cryptosvc:defa
online     Mar_26  svc:/system/device/fc-fabr
online     Mar_26  svc:/system/picl:default
online     Mar_26  svc:/system/manifest-impor
online     Mar_26  svc:/application/print/ppd
online     Mar_26  svc:/site/network/JPMCnets
online     Mar_26  svc:/milestone/network:def
online     Mar_26  svc:/network/initial:defau
online     Mar_26  svc:/network/service:defau
online     Mar_26  svc:/network/ldap/client:d
online     Mar_26  svc:/milestone/name-servic
online     Mar_26  svc:/network/ntp:default
online     Mar_26  svc:/system/vxvm/vxvm-star
online     Mar_26  svc:/system/vxvm/vxvm-star
online     Mar_26  svc:/milestone/devices:def
online     Mar_26  svc:/system/vxvm/vxvm-reco
online     Mar_26  svc:/milestone/single-user
online     Mar_26  svc:/system/filesystem/loc
online     Mar_26  svc:/network/shares/group:
online     Mar_26  svc:/system/cron:default
online     Mar_26  svc:/system/sysidtool:net
online     Mar_26  svc:/system/boot-archive-u
online     Mar_26  svc:/system/vxfs/vxfsldlic
online     Mar_26  svc:/network/routing-setup
online     Mar_26  svc:/network/rpc/bind:defa
online     Mar_26  svc:/system/sysidtool:syst
online     Mar_26  svc:/network/nfs/cbd:defau
online     Mar_26  svc:/network/nfs/status:de
online     Mar_26  svc:/network/nfs/mapid:def
online     Mar_26  svc:/milestone/sysconfig:d
online     Mar_26  svc:/network/nfs/nlockmgr:
online     Mar_26  svc:/system/sac:default
online     Mar_26  svc:/network/inetd:default
online     Mar_26  svc:/system/postrun:defaul
online     Mar_26  svc:/system/utmp:default
online     Mar_26  svc:/network/nfs/client:de
online     Mar_26  svc:/system/filesystem/aut
online     Mar_26  svc:/application/stosreg:d
online     Mar_26  svc:/system/dumpadm:defaul
online     Mar_26  svc:/system/system-log:def
online     Mar_26  svc:/application/managemen
online     Mar_26  svc:/system/auditd:default
online     Mar_26  svc:/system/console-login:
online     Mar_26  svc:/application/font/fc-c
online     Mar_26  svc:/system/fmd:default
online     Mar_26  svc:/network/rpc/meta:defa
online     Mar_26  svc:/network/stdiscover:de
online     Mar_26  svc:/network/stlisten:defa
online     Mar_26  svc:/network/bpcd/tcp:defa
online     Mar_26  svc:/network/vnetd/tcp:def
online     Mar_26  svc:/network/vopied/tcp:de
online     Mar_26  svc:/network/bpjava-msvc/t
online     Mar_26  svc:/network/ctmagent/tcp:
online     Mar_26  svc:/system/mdmonitor:defa
online     Mar_26  svc:/milestone/multi-user:
online     Mar_26  svc:/system/vxvm/vxvm-reco
online     Mar_26  svc:/application/cde-print
online     Mar_26  svc:/milestone/multi-user-
$ who -r
  .    run-level 3  Mar 26 08:06   3    0  S
A snippet when machine was going down as i know its reboots at 8.
Mar 26 08:00:09 slfappprd01 root[13307]: [ID 702911 user.crit] Stopping agent
Mar 26 08:00:09 slfappprd01 root[13307]: [ID 702911 user.crit] Stopping agent
Mar 26 08:00:33 slfappprd01 su: [ID 366847 auth.info] 'su ctmag' succeeded for root on /dev/???
Mar 26 08:00:33 slfappprd01 last message repeated 2 times
Mar 26 08:00:33 slfappprd01 boks_sshd[2839]: [ID 800047 auth.info] Received signal 15; terminating.
Mar 26 08:00:36 slfappprd01 syseventd[430]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
Mar 26 08:00:36 slfappprd01 syseventd[430]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
Mar 26 08:00:36 slfappprd01 vxesd[785]: [ID 477869 daemon.notice] Event source daemon stopped
Mar 26 08:00:36 slfappprd01 vxesd[785]: [ID 477869 daemon.notice] Event source daemon stopped
Mar 26 08:00:37 slfappprd01 syseventd[430]: [ID 661968 daemon.error] Daemon restarted
Mar 26 08:00:37 slfappprd01 syseventd[430]: [ID 661968 daemon.error] Daemon restarted
Mar 26 08:00:47 slfappprd01 rpc.metad: [ID 702911 daemon.error] Terminated
Mar 26 08:00:47 slfappprd01 rpc.metad: [ID 702911 daemon.error] Terminated
Mar 26 08:00:47 slfappprd01 syslogd: going down on signal 15
Mar 26 08:00:53 slfappprd01 rpcbind: [ID 564983 daemon.error] rpcbind terminating on signal.
Mar 26 08:00:53 slfappprd01 rpcbind: [ID 564983 daemon.error] rpcbind terminating on signal.
Mar 26 08:05:43 slfappprd01 ifconfig[28]: [ID 293258 user.warning] libsldap: Status: 2 Â Mesg: Unable to load configuration '/var/ldap/ldap_client_
file' ('').
I have seen in catalina.out that tomcat generally comes up around 8:13 so i checked /var/adm/messages for close to 8:13.
### su to application id
Mar 26 08:09:28 slfappprd01 su: [ID 366847 auth.info] 'su compass'' succeeded for root on /dev/???
Mar 26 08:09:29 slfappprd01 nrpe[6163]: [ID 320800 daemon.info] INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 26 08:09:27 slfappprd01 last message repeated 2 times
Mar 26 08:09:29 slfappprd01 nrpe[6164]: [ID 601491 daemon.notice] Starting up daemon
##all over in messages file at different places.
Mar 26 08:09:36 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:09:36 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
Mar 26 08:09:51 slfappprd01 last message repeated 5 times
Mar 26 08:09:51 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:09:51 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
## during 8:13 again su to compass
Mar 26 08:13:52 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
Mar 26 08:14:07 slfappprd01 last message repeated 5 times
Mar 26 08:14:07 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:14:37 slfappprd01 su: [ID 366847 auth.info] 'su aqueduct' succeeded for root on /dev/???
Mar 26 08:14:37 slfappprd01 <38>bgsud[5005]: bgsu monitoring su aqueduct by root
Mar 26 08:14:37 slfappprd01 last message repeated 1 time
Mar 26 08:20:02 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43904 ssh2
Mar 26 08:20:03 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] subsystem request for sftp
Mar 26 08:20:03 slfappprd01 boks_sshd[7154]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43963 ssh2
Mar 26 08:13:52 slfappprd01 last message repeated 3 times
Mar 26 08:13:52 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:13:52 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
Mar 26 08:14:07 slfappprd01 last message repeated 5 times
Mar 26 08:14:07 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:14:37 slfappprd01 su: [ID 366847 auth.info] 'su compass' succeeded for root on /dev/???
Mar 26 08:14:37 slfappprd01 <38>bgsud[5005]: bgsu monitoring su compass by root
Mar 26 08:14:37 slfappprd01 last message repeated 1 time
Mar 26 08:20:02 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43904 ssh2
Mar 26 08:20:03 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] subsystem request for sftp
Mar 26 08:20:03 slfappprd01 boks_sshd[7154]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43963 ssh2
Mar 26 08:13:41 slfappprd01 last message repeated 87 times
Mar 26 08:21:04 slfappprd01 boks_sshd[7188]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 44007 ssh2
Mar 26 08:21:04 slfappprd01 boks_sshd[7188]: [ID 800047 auth.info] subsystem request for sftp
it suppose to show that it tried to execute the script for starting up the tomcat?
Please suggest.
bash-3.00$ svcs
STATE Â Â Â Â Â STIME Â Â FMRI
legacy_run   Mar_26  lrc:/etc/rcS_d/S29wrsmcfg
legacy_run   Mar_26  lrc:/etc/rc2_d/S01nddconfi
legacy_run   Mar_26  lrc:/etc/rc2_d/S20sysetup
legacy_run   Mar_26  lrc:/etc/rc2_d/S42ncakmod
legacy_run   Mar_26  lrc:/etc/rc2_d/S45vxpbx_ex
legacy_run   Mar_26  lrc:/etc/rc2_d/S47pppd
legacy_run   Mar_26  lrc:/etc/rc2_d/S50vxvail
legacy_run   Mar_26  lrc:/etc/rc2_d/S64preisisd
legacy_run   Mar_26  lrc:/etc/rc2_d/S70vxatd
legacy_run   Mar_26  lrc:/etc/rc2_d/S73cachefs_
legacy_run   Mar_26  lrc:/etc/rc2_d/S73isisd
legacy_run   Mar_26  lrc:/etc/rc2_d/S750vxpal_g
legacy_run   Mar_26  lrc:/etc/rc2_d/S75vxpal_St
legacy_run   Mar_26  lrc:/etc/rc2_d/S75vxsmfd
legacy_run   Mar_26  lrc:/etc/rc2_d/S760vxpal_a
legacy_run   Mar_26  lrc:/etc/rc2_d/S79opsware-
legacy_run   Mar_26  lrc:/etc/rc2_d/S89PRESERVE
legacy_run   Mar_26  lrc:/etc/rc2_d/S91afbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91gfbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91ifbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91jfbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91kfbinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S91zuluinit
legacy_run   Mar_26  lrc:/etc/rc2_d/S94ncalogd
legacy_run   Mar_26  lrc:/etc/rc2_d/S94vxnm-vxn
legacy_run   Mar_26  lrc:/etc/rc2_d/S96vradmind
legacy_run   Mar_26  lrc:/etc/rc2_d/S96vxrsyncd
legacy_run   Mar_26  lrc:/etc/rc2_d/S98dealloca
legacy_run   Mar_26  lrc:/etc/rc2_d/S99bokcfg
legacy_run   Mar_26  lrc:/etc/rc2_d/S99boksm
legacy_run   Mar_26  lrc:/etc/rc2_d/S99eccmad
legacy_run   Mar_26  lrc:/etc/rc2_d/S99sneep
legacy_run   Mar_26  lrc:/etc/rc2_d/S99xboksm-a
legacy_run   Mar_26  lrc:/etc/rc3_d/S69esmrc
legacy_run   Mar_26  lrc:/etc/rc3_d/S75vxsmfd
legacy_run   Mar_26  lrc:/etc/rc3_d/S78bgsu
legacy_run   Mar_26  lrc:/etc/rc3_d/S80mipagent
legacy_run   Mar_26  lrc:/etc/rc3_d/S89jpmc-net
legacy_run   Mar_26  lrc:/etc/rc3_d/S95X11R6
legacy_run   Mar_26  lrc:/etc/rc3_d/S95dbora
legacy_run   Mar_26  lrc:/etc/rc3_d/S98control-
legacy_run   Mar_26  lrc:/etc/rc3_d/S99Tivoli_l
legacy_run   Mar_26  lrc:/etc/rc3_d/S99aqueduct
legacy_run   Mar_26  lrc:/etc/rc3_d/S99gcstartu
legacy_run   Mar_26  lrc:/etc/rc3_d/S99infra
legacy_run   Mar_26  lrc:/etc/rc3_d/S99startTig
legacy_run   Mar_26  lrc:/etc/rc3_d/S99vxtf_chk
online     Mar_26  svc:/system/svc/restarter:
online     Mar_26  svc:/network/pfil:default
online     Mar_26  svc:/network/tnctl:default
online     Mar_26  svc:/network/loopback:defa
online     Mar_26  svc:/system/installupdates
online     Mar_26  svc:/network/physical:defa
online     Mar_26  svc:/system/identity:node
online     Mar_26  svc:/system/metainit:defau
online     Mar_26  svc:/system/vxvm/vxvm-sysb
online     Mar_26  svc:/system/filesystem/roo
online     Mar_26  svc:/system/scheduler:defa
online     Mar_26  svc:/system/vxvm/vxvm-star
online     Mar_26  svc:/system/boot-archive:d
online     Mar_26  svc:/system/filesystem/usr
online     Mar_26  svc:/system/keymap:default
online     Mar_26  svc:/system/device/local:d
online     Mar_26  svc:/system/filesystem/min
online     Mar_26  svc:/system/resource-mgmt:
online     Mar_26  svc:/system/rmtmpfiles:def
online     Mar_26  svc:/system/name-service-c
online     Mar_26  svc:/system/identity:domai
online     Mar_26  svc:/site/JPMCsvc_prop_mod
online     Mar_26  svc:/system/coreadm:defaul
online     Mar_26  svc:/system/sysevent:defau
online     Mar_26  svc:/system/cryptosvc:defa
online     Mar_26  svc:/system/device/fc-fabr
online     Mar_26  svc:/system/picl:default
online     Mar_26  svc:/system/manifest-impor
online     Mar_26  svc:/application/print/ppd
online     Mar_26  svc:/site/network/JPMCnets
online     Mar_26  svc:/milestone/network:def
online     Mar_26  svc:/network/initial:defau
online     Mar_26  svc:/network/service:defau
online     Mar_26  svc:/network/ldap/client:d
online     Mar_26  svc:/milestone/name-servic
online     Mar_26  svc:/network/ntp:default
online     Mar_26  svc:/system/vxvm/vxvm-star
online     Mar_26  svc:/system/vxvm/vxvm-star
online     Mar_26  svc:/milestone/devices:def
online     Mar_26  svc:/system/vxvm/vxvm-reco
online     Mar_26  svc:/milestone/single-user
online     Mar_26  svc:/system/filesystem/loc
online     Mar_26  svc:/network/shares/group:
online     Mar_26  svc:/system/cron:default
online     Mar_26  svc:/system/sysidtool:net
online     Mar_26  svc:/system/boot-archive-u
online     Mar_26  svc:/system/vxfs/vxfsldlic
online     Mar_26  svc:/network/routing-setup
online     Mar_26  svc:/network/rpc/bind:defa
online     Mar_26  svc:/system/sysidtool:syst
online     Mar_26  svc:/network/nfs/cbd:defau
online     Mar_26  svc:/network/nfs/status:de
online     Mar_26  svc:/network/nfs/mapid:def
online     Mar_26  svc:/milestone/sysconfig:d
online     Mar_26  svc:/network/nfs/nlockmgr:
online     Mar_26  svc:/system/sac:default
online     Mar_26  svc:/network/inetd:default
online     Mar_26  svc:/system/postrun:defaul
online     Mar_26  svc:/system/utmp:default
online     Mar_26  svc:/network/nfs/client:de
online     Mar_26  svc:/system/filesystem/aut
online     Mar_26  svc:/application/stosreg:d
online     Mar_26  svc:/system/dumpadm:defaul
online     Mar_26  svc:/system/system-log:def
online     Mar_26  svc:/application/managemen
online     Mar_26  svc:/system/auditd:default
online     Mar_26  svc:/system/console-login:
online     Mar_26  svc:/application/font/fc-c
online     Mar_26  svc:/system/fmd:default
online     Mar_26  svc:/network/rpc/meta:defa
online     Mar_26  svc:/network/stdiscover:de
online     Mar_26  svc:/network/stlisten:defa
online     Mar_26  svc:/network/bpcd/tcp:defa
online     Mar_26  svc:/network/vnetd/tcp:def
online     Mar_26  svc:/network/vopied/tcp:de
online     Mar_26  svc:/network/bpjava-msvc/t
online     Mar_26  svc:/network/ctmagent/tcp:
online     Mar_26  svc:/system/mdmonitor:defa
online     Mar_26  svc:/milestone/multi-user:
online     Mar_26  svc:/system/vxvm/vxvm-reco
online     Mar_26  svc:/application/cde-print
online     Mar_26  svc:/milestone/multi-user-
$ who -r
  .    run-level 3  Mar 26 08:06   3    0  S
A snippet when machine was going down as i know its reboots at 8.
Mar 26 08:00:09 slfappprd01 root[13307]: [ID 702911 user.crit] Stopping agent
Mar 26 08:00:09 slfappprd01 root[13307]: [ID 702911 user.crit] Stopping agent
Mar 26 08:00:33 slfappprd01 su: [ID 366847 auth.info] 'su ctmag' succeeded for root on /dev/???
Mar 26 08:00:33 slfappprd01 last message repeated 2 times
Mar 26 08:00:33 slfappprd01 boks_sshd[2839]: [ID 800047 auth.info] Received signal 15; terminating.
Mar 26 08:00:36 slfappprd01 syseventd[430]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
Mar 26 08:00:36 slfappprd01 syseventd[430]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
Mar 26 08:00:36 slfappprd01 vxesd[785]: [ID 477869 daemon.notice] Event source daemon stopped
Mar 26 08:00:36 slfappprd01 vxesd[785]: [ID 477869 daemon.notice] Event source daemon stopped
Mar 26 08:00:37 slfappprd01 syseventd[430]: [ID 661968 daemon.error] Daemon restarted
Mar 26 08:00:37 slfappprd01 syseventd[430]: [ID 661968 daemon.error] Daemon restarted
Mar 26 08:00:47 slfappprd01 rpc.metad: [ID 702911 daemon.error] Terminated
Mar 26 08:00:47 slfappprd01 rpc.metad: [ID 702911 daemon.error] Terminated
Mar 26 08:00:47 slfappprd01 syslogd: going down on signal 15
Mar 26 08:00:53 slfappprd01 rpcbind: [ID 564983 daemon.error] rpcbind terminating on signal.
Mar 26 08:00:53 slfappprd01 rpcbind: [ID 564983 daemon.error] rpcbind terminating on signal.
Mar 26 08:05:43 slfappprd01 ifconfig[28]: [ID 293258 user.warning] libsldap: Status: 2 Â Mesg: Unable to load configuration '/var/ldap/ldap_client_
file' ('').
I have seen in catalina.out that tomcat generally comes up around 8:13 so i checked /var/adm/messages for close to 8:13.
### su to application id
Mar 26 08:09:28 slfappprd01 su: [ID 366847 auth.info] 'su compass'' succeeded for root on /dev/???
Mar 26 08:09:29 slfappprd01 nrpe[6163]: [ID 320800 daemon.info] INFO: SSL/TLS initialized. All network traffic will be encrypted.
Mar 26 08:09:27 slfappprd01 last message repeated 2 times
Mar 26 08:09:29 slfappprd01 nrpe[6164]: [ID 601491 daemon.notice] Starting up daemon
##all over in messages file at different places.
Mar 26 08:09:36 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:09:36 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
Mar 26 08:09:51 slfappprd01 last message repeated 5 times
Mar 26 08:09:51 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:09:51 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(0): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
## during 8:13 again su to compass
Mar 26 08:13:52 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
Mar 26 08:14:07 slfappprd01 last message repeated 5 times
Mar 26 08:14:07 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:14:37 slfappprd01 su: [ID 366847 auth.info] 'su aqueduct' succeeded for root on /dev/???
Mar 26 08:14:37 slfappprd01 <38>bgsud[5005]: bgsu monitoring su aqueduct by root
Mar 26 08:14:37 slfappprd01 last message repeated 1 time
Mar 26 08:20:02 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43904 ssh2
Mar 26 08:20:03 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] subsystem request for sftp
Mar 26 08:20:03 slfappprd01 boks_sshd[7154]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43963 ssh2
Mar 26 08:13:52 slfappprd01 last message repeated 3 times
Mar 26 08:13:52 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:13:52 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): fc packet to DID 0xfffffa failed state: Local Reject(0x3); reason: Il
legal Request(0xe)
Mar 26 08:14:07 slfappprd01 last message repeated 5 times
Mar 26 08:14:07 slfappprd01 fcsm: [ID 517869 kern.warning] WARNING: fcsm(1): ct_intr: CT command <0x118> to did 0xfffffa failed state: Local Rejec
t(0x3); reason: Undefined(0x0)
Mar 26 08:14:37 slfappprd01 su: [ID 366847 auth.info] 'su compass' succeeded for root on /dev/???
Mar 26 08:14:37 slfappprd01 <38>bgsud[5005]: bgsu monitoring su compass by root
Mar 26 08:14:37 slfappprd01 last message repeated 1 time
Mar 26 08:20:02 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43904 ssh2
Mar 26 08:20:03 slfappprd01 boks_sshd[7152]: [ID 800047 auth.info] subsystem request for sftp
Mar 26 08:20:03 slfappprd01 boks_sshd[7154]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 43963 ssh2
Mar 26 08:13:41 slfappprd01 last message repeated 87 times
Mar 26 08:21:04 slfappprd01 boks_sshd[7188]: [ID 800047 auth.info] Accepted publickey for kru_bank from 169.113.208.196 port 44007 ssh2
Mar 26 08:21:04 slfappprd01 boks_sshd[7188]: [ID 800047 auth.info] subsystem request for sftp
it suppose to show that it tried to execute the script for starting up the tomcat?
Please suggest.
ASKER
please read legacy_run   Mar_26  lrc:/etc/rc3_d/S99startTig
as        legacy_run   Mar_26  lrc:/etc/rc3_d/S99startLio
forgot to change the name before posting.
as        legacy_run   Mar_26  lrc:/etc/rc3_d/S99startLio
forgot to change the name before posting.
ASKER CERTIFIED SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
it looks like there may have been a fiber event around the time it was trying to bring up the app.
ASKER
Thank you for your prompt reply.
$ /usr/sbin/luxadm -e port
Permission denied - /devices/pci@1c,600000/lpf
Permission denied - /devices/pci@1e,600000/lpf
$ iostat -eE
      ---- errors ---
device   s/w h/w trn tot
md0 Â Â Â 0 Â 0 Â 0 Â 0
md1 Â Â Â 0 Â 0 Â 0 Â 0
md10 Â Â Â 0 Â 0 Â 0 Â 0
md11 Â Â Â 0 Â 0 Â 0 Â 0
md20 Â Â Â 0 Â 0 Â 0 Â 0
md21 Â Â Â 0 Â 0 Â 0 Â 0
ramdisk1 Â 0 Â 0 Â 0 Â 0
sd0 Â Â Â 0 Â 0 Â 0 Â 0
sd1 Â Â Â 0 Â 0 Â 0 Â 0
sd2 Â Â Â 0 Â 0 Â 0 Â 0
sd3 Â Â Â 1 Â 0 Â 1 Â 2
sd4 Â Â Â 0 Â 0 Â 0 Â 0
ssd0 Â Â Â 0 Â 0 Â 0 Â 0
ssd1 Â Â Â 0 Â 0 Â 0 Â 0
ssd2 Â Â Â 0 Â 0 Â 0 Â 0
ssd3 Â Â Â 0 Â 0 Â 0 Â 0
ssd4 Â Â Â 0 Â 0 Â 0 Â 0
ssd5 Â Â Â 0 Â 0 Â 0 Â 0
ssd6 Â Â Â 0 Â 0 Â 0 Â 0
ssd7 Â Â Â 0 Â 0 Â 0 Â 0
ssd8 Â Â Â 0 Â 0 Â 0 Â 0
ssd9 Â Â Â 0 Â 0 Â 0 Â 0
ssd10 Â Â 0 Â 0 Â 0 Â 0
ssd11 Â Â 0 Â 0 Â 0 Â 0
ssd12 Â Â 0 Â 0 Â 0 Â 0
ssd13 Â Â 0 Â 0 Â 0 Â 0
ssd14 Â Â 0 Â 0 Â 0 Â 0
ssd15 Â Â 0 Â 0 Â 0 Â 0
ssd16 Â Â 0 Â 0 Â 0 Â 0
ssd17 Â Â 0 Â 0 Â 0 Â 0
ssd18 Â Â 0 Â 0 Â 0 Â 0
ssd19 Â Â 0 Â 0 Â 0 Â 0
sd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZFNF
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZMNZ
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054642SNPE
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd3 Â Â Â Soft Errors: 1 Hard Errors: 0 Transport Errors: 1
Vendor: TOSHIBA Â Product: ODD-DVD SD-C2732 Revision: 1055 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
sd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842TR18
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd3 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd5 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd6 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd7 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd8 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd9 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd10 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd11 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd12 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd13 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd14 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd15 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd16 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd17 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd18 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd19 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
$ /usr/sbin/luxadm -e port
Permission denied - /devices/pci@1c,600000/lpf
Permission denied - /devices/pci@1e,600000/lpf
$ iostat -eE
      ---- errors ---
device   s/w h/w trn tot
md0 Â Â Â 0 Â 0 Â 0 Â 0
md1 Â Â Â 0 Â 0 Â 0 Â 0
md10 Â Â Â 0 Â 0 Â 0 Â 0
md11 Â Â Â 0 Â 0 Â 0 Â 0
md20 Â Â Â 0 Â 0 Â 0 Â 0
md21 Â Â Â 0 Â 0 Â 0 Â 0
ramdisk1 Â 0 Â 0 Â 0 Â 0
sd0 Â Â Â 0 Â 0 Â 0 Â 0
sd1 Â Â Â 0 Â 0 Â 0 Â 0
sd2 Â Â Â 0 Â 0 Â 0 Â 0
sd3 Â Â Â 1 Â 0 Â 1 Â 2
sd4 Â Â Â 0 Â 0 Â 0 Â 0
ssd0 Â Â Â 0 Â 0 Â 0 Â 0
ssd1 Â Â Â 0 Â 0 Â 0 Â 0
ssd2 Â Â Â 0 Â 0 Â 0 Â 0
ssd3 Â Â Â 0 Â 0 Â 0 Â 0
ssd4 Â Â Â 0 Â 0 Â 0 Â 0
ssd5 Â Â Â 0 Â 0 Â 0 Â 0
ssd6 Â Â Â 0 Â 0 Â 0 Â 0
ssd7 Â Â Â 0 Â 0 Â 0 Â 0
ssd8 Â Â Â 0 Â 0 Â 0 Â 0
ssd9 Â Â Â 0 Â 0 Â 0 Â 0
ssd10 Â Â 0 Â 0 Â 0 Â 0
ssd11 Â Â 0 Â 0 Â 0 Â 0
ssd12 Â Â 0 Â 0 Â 0 Â 0
ssd13 Â Â 0 Â 0 Â 0 Â 0
ssd14 Â Â 0 Â 0 Â 0 Â 0
ssd15 Â Â 0 Â 0 Â 0 Â 0
ssd16 Â Â 0 Â 0 Â 0 Â 0
ssd17 Â Â 0 Â 0 Â 0 Â 0
ssd18 Â Â 0 Â 0 Â 0 Â 0
ssd19 Â Â 0 Â 0 Â 0 Â 0
sd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZFNF
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZMNZ
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054642SNPE
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd3 Â Â Â Soft Errors: 1 Hard Errors: 0 Transport Errors: 1
Vendor: TOSHIBA Â Product: ODD-DVD SD-C2732 Revision: 1055 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
sd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842TR18
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd3 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd5 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd6 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd7 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd8 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd9 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd10 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd11 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd12 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd13 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd14 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd15 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd16 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd17 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd18 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd19 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
you have errors on the sd3 device maybe there were some issues upon reboot. It looks like it might map back to solaris volume manager
ASKER
I escalated to my UNIX team and they are saying they dont see any unusual thing and asking to wait on next reboot .Well lets see ..i am going to run all the commands provided by you guys and compare if it comes up fine and update this thread.
Thank you all of you for your precious time and guidance.
Thank you all of you for your precious time and guidance.
I would wait and see if it happens again, it would try to increase logging on the startup scripts.
ASKER
Hi All,
Server rebooted today again and application came up fine automatically. I ran the commands and result is similar to previous week output. sd3 device has the same error.
Dont know what exactly change.
Thank you guys for sharing your knowledge.
$svcs
STATE Â Â Â Â Â STIME Â Â FMRI
legacy_run    8:06:31 lrc:/etc/rcS_d/S29wrsmcfg
legacy_run    8:06:41 lrc:/etc/rc2_d/S01nddconfi
legacy_run    8:06:41 lrc:/etc/rc2_d/S20sysetup
legacy_run    8:06:41 lrc:/etc/rc2_d/S42ncakmod
legacy_run    8:06:42 lrc:/etc/rc2_d/S45vxpbx_ex
legacy_run    8:06:42 lrc:/etc/rc2_d/S47pppd
legacy_run    8:06:42 lrc:/etc/rc2_d/S50vxvail
legacy_run    8:06:42 lrc:/etc/rc2_d/S64preisisd
legacy_run    8:06:43 lrc:/etc/rc2_d/S70vxatd
legacy_run    8:06:43 lrc:/etc/rc2_d/S73cachefs_
legacy_run    8:06:53 lrc:/etc/rc2_d/S73isisd
legacy_run    8:06:54 lrc:/etc/rc2_d/S750vxpal_g
legacy_run    8:07:12 lrc:/etc/rc2_d/S75vxpal_St
legacy_run    8:07:12 lrc:/etc/rc2_d/S75vxsmfd
legacy_run    8:07:20 lrc:/etc/rc2_d/S760vxpal_a
legacy_run    8:07:21 lrc:/etc/rc2_d/S79opsware-
legacy_run    8:07:21 lrc:/etc/rc2_d/S89PRESERVE
legacy_run    8:07:21 lrc:/etc/rc2_d/S91afbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91gfbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91ifbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91jfbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91kfbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91zuluinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S94ncalogd
legacy_run    8:07:22 lrc:/etc/rc2_d/S94vxnm-vxn
legacy_run    8:07:22 lrc:/etc/rc2_d/S96vradmind
legacy_run    8:07:23 lrc:/etc/rc2_d/S96vxrsyncd
legacy_run    8:07:23 lrc:/etc/rc2_d/S98dealloca
legacy_run    8:07:24 lrc:/etc/rc2_d/S99bokcfg
legacy_run    8:07:55 lrc:/etc/rc2_d/S99boksm
legacy_run    8:07:55 lrc:/etc/rc2_d/S99eccmad
legacy_run    8:08:04 lrc:/etc/rc2_d/S99sneep
legacy_run    8:08:04 lrc:/etc/rc2_d/S99xboksm-a
legacy_run    8:08:06 lrc:/etc/rc3_d/S69esmrc
legacy_run    8:08:06 lrc:/etc/rc3_d/S75vxsmfd
legacy_run    8:08:09 lrc:/etc/rc3_d/S78bgsu
legacy_run    8:08:09 lrc:/etc/rc3_d/S80mipagent
legacy_run    8:08:15 lrc:/etc/rc3_d/S89jpmc-net
legacy_run    8:08:15 lrc:/etc/rc3_d/S95X11R6
legacy_run    8:09:02 lrc:/etc/rc3_d/S95dbora
legacy_run    8:09:04 lrc:/etc/rc3_d/S98control-
legacy_run    8:09:04 lrc:/etc/rc3_d/S99Tivoli_l
legacy_run    8:09:05 lrc:/etc/rc3_d/S99aqueduct
legacy_run    8:09:05 lrc:/etc/rc3_d/S99gcstartu
legacy_run    8:09:05 lrc:/etc/rc3_d/S99infra
legacy_run    8:14:15 lrc:/etc/rc3_d/S99startTig
legacy_run    8:14:15 lrc:/etc/rc3_d/S99vxtf_chk
online      8:05:39 svc:/system/svc/restarter:
online      8:05:40 svc:/network/pfil:default
online      8:05:40 svc:/network/tnctl:default
online      8:05:40 svc:/network/loopback:defa
online      8:05:41 svc:/system/installupdates
online      8:05:52 svc:/network/physical:defa
online      8:05:53 svc:/system/identity:node
online      8:05:53 svc:/system/metainit:defau
online      8:05:55 svc:/system/vxvm/vxvm-sysb
online      8:05:56 svc:/system/filesystem/roo
online      8:05:56 svc:/system/scheduler:defa
online      8:05:57 svc:/system/vxvm/vxvm-star
online      8:05:57 svc:/system/boot-archive:d
online      8:05:58 svc:/system/filesystem/usr
online      8:05:59 svc:/system/keymap:default
online      8:05:59 svc:/system/device/local:d
online      8:05:59 svc:/system/filesystem/min
online      8:05:59 svc:/system/coreadm:defaul
online      8:05:59 svc:/system/resource-mgmt:
online      8:05:59 svc:/site/JPMCsvc_prop_mod
online      8:05:59 svc:/system/rmtmpfiles:def
online      8:05:59 svc:/system/name-service-c
online      8:05:59 svc:/system/identity:domai
online      8:05:59 svc:/system/sysevent:defau
online      8:05:59 svc:/system/cryptosvc:defa
online      8:05:59 svc:/system/device/fc-fabr
online      8:05:59 svc:/system/picl:default
online      8:06:00 svc:/system/manifest-impor
online      8:06:17 svc:/application/print/ppd
online      8:06:28 svc:/site/network/JPMCnets
online      8:06:28 svc:/milestone/network:def
online      8:06:29 svc:/network/initial:defau
online      8:06:29 svc:/network/service:defau
online      8:06:30 svc:/system/vxvm/vxvm-star
online      8:06:31 svc:/system/vxvm/vxvm-star
online      8:06:31 svc:/milestone/devices:def
online      8:06:31 svc:/system/vxvm/vxvm-reco
online      8:06:31 svc:/milestone/single-user
online      8:06:32 svc:/network/ldap/client:d
online      8:06:33 svc:/milestone/name-servic
online      8:06:34 svc:/system/filesystem/loc
online      8:06:35 svc:/network/shares/group:
online      8:06:35 svc:/system/boot-archive-u
online      8:06:35 svc:/system/cron:default
online      8:06:35 svc:/system/sysidtool:net
online      8:06:36 svc:/network/routing-setup
online      8:06:36 svc:/system/vxfs/vxfsldlic
online      8:06:36 svc:/network/ntp:default
online      8:06:36 svc:/network/rpc/bind:defa
online      8:06:37 svc:/network/nfs/cbd:defau
online      8:06:37 svc:/system/sysidtool:syst
online      8:06:37 svc:/network/nfs/mapid:def
online      8:06:37 svc:/network/nfs/status:de
online      8:06:37 svc:/network/nfs/nlockmgr:
online      8:06:37 svc:/milestone/sysconfig:d
online      8:06:37 svc:/system/sac:default
online      8:06:37 svc:/network/inetd:default
online      8:06:37 svc:/system/utmp:default
online      8:06:37 svc:/network/nfs/client:de
online      8:06:37 svc:/system/postrun:defaul
online      8:06:37 svc:/application/stosreg:d
online      8:06:37 svc:/system/filesystem/aut
online      8:06:38 svc:/system/dumpadm:defaul
online      8:06:38 svc:/system/system-log:def
online      8:06:38 svc:/application/managemen
online      8:06:38 svc:/system/auditd:default
online      8:06:38 svc:/system/console-login:
online      8:06:39 svc:/application/font/fc-c
online      8:06:39 svc:/system/fmd:default
online      8:06:39 svc:/network/rpc/meta:defa
online      8:06:39 svc:/network/stdiscover:de
online      8:06:39 svc:/network/stlisten:defa
online      8:06:39 svc:/network/bpcd/tcp:defa
online      8:06:39 svc:/network/vnetd/tcp:def
online      8:06:39 svc:/network/vopied/tcp:de
online      8:06:39 svc:/network/bpjava-msvc/t
online      8:06:39 svc:/network/ctmagent/tcp:
online      8:06:40 svc:/system/mdmonitor:defa
online      8:08:04 svc:/milestone/multi-user:
online      8:08:04 svc:/system/vxvm/vxvm-reco
online      8:08:05 svc:/application/cde-print
online      8:14:15 svc:/milestone/multi-user-
$ iostat -eE
      ---- errors ---
device  s/w h/w trn tot
md0 Â Â 0 Â 0 Â 0 Â 0
md1 Â Â 0 Â 0 Â 0 Â 0
md10 Â Â 0 Â 0 Â 0 Â 0
md11 Â Â 0 Â 0 Â 0 Â 0
md20 Â Â 0 Â 0 Â 0 Â 0
md21 Â Â 0 Â 0 Â 0 Â 0
sd0 Â Â 0 Â 0 Â 0 Â 0
sd1 Â Â 0 Â 0 Â 0 Â 0
sd2 Â Â 0 Â 0 Â 0 Â 0
sd3 Â Â 1 Â 0 Â 1 Â 2
sd4 Â Â 0 Â 0 Â 0 Â 0
ssd0 Â Â 0 Â 0 Â 0 Â 0
ssd1 Â Â 0 Â 0 Â 0 Â 0
ssd2 Â Â 0 Â 0 Â 0 Â 0
ssd3 Â Â 0 Â 0 Â 0 Â 0
ssd4 Â Â 0 Â 0 Â 0 Â 0
ssd5 Â Â 0 Â 0 Â 0 Â 0
ssd6 Â Â 0 Â 0 Â 0 Â 0
ssd7 Â Â 0 Â 0 Â 0 Â 0
ssd8 Â Â 0 Â 0 Â 0 Â 0
ssd9 Â Â 0 Â 0 Â 0 Â 0
ssd10 Â 0 Â 0 Â 0 Â 0
ssd11 Â 0 Â 0 Â 0 Â 0
ssd12 Â 0 Â 0 Â 0 Â 0
ssd13 Â 0 Â 0 Â 0 Â 0
ssd14 Â 0 Â 0 Â 0 Â 0
ssd15 Â 0 Â 0 Â 0 Â 0
ssd16 Â 0 Â 0 Â 0 Â 0
ssd17 Â 0 Â 0 Â 0 Â 0
ssd18 Â 0 Â 0 Â 0 Â 0
ssd19 Â 0 Â 0 Â 0 Â 0
sd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZFNF
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZMNZ
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054642SNPE
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd3 Â Â Â Soft Errors: 1 Hard Errors: 0 Transport Errors: 1
Vendor: TOSHIBA Â Product: ODD-DVD SD-C2732 Revision: 1055 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
sd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842TR18
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd3 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd5 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd6 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd7 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd8 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd9 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd10 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd11 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd12 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd13 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd14 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd15 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd16 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd17 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd18 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd19 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
Server rebooted today again and application came up fine automatically. I ran the commands and result is similar to previous week output. sd3 device has the same error.
Dont know what exactly change.
Thank you guys for sharing your knowledge.
$svcs
STATE Â Â Â Â Â STIME Â Â FMRI
legacy_run    8:06:31 lrc:/etc/rcS_d/S29wrsmcfg
legacy_run    8:06:41 lrc:/etc/rc2_d/S01nddconfi
legacy_run    8:06:41 lrc:/etc/rc2_d/S20sysetup
legacy_run    8:06:41 lrc:/etc/rc2_d/S42ncakmod
legacy_run    8:06:42 lrc:/etc/rc2_d/S45vxpbx_ex
legacy_run    8:06:42 lrc:/etc/rc2_d/S47pppd
legacy_run    8:06:42 lrc:/etc/rc2_d/S50vxvail
legacy_run    8:06:42 lrc:/etc/rc2_d/S64preisisd
legacy_run    8:06:43 lrc:/etc/rc2_d/S70vxatd
legacy_run    8:06:43 lrc:/etc/rc2_d/S73cachefs_
legacy_run    8:06:53 lrc:/etc/rc2_d/S73isisd
legacy_run    8:06:54 lrc:/etc/rc2_d/S750vxpal_g
legacy_run    8:07:12 lrc:/etc/rc2_d/S75vxpal_St
legacy_run    8:07:12 lrc:/etc/rc2_d/S75vxsmfd
legacy_run    8:07:20 lrc:/etc/rc2_d/S760vxpal_a
legacy_run    8:07:21 lrc:/etc/rc2_d/S79opsware-
legacy_run    8:07:21 lrc:/etc/rc2_d/S89PRESERVE
legacy_run    8:07:21 lrc:/etc/rc2_d/S91afbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91gfbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91ifbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91jfbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91kfbinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S91zuluinit
legacy_run    8:07:21 lrc:/etc/rc2_d/S94ncalogd
legacy_run    8:07:22 lrc:/etc/rc2_d/S94vxnm-vxn
legacy_run    8:07:22 lrc:/etc/rc2_d/S96vradmind
legacy_run    8:07:23 lrc:/etc/rc2_d/S96vxrsyncd
legacy_run    8:07:23 lrc:/etc/rc2_d/S98dealloca
legacy_run    8:07:24 lrc:/etc/rc2_d/S99bokcfg
legacy_run    8:07:55 lrc:/etc/rc2_d/S99boksm
legacy_run    8:07:55 lrc:/etc/rc2_d/S99eccmad
legacy_run    8:08:04 lrc:/etc/rc2_d/S99sneep
legacy_run    8:08:04 lrc:/etc/rc2_d/S99xboksm-a
legacy_run    8:08:06 lrc:/etc/rc3_d/S69esmrc
legacy_run    8:08:06 lrc:/etc/rc3_d/S75vxsmfd
legacy_run    8:08:09 lrc:/etc/rc3_d/S78bgsu
legacy_run    8:08:09 lrc:/etc/rc3_d/S80mipagent
legacy_run    8:08:15 lrc:/etc/rc3_d/S89jpmc-net
legacy_run    8:08:15 lrc:/etc/rc3_d/S95X11R6
legacy_run    8:09:02 lrc:/etc/rc3_d/S95dbora
legacy_run    8:09:04 lrc:/etc/rc3_d/S98control-
legacy_run    8:09:04 lrc:/etc/rc3_d/S99Tivoli_l
legacy_run    8:09:05 lrc:/etc/rc3_d/S99aqueduct
legacy_run    8:09:05 lrc:/etc/rc3_d/S99gcstartu
legacy_run    8:09:05 lrc:/etc/rc3_d/S99infra
legacy_run    8:14:15 lrc:/etc/rc3_d/S99startTig
legacy_run    8:14:15 lrc:/etc/rc3_d/S99vxtf_chk
online      8:05:39 svc:/system/svc/restarter:
online      8:05:40 svc:/network/pfil:default
online      8:05:40 svc:/network/tnctl:default
online      8:05:40 svc:/network/loopback:defa
online      8:05:41 svc:/system/installupdates
online      8:05:52 svc:/network/physical:defa
online      8:05:53 svc:/system/identity:node
online      8:05:53 svc:/system/metainit:defau
online      8:05:55 svc:/system/vxvm/vxvm-sysb
online      8:05:56 svc:/system/filesystem/roo
online      8:05:56 svc:/system/scheduler:defa
online      8:05:57 svc:/system/vxvm/vxvm-star
online      8:05:57 svc:/system/boot-archive:d
online      8:05:58 svc:/system/filesystem/usr
online      8:05:59 svc:/system/keymap:default
online      8:05:59 svc:/system/device/local:d
online      8:05:59 svc:/system/filesystem/min
online      8:05:59 svc:/system/coreadm:defaul
online      8:05:59 svc:/system/resource-mgmt:
online      8:05:59 svc:/site/JPMCsvc_prop_mod
online      8:05:59 svc:/system/rmtmpfiles:def
online      8:05:59 svc:/system/name-service-c
online      8:05:59 svc:/system/identity:domai
online      8:05:59 svc:/system/sysevent:defau
online      8:05:59 svc:/system/cryptosvc:defa
online      8:05:59 svc:/system/device/fc-fabr
online      8:05:59 svc:/system/picl:default
online      8:06:00 svc:/system/manifest-impor
online      8:06:17 svc:/application/print/ppd
online      8:06:28 svc:/site/network/JPMCnets
online      8:06:28 svc:/milestone/network:def
online      8:06:29 svc:/network/initial:defau
online      8:06:29 svc:/network/service:defau
online      8:06:30 svc:/system/vxvm/vxvm-star
online      8:06:31 svc:/system/vxvm/vxvm-star
online      8:06:31 svc:/milestone/devices:def
online      8:06:31 svc:/system/vxvm/vxvm-reco
online      8:06:31 svc:/milestone/single-user
online      8:06:32 svc:/network/ldap/client:d
online      8:06:33 svc:/milestone/name-servic
online      8:06:34 svc:/system/filesystem/loc
online      8:06:35 svc:/network/shares/group:
online      8:06:35 svc:/system/boot-archive-u
online      8:06:35 svc:/system/cron:default
online      8:06:35 svc:/system/sysidtool:net
online      8:06:36 svc:/network/routing-setup
online      8:06:36 svc:/system/vxfs/vxfsldlic
online      8:06:36 svc:/network/ntp:default
online      8:06:36 svc:/network/rpc/bind:defa
online      8:06:37 svc:/network/nfs/cbd:defau
online      8:06:37 svc:/system/sysidtool:syst
online      8:06:37 svc:/network/nfs/mapid:def
online      8:06:37 svc:/network/nfs/status:de
online      8:06:37 svc:/network/nfs/nlockmgr:
online      8:06:37 svc:/milestone/sysconfig:d
online      8:06:37 svc:/system/sac:default
online      8:06:37 svc:/network/inetd:default
online      8:06:37 svc:/system/utmp:default
online      8:06:37 svc:/network/nfs/client:de
online      8:06:37 svc:/system/postrun:defaul
online      8:06:37 svc:/application/stosreg:d
online      8:06:37 svc:/system/filesystem/aut
online      8:06:38 svc:/system/dumpadm:defaul
online      8:06:38 svc:/system/system-log:def
online      8:06:38 svc:/application/managemen
online      8:06:38 svc:/system/auditd:default
online      8:06:38 svc:/system/console-login:
online      8:06:39 svc:/application/font/fc-c
online      8:06:39 svc:/system/fmd:default
online      8:06:39 svc:/network/rpc/meta:defa
online      8:06:39 svc:/network/stdiscover:de
online      8:06:39 svc:/network/stlisten:defa
online      8:06:39 svc:/network/bpcd/tcp:defa
online      8:06:39 svc:/network/vnetd/tcp:def
online      8:06:39 svc:/network/vopied/tcp:de
online      8:06:39 svc:/network/bpjava-msvc/t
online      8:06:39 svc:/network/ctmagent/tcp:
online      8:06:40 svc:/system/mdmonitor:defa
online      8:08:04 svc:/milestone/multi-user:
online      8:08:04 svc:/system/vxvm/vxvm-reco
online      8:08:05 svc:/application/cde-print
online      8:14:15 svc:/milestone/multi-user-
$ iostat -eE
      ---- errors ---
device  s/w h/w trn tot
md0 Â Â 0 Â 0 Â 0 Â 0
md1 Â Â 0 Â 0 Â 0 Â 0
md10 Â Â 0 Â 0 Â 0 Â 0
md11 Â Â 0 Â 0 Â 0 Â 0
md20 Â Â 0 Â 0 Â 0 Â 0
md21 Â Â 0 Â 0 Â 0 Â 0
sd0 Â Â 0 Â 0 Â 0 Â 0
sd1 Â Â 0 Â 0 Â 0 Â 0
sd2 Â Â 0 Â 0 Â 0 Â 0
sd3 Â Â 1 Â 0 Â 1 Â 2
sd4 Â Â 0 Â 0 Â 0 Â 0
ssd0 Â Â 0 Â 0 Â 0 Â 0
ssd1 Â Â 0 Â 0 Â 0 Â 0
ssd2 Â Â 0 Â 0 Â 0 Â 0
ssd3 Â Â 0 Â 0 Â 0 Â 0
ssd4 Â Â 0 Â 0 Â 0 Â 0
ssd5 Â Â 0 Â 0 Â 0 Â 0
ssd6 Â Â 0 Â 0 Â 0 Â 0
ssd7 Â Â 0 Â 0 Â 0 Â 0
ssd8 Â Â 0 Â 0 Â 0 Â 0
ssd9 Â Â 0 Â 0 Â 0 Â 0
ssd10 Â 0 Â 0 Â 0 Â 0
ssd11 Â 0 Â 0 Â 0 Â 0
ssd12 Â 0 Â 0 Â 0 Â 0
ssd13 Â 0 Â 0 Â 0 Â 0
ssd14 Â 0 Â 0 Â 0 Â 0
ssd15 Â 0 Â 0 Â 0 Â 0
ssd16 Â 0 Â 0 Â 0 Â 0
ssd17 Â 0 Â 0 Â 0 Â 0
ssd18 Â 0 Â 0 Â 0 Â 0
ssd19 Â 0 Â 0 Â 0 Â 0
sd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZFNF
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842ZMNZ
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054642SNPE
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd3 Â Â Â Soft Errors: 1 Hard Errors: 0 Transport Errors: 1
Vendor: TOSHIBA Â Product: ODD-DVD SD-C2732 Revision: 1055 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
sd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Â Product: ST314670LSUN146G Revision: 045A Serial No: 054842TR18
Size: 146.80GB <146800115712 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd0 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd1 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd2 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd3 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd4 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd5 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd6 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd7 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd8 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd9 Â Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd10 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd11 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd12 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd13 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 0.00GB <2949120 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd14 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd15 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd16 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd17 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd18 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd19 Â Â Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: EMC Â Â Â Product: SYMMETRIX Â Â Â Â Revision: 5773 Serial No:
Size: 31.95GB <31947816960 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
If the app came up as expected it looks like it might have been a fluke issue that you might not be able to reproduce.
Are your reboots scheduled? Or are these crashes?
To see what device is using the disk do an metastat
Are your reboots scheduled? Or are these crashes?
To see what device is using the disk do an metastat
ASKER
yes,reboot is scheduled for Every week.
Here is the output:
$ /usr/sbin/metastat -p
d1 -m d11 d21 1
d11 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d0 -m d10 d20 1
d10 1 1 c1t0d0s0
d20 1 1 c1t1d0s0
$ /usr/sbin/metastat
d1: Mirror
  Submirror 0: d11
   State: Okay
  Submirror 1: d21
   State: Okay
  Pass: 1
  Read option: roundrobin (default)
  Write option: parallel (default)
  Size: 40968576 blocks (19 GB)
d11: Submirror of d1
  State: Okay
  Size: 40968576 blocks (19 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t0d0s1      0   No       Okay  Yes
d21: Submirror of d1
  State: Okay
  Size: 40968576 blocks (19 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t1d0s1      0   No       Okay  Yes
d0: Mirror
  Submirror 0: d10
   State: Okay
  Submirror 1: d20
   State: Okay
  Pass: 1
  Read option: roundrobin (default)
  Write option: parallel (default)
  Size: 245506176 blocks (117 GB)
d10: Submirror of d0
  State: Okay
  Size: 245506176 blocks (117 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t0d0s0      0   No       Okay  Yes
d20: Submirror of d0
  State: Okay
  Size: 245506176 blocks (117 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t1d0s0      0   No       Okay  Yes
Device Relocation Information:
Device  Reloc  Device ID
c1t1d0  Yes   id1,sd@SSEAGATE_ST314670LS
c1t0d0  Yes   id1,sd@SSEAGATE_ST314670LS
$
thanks
Here is the output:
$ /usr/sbin/metastat -p
d1 -m d11 d21 1
d11 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d0 -m d10 d20 1
d10 1 1 c1t0d0s0
d20 1 1 c1t1d0s0
$ /usr/sbin/metastat
d1: Mirror
  Submirror 0: d11
   State: Okay
  Submirror 1: d21
   State: Okay
  Pass: 1
  Read option: roundrobin (default)
  Write option: parallel (default)
  Size: 40968576 blocks (19 GB)
d11: Submirror of d1
  State: Okay
  Size: 40968576 blocks (19 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t0d0s1      0   No       Okay  Yes
d21: Submirror of d1
  State: Okay
  Size: 40968576 blocks (19 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t1d0s1      0   No       Okay  Yes
d0: Mirror
  Submirror 0: d10
   State: Okay
  Submirror 1: d20
   State: Okay
  Pass: 1
  Read option: roundrobin (default)
  Write option: parallel (default)
  Size: 245506176 blocks (117 GB)
d10: Submirror of d0
  State: Okay
  Size: 245506176 blocks (117 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t0d0s0      0   No       Okay  Yes
d20: Submirror of d0
  State: Okay
  Size: 245506176 blocks (117 GB)
  Stripe 0:
    Device   Start Block  Dbase     State Reloc Hot Spare
    c1t1d0s0      0   No       Okay  Yes
Device Relocation Information:
Device  Reloc  Device ID
c1t1d0  Yes   id1,sd@SSEAGATE_ST314670LS
c1t0d0  Yes   id1,sd@SSEAGATE_ST314670LS
$
thanks
ASKER
Didnt find the exact solution but learned a lot of things.
Thank you guys.
Thank you guys.
ASKER
Is it possible to use one rc script to stop and start.
Because  when i grepped for my application,compass using i see only one file.
Or is it possible to have another rc script in higher level for starting the tomcat?
bash-3.00$ pwd
/etc/rc2.d
bash-3.00$ grep -i compass  *
bash-3.00$ more K99stopLionnPRODEnv
#!/bin/sh
case "$1" in
'start')
     su - compass -c "/apps/compass/startStopJa
     sleep 180
     /apps/compass/startStopLio
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
    su - compass -c "/apps/compass/startStopLi
    ;;
'stop')
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     su - compass -c "/apps/compass/startStopLi
     /apps/compass/startStopLio
     su - compass -c "/apps/compass/startStopJa
    ;;
*)
    echo "Usage: $0 { start | stop }"
    exit