Link to home
Start Free TrialLog in
Avatar of questil
questil

asked on

Solaris 9 keep crashing

Hello experts!

I have a Solaris 9 system that after power outage I started him in maintenance mode and run fsck –y.
One of the file systems (oracle database file system) return an error:
Can't open /dev/rdsk/c2t0d0s0: I/O error
So I commented out this file system on /etc/vfstab and booted the system.

It came up ok, but since then every 30~ it crashes and reset.
Any idea how to find the cause of these crashes?
/var/adm/messages attached, can someone help me figure it out or any other place I should look?

Thanks!
Tal
sol09
Avatar of Joseph Gan
Joseph Gan
Flag of Australia image

Please list outputs of following:

# iostat -En
# df -k
Avatar of questil
questil

ASKER

bash-2.05# iostat -En
c0t6d0          Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: TOSHIBA  Product: DVD-ROM SD-M1401 Revision: 1009 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c2t0d0          Soft Errors: 0 Hard Errors: 18 Transport Errors: 0
Vendor: IBM      Product:       Revision:  Serial No:  Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 18 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c2t1d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: IBM      Product: IC35L036UWDY10-0 Revision: S29A Serial No:
Size: 36.70GB <36702124032 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c2t2d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: FUJITSU  Product: MAP3735NP        Revision: 0108 Serial No:
Size: 73.43GB <73425438720 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c2t3d0          Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE  Product: ST373307LW       Revision: 0006 Serial No:
Size: 73.41GB <73405071360 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c2t4d0          Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE  Product: ST3146707LW      Revision: 0005 Serial No:
Size: 146.81GB <146811883520 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c1t0d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE  Product: ST336607FSUN36G  Revision: 0207 Serial No: 0302A0PNK1
Size: 36.42GB <36418595328 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c1t1d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE  Product: ST336607FSUN36G  Revision: 0207 Serial No: 0302A0PAXH
Size: 36.42GB <36418595328 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0


bash-2.05# df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    2437448 2219622  169078    93%    /
/proc                      0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
fd                         0       0       0     0%    /dev/fd
swap                 10997872      40 10997832     1%    /var/run
swap                 10997832       0 10997832     0%    /tmp
/dev/dsk/c2t1d0s6    35019617 12267057 22402364    36%    /inst_qa
/dev/dsk/c2t3d0s3    23522848 8555458 14732162    37%    /db2lv
/dev/dsk/c2t4d0s1    10328704 3147289 7078128    31%    /sybaselv01
/dev/dsk/c2t3d0s1    23521408 21211316 2074878    92%    /inst_dev
/dev/dsk/c2t3d0s0    23521408 20373437 2912757    88%    /oravl03
/dev/dsk/c2t4d0s0    41309179 36415100 4480988    90%    /oravl01
/dev/dsk/c1t0d0s7    28526648 25993979 2247403    93%    /export/home


The file system that I commented out from /etc/vfstab is: /dev/dsk/c2t0d0s0      /oravl02
Yep, the disk c2t0d0 was fauty, should replace/remove it.

c2t0d0          Soft Errors: 0 Hard Errors: 18 Transport Errors: 0

You can disable disk c2t0d0 in the system by
# cfgadm -c unconfigures c2::dsk/c2t0d0

Also root / filesystem is 93% full, this will triger system reboot if it becomes 100%. You should clean it up.
You also have c2t4d0 that might be failing as well.  In future setup, you might want to consider RAID based setup which will provide you with some fault tolerance while minimizing/reducing the likelihood of data loss.
Solaris volume manager, metainit, metadb, metastat, etc.
http://docs.oracle.com/cd/E19082-01/819-2789/about-mirrors-2/index.html
You have a power supply ps1 either has a bad power connection, or the supply itself is failing.

You have a qlogic HBA that is trying to mount/overlay the root filesystem and fails.
ASKER CERTIFIED SOLUTION
Avatar of questil
questil

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of questil

ASKER

Other answers was not enough.