Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

The shared partition can not be mounted

Posted on 2011-04-21
3
Medium Priority
?
1,109 Views
Last Modified: 2012-05-11
I have 2 sun machines of the following info:
root@rotwxza01 # uname -a
SunOS rotwxza01 5.9 Generic_122300-15 sun4u sparc SUNW,Sun-Fire-V490

root@rotwxza02 # uname -a
SunOS rotwxza02 5.9 Generic_122300-15 sun4u sparc SUNW,Sun-Fire-V490


Last week we had a network problem so the cluster is not working, once the network is back we noticed that the cluster still not working so we rebooted both nodes, but we found that shared partition /wxz is not mounted, we tried to mount it manually on both nodes, but we are getting the following errors:
Node 1
root@rotwxza01 # mount /xtc
mount: /dev/vx/dsk/WXZ/wxz_app no such device

Node 2
root@rotwxza02 # mount /xtc
mount: /dev/vx/dsk/WXZ/xtc_app no such device

Here are the output of scstat, and df -k on both nodes:
Node 1

root@rotwxza01 # scstat
------------------------------------------------------------------

-- Cluster Nodes --

                    Node name           Status
                    ---------           ------
  Cluster node:     rotwxza01           Online
  Cluster node:     rotwxza02           Online

------------------------------------------------------------------

-- Cluster Transport Paths --

                    Endpoint               Endpoint               Status
                    --------               --------               ------
  Transport path:   rotwxza01:ce1          rotwxza02:ce1          Path online
  Transport path:   rotwxza01:ce0          rotwxza02:ce0          Path online

------------------------------------------------------------------

-- Quorum Summary --

  Quorum votes possible:      3
  Quorum votes needed:        2
  Quorum votes present:       2


-- Quorum Votes by Node --

                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       rotwxza01           1        1       Online
  Node votes:       rotwxza02           1        1       Online


-- Quorum Votes by Device --

                    Device Name         Present Possible Status
                    -----------         ------- -------- ------
  Device votes:     /dev/did/rdsk/d23s2 0        1       Offline

------------------------------------------------------------------

-- Device Group Servers --

                         Device Group        Primary             Secondary
                         ------------        -------             ---------
  Device group servers:  WXZ                 rotwxza01           rotwxza02


-- Device Group Status --

                              Device Group        Status
                              ------------        ------
  Device group status:        WXZ                 Online


-- Multi-owner Device Groups --

                              Device Group        Online Status
                              ------------        -------------

------------------------------------------------------------------

-- Resource Groups and Resources --

            Group Name          Resources
            ----------          ---------
 Resources: service-rg            rotwxzavip-lh-rs rotwxzait-lh-rs legato-rs service-rs cascon-rs
 Resources: rac-rg              rac-framework-rs rac-udlm-rs rac-cvm-rs
 Resources: wxzac1-rg           instance-wxzac1-rs lsnr-wxzac1-rs
 Resources: wxzac2-rg           instance-wxzac2-rs lsnr-wxzac2-rs


-- Resource Groups --

            Group Name          Node Name           State
            ----------          ---------           -----
     Group: service-rg            rotwxza01           Offline
     Group: service-rg            rotwxza02           Offline

     Group: rac-rg              rotwxza01           Online
     Group: rac-rg              rotwxza02           Online

     Group: wxzac1-rg           rotwxza01           Online

     Group: wxzac2-rg           rotwxza02           Online


-- Resources --

            Resource Name       Node Name           State     Status Message
            -------------       ---------           -----     --------------
  Resource: rotwxzavip-lh-rs    rotwxza01           Offline   Offline - LogicalHostname offline.
  Resource: rotwxzavip-lh-rs    rotwxza02           Offline   Offline - LogicalHostname offline.

  Resource: rotwxzait-lh-rs     rotwxza01           Offline   Offline - LogicalHostname offline.
  Resource: rotwxzait-lh-rs     rotwxza02           Offline   Offline - LogicalHostname offline.

  Resource: legato-rs           rotwxza01           Offline   Offline
  Resource: legato-rs           rotwxza02           Offline   Offline

  Resource: service-rs            rotwxza01           Offline   Offline
  Resource: service-rs            rotwxza02           Offline   Offline

  Resource: cascon-rs           rotwxza01           Offline   Offline
  Resource: cascon-rs           rotwxza02           Offline   Offline

  Resource: rac-framework-rs    rotwxza01           Online    Online
  Resource: rac-framework-rs    rotwxza02           Online    Online

  Resource: rac-udlm-rs         rotwxza01           Online    Online
  Resource: rac-udlm-rs         rotwxza02           Online    Online

  Resource: rac-cvm-rs          rotwxza01           Online    Online
  Resource: rac-cvm-rs          rotwxza02           Online    Online

  Resource: instance-wxzac1-rs  rotwxza01           Online    Online

  Resource: lsnr-wxzac1-rs      rotwxza01           Online    Online

  Resource: instance-wxzac2-rs  rotwxza02           Online    Online

  Resource: lsnr-wxzac2-rs      rotwxza02           Online    Online

------------------------------------------------------------------

-- IPMP Groups --

              Node Name           Group   Status         Adapter   Status
              ---------           -----   ------         -------   ------
  IPMP Group: rotwxza01           sc_ipmp1 Online         ce3       Online
  IPMP Group: rotwxza01           sc_ipmp0 Online         ce6       Standby
  IPMP Group: rotwxza01           sc_ipmp0 Online         ce2       Online

  IPMP Group: rotwxza02           sc_ipmp1 Online         ce3       Online
  IPMP Group: rotwxza02           sc_ipmp0 Online         ce6       Online
  IPMP Group: rotwxza02           sc_ipmp0 Online         ce2       Offline

------------------------------------------------------------------
root@rotwxza01 #


root@rotwxza01 # df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d10      41311843 7525988 33785855    19%    /
/proc                      0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
fd                         0       0       0     0%    /dev/fd
/dev/md/dsk/d30      30988774 17400626 13588148    57%    /var
swap                 9732680     184 9732496     1%    /var/run
dmpfs                9732496       0 9732496     0%    /dev/vx/dmp
dmpfs                9732496       0 9732496     0%    /dev/vx/rdmp
swap                 9732712     216 9732496     1%    /tmp
/dev/md/dsk/d40      30988774 2661033 28327741     9%    /mnt
/dev/md/dsk/d50      33103653 23095708 10007945    70%    /opt/oracle
/dev/md/dsk/d60       498039    5163  492876     2%    /global/.devices/node@1
/dev/md/dsk/d63       498039    5163  492876     2%    /global/.devices/node@2
/vol/dev/dsk/c0t0d0/sc_agents_sc31u3_sparc
                       81024   81024       0   100%    /cdrom/sc_agents_sc31u3_sparc


Node 2

root@rotwxza02 # scstat
------------------------------------------------------------------

-- Cluster Nodes --

                    Node name           Status
                    ---------           ------
  Cluster node:     rotwxza01           Online
  Cluster node:     rotwxza02           Online

------------------------------------------------------------------

-- Cluster Transport Paths --

                    Endpoint               Endpoint               Status
                    --------               --------               ------
  Transport path:   rotwxza01:ce1          rotwxza02:ce1          Path online
  Transport path:   rotwxza01:ce0          rotwxza02:ce0          Path online

------------------------------------------------------------------

-- Quorum Summary --

  Quorum votes possible:      3
  Quorum votes needed:        2
  Quorum votes present:       2


-- Quorum Votes by Node --

                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       rotwxza01           1        1       Online
  Node votes:       rotwxza02           1        1       Online


-- Quorum Votes by Device --

                    Device Name         Present Possible Status
                    -----------         ------- -------- ------
  Device votes:     /dev/did/rdsk/d23s2 0        1       Offline

------------------------------------------------------------------

-- Device Group Servers --

                         Device Group        Primary             Secondary
                         ------------        -------             ---------
  Device group servers:  WXZ                 rotwxza01           rotwxza02


-- Device Group Status --

                              Device Group        Status
                              ------------        ------
  Device group status:        WXZ                 Online


-- Multi-owner Device Groups --

                              Device Group        Online Status
                              ------------        -------------

------------------------------------------------------------------

-- Resource Groups and Resources --

            Group Name          Resources
            ----------          ---------
 Resources: service-rg            rotwxzavip-lh-rs rotwxzait-lh-rs legato-rs service                                                                              -rs cascon-rs
 Resources: rac-rg              rac-framework-rs rac-udlm-rs rac-cvm-rs
 Resources: wxzac1-rg           instance-wxzac1-rs lsnr-wxzac1-rs
 Resources: wxzac2-rg           instance-wxzac2-rs lsnr-wxzac2-rs


-- Resource Groups --

            Group Name          Node Name           State
            ----------          ---------           -----
     Group: service-rg            rotwxza01           Offline
     Group: service-rg            rotwxza02           Offline

     Group: rac-rg              rotwxza01           Online
     Group: rac-rg              rotwxza02           Online

     Group: wxzac1-rg           rotwxza01           Online

     Group: wxzac2-rg           rotwxza02           Online


-- Resources --

            Resource Name       Node Name           State     Status Message
            -------------       ---------           -----     --------------
  Resource: rotwxzavip-lh-rs    rotwxza01           Offline   Offline - LogicalH                                                                              ostname offline.
  Resource: rotwxzavip-lh-rs    rotwxza02           Offline   Offline - LogicalH                                                                              ostname offline.

  Resource: rotwxzait-lh-rs     rotwxza01           Offline   Offline - LogicalH                                                                              ostname offline.
  Resource: rotwxzait-lh-rs     rotwxza02           Offline   Offline - LogicalH                                                                              ostname offline.

  Resource: legato-rs           rotwxza01           Offline   Offline
  Resource: legato-rs           rotwxza02           Offline   Offline

  Resource: service-rs            rotwxza01           Offline   Offline
  Resource: service-rs            rotwxza02           Offline   Offline

  Resource: cascon-rs           rotwxza01           Offline   Offline
  Resource: cascon-rs           rotwxza02           Offline   Offline

  Resource: rac-framework-rs    rotwxza01           Online    Online
  Resource: rac-framework-rs    rotwxza02           Online    Online

  Resource: rac-udlm-rs         rotwxza01           Online    Online
  Resource: rac-udlm-rs         rotwxza02           Online    Online

  Resource: rac-cvm-rs          rotwxza01           Online    Online
  Resource: rac-cvm-rs          rotwxza02           Online    Online

  Resource: instance-wxzac1-rs  rotwxza01           Online    Online

  Resource: lsnr-wxzac1-rs      rotwxza01           Online    Online

  Resource: instance-wxzac2-rs  rotwxza02           Online    Online

  Resource: lsnr-wxzac2-rs      rotwxza02           Online    Online

------------------------------------------------------------------

-- IPMP Groups --

              Node Name           Group   Status         Adapter   Status
              ---------           -----   ------         -------   ------
  IPMP Group: rotwxza01           sc_ipmp1 Online         ce3       Online
  IPMP Group: rotwxza01           sc_ipmp0 Online         ce6       Standby
  IPMP Group: rotwxza01           sc_ipmp0 Online         ce2       Online

  IPMP Group: rotwxza02           sc_ipmp1 Online         ce3       Online
  IPMP Group: rotwxza02           sc_ipmp0 Online         ce6       Online
  IPMP Group: rotwxza02           sc_ipmp0 Online         ce2       Offline

------------------------------------------------------------------
root@rotwxza02 #
root@rotwxza02 #
root@rotwxza02 # df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d10      41311843 7590800 33721043    19%    /
/proc                      0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
fd                         0       0       0     0%    /dev/fd
/dev/md/dsk/d30      30988774 12475034 18513740    41%    /var
swap                 9724960     120 9724840     1%    /var/run
dmpfs                9724840       0 9724840     0%    /dev/vx/dmp
dmpfs                9724840       0 9724840     0%    /dev/vx/rdmp
swap                 9724912      72 9724840     1%    /tmp
/dev/md/dsk/d40      30988774 2559221 28429553     9%    /mnt
/dev/md/dsk/d50      33103653 8579250 24524403    26%    /opt/oracle
/dev/md/dsk/d60       498039    5163  492876     2%    /global/.devices/node@1
/dev/md/dsk/d63       498039    5163  492876     2%    /global/.devices/node@2
/dev/vx/dsk/WXZ/wxz_app
                     70301401 18584389 51013998    27%    /test
root@rotwxza02 #
I attach as well the part of the messages file during the reboot.
I would be very gratefull if someone can tell me how to solve this problem without loosing the data on /wxz and get back the cluster up.


messages.txt
0
Comment
Question by:Ashraf-Hassan
  • 2
3 Comments
 
LVL 38

Expert Comment

by:wesly_chen
ID: 35445697
On rotwxza02, I saw from "df -k"
/dev/vx/dsk/WXZ/wxz_app                70301401 18584389 51013998    27%    /test

Could you
root@rotwxza02 # umount /test
root@rotwxza02 # mount /dev/vx/dsk/WXZ/wxz_app /xtc

By the way, rotwxza02 can see /dev/vx/dsk/WXZ/wxz_app but rotwxza01 can not see it.
Is something wrong with the cluster?
0
 

Author Comment

by:Ashraf-Hassan
ID: 35446533
I have dismounted but I have the same problem
I have contacted SUN, and they told me that the problem might be in the Oracle 9i RAC, as part of the cluster configuration for the cluster is there.
I have no experience in this area. any help please?
0
 
LVL 38

Accepted Solution

by:
wesly_chen earned 2000 total points
ID: 35447061
I'm not familar with Oracle RAC.
However, from the messages file on rotwxza01
> Apr 15 11:59:53 rotwxza01 Cluster.CCR: [ID 192619 daemon.warning] reservation error(node_join) - Unable to open device /dev/did/rdsk/d22s2
> Apr 15 11:59:53 rotwxza01 Cluster.CCR: [ID 192619 daemon.warning] reservation error(node_join) - Unable to open device /dev/did/rdsk/d21s2
> Apr 15 11:59:57 rotwxza01 Cluster.Framework: [ID 801593 daemon.notice] stdout: resetting scsi buses shared with non-cluster nodes
> Apr 15 11:59:58 rotwxza01 Cluster.CCR: [ID 869698 daemon.warning] scvxvmlg warning - found no matching volume for device node /global/.devices/node@1/dev/vx/dsk/WXZ/wxz_app, removing it

It seems to me is the cluster problem. Some of disk splice failed to open thus the cluster fail to start.

Besides,
> Apr 15 12:37:55 rotwxza01 Cluster.PNM: [ID 544775 daemon.error] libpnm system error: SIOCLIFADDIF failed.: Network is down
> Apr 15 12:37:55 rotwxza01 Cluster.PNM: [ID 544775 daemon.error] libpnm system error: SIOCLIFADDIF failed.: Network is down
> Apr 15 12:37:55 rotwxza01 Cluster.PNM: [ID 405989 daemon.error] sc_ipmp0 can't plumb 10.119.80.38.
> Apr 15 12:37:55 rotwxza01 Cluster.PNM: [ID 122188 daemon.error] IPMP logical interface configuration operation failed with <-1>.
> Apr 15 12:37:55 rotwxza01 Cluster.PNM: [ID 405989 daemon.error] sc_ipmp0 can't plumb 10.119.80.38.
> Apr 15 12:37:55 rotwxza01 Cluster.PNM: [ID 122188 daemon.error] IPMP logical interface configuration operation failed with <-1>.
It seems that the SCSI (or iSCSI) failure.

So there are a couple "offline" status in "Resources" from scstat command.

By the way, Sun is accquired by Oracle, they are in one company.
They should be able to redirect you to the proper support team to help you to resolve the issue.
0

Featured Post

Receive 1:1 tech help

Solve your biggest tech problems alongside global tech experts with 1:1 help.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Suggested Courses
Course of the Month12 days, 9 hours left to enroll

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question