Expiring Today—Celebrate National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


How can i fix following HACMP error ?

Posted on 2009-04-10
Medium Priority
Last Modified: 2013-11-17
Dear all !
I have created the two node cluster on AIX (configuration file is attached ). Cluster has one resource group including 1 Service IP label and 1 volume group.
The status of the cluster as follow

bash-3.00# /usr/es/sbin/cluster/utilities/cldump

Obtaining information via SNMP from Node: node1...

Cluster Name: crmcluster
Cluster State: UP
Cluster Substate: STABLE

Node Name: node1                State: UP

  Network Name: net_diskhb_01      State: UP

    Address:                 Label: hdisk6             State: UP

  Network Name: net_ether_01       State: UP

    Address:    Label: vnpcrm05           State: UP

  Network Name: net_ether_02       State: UP

    Address:         Label: node1-boot         State: UP
    Address:    Label: vnpcluscrm         State: UP

Node Name: node2                State: UP

  Network Name: net_diskhb_01      State: DOWN

  Network Name: net_ether_01       State: UP

    Address:    Label: vnpcrm06           State: UP

  Network Name: net_ether_02       State: UP

    Address:         Label: node2-boot         State: UP

Cluster Name: crmcluster

Resource Group Name: test
Startup Policy: Online On First Available Node
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Fallback To Higher Priority Node In The List
Site Policy: ignore
Primary instance(s):
The following node temporarily has the highest priority for this instance:
node1, user-requested rg_move performed on Wed Apr  8 16:42:18 2009

Node                         Group State      
---------------------------- ---------------
node1                        ONLINE          
node2                        OFFLINE

When I excute fallover test by using Shutdown Fr on one node, or using Moving resource group function on HACMP, it has done. But if I unplug the ethernet cable or the fibre cable, nothing happens.

Using Cluster Test Tool, I encouter some errors (some tests failed)

|| Test 1 Complete - NODE_UP: Start cluster services on all available nodes
08/04/2009_14:43:37: || Test Completion Status: NOT RATIONAL

|| Test 2 Complete - NETWORK_UP_GLOBAL: Bring up network1 globally
08/04/2009_14:52:23: || Test Completion Status: FAILED

|| Test 1 Complete - NETWORK_DOWN_GLOBAL: Bring down non-ip network
08/04/2009_14:54:01: || Test Completion Status: FATAL

How can i fix these errors?

Question by:vinhbt
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
LVL 68

Expert Comment

ID: 24121323

there seems to be a problem with your disk heartbeat network net_diskhb_01.

Consider deleting and redefining it. When selecting hdisks, carefully compare the PVids (not the hdiskn names) to be sure to have the same disk defined at both ends. Best let HACMP discover the resources, then select the appropriate devices.

Additionally, please post the output of

lssrc -ls topsvcs

... and please explain the purpose of the IP interface   (vnpcluscrm). Seems a bit strange that it is a member of the same network as,  What are your netmask settings?



Author Comment

ID: 24213104
Hi woolmilkporc:
I have reconfigured HACMP with following change:
################HACMP IP ADDRESS##########################     vnpcrm05_boot        vnpcrm05_standby    vnpcrm05     vnpcrm06_boot        vnpcrm06_standby    vnpcrm06    vnpcrm_virtual

# ifconfig -a
        inet netmask 0xffffffe0 broadcast
         tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
        inet netmask 0xffffffe0 broadcast
        inet netmask 0xffffff00 broadcast
         tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
        inet netmask 0xff000000 broadcast
        inet6 ::1/0
         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
I also exported the configuration to the attach file (crmcluster.haw)
I still have the same problem. HACMP can not fail-over when I unplug the ethernet cable, unplug the fibre cable&
Running the cluster testtool, I get the log file (cl_testtool)
I excuted the command  lssrc ls  topsvcs and the output is
lssrc -ls topsvcs
0513-036 The request could not be passed to the topsvcs subsystem.
Start the subsystem and try your command again.
Do  you have any ideal ?

LVL 68

Accepted Solution

woolmilkporc earned 1500 total points
ID: 24213341
lssrc -ls topsvcs works only when the cluster is up. But no problem, the test tool log contains that output.
Did you pull both ethernet cables? From your output I see that there is a standby configuration.
Did you unplug ethernet and fibre (SAN, I assume) at the same time? At least in that case a failover should occur!
Did you wait long enough? Check actual grace periods and failure detection rate with
/usr/es/sbin/cluster/utilities/cllsnim -g -n 'ether'  
/usr/es/sbin/cluster/utilities/cllsnim -g -n 'diskhb'

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

My previous tech tip, Installing the Solaris OS From the Flash Archive On a Tape (http://www.experts-exchange.com/articles/OS/Unix/Solaris/Installing-the-Solaris-OS-From-the-Flash-Archive-on-a-Tape.html), discussed installing the Solaris Operating S…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Suggested Courses

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question