How can i fix following HACMP error ?

Posted on 2009-04-10
Last Modified: 2013-11-17
Dear all !
I have created the two node cluster on AIX (configuration file is attached ). Cluster has one resource group including 1 Service IP label and 1 volume group.
The status of the cluster as follow

bash-3.00# /usr/es/sbin/cluster/utilities/cldump

Obtaining information via SNMP from Node: node1...

Cluster Name: crmcluster
Cluster State: UP
Cluster Substate: STABLE

Node Name: node1                State: UP

  Network Name: net_diskhb_01      State: UP

    Address:                 Label: hdisk6             State: UP

  Network Name: net_ether_01       State: UP

    Address:    Label: vnpcrm05           State: UP

  Network Name: net_ether_02       State: UP

    Address:         Label: node1-boot         State: UP
    Address:    Label: vnpcluscrm         State: UP

Node Name: node2                State: UP

  Network Name: net_diskhb_01      State: DOWN

  Network Name: net_ether_01       State: UP

    Address:    Label: vnpcrm06           State: UP

  Network Name: net_ether_02       State: UP

    Address:         Label: node2-boot         State: UP

Cluster Name: crmcluster

Resource Group Name: test
Startup Policy: Online On First Available Node
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Fallback To Higher Priority Node In The List
Site Policy: ignore
Primary instance(s):
The following node temporarily has the highest priority for this instance:
node1, user-requested rg_move performed on Wed Apr  8 16:42:18 2009

Node                         Group State      
---------------------------- ---------------
node1                        ONLINE          
node2                        OFFLINE

When I excute fallover test by using Shutdown Fr on one node, or using Moving resource group function on HACMP, it has done. But if I unplug the ethernet cable or the fibre cable, nothing happens.

Using Cluster Test Tool, I encouter some errors (some tests failed)

|| Test 1 Complete - NODE_UP: Start cluster services on all available nodes
08/04/2009_14:43:37: || Test Completion Status: NOT RATIONAL

|| Test 2 Complete - NETWORK_UP_GLOBAL: Bring up network1 globally
08/04/2009_14:52:23: || Test Completion Status: FAILED

|| Test 1 Complete - NETWORK_DOWN_GLOBAL: Bring down non-ip network
08/04/2009_14:54:01: || Test Completion Status: FATAL

How can i fix these errors?

Question by:vinhbt
  • 2
LVL 68

Expert Comment

ID: 24121323

there seems to be a problem with your disk heartbeat network net_diskhb_01.

Consider deleting and redefining it. When selecting hdisks, carefully compare the PVids (not the hdiskn names) to be sure to have the same disk defined at both ends. Best let HACMP discover the resources, then select the appropriate devices.

Additionally, please post the output of

lssrc -ls topsvcs

... and please explain the purpose of the IP interface   (vnpcluscrm). Seems a bit strange that it is a member of the same network as,  What are your netmask settings?



Author Comment

ID: 24213104
Hi woolmilkporc:
I have reconfigured HACMP with following change:
################HACMP IP ADDRESS##########################     vnpcrm05_boot        vnpcrm05_standby    vnpcrm05     vnpcrm06_boot        vnpcrm06_standby    vnpcrm06    vnpcrm_virtual

# ifconfig -a
        inet netmask 0xffffffe0 broadcast
         tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
        inet netmask 0xffffffe0 broadcast
        inet netmask 0xffffff00 broadcast
         tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
        inet netmask 0xff000000 broadcast
        inet6 ::1/0
         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
I also exported the configuration to the attach file (crmcluster.haw)
I still have the same problem. HACMP can not fail-over when I unplug the ethernet cable, unplug the fibre cable&
Running the cluster testtool, I get the log file (cl_testtool)
I excuted the command  lssrc ls  topsvcs and the output is
lssrc -ls topsvcs
0513-036 The request could not be passed to the topsvcs subsystem.
Start the subsystem and try your command again.
Do  you have any ideal ?

LVL 68

Accepted Solution

woolmilkporc earned 500 total points
ID: 24213341
lssrc -ls topsvcs works only when the cluster is up. But no problem, the test tool log contains that output.
Did you pull both ethernet cables? From your output I see that there is a standby configuration.
Did you unplug ethernet and fibre (SAN, I assume) at the same time? At least in that case a failover should occur!
Did you wait long enough? Check actual grace periods and failure detection rate with
/usr/es/sbin/cluster/utilities/cllsnim -g -n 'ether'  
/usr/es/sbin/cluster/utilities/cllsnim -g -n 'diskhb'

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. Please see for the updated article. It is avail…
A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

929 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now