How do I run a script at failover/failback on an AIX HACMP cluster?

babyb00mer
babyb00mer used Ask the Experts™
on
I have a 2-node AIX HACMP cluster in which I'd like the active node to run a script when it switches to the other (standby) node. I've done a little research and found references to a startserver script and a stopserver script, but I'm not familiar with the details of the process so I'm not sure of their relevance. Ideally I'd like to get cluster services to manage this process (i.e., run the script). I imagine this would happen as part of the process that moves the resource group. Any ideas?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Most Valuable Expert 2013
Top Expert 2013

Commented:
Hi,

we will have to distinguish two things here:

First, there are the "startserver" and "stopserver" scripts you mentioned.
These scripts (made by you) are part of the application server configuration
(see "smitty clchserv.extended.select" -> (select application server) -> <enter> to verify your settings).
The scripts are executed whenever the resource group containing this application server goes online, regardless of whether it's due to regular startup at the home node or due to a failover to the standby node.

Second, you have the opportunity to create scripts which you can associate with predefined HACMP events.
These events include things like " rg_move_complete" which could be the one you want to configure. The events are associated with HACMP-internal scripts, but you can add commands (scripts) to be processed pre-event, post-event, for recovery and for notification.
You will have to experiment a bit to find out whether a given event is the right trigger for your script.

Issue "smitty clcsclev.select" -> (select the desired event, e.g. " rg_move_complete") ->
fill in values (script names) as desired.

Using the scripts I mentioned first above is quite common practice (you do not just want to make resources available to a node or move/failover resources between nodes, you also want to start your application on the node owning the given resources, don't you?), whereas the other event scripts are only used when there are very special requirements, or when there is the need for notifications, e.g. via email).

wmp


Author

Commented:
What I need to do is push files to the alternate node when the resource group is moved. It sounds as though the rg_move_complete might be more appropriate.
Most Valuable Expert 2013
Top Expert 2013

Commented:
Yes, maybe,

but I could think of three alternatives:

1) Put the pushing of files (from where? The primary node might be down!) into the regular start script.

2) Put the files in question in a shared filesystem (a filesystem belonging to the volume group which is taken over). You will then have them available on whatever node is the active one.

3) Put the files on an NFS server, mount the share during regular application startup on the active node.

But yes, a post-event script of rg_move_complete would most probably do the job as well.

OWASP: Threats Fundamentals

Learn the top ten threats that are present in modern web-application development and how to protect your business from them.

Author

Commented:
These files/directories are in the root volume, and won't be in a shared volume group. And as big a fan as I am, NFS is out of the question.

The push will always be from the active node (i.e., the one with the application/resource group). Ideally, these files and directories should follow the application, and there's no reason to move them when there's no place to go (i.e., the alternate node is unavailable). The idea is that when the active node realizes it's going down, it tosses these files to the one that's taking over - so that the active node will always be current.

I'm kinda leanin' towards the application server approach. However, I don't "own" that process. If I can find a suitable alternative - one that doesn't require convincing someone to let me modify their script - I'd prefer it. Having said that, the application server approach seems ideal.
Most Valuable Expert 2013
Top Expert 2013

Commented:
>> And as big a fan as I am, NFS is out of the question. << 

Well, I see the irony, and I think you're right. I mentioned NFS just for completeness.

But I must say, I can't fully understand what you're after.

>>Ideally, these files and directories should follow the application ..<<

Yes, and that's the best reason I could imagine for putting them into the shared volume group!

>> when the active node realizes it's going down, it tosses these files to the one that's taking over << 

How can you be sure that the "active" node is still able to do anything when it's going down? Think of power supply failures, network outages, kernel panic or the like - all good reasons for the passive node to take over, but the ex-primary node will be cut off!

Short, I'd always take a scenario into account where the failing node is really failing - dead and gone, in other words! Don't rely on stuff residing in rootvg of a participating node.
Why would you setup a cluster when you're convinced that the "failing" node will never really fail?

wmp

Author

Commented:
You are right! I'm going to abandon this path in favor of my original strategy, which was to synchronize the files while both nodes are healthy. I'm glad you're out there to bounce these (crazy) ideas off of. In the meantime, I did learn something from which my future endeavors might benefit.
Most Valuable Expert 2013
Top Expert 2013
Commented:
Sounds better!

Synchronizing files is the best choice if you can't put your data into the shared VG.

Good news: HACMP has its own file synchronization/propagation fetaure!

Just run "smitty cm_filecollection_add" and create an empty file collection. You will be given the choice to propagate files during cluster synchronization and/or to propagate them automatically when changes are detected (you can also configure the update interval, from 10 to 1440 minutes - use "smitty cm_filecollection_time").

Next, run "smitty cm_filesinfilecollection_add" to add files to your collection.

Please note that you must do the above on only one node. Changes will be made known to other nodes with the next cluster synchronization. Just take care to start the synchronization on the node where you defined the collection.

That's all. If you chose one of the automatic options the files will be kept in sync using the paradigm "Newest file wins".

If you want to propagate files by hand from a selected node to other nodes (paradigm: "File coming from the propagating node wins") issue

/usr/es/sbin/cluster/utilities/clfileprop -m my_file_collection
or
"smitty cm_filecollection_menu" -> "Propagate ..." -> (select collection) -> Go!

Good luck!

wmp

Author

Commented:
WMP!

You've been holding out on me!

Seriously, though, you wouldn't believe the hours I've burned writing my own Korn shell wrapper around the rsync command. I should have come here first!

Thanks again.
Most Valuable Expert 2013
Top Expert 2013

Commented:
Glad I could help!

I'm looking forward to your next question (hopefully in the AIX zone - that's where I always look first here at EE.)

Have fun with AIX, the best proprietary Unix on the planet!

Cheers: The wmp.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial