Run shutdown python script on AWS autoscale scale In event

Hi everyone,
I have a python script which removed a AWS Ec2 instance from Zabbix monitoring.
I need to have this script run prior to the instance terminating on a auto-scale scale in event.

I have tried placing the script in /etc/init.d and ln -s to /etc/rc0.d

I have also tried chkconfig 0 1 99 and
chkconfig --add script.py --level 0
None of the above methods cause the script to execute when entering runlevel 0

I have created a auto-scale life cycle hook which waits 90 seconds in order to give the script time to execute.

I then created the following bash script which is polled by cron every minute waiting for the Terminating:Wait signal

This also does not work. Nothing executes.

 #!/bin/bash INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
 aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCE_ID | grep -q 'Terminating:Wait'
 if [ $? -eq 0 ]; then
 /usr/local/sbin/remove_zabbix.py
 aws autoscaling complete-lifecycle-action --lifecycle-hook-name Zabbix-shutdown-wait --auto-scaling-group-name 5sight-prod --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID
  fi

So my question is, HOW can I get a aws Linux instance to execute a python script on a auto_scale Termination event.

Thanks
LVL 1
petersystems engineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Phil PhillipsDevOps ArchitectCommented:
Hmm, init level 0 is usually where that goes.  What did you name it in /etc/rc0.d? You usually have to make sure it comes before "S01halt", so something like:

ls -s /usr/local/sbin/remove_zabbix.py /etc/rc0.d/S01ec2-terminate

I feel like that *should* work, but I haven't done that in a while.  Obviously getting it to work that way would be the ideal solution.  An uglier solution would be maybe using autoscaling lifecycle hooks to invoke a lambda function that runs the script.

One other thing to keep in mind is that scripts at init level 0 lose a lot of the usual environment variables, so the scripts can end up breaking.  For example, if the PATH variable is lost, you have to make sure to specify the full path to any commands.
0
petersystems engineerAuthor Commented:
I finally got this configured out.
On AWS, when a instance is terminated, no shot down signal is sent to the instance, it's simply as if the power was pulled.
The solution. Is to implement a auto scale life cycle hook which puts the instance into a terminacting:wait status for as much time as you wish typically 300 seconds dsl but can be up to 24 hours, after which time the instance will be terminated. Terminating wait allows for doing something to the I stance prior to its termination.

For this use case, I would suggest using Lifecycle hooks [1]. Hooks are applied to AutoScaling groups and affect every instance on that group. After setting up the hook, If AutoScaling needs to scale-in, it will select an instance to be terminated and put it in a "Wait" state. Having an instance in a Wait state will give you time to run any custom script. After you are done with the script, you can execute the complete-lifecycle-action [2] API call to release the hook and allow the instance to be terminated. To run a script on an instance when it enters the Wait state, you can set up a cron running every 5 or 15 minutes. The cron script will poll the AutoScaling API [3] and will check if the instance status is 'Terminating:Wait'. If it is, then you can run the required commands (to shut down a process for example). Here is a sample script to be used on a cron job: --- #!/bin/bash INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id) aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCE_ID | grep -q 'Terminating:Wait' if [ $? -eq 0 ]; then # RUN ZABBIX CUSTOM SCRIPT HERE TO REMOVE INSTANCE FROM BEING MONITORED aws autoscaling complete-lifecycle-action --lifecycle-hook-name hook_name --auto-scaling-group-name asg_name --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID fi --- Please use it as a base and adapt it to your needs. Make sure to replace the values 'asg_name' and 'hook_name'. Note: Instances will require the AWS CLI and an instance role giving permissions to run describe-auto-scaling-instances and complete-lifecycle-action. Additionally, the hook will need to be created selecting "Instance Terminate" transition.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
petersystems engineerAuthor Commented:
Again, here is the check.sh script which checks for the terminating wait signal of the auto scale group

#!/bin/bash  
#checks the auto scale groups lifecycle for Terminating:Wait signal
#if recieved, executes whatever.sh
INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
EC2_AVAIL_ZONE=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`
export AWS_DEFAULT_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCE_ID | grep -q 'Terminating:Wait'
if [ $? -eq 0 ]; then
#add your commands below
/etc/init.s/httpd stop
python /home/ec2-user/whatever.sh
aws autoscaling complete-lifecycle-action --lifecycle-hook-name zabbix-terminate-delay --auto-scaling-group-name 5sight-test --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID
sleep 5
sudo init 0
fi
0
petersystems engineerAuthor Commented:
Here is the life cycle hook

aws autoscaling put-lifecycle-hook \
  --lifecycle-hook-name shutdown-delay \
  --auto-scaling-group-name myserver-test \
  --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --role-arn arn:aws:iam::123456789:role/xxx-app-server \
  --notification-target-arn arn:aws:sns:us-east-1:123456789:instance_shutdowns \
  --heartbeat-timeout 300 \
  --default-result ABANDON
0
petersystems engineerAuthor Commented:
This was a special case that I evolved laws shut down processes which h no o e replied to. I had to resolve myself. But I provided all documentation for anyone e else who may run I to this.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Python

From novice to tech pro — start learning today.