• Status: Solved
  • Priority: Medium
  • Security: Private
  • Views: 89
  • Last Modified:

Run shutdown python script on AWS autoscale scale In event

Hi everyone,
I have a python script which removed a AWS Ec2 instance from Zabbix monitoring.
I need to have this script run prior to the instance terminating on a auto-scale scale in event.

I have tried placing the script in /etc/init.d and ln -s to /etc/rc0.d

I have also tried chkconfig 0 1 99 and
chkconfig --add script.py --level 0
None of the above methods cause the script to execute when entering runlevel 0

I have created a auto-scale life cycle hook which waits 90 seconds in order to give the script time to execute.

I then created the following bash script which is polled by cron every minute waiting for the Terminating:Wait signal

This also does not work. Nothing executes.

 #!/bin/bash INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
 aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCE_ID | grep -q 'Terminating:Wait'
 if [ $? -eq 0 ]; then
 /usr/local/sbin/remove_zabbix.py
 aws autoscaling complete-lifecycle-action --lifecycle-hook-name Zabbix-shutdown-wait --auto-scaling-group-name 5sight-prod --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID
  fi

So my question is, HOW can I get a aws Linux instance to execute a python script on a auto_scale Termination event.

Thanks
0
peter
Asked:
peter
  • 4
1 Solution
 
Phil PhillipsDirector of DevOps & Quality AssuranceCommented:
Hmm, init level 0 is usually where that goes.  What did you name it in /etc/rc0.d? You usually have to make sure it comes before "S01halt", so something like:

ls -s /usr/local/sbin/remove_zabbix.py /etc/rc0.d/S01ec2-terminate

I feel like that *should* work, but I haven't done that in a while.  Obviously getting it to work that way would be the ideal solution.  An uglier solution would be maybe using autoscaling lifecycle hooks to invoke a lambda function that runs the script.

One other thing to keep in mind is that scripts at init level 0 lose a lot of the usual environment variables, so the scripts can end up breaking.  For example, if the PATH variable is lost, you have to make sure to specify the full path to any commands.
0
 
petersystems engineerAuthor Commented:
I finally got this configured out.
On AWS, when a instance is terminated, no shot down signal is sent to the instance, it's simply as if the power was pulled.
The solution. Is to implement a auto scale life cycle hook which puts the instance into a terminacting:wait status for as much time as you wish typically 300 seconds dsl but can be up to 24 hours, after which time the instance will be terminated. Terminating wait allows for doing something to the I stance prior to its termination.

For this use case, I would suggest using Lifecycle hooks [1]. Hooks are applied to AutoScaling groups and affect every instance on that group. After setting up the hook, If AutoScaling needs to scale-in, it will select an instance to be terminated and put it in a "Wait" state. Having an instance in a Wait state will give you time to run any custom script. After you are done with the script, you can execute the complete-lifecycle-action [2] API call to release the hook and allow the instance to be terminated. To run a script on an instance when it enters the Wait state, you can set up a cron running every 5 or 15 minutes. The cron script will poll the AutoScaling API [3] and will check if the instance status is 'Terminating:Wait'. If it is, then you can run the required commands (to shut down a process for example). Here is a sample script to be used on a cron job: --- #!/bin/bash INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id) aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCE_ID | grep -q 'Terminating:Wait' if [ $? -eq 0 ]; then # RUN ZABBIX CUSTOM SCRIPT HERE TO REMOVE INSTANCE FROM BEING MONITORED aws autoscaling complete-lifecycle-action --lifecycle-hook-name hook_name --auto-scaling-group-name asg_name --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID fi --- Please use it as a base and adapt it to your needs. Make sure to replace the values 'asg_name' and 'hook_name'. Note: Instances will require the AWS CLI and an instance role giving permissions to run describe-auto-scaling-instances and complete-lifecycle-action. Additionally, the hook will need to be created selecting "Instance Terminate" transition.
0
 
petersystems engineerAuthor Commented:
Again, here is the check.sh script which checks for the terminating wait signal of the auto scale group

#!/bin/bash  
#checks the auto scale groups lifecycle for Terminating:Wait signal
#if recieved, executes whatever.sh
INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
EC2_AVAIL_ZONE=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`
export AWS_DEFAULT_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
aws autoscaling describe-auto-scaling-instances --instance-ids $INSTANCE_ID | grep -q 'Terminating:Wait'
if [ $? -eq 0 ]; then
#add your commands below
/etc/init.s/httpd stop
python /home/ec2-user/whatever.sh
aws autoscaling complete-lifecycle-action --lifecycle-hook-name zabbix-terminate-delay --auto-scaling-group-name 5sight-test --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID
sleep 5
sudo init 0
fi
0
 
petersystems engineerAuthor Commented:
Here is the life cycle hook

aws autoscaling put-lifecycle-hook \
  --lifecycle-hook-name shutdown-delay \
  --auto-scaling-group-name myserver-test \
  --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --role-arn arn:aws:iam::123456789:role/xxx-app-server \
  --notification-target-arn arn:aws:sns:us-east-1:123456789:instance_shutdowns \
  --heartbeat-timeout 300 \
  --default-result ABANDON
0
 
petersystems engineerAuthor Commented:
This was a special case that I evolved laws shut down processes which h no o e replied to. I had to resolve myself. But I provided all documentation for anyone e else who may run I to this.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now