How I automated windows AMI patching & update in AWS
AWS Systems Manager
Problem
Keep my bamboo elastic agents up to date with OS updates and other 3rd party applications.
Initial Attempt
I tried to solve this with packer, using ansible provisioner. It all goes well until I update the ansible version. Then I came across this issue and it pushes me back to look for some other available options. ( side note: at this moment packer ansible provisioner issue is fixed. @hashicorp developers did a great job to fix this issue as soon as we escalated to them even though this issue related to community-supported provisioner )
Using AWS Systems Manager
When I explore other available avenues, I came across a couple of articles about AWS SSM. Then I start to read and try out things with AWS SSM and I was able to find a solid answer for my above-defined use case.
It’s a really cool tool you can use to view and control your infrastructure on AWS. Actually later I feel glad that I moved on with packer and automate complete workflow using SSM. One of the major benefits you are getting is you no longer need to maintain tooling servers or run scripts locally on your computer. You can automate or schedule most of the maintenance and deployment tasks as a run-book style SSM documents that define the actions to perform.
For most of the common use cases, you can find pre-configured documents that you can use by specifying parameters at run time.
For example, for my use case, I can choose “AWS-UpdateWindowsAMI” pre-configured template as a baseline. As you can see below it gives us a lot of flexibility to control what we need by specifying parameters at run time.
However I need more, I need to update some custom 3rd party tools, and also each time I need to update the bamboo Elastic agents AMI ID’s. So What I have done was create a custom automation document so I can add more features as I need.
As you can see Documents support JSON or YAML formats. You can just copy over whatever the steps you need from the “AWS-UpdateWindowsAMI” document. Then you can add additional steps as you need.
I’ve used “AWS-RunPowerShellScript” document to deal with my 3rd party tool updates. I can call “AWS-RunPowerShellScript” document inside my newly created custom document as a ‘aws:runCommand’ action.
here we can provide custom power shell script or script path as a parameter.
OK, next step is to deal with update bamboo elastic instance AMI ID. For this, I’ve use SSM parameter store to keep track of current AMI ID. and AWS lambda function to call bamboo API to update AMI ID.
Also, I’ve used another Lambda function to update my SSM parameter for the current AMI ID.
Then I wanted to add a couple of new tags for the newly build AMI. for that I’ve used ‘aws:executeAwsApi’ action.
Furthermore, I needed to share this AMI with some other AWS accounts as well. Again I’ve used ‘aws:executeAwsApi’ action.
OK, now I’m done with my custom document.
Then I wanted to schedule this workflow weekly for a couple of AMIs.
For that, you got another cool feature called “Maintenance Windows”. You can simply define the maintenance window and the schedule.
Once you define the maintenance window you can register the tasks you need to execute during the window. for our case, we can register our newly created AWS document as a task. If you need to run multiple instances of the document with different parameters. you can register the same doc as different tasks with different parameters.
Now it’s almost 2 months I have handover all my manual patching and update tasks to AWS System Manager. So far it’s working nicely and I never had to do any manual intervention.
Tips :
How to register dummy target
Sometimes you may need to create a task in a maintenance window without a target. (for non-instance based operations). If you check the AWS console you can’t register tasks without a target. In that case, you can use the following workaround to get over it.
create dummy instance with instance id “i-00000000000000000” from the AWS CLI
aws ssm register-target-with-maintenance-window \
--window-id "mw-ab12cd34ef56gh78" \
--target "Key=InstanceIds,Values=i-00000000000000000" \
--resource-type "INSTANCE"References
https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-maintenance.html