CloudFormation is the AWS native way of describing Infrastructure as Code, it is a simple and easy way to deploy almost all types of resources while AWS takes care of their lifecycle. For example, we can easily create, delete or update an EC2 instance by changing properties in a CloudFormation template. Unfortunately, in some circumstances, AWS resources cannot be deleted by CloudFormation, an S3 Bucket that contains objects for example.
See here and here for reference.
At AL, we like to think we go above and beyond for our clients. With this in mind, when a client team is having an issue, our engineer’s curiosity comes into play and they will always do everything they can to find a resolution. A number of internal teams were extensively using CloudFormation and were re-writing the same code across the company to clean up their resources. This manual effort was introducing inconsistencies so the teams asked Automation Logic to develop a better system to solve this issue.
One of the most common development workflows used by developers is gitflow. This is based on feature branches and the client teams run tests for each one of them independently by deploying a CloudFormation stack for each branch and running a suite of tests before merging. This can be used to prevent broken code from being merged or to allow developers to have their own stacks so they can develop without any entropy from other developments on shared infrastructure.
So far this is a well-adopted workflow but there are 3 edge cases:
To solve this, multiple project teams created custom resources within their CloudFormation templates so they could clean up all these resources left behind. Initially, this was a snippet of code in 1 or 2 projects running in a lambda function. Nowadays it’s in dozens of CloudFormation templates. However, there are still a few drawbacks:
To begin with, we reviewed several projects and spoke with multiple teams across the company to get as broad a view as possible, specifically those starting new projects, to find what could be done to help them.
We came up with a few requirements:
Based on these requirements, we started experimenting. First, we combined all the versions available and created a Serverless App. Then we shared our work with one of the client’s project teams who requested this work to get feedback as we went. After this we were able to distribute the feature and since we started with a Serverless App, AWS SAR was the most obvious solution.
When we planned to share this amongst everyone, the initial plan was to deploy the lambda function in one central account and allow everyone to invoke it. This would make it easier to manage and upgrade however it would require the central lambda function to have access to every account and that could be exploited to delete resources in other accounts. We solved this by adding a new requirement to our list: The lambda function needs to rely on IAM Roles & policies already in place and any custom security logic should be avoided.
After this experimentation, we realized that the best way would be to deploy the lambda function in every single account and for that, we could leverage AWS CloudFormation StackSets.
It’s worth pointing out again that shared components can also be exploited to bypass security, in this shared component we had to rely on IAM roles and their trust relationships. The best advice is to rely on policies setup in AWS and not try to implement them in any shape or form in the code itself.
Extending cloudformation in one template is simple and straightforward using lambda functions although maintaining it in a big organization can be harder.
AWS SAR is a good way to share lambda functions across the whole organization, to have good code governance and allow every team to have the freedom to use shared components as well as contribute to them.
The whole journey of writing and sharing this functionality also helped us to provide better support and new ideas for shared components emerged, for instance, code to handle webhooks from GitHub which was used by several teams.
Any questions? Feel free to get in touch via firstname.lastname@example.org or reach out to us on social media here or here.
We work with our clients to de-risk and accelerate their business goals realisation. Our approach is based on tailoring our services to fit your needs leveraging our portfolio of strategy, execution, innovation and service delivery offerings to help you reach your objectives
We’re always on the lookout for exceptional talent and people who share our values. Even as we continue to grow, we maintain a family environment with respect and teamwork core to our culture.
Piotr Grześkowiak has been at Automation Logic for just over five years, starting out in our DevOps Academy after graduating in Computer Science with Information Security. During those 5 years, he’s gone from an engineer in training to a well respected senior engineer, trusted by the whole company. Piotr’s been on three Central Government client […]
We interviewed AL’s co-founders Kris & Norm about their journey building Automation Logic into the business it is over the last 12 years. From the values they’ve set in place, to the struggles they’ve faced. And obviously, because it’s pretty difficult not to mention it these days, the impact covid had.
A memoir of a Workload Migration engineer by Liam Rae-McLauchlan We’ve all read the blogs and articles about migrating to the Cloud and its benefits, but in practice it can be a daunting task. Maybe your organisation is planning to move to the cloud, or is already trying – up to 85% of enterprises are […]