CloudFormation is the AWS native way of describing Infrastructure as Code, it is a simple and easy way to deploy almost all types of resources while AWS takes care of their lifecycle. For example, we can easily create, delete or update an EC2 instance by changing properties in a CloudFormation template. Unfortunately, in some circumstances, AWS resources cannot be deleted by CloudFormation, an S3 Bucket that contains objects for example.
See here and here for reference.
At AL, we like to think we go above and beyond for our clients. With this in mind, when a client team is having an issue, our engineer’s curiosity comes into play and they will always do everything they can to find a resolution. A number of internal teams were extensively using CloudFormation and were re-writing the same code across the company to clean up their resources. This manual effort was introducing inconsistencies so the teams asked Automation Logic to develop a better system to solve this issue.
One of the most common development workflows used by developers is gitflow. This is based on feature branches and the client teams run tests for each one of them independently by deploying a CloudFormation stack for each branch and running a suite of tests before merging. This can be used to prevent broken code from being merged or to allow developers to have their own stacks so they can develop without any entropy from other developments on shared infrastructure.
So far this is a well-adopted workflow but there are 3 edge cases:
To solve this, multiple project teams created custom resources within their CloudFormation templates so they could clean up all these resources left behind. Initially, this was a snippet of code in 1 or 2 projects running in a lambda function. Nowadays it’s in dozens of CloudFormation templates. However, there are still a few drawbacks:
To begin with, we reviewed several projects and spoke with multiple teams across the company to get as broad a view as possible, specifically those starting new projects, to find what could be done to help them.
We came up with a few requirements:
Based on these requirements, we started experimenting. First, we combined all the versions available and created a Serverless App. Then we shared our work with one of the client’s project teams who requested this work to get feedback as we went. After this we were able to distribute the feature and since we started with a Serverless App, AWS SAR was the most obvious solution.
When we planned to share this amongst everyone, the initial plan was to deploy the lambda function in one central account and allow everyone to invoke it. This would make it easier to manage and upgrade however it would require the central lambda function to have access to every account and that could be exploited to delete resources in other accounts. We solved this by adding a new requirement to our list: The lambda function needs to rely on IAM Roles & policies already in place and any custom security logic should be avoided.
After this experimentation, we realized that the best way would be to deploy the lambda function in every single account and for that, we could leverage AWS CloudFormation StackSets.
It’s worth pointing out again that shared components can also be exploited to bypass security, in this shared component we had to rely on IAM roles and their trust relationships. The best advice is to rely on policies setup in AWS and not try to implement them in any shape or form in the code itself.
Extending cloudformation in one template is simple and straightforward using lambda functions although maintaining it in a big organization can be harder.
AWS SAR is a good way to share lambda functions across the whole organization, to have good code governance and allow every team to have the freedom to use shared components as well as contribute to them.
The whole journey of writing and sharing this functionality also helped us to provide better support and new ideas for shared components emerged, for instance, code to handle webhooks from GitHub which was used by several teams.
Any questions? Feel free to get in touch via email@example.com or reach out to us on social media here or here.
Interested in the cloud? Check out our services.
We work with our clients to de-risk and accelerate their business goals realisation. Our approach is based on tailoring our services to fit your needs leveraging our portfolio of strategy, execution, innovation and service delivery offerings to help you reach your objectives
We’re always on the lookout for exceptional talent and people who share our values. Even as we continue to grow, we maintain a family environment with respect and teamwork core to our culture.
Automation Logic were recognised as one of the UK’s best Workplaces again for the 5th year running! We also placed 29th in best workplaces for women. This highlights the great work that we are doing as a business, including all the wonderful work being put in by the employee-led DE&I groups. This award means so […]
Once a year we get together for ‘AL fest’ to thank all of our employees for the amazing work they do and allow them a weekend of bonding. There’s something for everyone, from our internal talent show allowing people to showcase their talents, from musical, comedy, and anything else they excel at outside of the […]
Ynyr has been with Automation Logic for several years now and his client feedback is consistently glowing. Ynyr is somewhat of a security superhero in his field, helping clients with high-impact work, and trusted implicity to do so. We can’t think of anyone more deserving of the Young Digital Professional award. Congratulations Ynyr, and thank […]