Automation for the Win

Imagine a bitter, cold January day. People at the office are just starting to settle into the post-holiday work-life, still recuperating from the many holiday food comas and parties. Then suddenly, panic – a large group of about 600 caseworkers are not able to carry out their vital work because of an internal network issue. 

 The Situation

At the time of this occurrence, the Automation Logic WebOps team’s main responsibility was supporting developers by improving infrastructure, facilitating development, releases and keeping a keen eye on processes that could be automated. One of the tools we provided for developers to quickly test application changes was AWS Workspaces. For those who are not familiar with the service, here’s a quick-ish summary: Amazon Workspaces is a managed, cloud-based, virtual Desktop-as-a-Service solution that allows users to provision Microsoft Windows or Amazon Linux desktops. There is no hardware overhead and the Workspaces are accessible from various devices and web browsers. All Workspaces are associated with a directory where information about users and their Workspaces is kept and managed. The directory is used to authenticate users. When users log in to their Workspaces with their directory credentials the information is sent to an authentication gateway, traffic is then sent to the corresponding directory. After the user is authenticated, streaming traffic is initiated through the streaming gateway and users can then use their virtual desktop – happy days.

https://docs.aws.amazon.com/workspaces/latest/adminguide/amazon-workspaces.html

After explaining all the wonderful ways in which AWS Workspaces work you can see it was no surprise our team was approached to try and resolve the pickle caseworkers were in by implementing a solution using Workspaces. 

Err…AWS API, We Have a Problem

The way our team deployed Workspaces for developers was largely manual. Mainly because this particular use case only required the occasional Workspace build every few weeks or months so the need to automate the process was not critical. Creating a Workspace was simple enough – add a user to the Active Directory using the AWS console, update CloudFormation stack to build the resource and done. However, the team quickly discovered that attempting to enter 600 names, one by one, to an Active Directory through the AWS web console was not going to cut it. To make matters worse the team also discovered creating hundreds of Workspaces through a CloudFormation stack had major drawbacks. Only a handful of Workspaces could be created in parallel, so the creation had to be separated into batches. Each batch took between 1 to 2 hours to create. Additionally, if an error occurred while creating one of the Workspaces, the stack would roll back deleting all the Workspaces – even the ones created successfully! With all this being said, the team still managed to resume service to caseworkers in a couple of days. 

The process of creating Active Directory users was automated shortly after the incident began by using Powershell scripts but the CloudFormation issue persisted and needed to be fixed. The whole creation process was error-prone, repetitive, and slow. In other words, it was a prime candidate for automation.

The Bash of My Dreams

Following the success of the incident resolution, the business decided Workspaces would be a perfect fit for a new caseworker use case. The team began designing a new automated process of creating Workspaces. We knew where improvements could be made, and we got to work.

The first step in overcoming the CloudFormation stumbling block was deciding to not use CloudFormation and instead write a Bash script tool that would call the AWS API to create the Workspaces. In order to manage Workspace users, the script used a list of users where addition to the list meant create a Workspace and deletion of a user meant remove Workspace. The process worked. Unlike with CloudFormation, using the Bash scripts/AWS CLI, seamlessly skipped a failed Workspace creation, returned the cause of the error, and continued creating the rest. Although using the AWS CLI had its limitations, like only having the ability to create 25 Workspaces per API call, the limitation was bypassed by just doing continuous calls one after the other with a simple loop. Crucially, the necessary API calls to build the 700 Workspaces necessary took only seconds compared to 1 to 2 hours for a handful of Workspaces using CloudFormation.

The Bash tool worked great – at first. Keeping user data became more difficult to manage as the user list kept expanding. Error handling was virtually non-existent and attempting data validation made the script more complex – regex, grep, awk, sed, all ruled the script. As someone on Reddit once said, “in order to debug Bash you need to be at least 20% better than the person who wrote the script.” 

So, with that, and in order to save any future teams the headache of maintaining esoteric scripts, a decision was made to re-write the tool. The goals were simple: have a single, reliable, source of truth for user data and make the script more user-friendly and maintainable.

Domo Arigato, Mr RoBoto3

Python was chosen to do the tool re-write. The main reasons for choosing it were pretty simple – Python is easy to understand and write, it has an extensive library, and we could use Boto3, the AWS SDK for Python, which allows developers to write scripts that makes use of AWS services, similar to the AWS CLI.

YAML was chosen to generate a configuration file to serve as a single source of truth for the user data. Choosing YAML was an obvious choice because it provides well-structured data with key, value pairs, is has an easy to read and maintain data-structure hierarchy. Plus, AWS returns service data using JSON which also uses key-value pairs, so it made sense.  

Image of a user data structure

To make the tool more user-friendly argument flags were included to make creating and deleting Workspaces easy. Flags to enable users to query Workspaces information were also included. And the generation and validation of a Powershell script to create users in the Active Directory was included to integrate the process as much as possible. Previous iterations of the tool had users create and validate the Powershell output in a separate step, which made it error-prone.

Image of Workspace tool help menu  

This current version of the tool has been tried and tested by various team members with varying degrees of Python experience and it seems to be a hit!

Très CHIC

The AL team continuously aim to embody the core values of the company (Curiosity, Humility, Integrity, Collaboration) while on client sites to try and offer the best service possible. And these values are in large part the reason why the Workspaces tool was successfully created and used in various use cases since. The AL WebOps team now support a total of 5 different Workspaces uses cases totalling approximately 1,500 users. User feedback reports cite better productivity and better desktop performance. One example, an update process using a different service took 25 minutes for 8 updates. With Workspaces, users can do 24 updates in 15 minutes. The same users also reported massive improvement in opening records. Workspaces opened individual records in a little over a minute, whereas the old service took 20 minutes. 

The AL team takes pride in delivering automated solutions that have a real, tangible impact on users. Check out our AWS Competency.