On the ground, in the cloud

A memoir of a Workload Migration engineer by Liam Rae-McLauchlan

We’ve all read the blogs and articles about migrating to the Cloud and its benefits, but in practice it can be a daunting task. Maybe your organisation is planning to move to the cloud, or is already trying – up to 85% of enterprises are adopting a “cloud-first” strategy (Gartner, Nov 2021).

However, the aim of this post is to give an inside perspective, from someone involved in a large-scale migration to AWS; the triumphs, struggles and learning that we encountered along the way. The view from the coalface and in the trenches.

I’m a Senior Consulting Engineer at Automation Logic, currently leading a full-scale migration of legacy applications and databases into the cloud, for a large government department. This is my experience, and the lessons I learned.

Why move to the cloud in the first place? Isn’t it just easier to stay where you are?

There’s a storm coming

At Automation Logic we work closely with AWS in order to accelerate our clients’ cloud migrations. We have years of experience in helping enterprise organisations migrate to the cloud. In partnership with AWS, providing our services through their MAP accreditation programme, we’ve helped customers successfully migrate all types of workloads to AWS, including large-scale migrations of hundreds, if not thousands, of servers.

However, migration is no easy feat – our independent research, with over 50 IT leaders in the UK, found the most common issue cited when it came to migrations was lack of skills within the business. That’s where my team comes in – through the use of automation, best practices and data-driven results, we are able to simplify the migration process. Simplification leads to quicker and easier migrations, reducing complexity and effort for the engineers involved. It also brings incredible benefits to the organisation, some of which are outlined in the diagram below.

The benefits of cloud migrations often speak for themselves, but for those unfamiliar with them, the diagram above outlines the major benefits organisations see: reduced costs, increased security, and huge boosts in productivity and delivery.

Right, so let’s get started then?

When it rains, it pours

It’s tempting to just pick up tools and start throwing things into the cloud – why waste time, when every day sitting in your current hosting solution is costing you money and holding you back from all those promised benefits.
But we’ve seen that a panicked beginning without a solid roadmap and process leads to more expense, stress and issues down the road. So rather than rushing in, all guns blazing, we take a step back and take the time to break things down into manageable chunks.

Everyone loves a 5 point plan

Making sure its not a wash out

We’ve taken the literature available across all cloud providers as well as our own expertise and experience to devise a best-practice, evidence-based 5 phase migration process – one that is fully aligned with AWS’s own migration methodology. We put significant emphasis on discovery and planning; we set our clients up for success, not stress.

Our Discover phase captures the technical and organisational information needed to plan a successful, de-risked migration. It assesses people, process and tools, with the key activities being workshops, cataloging of workloads, and uncovering their dependencies to provide a prioritisation of work.

Our Plan phase lays out the approach for migrating workloads. It’s how we map the steps needed to bring the current state to the target state. We lay out the roadmap, plot the design and architecture and decide on measurable performance indicators. For the engineers, this is the phase where we start to get excited, thinking about all the shiny tools we’ll get to play with and the puzzles we get to solve.

With our appetites sufficiently whetted, our interests sufficiently piqued, Prepare and Deploy are the “hands-on” stages. We now know what we need to prepare and how to deploy it. These phases are when we actually get to do it – and we support and encourage your team to do it alongside us.

Our Prepare phase is purposefully decoupled from the Deploy phase; during this phase we set the groundwork for the migration. One of the ways we do this is by creating and securing a landing zone – something I’ll cover in more detail later – and enabling the customer teams through workshops and up-skilling opportunities. We also finalise dependencies and ensure everything is in place to ensure a smooth deployment.

The Deploy phase is where we get our hands dirty. This is what most people think of when they think of “migration”. The deployment of servers, databases, networks and storage in the cloud. The migration of applications and data from one place to another.

Finally we move onto the Improve phase, optimising all your hard work for continuous savings and day-to-day improvements.

The final three stages are the ones that deliver that measurable value: the tangible migration of resources and services – but the importance of the first two stages in this process cannot be overstated. We only made it this far because we focussed on identifying risk and pain points early on, and ensured we knew as much as we could before we jumped in.

So, what does this look like on the ground?

Getting our feet wet

Having been involved in a number of migrations, from single applications to large datasets and services, they all have one thing in common – a very specific kind of laziness. A “lazy” engineer is a great engineer – we always want to find ways to do things easier and faster, and we never want to repeat ourselves.
The more “technical” term for this is Automation – the cornerstone of any migration.

There are a number of tools available to help engineers automate and accelerate migrations to the cloud. AWS offers a number of services; including Application Discovery Services, Amazon Database Migration Accelerator, and AWS Landing Zone, that help an organisation streamline their migration process. These services help reduce the load on migration engineers, by providing automated tooling and best-practice designed systems.

We began our Prepare phase of the migration by implementing an AWS Landing Zone solution – a solution that helps customers quickly set up a secure, multi-account AWS environment based on AWS best practices. The Landing Zone implementation saved us time and effort, by automating the set-up of multi-account architecture, identity and access management, governance, data security, network design, and logging. With a baseline environment established, we could rapidly begin migrating our core infrastructure and services.

Surely it can’t all be that easy?

Every cloud has a silver lining

With any migration, especially one dealing with legacy applications and databases, there’s always an element of the unknown. Alongside the usual Infrastructure as Code, Deployment Pipelines and Monitoring solutions, we were tasked with creating a number of outside-the-box solutions to address issues that arose from the legacy nature of the services we were migrating.

As the emphasis for this migration was on lift-and-shift, due to pressing time constraints (with modernisation to come at a later date), we had to adapt quickly and find solutions that worked when nothing else would. Early on in the migration, we identified a number of manual tasks that were used to provision replica databases, on a fixed schedule. In the previous hosting solution, DBAs (Database Administrators) or hosting engineers would perform these tasks. Sometimes these tasks were missed, or incorrectly executed – causing downtime/outages or performance issues.

We selected AWS Lambda (a serverless, event-driven compute service that lets you run code without provisioning or managing servers) as a tool that could help us replicate these requirements in our cloud-based environments. We knew the process and the steps that needed to be taken, and it was just a matter of crafting code (in this case Python) to execute them in an automated, repeatable and scheduled way – without any human intervention. We were also able to implement monitoring and alerting, for the unlikely event things with the Lambda didn’t go as planned – thus reducing the likelihood of extended disruption to service.

Towards the end of migrations, a major area for improvement was identified: the need for increased security on internet-facing applications. As with any internet-facing application these days, there is an ever changing landscape of exploits and vulnerabilities – the recent Log4J vulnerability for example.

Previously these applications were either unsecured, or used a simple whitelist to protect them from unauthorized access. We explored a number of tools, and settled on the AWS WAF – a fully managed Web Application Firewall tool. AWS WAF allowed us to quickly and easily deploy a highly-scalable and resilient firewall to protect our most vulnerable applications. AWS offers a number of pre-configured rules for protecting applications, all of them managed and updated by AWS regularly to ensure we were always protected against the latest exploits.

With the simple flick of a switch, we were able to secure a number of applications that had previously been at risk of breaches and exploits. Needless to say the client was impressed, which is always a good thing!

Okay great, but now what?

Cloudy with a chance of savings?

Since migrating to AWS, we’ve seen marked improvements in a number of areas. The biggest win is the significant reduction in spending for the client – we’ve seen roughly 35% less spend in the first few months of migration, with even more to come during the Improve phase. With help from AWS’s Compute Optimizer tool, my team has right-sized a number of over-provisioned resources and we’ve moved a number of applications to container hosted solutions (AWS ECS). Due to the new comprehensive monitoring and alerting, and tighter security, downtime has been reduced, and the applications are now more resilient.

Looking back, the thing I take the most pride in though is the growth I’ve seen in the client engineers we’ve worked alongside. Many of these engineers had never used AWS (or any cloud technology) before we arrived. A key focus of this engagement was the upskilling of these engineers, who would be responsible for the management and ownership of the cloud platform once the migration was complete. Through a culture of peer-programming, constant knowledge sharing and repeated collaboration we were able to prepare these engineers with the skills and experience they needed to be fully fledged Migration Engineers. I’m incredibly proud of their achievements and know that the platform we built together is in good hands.

Would you do it all again?

Next time, take an umbrella.

Absolutely. Of course you’d expect me to say that, but I honestly mean it. This experience has been tough, tiring and time consuming but we’re now close to the end and we can see the light at the end of the tunnel. The client is incredibly happy with what we’ve done, and never fails to remind us of the outstanding work we’ve all done over these past months. This was my first time actually leading a migration, and I’ve learnt more than I could have ever imagined. Most of it thanks to my incredibly talented team of civil servants and AL engineers.

As it comes to an end, I can look back at all the great work my team has done and know we did the best we knew how. The Improvement phase never really ends; there are things we will do differently next time, but that’s the nature of engineering and especially Cloud Migrations – the landscape is constantly changing and we need to constantly improve to keep up!

So, in closing, another last big shoutout to my team – both AL and civil servants. We did it!

Want to learn about Automation Logics Workload Migration offer? Click here