Raspberry Pi

The Graduate DevOps Academy Final Project

To complete the 12-week intensive training our DevOps graduates go through to become junior DevOps Engineers, they are all tasked with a final project. This project is undertaken in small groups, and we continuously evolve the ask in these projects to match current business needs. For example this group were focused on Raspberry Pi because we as a company are currently launching our Secure Container Platforms offerings. 

Now that we can very happily say that every graduate has passed this Academy and we have 6 more Junior DevOps Engineers, take a look at the work they did around Secure Container Platforms using Raspberry Pie. 

Offering & Deliverables

Our final project in the academy was based on Automation Logic’s Secure Container Platform (SCP) offering. As part of the project, we were primarily tasked with creating a SCP that would be deployed onto a cluster of Raspberry Pis (RPIs) – adding further value to the current offering. Our deliverables consisted of (1) Auditing existing Kubernetes (Kubespray) documentation, (2) Creating a simulated offline Kubernetes cluster, and (3) Building an offline Kubernetes cluster in Raspberry Pi. 

First, let’s cover the documentation audit.

Kubespray Audit

Innovation Lab had built air-gapped environments and uploaded the guides on how to recreate these onto Confluence. However, there was not much mileage on the existing documentation, which meant that the AL lead engineer for this project did not have full confidence that the information was solid and that we have enough resources to deliver the offering.

We acted as a fresh set of eyes to build the system using the documentation as a guide. We were tasked with identifying any gaps in the existing documentation as well as writing more documentation, so that we know that the offering can be delivered. This brings value to AL as we evaluate how helpful and easy it is to follow the documentation, as we are new engineers who do not have much experience or knowledge with the tools required to build the simulated air-gapped system.

We learned that the documentation did indeed have a few gaps, which meant that it would not be easy to follow those steps and set up a SCP on a RPI cluster. This led to a change of plans, and we needed to decide on a different RPI compatible Kubernetes distribution to employ in the cluster.

MicroK8s

This is where ‘microk8s’ came in – a lightweight and quick-to-start kubernetes solution – perfect as a step towards physical offline kubernetes. However, prior to setting up this RPI environment, we needed to test the process – how difficult would it be to use Microk8s offline? For this, we employed a simulated offline Mircok8s environment using Amazon Web Services (AWS).

Imagine an offline system consisting of multiple computers – the networking would be dependent on real components. Virtual machines (VMs) on the other hand, can simulate such environments…all on the cloud (perhaps not perfectly). Using VMs through AWS, we simulated an offline Microk8s cluster totalling three machines (one acting as the control plane) – the fourth, being a bastion we configured independently in an online subnet, which acted as the only internet-facing server. We utilised the online server to download Microk8s and Docker snaps/files, which we secure-copied to the offline cluster for installation. Through this method, we made available application images (i.e. Nginx) to the offline cluster. Then, using the Microk8s kubernetes features, we attempted to deploy the application.

We then directed our focus to the RPIs. This step involved (1) Building the RPIs, (2) Configuring the RPIs, and (3) Using the Microk8s code and implementing it in the RPI cluster setup, with minor adjustments.

Raspberry Pi SCP

We began by building the cluster of four RPis into a case and installing an operating system. Three of the four RPis were offline (not connected to the internet) which makes them a more secure system as it is far harder for malicious actors to gain access, hence the secure platform. The last RPi acted as the bastion, this allowed us to download the necessary software and then transfer the required files to the offline machines. We explored various Kubernetes distributions including Kubespray and Openshift but decided that MicroK8s provided the low system requirements that a RPi Kubernetes cluster would need. We then installed MicroK8s onto the offline machines and started. We ran a simple NGINX deployment with Cluster IP service to demonstrate that the cluster was fully running.

Challenges

Challenges are in most cases inevitable. With the simulated Kubernetes, we found issues with deploying the Nginx application, oddly the code produced for this section (with minor adjustments) worked successfully in the RPI offline cluster. Hence, we learned to prepare for a plethora of problems. The foremost, being that solutions in a specific setup are not always generalisable to other environments and vice versa. In our case, we found a more perplexing situation, whereby, code created for a specific environment worked in another environment, but not in the environment it was originally intended for! Additionally, we learned to cooperate on projects with more complex deliverables, in conjunction with experiencing real-world consulting scenarios. 

During the project COVID became a more significant risk and we had to make changes to allow some of us to work remotely. We did this by using the bastion RPi as an access point and allowing remote access through a programme called tmate. This allowed us to continue to work collaboratively on a system that in general requires direct in person access.

Outcome

The project was a great opportunity for us as Academy members to get hands-on experience deploying a Kubernetes cluster in an offline environment and learn the techniques for installing software in these conditions. We were also able to create documentation on offline installation of different distributions of Kubernetes including Kubespray which will be a key resource for less experienced engineers when working more with the new SCP offering. The documentation of MicroK8s will hopefully find uses in more niche applications including edge computing and Internet of Things (IoT) which are gaining popularity.

The audit portion of the project was a good learning experience for us as we learned about the significance of SCPs, and more about the reviewing process in engineering applications. We encountered errors whilst navigating the existing documentation, which required us to collaborate with senior engineers at AL to solve. This was useful because we learned how experienced engineers debug errors in real time, as well as learn how to articulate the issues we encountered in a professional manner. It was nice to learn that the AL engineers are great at solving problems, and are always happy to help where they can. We also gained experience in the handover process at the end of a project.

Overall we were able to improve the Confluence page for SCP by removing outdated documentation and writing a prerequisite guide to ensure that future engineers who follow the guide will be able to move past the blockers we had encountered much quicker. Our review also means that the AL lead engineer knows the documentation needs further improvements.

Written by Steven Blount