Kubernetes is a container management platform – an open source and inspired by Google’s internal container platform (Borg). We’re going to use it to do some ChatOps, building and deploying containers to a Kubernetes cluster using a bot on Slack. This is a write up of a talk I gave at a recent Infracoders meetup which you can find here. So, to get started we need a Kubernetes cluster! Luckily there are some helpful scripts to get started on several cloud providers. I’m choosing AWS, although Google’s own cloud probably has the best compatibility with Kubernetes. First we’ll set some environment variables, then we’ll curl the installer script (obviously you should really download and check the script before piping it to bash!):
curl -sS https://get.k8s.io | bash
You need an AWS account, the script will use the default one if you’ve already configured it with the AWS command line tools. Otherwise, you can specify the profile as above. Once the script has finished, check that it worked by using kubectl to find the Kubernetes nodes:
$ ./kubernetes/cluster/kubectl.sh get node
NAME LABELS STATUS AGE
ip-172-20-0-59.eu-west-1.compute.internal kubernetes.io/hostname=ip-172-20-0-59.eu-west-1.compute.internal Ready 2m
ip-172-20-0-60.eu-west-1.compute.internal kubernetes.io/hostname=ip-172-20-0-60.eu-west-1.compute.internal Ready 2m
ip-172-20-0-61.eu-west-1.compute.internal kubernetes.io/hostname=ip-172-20-0-61.eu-west-1.compute.internal Ready 2m
Now we have a cluster we need to set up some infrastructure for building containers and a registry in which to store our Docker images. We’ll start by creating a Docker registry using a replication controller and service that I’ve defined for you.
git clone https://bitbucket.org/automationlogic/k8s.git
./kubernetes/cluster/kubectl.sh create -f ./registry/registry-rc.yml
./kubernetes/cluster/kubectl.sh create -f ./registry/registry-service.yml
Immediately, we have a problem to solve. Whilst Kubernetes is good at exposing and connecting services between containers – we need our docker registry to be exposed on the Kubernetes minions themselves. To make this work we can use a small proxy container on each host that exposes the registry to the host:
./kubernetes/cluster/kubectl.sh create -f ./registry/reg-proxy-rc.yml
This exposes the registry on port 5000 on all the hosts. Now we’re going to create a Jenkins pod and service. We’ll want the Jenkins config to persist, so we’ll use a Kubernetes persistent volume (backed by EBS). First we have to create the EBS volume, either using the AWS console or through the command line:
aws ec2 create-volume --availability-zone eu-west-1b --size 10 --volume-type gp2
Make a note of the volume ID that is returned and also be careful to match the availability zone to the Kubernetes nodes’ availability zone, otherwise AWS can’t attach the volume. Now open the persistent volume definition (./jenkins/jenkins-pv.yml) in your editor and update the volume ID to the one we just created:
Now we can create a persistent volume, a claim to that volume, the pod to run the jenkins container and a service to expose it like so:
./kubernetes/cluster/kubectl.sh create -f ./jenkins
If you check the definition of the Jenkins pod:
- name: jenkins
- containerPort: 8080
- mountPath: "/var/jenkins_home"
- mountPath: "/var/run/docker.sock"
- mountPath: "/bin/docker"
You’ll notice that we’re exposing the docker socket and docker binary from the host into the container. This isn’t exactly recommended practice because it essentially gives the Jenkins container root on the Kubernetes node that it is running on. However, it is the easiest way to give the Jenkins pod the ability to build containers. It does create another problem, Kubernetes will mount these volumes as root, so they won’t be accessible to the Jenkins user. To get round this we can build our own Jenkins container and push it to the registry we just created. Our container will run Jenkins as root, again not a very good idea but an easy way to make things work. Hopefully an update to Kubernetes will let us mount volumes as another user at some point in the future. For now, create the following Dockerfile:
docker build -t jenkins .
docker tag jenkins <myregistry_host>:5000/jenkins
docker push <myregistry_host>:5000/jenkins
You can find the host for your registry like so:
./kubernetes/cluster/kubectl.sh get svc local-reg -o json| jq '.["status"]["loadBalancer"]["ingress"]["hostname"]'
You’ll need jq do:
brew install jq
if you don’t already have it installed.
brew install jq
Once you’ve pushed your container, the Kubernetes cluster should detect the image that was missing and start the jenkins pod. Then the service should become available once the health check on the ELB passes. You can get the URL in the same way as for the registry:
./kubernetes/cluster/kubectl.sh get svc jenkins -o json| jq '.["status"]["loadBalancer"]["ingress"]["hostname"]'
In the next part, we’ll deploy an example web application and a chat bot to our new cluster.
We work with our clients to de-risk and accelerate their business goals realisation. Our approach is based on tailoring our services to fit your needs leveraging our portfolio of strategy, execution, innovation and service delivery offerings to help you reach your objectives
We’re always on the lookout for exceptional talent and people who share our values. Even as we continue to grow, we maintain a family environment with respect and teamwork core to our culture.
Automation Logic were recognised as one of the UK’s best Workplaces again for the 5th year running! We also placed 29th in best workplaces for women. This highlights the great work that we are doing as a business, including all the wonderful work being put in by the employee-led DE&I groups. This award means so […]
Once a year we get together for ‘AL fest’ to thank all of our employees for the amazing work they do and allow them a weekend of bonding. There’s something for everyone, from our internal talent show allowing people to showcase their talents, from musical, comedy, and anything else they excel at outside of the […]
Ynyr has been with Automation Logic for several years now and his client feedback is consistently glowing. Ynyr is somewhat of a security superhero in his field, helping clients with high-impact work, and trusted implicity to do so. We can’t think of anyone more deserving of the Young Digital Professional award. Congratulations Ynyr, and thank […]