Kubernetes under my desk

I’m diving into Kubernetes for a couple of months now. Discovering the possibilities and philosophy behind the hype definitely changed my mind. Yes, it is huge (in every sense ;) ) and it does change the way we, ex-sysops / ops / syasdmins do our work. Not tomorrow, not soon, now.

I’ve had my hands on various managed kubernetes clusters like GKE (Google Container Engine), EKS (AWS Elastic Container Service) or the more humble minikube but I’m not happy when I don’t understand what a technology is made of. So I googled and googled (yeah sorry Qwant and duckduckgo I needed actual answers), until I found >many >incredibly >useful resources.

Finally, after hours of reading, I decided to fire up my own k8s cluster “on premise”, or better said, under my desk ;).
With a couple of hardware I had here and there I built a good old Debian GNU/Linux 9 machine which will be my trustful home-datacenter.
There’s a shitton of resources on how to build up a k8s cluster, but many pointers and the experience of friends like @kedare and @Jean-Alexis put Kubespray on top of the list.
Long story short, Kubespray is a huge ansible playbook with all the bits and pieces necessary to build a high-available, up-to-date kubernetes cluster. It also comes with a Vagrantfile which helps with the creation of the needed nodes.

By default, vagrant uses VirtualBox as its virtual machine provider, using kvm makes it faster and well integrated into our Debian system.
Here is a great tutorial on setting up such a combo.

Contrary to official Kubespray documentation guidelines, I used virtualenv to install python bits, which is cleaner.

Some notes about how to run or tune Vagrantfile:

• Can’t have less than 3 nodes, the playbook will fail at some point
• Each node can’t have less than 1.5G RAM
• CoreOS has no libvirt vagrant box, stick with ubuntu1804
• When the cluster is created with vagrant, ansible inventory is available at inventory/sample/vagrant_ansible_inventory
• Even if disable_swap is set to true in roles/kubespray-defaults/defaults/main.yaml it remains active, preventing kubelet to start. journalctl showed the following :

Oct 15 06:17:36 k8s-01 kubelet[2140]: F1015 06:17:36.672113
2140 server.go:262] failed to run Kubelet: Running with swap on is not supported, please disable swap ! or set --fail-swap-on flag to false.
/proc/swaps contained:
[Filename           Type                Size        Used        Priority /dev/sda
2                               partition        1999868        0        -2]

Which seems caused by those previous warnings:

Oct 15 06:17:47 k8s-01 kubelet[2369]: Flag --fail-swap-on has been deprecated,
This parameter should be set via the config file specified by the Kubelet's --config flag.
See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

Simply fix this by executing:

$ ansible -i inventory/sample/vagrant_ansible_inventory kube-node -a "sudo swapoff -a"
$ ansible -i inventory/sample/vagrant_ansible_inventory kube-node -a "sudo systemctl restart kubelet
$ ansible -i inventory/sample/vagrant_ansible_inventory kube-node -a "sudo sed -i'.bak' '/swap/d' /etc/fstab

Enable localhost kubectl in inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml

kubeconfig_localhost: true
kubectl_localhost: true

This will populate the inventory/sample/artifacts/ directory with the kubectl binary and a proper admin.conf file in order to use kubectl from a client able to reach the cluster. Usually, you’d copy it like this:

$ mkdir -p $HOME/.kube
$ cp inventory/sample/artifacts/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

From here, you’ll be able to use kubectl in the actual host.
You may want to connect from another host which has no direct route to the cluster, in such case simply use kubectl as a proxy:

$ kubectl proxy --address='0.0.0.0' --accept-hosts='.*'

And voila:

$ kubectl get nodes                               
NAME      STATUS    ROLES         AGE       VERSION
k8s-01    Ready     master,node   1d        v1.12.1
k8s-02    Ready     master,node   1d        v1.12.1
k8s-03    Ready     node          1d        v1.12.1

(yeah, keyboard needed, mandatory F2 at boot because of missing fan…)