Introduction

In this tutorial,  we'll walk through the process of deploying a Kubernetes multi-node cluster with microk8s tool.

The version of Kubernetes we'll be deploying is 1.27.

Setting up a multi-node cluster is great if you want  to experiment with more advanced features, but don't want to pay for cloud Kubernetes services provided by the cloud providers 

While single-node clusters are adequate option for many things like learning, multi-node clusters allow for a more sophisticated testing environment that better mirrors real-world production conditions.

For instance, testing Pod affinity and anti affinity rules wouldn't be possible on a single node cluster.

In addition, this tutorial will show how to add more nodes to your Kubernetes cluster using a tool called Multipass if you're on macOS or Windows. 

Prerequisites

1. At least 16GB of free RAM and approx. 30GB of free disk space. You can adjust the size of the implementation to fit your machine's spec. 
2. Basic knowledge of Kubernetes, terminal commands, and networking concepts.

Later you could also do the following:

Set up ingress controller:
https://devoriales.com/post/342/how-to-configure-ingress-nginx-for-local-kubernetes-development

Install MicroK8s 

MicroK8s is a lightweight, fully conformant Kubernetes distribution from Canonical, which is easy to install and operate.

Install the MicroK8s:

brew install ubuntu/microk8s/microk8s

Install Multipass

Multipass is a lightweight VM manager for Linux, Windows and MacOS. It's used to create and manage Ubuntu instances. You can use it to create a multi-node Kubernetes cluster.

If you're running MicroK8s directly on a Linux system, you don't need Multipass. You can just install MicroK8s directly on the host system. But for MacOS and Windows, some kind of virtualization software (like Multipass, VirtualBox, etc.) is needed. 

To install Multipass, use the following Homebrew command:

brew install --cask multipass

Verify the installation with the following command:

$ multipass version

multipass   1.12.0+mac
multipassd  1.12.0+mac

Install Cluster With Three Nodes

We're going to install one control plane node and two worker nodes with low spec. Run the following command in your terminal:

Please note, the first node is named microk8s-vm which will be the control plane node.

Control Plane node:

$ microk8s install --cpu 2 --mem 2 --disk 10 --channel=1.27

with --channel we can specify the upstream Kubernetes version.

Official microk8s doc: https://microk8s.io/docs/setting-snap-channel

Check the status:

microk8s status --wait-ready

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
...

Worker nodes:

$ multipass launch --name microk8s-vm2 --cpus 1 --mem 2G --disk 10G
multipass launch --name microk8s-vm3 --cpus 1 --mem 2G --disk 10G

check ip addresses after the installation:

$ multipass list

microk8s-vm            Running           192.168.64.25    Ubuntu 22.04 LTS
                                          10.1.14.192
microk8s-vm2            Running           192.168.64.26    Ubuntu 22.04 LTS
                                          10.1.169.192
microk8s-vm3            Running           192.168.64.27    Ubuntu 22.04 LTS
                                          10.1.83.0

We will need the following for updating the /etc/hosts file on each node (you will probably have other IP addresses)

192.168.64.25 microk8s-vm
192.168.64.26 microk8s-vm2
192.168.64.27 microk8s-vm3

Interact with nodes

To enter a node, you simply run the following from your local machine:

multipass shell <your-node-hostname>

The following command allow us to enter the nodes. We will begin with entering the control plane node.

Run the following commands on each node.

microk8s-vm

$ sudo snap install multipass

$ sudo usermod -a -G microk8s ubuntu
$ newgrp microk8s

# Name Resolution:
$ sudo vi /etc/hosts

"/etc/hosts" 17L, 628B                                        12,26         All
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 microk8s-vm
127.0.0.1 localhost
192.168.64.25 microk8s-vm
192.168.64.26 microk8s-vm2
192.168.64.27 microk8s-vm3
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes

microk8s-vm2

$ sudo snap install multipass

$ sudo snap install microk8s --classic --channel=1.27/stable
$ sudo usermod -a -G microk8s ubuntu
$ newgrp microk8s

# Name Resolution:
$ sudo vi /etc/hosts

"/etc/hosts" 17L, 628B                                        12,26         All
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 microk8s-vm microk8s-vm2
127.0.0.1 localhost
192.168.64.25 microk8s-vm
192.168.64.26 microk8s-vm2
192.168.64.27 microk8s-vm3
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes

microk8s-vm3

$ sudo snap install multipass

$ sudo snap install microk8s --classic --channel=1.27/stable
$ sudo usermod -a -G microk8s ubuntu
$ newgrp microk8s

# Name Resolution:
$ sudo vi /etc/hosts

"/etc/hosts" 17L, 628B                                        12,26         All
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 microk8s-vm3
127.0.0.1 localhost
192.168.64.25 microk8s-vm
192.168.64.26 microk8s-vm2
192.168.64.27 microk8s-vm3
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes

Command descriptions:

1. sudo snap install multipass: We need to install multipass software which is needed to be able to join the worker node to the cluster

2. sudo snap install microk8s --classic --channel=1.27/stable:  we are installing the stable microk8s(K8s) release 1.27.

3. sudo usermod -a -G microk8s ubuntu: This command adds the user ubuntu to the microk8s group. The -a flag appends the user to the supplementary group(s), and -G specifies the group. After running this command, the ubuntu user will have the necessary permissions to run microk8s commands.

4. newgrp microk8s: This command logs the user into the microk8s group. This is done because group changes don't take effect in the current shell session, so this command helps to apply the changes without needing to completely log out and log back in. After running this command, the ubuntu user will be able to run microk8s commands without needing to use sudo.

Join the nodes to the cluster

On the control plane node, we will begin with generating tokens for workers to join the cluster.

Enter the shell of the control plane node:

$ multipass shell microk8s-vm

Generate tokens:

$ microk8s add-node

From the node you wish to join to this cluster, run the following:
microk8s join 192.168.64.25:25000/f8e01dc1a2c88ea9d2aff8f4b8a2c76b/36d8df8259d5

Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.64.25:25000/f8e01dc1a2c88ea9d2aff8f4b8a2c76b/36d8df8259d5 --worker

If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.64.25:25000/f8e01dc1a2c88ea9d2aff8f4b8a2c76b/36d8df8259d5
microk8s join fdab:459d:e7ff:ac51:5054:ff:fe6b:2974:25000/f8e01dc1a2c88ea9d2aff8f4b8a2c76b/36d8df8259d5

On microk8s-vm2 node, run the following command, choose the line with the flag: --worker

❗Recommendation: open the shell in another terminal window

$ multipass shell microk8s-vm2
$ microk8s join 192.168.64.25:25000/f8e01dc1a2c88ea9d2aff8f4b8a2c76b/36d8df8259d5 --worker

❗Generate a new token on control plane node (microk8s-vm) since you cannot use the same token. This has to be done for each node you want to join to the cluster

You need to repeat this process by running microk8s add-node on the microk8s-vm control plane node and then run microk8s join with the new token on the next node and so on until you have joined all worker nodes. 

Verification

We can verify the nodes in the cluster by running the following:

$ microk8s kubectl get nodes

NAME           STATUS   ROLES    AGE     VERSION
microk8s-vm    Ready    <none>   9m28s   v1.27.2
microk8s-vm3   Ready    <none>   40s     v1.27.2
microk8s-vm2   Ready    <none>   10s     v1.27.2

We now have a three node cluster running Kubernetes verion 1.27.

We can also check the distribution of the pods:

microk8s kubectl get pods -o wide --all-namespaces

NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE    IP              NODE           NOMINATED NODE   READINESS GATES
kube-system   calico-node-29fns                          1/1     Running   0             78m    192.168.64.47   microk8s-vm2   <none>           <none>
kube-system   calico-node-5wq5c                          1/1     Running   0             78m    192.168.64.48   microk8s-vm3   <none>           <none>
kube-system   coredns-7745f9f87f-x2h6v                   1/1     Running   1 (11m ago)   148m   10.1.254.68     microk8s-vm    <none>           <none>
kube-system   calico-kube-controllers-6c99c8747f-pftpw   1/1     Running   1 (11m ago)   148m   10.1.254.67     microk8s-vm    <none>           <none>
kube-system   calico-node-vsxll                          1/1     Running   1 (11m ago)   78m    192.168.64.46   microk8s-vm    <none>           <none>

Pods are distributed across all nodes. This is of course not guaranteed.

Upgrade Kubernetes on nodes

Sometimes we need to update nodes.

In the following example, we have one node, which is the control plane, on version v1.28.4 and two worker nodes on lower version:

➜  ~ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
microk8s-vm    Ready    <none>   59d   v1.28.4
microk8s-vm2   Ready    <none>   92s   v1.27.8
microk8s-vm3   Ready    <none>   56s   v1.27.8

We can enter the node that we want to upgrade and run the following command:

multipass shell microk8s-vm2

$ sudo snap refresh microk8s --channel=1.28/stable

Repeat on all nodes and verify again:

➜  ~ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
microk8s-vm    Ready    <none>   59d   v1.28.4
microk8s-vm3   Ready    <none>   11m   v1.28.4
microk8s-vm2   Ready    <none>   12m   v1.28.4

❗Don't forget to upgrade kubectl client to be on the same version as the server: https://kubernetes.io/docs/tasks/tools/

Alias

It can be annoying to type microk8s kubectl to interact with the cluster, instead you could create an alias. 

Bash shell

You can set the alias in the following way if you use th bash shell:

vim ~/.bash_aliases) 

# add
alias kubectl='microk8s kubectl'

# save the file and reload
source ~/.bash_aliases

Z Shell

The following is if you use the Z shell:

vim ~/.zshrc
# add
alias kubectl='microk8s kubectl'

# save the file and reload
source ~/.zshrc

Configuring MicroK8s config and local kube config to work together

In this section, we'll walk through how to configure kubectl to work seamlessly with both a standard Kubernetes cluster and a MicroK8s cluster. This is particularly useful if you're developing on MacOS or Windows, where MicroK8s runs within a lightweight VM, and you also have a remote Kubernetes cluster.

On MacOS or Windows, MicroK8s runs inside a VM. You can access this VM using Multipass:

multipass shell microk8s-vm

Replace microk8s-vm with the name of your VM if it's different.

Get Your MicroK8s Kubeconfig

Inside the VM, you can find the kubeconfig file at /var/snap/microk8s/current/credentials/client.config. Display its content with:

cat /var/snap/microk8s/current/credentials/client.config

Copy this content and paste it into a new file on your host machine, for example ~/.kube/microk8s-config.

❗UPDATE❗ You can also get the config with the following command:

 microk8s config

Set KUBECONFIG for MicroK8s

Tell kubectl to use the MicroK8s kubeconfig file by setting the KUBECONFIG environment variable:

export KUBECONFIG=~/.kube/microk8s-config

Now kubectl should be able to interact with your MicroK8s cluster.

Merge Kubeconfig Files

KUBECONFIG=~/.kube/config:~/.kube/microk8s-config kubectl config view --flatten > ~/.kube/merged_kubeconfig

This command merges your original kubeconfig file (~/.kube/config) with the MicroK8s kubeconfig file and saves the result to ~/.kube/merged_kubeconfig.

Use the Merged Kubeconfig File

Set KUBECONFIG to point to the merged kubeconfig file:

If you're also working with a standard Kubernetes cluster, you can merge the kubeconfig files together:

export KUBECONFIG=~/.kube/merged_kubeconfig

Switch Between The Clusters

Now that you have a kubeconfig file that includes both clusters, you can switch between them using contexts. List your contexts with:

kubectl config get-contexts

And switch to a context with:

kubectl config use-context my-context

Replace my-context with the name of the context you want to use.

Clean up 

Here is how to delete the nodes and the cluster:

$ microk8s stop
multipass delete microk8s-vm
multipass delete microk8s-vm2
multipass delete microk8s-vm3
multipass purge

Uninstall multipass

brew uninstall --zap multipass # to destroy all data

Uninstall microk8s

brew uninstall microk8s

Where to find things in microk8s

Here is the official page that lists different things that you can configure, like the controllers.

https://microk8s.io/docs/configuring-services

Summary

In this tutorial we have explored the process of setting up a node Kubernetes cluster on a MacOS machine using microK8s and Multipass. By following this guide you have gained knowledge on creating a testing environment that closely mimics real world production conditions enabling you to experiment and test your applications effectively.

The tutorial provided instructions, on installing MicroK8s and Multipass guiding you through the creation of a cluster consisting of three nodes. It also explained how to interact with these nodes and demonstrated the steps to join them into the cluster.

As we have observed, microK8s and Multipass offer an powerful approach to working with Kubernetes. They make it effortless to set up node clusters making it suitable for both beginners and experienced users. This setup provides an scalable platform for developing and testing applications, in an environment.

Remember, Devoriales is here to support your learning journey. In the world of Kubernetes, things move fast, but don't worry, we'll keep you up to date. Stay tuned for more tutorials and deep dives into the exciting world of Kubernetes and its ecosystem.