Step-by-Step Guide to Multi-node MicroK8s Installation on Linux (Red Hat and Ubuntu)

Introduction:

Having previously explored MicroK8s installations on Windows and delved into a single-node setup on Linux, we explore setting up multi-node MicroK8s on RedHat and Ubuntu environments. Setting up a multi-node cluster is complex, especially when deployed on-premises.
‍

Need for Multi-Node Architecture:

While single-node clusters are easy to deploy and test for developing and testing smaller applications, they need to scale to handle the high availability and resiliency needed for large enterprise applications. The multi-node architecture provides the following benefits:

• Availability: The multi-node architecture safeguards application continuity, mitigating the risk of system downtime by functioning seamlessly amidst node failures.
• Scalability: Designed to handle a surge in application requests while distributing the load across multiple nodes.
• Resource Management: Dedicated nodes can be optimized for specific tasks or applications, allocating resources efficiently, preventing bottlenecks, and facilitating consistent performance.

Prerequisites:

Update the server with the latest updates and install snap:

• Run all commands as root.
• At least three nodes are required to achieve high availability in Microk8s.
• Clone or download files from GitHub link: https://github.com/arysdev/microk8s.git

For Redhat and Ubuntu

Update the hostname:

Update the hostname in each node by editing the below file in the given path. Each node should have a different hostname.

vi /etc/hostname

Update the host file and mention the nodes’ IP address and hostname. An example below:

vi /etc/hosts

For example, the entries would look like:

192.168.0.1 node1 
192.168.0.2 node2
192.168.0.3 node3

Reboot each node once the hosts and hostname files are updated.

‍

Firewall Configuration

Redhat:

When running Microk8s on Redhat Linux, SELinux can prevent the proper functioning of the Microk8s components by blocking network communications or preventing access to specific files or directories.

sudo sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
sudo setenforce 0

Open the ports below if the firewall is running. Run the command on each node. Enabling masquerading is often required to allow traffic to flow correctly between pods and external networks.

sudo firewall-cmd --add-port={16443/tcp,10250/tcp,10255/tcp,25000/tcp,12379/tcp,10257/tcp,10259/tcp,19001/tcp,4789/udp,10248/tcp,10249/tcp,10251/tcp,10252/tcp,10256/tcp,10257/tcp,10259/tcp,2380/tcp,1338/tcp,80/tcp,443/tcp,4443/tcp,6443/tcp, 7946/tcp,7472/tcp } ––permanent

sudo firewall-cmd --add-masquerade --permanent

sudo firewall-cmd --reload

sudo systemctl restart firewalld.service

‍

Ubuntu:

sudo ufw allow in on cni0 && sudo ufw allow out on cni0
sudo ufw default allow routed

‍

Install Snap

Redhat:

Add a repo if you are using rhel9 for Snapd installation.

sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo dnf upgrade
sudo dnf -y install snapd

Once the package has finished installing, the systemd unit that manages the main snap communication socket needs to be enabled as follows:

sudo systemctl enable --now snapd.socket

Furthermore, to enable classic snap support, enter the following to create a symbolic link between /var/lib/snapd/snap and /snap, then add snap to the PATH variable.

sudo ln -s /var/lib/snapd/snap /snap
echo 'export PATH=$PATH:/var/lib/snapd/snap/bin' | sudo tee -a /etc/profile.d/mysnap.sh

Ubuntu:

Use the below command to update the Ubuntu server before installing microk8s. Once the system is back, install microk8s.

sudo apt update&& sudo apt upgrade
sudo reboot

‍

Microk8s Installation Steps ( v1.27)

Redhat

Install Microk8s on all nodes:

sudo snap install microk8s --classic  --channel=1.27/stable ; sudo usermod -aG microk8s $USER ;  sudo chown -f -R $USER ~/.kube

Log out and log in from the server.

Ubuntu:

sudo usermod -aG microk8s $USER 
sudo chown -f -R $USER ~/.kube
sudo snap install microk8s --classic --channel=1.27/stable

Log out and log in from the server.

Add Nodes to your Cluster:

Run this on the first instance of the cluster.

microk8s add-node

Copy the first “microk8s join” command and execute that on the first node.

‍

microk8s join 172.26.24.237:25000/dbb2fa9e23dfbdda83c8cb2ae53eaa53

‍

Repeat this on each additional node you would like to add.

Note: Don’t use the same token for every node. Create a separate token for each node.

Check the status by using the below command:

microk8s status

The status should show that microk8s is running with a high-availability status as yes.

‍

Balancing Load with MetalLB and Steering Traffic with Ingress Controllers

Within Kubernetes, particularly in bare-metal deployments, orchestrating external service access is vital. By default, microk8s does not provide a load-balancer implementation, making exposing services to the external network difficult. By utilizing MetalLB in microk8s, you can enable load-balancing capabilities for your services and make them accessible from the external network. To use Metallb, use the IP address you have provided in the IP-address-pool after enabling the metallb service.

Run the below command to enable metallb.

microk8s enable metallb

Provide the range of the one IP address allocated for ingress. Use an IP that is not an IP of one of the nodes in the cluster. Use an IP address in the same subnet for metallb.

Eg. 192.168.0.4–192.168.0.10

Microk8s Ingress Controller (Nginx ingress Controller)

Microk8s Ingress Controller is a Kubernetes resource that manages external access to services in a Kubernetes cluster. It routes incoming requests to the correct services based on their host names and paths. It acts as a reverse proxy and routes incoming requests to the appropriate Kubernetes services based on the rules defined in the Ingress resource.

The Ingress Controller continuously monitors the Kubernetes API server for changes to the Ingress resources. When a new Ingress resource is created or updated, the Ingress Controller reads the rules and uses them to configure the external load balancer. This load balancer then routes the incoming requests to the appropriate Kubernetes service.

The Microk8s Ingress Controller can expose multiple Kubernetes services using a single IP address and port number. This simplifies the management of external access to Kubernetes services and provides a consistent way to access the services across the entire cluster.

To expose the ingress controller service to the outside cluster, change the service type from ClusterIP to Load Balancer). We are using Metallb to expose the ingress controller.

In microk8s, expose the ingress controller service to the load balancer.

Copy and apply the below yaml file. Create a file name ingress-service.yaml and paste the below content.

https://github.com/arysdev/microk8s/tree/7ff13698e6e91378336bcb84dcaef75df1c60412/ingress

Run the below command to apply the ingress service yaml file.

microk8s kubectl apply -f ingress-service.yaml

Kubernetes Dashboard

Enable Dashboard

microk8s enable dashboard

Change the type from ClusterIP to NodePort. Change the last line from type: ClusterIp to NodePort. Run the below in cmd.

microk8s kubectl -n kube-system edit service kubernetes-dashboard

You can get the nodeport from the below command. It starts with a higher number port from 30000.

microk8s kubectl get svc -n kube-system

Allow firewall rule to access Microk8s Dashboard.

sudo firewall-cmd --add-port=<NodePort>/tcp --permanent
sudo firewall-cmd --reload

Check for the Port on which Dashboard is listening. Access it using https://<ipaddress>:NodePort

For Token

microk8s kubectl -n kube-system describe secret $(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)

Copy and paste the token into the browser to access the Kubernetes Dashboard.

Add a policy to the metric server. Run the below command.

microk8s kubectl patch deployment metrics-server -n kube-system --patch '{"spec":{"template":{"spec":{"hostNetwork": true}}}}'

Deploy the Nginx pod in microk8s and access it via ingress.

Create a deployment for the nginx webserver. Run the below command.

microk8s kubectl create deployment nginx --image=nginx

Expose service for nginx webserver as clusterip.

microk8s kubectl create service clusterip nginx --tcp=80:80

Create an ingress file for Nginx Webserver. To access the nginx webserver publicly on-premises environments.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-webserver
spec:
  ingressClassName: nginx
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

Save the file as nginx-ingress.yaml. Run the below command to apply it.

microk8s kubectl apply -f nginx-ingress.yaml

Go to your Web-browser to access http://nginx.example.com. You will see the nginx welcome page.

Conclusion:

Transitioning from a single-node to a multi-node MicroK8s cluster is an essential strategic move for enterprises aiming for scalability, high availability, and efficient resource utilization. Utilizing MetalLB and Ingress Controllers enables businesses to maintain high-performance applications, irrespective of scale.

‍