Setting Up a Docker Registry on Kubernetes

Written By Aleksandro Matejic
Published 2023-10-17 21:53:52

Setting Up a Docker Registry on Kubernetes

TLDR;

There might be many reasons why you would like to have your images in your own private container image registry.

Having a private Docker registry can be a great asset in your development pipeline, allowing for more controlled and secure image management. This tutorial walks you through deploying a Docker Registry on a MicroK8s cluster, exposing it, and pushing images to it.

In another post, we learned how to set up a multinode cluster with microk8s which I personally use it as my local development environment. However, this doesn't restrict you from deploying the registry to any cluster of your choice.

❗Please note, MicroK8s does offer a very simple method for enabling a built-in registry. However, this article is not focused solely on enabling a registry in MicroK8s, but on doing so in any Kubernetes cluster.

You can find how to enable the registry in MicroK8s in the official documentation here

Prerequisites

MicroK8s (or any K8s) installed on your machine. We have a great guide how to install microk8s here
Docker installed on your machine. Official guide can be found here
Docker distribution project

What is Docker Distribution project

The Docker Distribution project, known as the Registry, is a Kubernetes application that stores container images.

Worth mentioning, Distribution also complies with OCI Distribution Specification.

The Distribution project supports a diverse range of storage backends, making it a flexible choice for different infrastructure setups.

Ref: https://github.com/distribution/distribution

Deploying Docker Registry on MicroK8s

We’ll begin by creating Kubernetes manifests for our Docker Registry. We need a PersistentVolume, a PersistentVolumeClaim, and a Deployment.

Create a file named registry-manifest.yaml with the following content:

apiVersion: v1
kind: Namespace
metadata:
  name: registry
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/var/lib/registry"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-pvc
  namespace: registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers:
      - name: registry
        image: registry:2
        env:
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: "/var/lib/registry"
        volumeMounts:
        - name: registry-storage
          mountPath: "/var/lib/registry"
      volumes:
      - name: registry-storage
        persistentVolumeClaim:
          claimName: registry-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: registry
spec:
  selector:
    app: registry
  ports:
    - protocol: TCP
      port: 5000
      nodePort: 30500
  type: NodePort

❗REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY environment variable is used to specify the root directory where the Docker Registry stores its data. When you set up a Docker Registry, it needs a location on the filesystem to store Docker image layers, manifest files, and other data related to the Docker images it is managing.

In the context of our Kubernetes deployment, this environment variable is critical because it informs the Docker Registry container where to read and write data.

Apply this manifest to your MicroK8s cluster:

microk8s kubectl apply -f registry-manifest.yaml

Understanding hostPath

In our registry-manifest.yaml, we've defined a PersistentVolume with a hostPath type. Here's a brief excerpt from the manifest:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/lib/registry"

The hostPath field is used to reference a directory (/var/lib/registry in this case) on the node's filesystem. When a Pod references this PersistentVolume, Kubernetes mounts this directory onto the Pod, allowing the Pod to read from and write to this directory.

Here are some key points about hostPath:

Local Storage: hostPath provides a simple way to use local storage on the node. It's straightforward but has limitations, especially in multi-node clusters where Pods might be scheduled on different nodes with different local storage setups.
Persistence: Data in a hostPath volume is preserved across Pod restarts. However, if the node goes down, the data can be lost, depending on the directory and the node's setup.
Use Cases: While hostPath is useful for development and testing, in a production environment, you might want to use more robust and distributed storage solutions like NFS, AWS EBS/EFS etc.

This hostPath setup allows our Docker Registry to store image data directly on the node's filesystem, making it a convenient choice for our local setup.

Pushing Your Image

Find the IP address of the MicroK8s VM. You can use the multipass list command if you're using Multipass:

multipass list

You might get an output like the following:

Name                    State             IPv4             Image
microk8s-vm             Running           192.168.64.51    Ubuntu 22.04 LTS
                                          10.11.3.1
                                          10.1.254.64
microk8s-vm2            Running           192.168.64.52    Ubuntu 22.04 LTS
                                          10.253.0.1
                                          10.1.169.192
microk8s-vm3            Running           192.168.64.53    Ubuntu 22.04 LTS
                                          10.41.196.1

We're assuming that we have an image called devoriales-demo:v1.0
Also we assume that the IP address is 192.168.64.51( you need to change this to your ip address).
Tag a Docker image with the new registry address:

docker tag devoriales-demo:v1.0 192.168.64.51:30500/devoriales-demo:v1.0

Configure Docker to allow insecure registry communication. Add the following to your Docker daemon configuration file (/etc/docker/daemon.json or via Docker Desktop settings):

{
  "insecure-registries": ["192.168.64.51:30500"]
}

Restart your Docker daemon, then push your image:

docker push 192.168.64.51:30500/devoriales-demo:v1.0

Pulling Images from Your Registry

With the following command, you can pull the image:

docker pull 192.168.64.51:30500/devoriales-demo:v1.0

List Images

There are several ways to list images in your registry, but I find using old good curl the easiest:

curl -X GET http://\192.168.64.51:30500\/v2/_catalog

Output:
{"repositories":["devoriales-demo"]}

If you want to check all the tags for your image, you can simply run:

curl -X GET http://192.168.64.51:30500/v2/devoriales-demo/tags/list

Output:
{"name":"devoriales-demo","tags":["v1.0"]}

❗Your pods might have issues pulling the images from the local registry due to certificate problem. Inside your microm8s vm you might need to state the registry as the insecure registry so your pods can pull the images.
Here is how you fix it.

Enter the vm (you can find your vms with multipass list command if you're on Mac):

multipass shell microk8s-vm

Locate the containerd config file inside your vm:

sudo vim /var/snap/microk8s/current/args/containerd-template.toml

Under the [plugins."io.containerd.grpc.v1.cri".registry.mirrors] section, add your registry and specify it as an insecure registry. Your entry should look something like this:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.64.54:30500"]
  endpoint = ["http://192.168.64.54:30500"]

Make sure to change the IP address of your VM.

Save the file, exit your microk8s vm and restart the service:

multipass restart microk8s-vm

Use hostname instead of IP address

Imagine that you want to tag your image as microk8s-vm:30500/devoriales-demo:v.1.0.

So far, we have used an IP address, which should work fine. However, if you try to use a hostname instead, you might encounter an Error: ErrImagePull when your pods attempt to pull the image.

If you prefer to use a hostname rather than an IP address, you will likely need to update the CoreDNS configuration to ensure the hostname resolves correctly. Additionally, you should add the hostname to the hosts file on the MicroK8s node.

coredns config

kubectl edit configmap coredns -n kube-system

Add the hosts section with your node:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
          class error
        }
        hosts {
          192.168.64.54 microk8s-vm <<<<<< This you need to add, adjust according to your environment 
          fallthrough
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system

Enter your multipass vm

multipass shell microk8s-vm

Edit template file for hosts file

 sudo vim /etc/cloud/templates/hosts.debian.tmpl

add the following:

# Custom static IP address for microk8s-vm
192.168.64.54 microk8s-vm

In the file /var/snap/microk8s/current/args/containerd-template.toml, add your hostname to the list of insecure registries. Here, we have included both the IP address and the hostname.

Ensure that you add both your VM's IP address and its hostname, be aware you need to have the following containerd configuration on all your nodes:

  # 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
  [plugins."io.containerd.grpc.v1.cri".cni]
    # bin_dir is the directory in which the binaries for the plugin is kept.
    bin_dir = "${SNAP_DATA}/opt/cni/bin"

    # conf_dir is the directory in which the admin places a CNI conf.
    conf_dir = "${SNAP_DATA}/args/cni-network"

  # 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to the registry
  [plugins."io.containerd.grpc.v1.cri".registry]
  # config_path = "${SNAP_DATA}/args/certs.d"
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.64.54:30500"]
    endpoint = ["http://192.168.64.54:30500"]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."microk8s-vm:30500"]
    endpoint = ["http://microk8s-vm:30500"]

Save and exit.

Restart the containerd service:

sudo systemctl restart snap.microk8s.daemon-containerd.service

Check the containerd status:

$ journalctl -xeu snap.microk8s.daemon-containerd.service


Output example:
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.938601369+01:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemet>
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.938668702+01:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.>
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.938824119+01:00" level=warning msg="`mirrors` is deprecated, please use `config_path` instead"
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.938891536+01:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter>
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.939405869+01:00" level=info msg="Connect containerd service"
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.939582661+01:00" level=info msg="Get image filesystem path \"/var/snap/microk8s/common/var/lib/containerd/>
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.940255119+01:00" level=info msg=serving... address="127.0.0.1:1338"
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.940411036+01:00" level=info msg=serving... address=/var/snap/microk8s/common/run/containerd.sock.ttrpc
Dec 26 20:48:56 microk8s-vm2 microk8s.daemon-containerd[18068]: time="2023-12-26T20:48:56.940498827+01:00" level=info msg=serving... address=/var/snap/microk8s/common/run/containerd.sock
Dec 26 20:48:56 microk8s-vm2 systemd[1]: Started Service for snap application microk8s.daemon-containerd.
░░ Subject: A start job for unit snap.microk8s.daemon-containerd.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit snap.microk8s.daemon-containerd.service has finished successfully.
░░
░░ The job identifier is 4952.

Exit your microk8s vm and restart it with following command (enter the your microk8s node name):

 multipass restart microk8s-vm

Verify if your registry is accessable from any pod:

kubectl exec -it <some-pod> -- curl -v http://microk8s-vm:30500/v2/

You should get something like the following:

Trying 192.168.64.54:30500...
* Connected to microk8s-vm (192.168.64.54) port 30500 (#0)
> GET /v2/ HTTP/1.1
> Host: microk8s-vm:30500
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Thu, 02 Nov 2023 19:27:51 GMT
< 
* Connection #0 to host microk8s-vm left intact

Now you can tag your image as microk8s-vm:30500/<image-name:<tag> and push it to your registry.

In your deployment, you can now use the image name that has hostname instead of IP address.

Your pods should be able to pull the image with the same image name.

Wrap Up

You've now set up a Docker Registry on your MicroK8s cluster, exposed it, and pushed an image to it 👏. This setup is suitable for development and testing environments.

About the Author

Aleksandro Matejic, a Cloud Architect, began working in the IT industry over 21 years ago as a technical specialist, right after his studies. Since then, he has worked in various companies and industries in various system engineer and IT architect roles. He currently works on designing Cloud solutions, Kubernetes, and other DevOps technologies.

In his spare time, Aleksandro works on different development projects such as developing devoriales.com, a blog and learning platform launching in 2022/2023. In addition, he likes to read and write technical articles about software development and DevOps methods and tools.

You can contact Aleksandro by visiting his LinkedIn Profile

Commented by Aleksandro Matejic on 2023-11-01 14:37:57 Author

@Aarav, thanks for your feedback. I found this documentation on setting up S3 as the storage backend for Docker Registry:
https://github.com/distribution/distribution/blob/ecb475a2324f7b5ed28b8cfec25798fd4a07273e/docs/content/storage-drivers/s3.md

Based on the doc, here's a simplified ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: registry-config
  namespace: registry
data:
  config.yml: |
    version: 0.1
    storage:
      s3:
        accesskey: your-access-key
        secretkey: your-secret-key
        region: us-west-1
        bucket: your-bucket-name
    http:
      addr: :5000

Now, mount the ConfigMap in your registry deployment.

I hope this should guide you in the right direction.

Join Devoriales Today!