How to Configure Ingress-Nginx for Local Kubernetes Development

Written By Aleksandro Matejic
Published 2024-05-10 14:07:50

How to Configure Ingress-Nginx for Local Kubernetes Development

Introduction

Are you tired of the complexities and limitations of exposing your services via node ports and port-forwarding in your local Kubernetes environment?

Developing applications locally with Kubernetes often benefits from a setup that resembles the production environment as closely as possible. Using a custom domain name instead of an IP address makes our setups more realistic.

This tutorial will guide through configuring your local Kubernetes cluster to handle traffic using a custom DNS name.

Additionally, we'll cover how to create and use a self-signed certificate. We'll learn how to set up a local trust store and integrate it with an Ingress, enhancing the security and mimic the real world setup.

Prerequisites

A Kubernetes cluster installed on your local machine or development environment.
kubectl command-line tool installed and configured.

Setting Up Ingress-Nginx

Install SIG nginx controller using their helm chart:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install my-nginx ingress-nginx/ingress-nginx \
  --namespace ingress \
  --create-namespace \
  --set controller.publishService.enabled=true

If you run microk8s as I do, you can enable ingress-nginx by the following command:

microk8s enable ingress

Output:

 microk8s enable ingress

Infer repository core for addon ingress
Enabling Ingress
ingressclass.networking.k8s.io/public created
ingressclass.networking.k8s.io/nginx created
Warning: resource namespaces/ingress is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/ingress configured
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
configmap/nginx-ingress-tcp-microk8s-conf created
configmap/nginx-ingress-udp-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled

This command sets up the ingress-nginx controller within a MicroK8s cluster, ready to manage Ingress resources.

When you install the Ingress-Nginx controller via Helm on MicroK8s, the manual configuration might not align perfectly with microk8s simplified networking and security model. For instance:

Network Policies: MicroK8s has a unique way of handling network isolation that may not be automatically configured by the Helm chart.
Service Exposure: MicroK8s often uses different mechanisms (like its built-in LoadBalancer) to expose services, which may need specific configuration not handled by default Helm deployments.
Resource Permissions: Helm charts might require additional RBAC configuration to work seamlessly with MicroK8s, especially regarding how resources are accessed and managed across namespaces.

Find the Cluster IP To direct traffic to our services, we need to identify the cluster's IP:
```
kubectl cluster-info
```
Look for the "Kubernetes control plane" line, which shows the IP address, e.g., 192.168.64.54.
Update /etc/hosts on Your Local Machine Map the domain dummy-app.devoriales.local to the cluster IP to make it accessible via your browser:
```
echo "192.168.64.54 dummy-app.devoriales.local" | sudo tee -a /etc/hosts
```
Change ip to match your cluster IP.

❗Unfortunately, we'll need to add the full application host name to the list since the hosts file does not support wild cards. You could install the dnsmasq, but that is out of scope of this article.

Deploy a Dummy Nginx Application

Create a namespace

kubectl create ns dummy-app

Create a ConfigMap

If you want to have a custom home page, we could simply do it by creating a ConfigMap like the following:

kubectl create configmap nginx-html --from-literal=index.html='<html><body><h1>This is devoriales.local</h1></body></html>' -n dummy-app --dry-run=client -o yaml | kubectl apply -f -

In the deployment manifest, we need to map this ConfigMap, see the next section. If you want to stick to the Nginx default page, you could simple skip this step.

Create a Deployment

If you skipped the previous step where we created a custom home page, please exclude the VolumeMount and the volume for the ConfigMap.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: dummy-app
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: html-volume
          mountPath: /usr/share/nginx/html
      volumes:
      - name: html-volume
        configMap:
          name: nginx-html
EOF

Create the service

Since we aim to expose our dummy app, let's create a service:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: dummy-app-service
  namespace: dummy-app
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
EOF

Create an Ingress rule

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: http-ingress
  namespace: dummy-app
spec:
  ingressClassName: nginx
  rules:
  - host: dummy-app.devoriales.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-app-service
            port:
              number: 80
EOF

Test Access

Verify the setup by accessing http://dummy-app.devoriales.local via a browser or using curl:

curl http://dummy-app.devoriales.local -v

Output example:

 Trying 192.168.64.54:80...
* Connected to dummy-app.devoriales.local (192.168.64.54) port 80
> GET / HTTP/1.1
> Host: dummy-app.devoriales.local
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 12 May 2024 09:11:48 GMT
< Content-Type: text/html
< Content-Length: 59
< Connection: keep-alive
< Last-Modified: Fri, 10 May 2024 12:09:28 GMT
< ETag: "663e0e78-3b"
< Accept-Ranges: bytes
<
* Connection #0 to host dummy-app.devoriales.local left intact
<html><body><h1>This is devoriales.local</h1></body></html>%

Great! We've successfully exposed our application.

Setting Up TLS Locally with a Self-Signed Certificate

For local development environments, securing your applications with TLS can enhance security by encrypting traffic to and from your services. Here’s how you can set up TLS using locally trusted certificates generated by mkcert, a simple tool that simplifies the creation of valid certificates.

💁‍♂️ mkcert is a simple tool designed to make it easier to create locally-trusted development certificates. It simplifies the process of generating self-signed certificates that are trusted by your operating system and browsers, eliminating common security warnings that typically occur when using self-signed certificates.

When you're developing websites or web applications that use HTTPS, browsers require a certificate to secure the connection. Normally, certificates are issued by Certificate Authorities (CAs) which browsers trust by default. However, for local development, obtaining a certificate from a CA every time can be impractical.

Official git repository: https://github.com/FiloSottile/mkcert

Set Up TLS

# Install mkcert on macOS
brew install mkcert

# Install and set up the local CA
mkcert -install

# Install mkcert on Debian, example:
sudo snap install mkcert

# or manual installation on linux:
wget https://github.com/FiloSottile/mkcert/releases/download/<release>/mkcert-<release>-linux-amd64

Now, we'll generate Locally-Trusted Certificates:

mkcert dummy-app.devoriales.local

This command will create two files: dummy-app.devoriales.local.pem (the certificate) and dummy-app.devoriales.local-key.pem (the private key), which are automatically trusted by your system.

Create a Kubernetes Secret for the Certificate: Store the certificate and key in your Kubernetes cluster as a secret.

kubectl create secret tls dummy-app-tls --cert=dummy-app.devoriales.local.pem --key=dummy-app.devoriales.local-key.pem -n dummy-app

Configure the Ingress to Use the TLS Secret: Update your Ingress resource to reference the newly created secret for TLS termination:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: http-ingress
  namespace: dummy-app
spec:
  ingressClassName: nginx  # Specifies which Ingress Controller to use
  tls:
  - hosts:
    - dummy-app.devoriales.local
    secretName: devoriales-local-tls
  rules:
  - host: dummy-app.devoriales.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-app-service
            port:
              number: 80
EOF

Test HTTPS Access: Access the application via a browser or using curl to ensure HTTPS is working:

curl https://dummy-app.devoriales.local -v

Output example:

*   Trying 192.168.64.54:80...
* Connected to dummy-app.devoriales.local (192.168.64.54) port 80
> GET / HTTP/1.1
> Host: dummy-app.devoriales.local
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Date: Sun, 12 May 2024 09:40:56 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: https://dummy-app.devoriales.local
<
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host dummy-app.devoriales.local left intact
➜  ~ curl https://dummy-app.devoriales.local
<html><body><h1>This is devoriales.local</h1></body></html>%
➜  ~ curl https://dummy-app.devoriales.local -v
*   Trying 192.168.64.54:443...
* Connected to dummy-app.devoriales.local (192.168.64.54) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: O=mkcert development certificate; OU=alma@Aleksandros-MBP (Aleksandro Matejic)
*  start date: May 12 09:35:06 2024 GMT
*  expire date: Aug 12 09:35:06 2026 GMT
*  subjectAltName: host "dummy-app.devoriales.local" matched cert's "dummy-app.devoriales.local"
*  issuer: O=mkcert development CA; OU=alma@Aleksandros-MBP (Aleksandro Matejic); CN=mkcert alma@Aleksandros-MBP (Aleksandro Matejic)
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://dummy-app.devoriales.local/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: dummy-app.devoriales.local]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: dummy-app.devoriales.local
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 200
< date: Sun, 12 May 2024 09:42:07 GMT
< content-type: text/html
< content-length: 59
< last-modified: Fri, 10 May 2024 12:09:28 GMT
< etag: "663e0e78-3b"
< accept-ranges: bytes
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host dummy-app.devoriales.local left intact
<html><body><h1>This is devoriales.local</h1></body></html>%

We have successfully set up TLS for our dummy-app, ensuring that our development environment mirrors production security configurations as closely as possible. This setup gives us a hands-on experience with managing TLS within Kubernetes.

Troubleshooting the Ingress Controller

When the Kubernetes Ingress controller is not behaving as expected, such as not routing traffic properly or showing errors, we could follow the following troubleshooting steps to diagnose and resolve issues.

Check Pod Status

First, we could ensure that all the Ingress controller pods are running without frequent restarts:

kubectl get pods -n ingress

Output example:

NAME                                      READY   STATUS    RESTARTS   AGE
nginx-ingress-microk8s-controller-hdqft   1/1     Running   0          69m
nginx-ingress-microk8s-controller-rs52q   1/1     Running   0          69m
nginx-ingress-microk8s-controller-zr4cj   1/1     Running   0          69m

Inspect Logs

Logs provide insights into what the Ingress controller is doing and any errors it may be encountering:

kubectl logs -n ingress -l name=nginx-ingress-microk8s --tail=50

❗Make sure to set the correct label of your ingress controller pods. You can check it by running:

kubectl describe daemonset nginx-ingress-microk8s-controller -n ingress

Name:           nginx-ingress-microk8s-controller
Selector:       name=nginx-ingress-microk8s
Node-Selector:  <none>
Labels:         microk8s-application=nginx-ingress-microk8s

Logs example:

I0512 09:35:25.193004       7 store.go:585] "Secret was added and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"
I0512 09:35:25.200647       7 backend_ssl.go:67] "Adding secret to local store" name="dummy-app/devoriales-local-tls"
W0512 09:35:28.498429       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.
I0512 09:35:45.718672       7 store.go:658] "secret was deleted and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"
W0512 09:35:45.721568       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.
W0512 09:35:45.744381       7 controller.go:1333] Error getting SSL certificate "dummy-app/devoriales-local-tls": local SSL certificate dummy-app/devoriales-local-tls was not found. Using default certificate
I0512 09:35:45.798520       7 store.go:585] "Secret was added and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"
I0512 09:35:45.799176       7 backend_ssl.go:67] "Adding secret to local store" name="dummy-app/devoriales-local-tls"
I0512 09:35:47.899768       7 store.go:658] "secret was deleted and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"
I0512 09:35:47.953676       7 store.go:585] "Secret was added and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"
I0512 09:35:47.954332       7 backend_ssl.go:67] "Adding secret to local store" name="dummy-app/devoriales-local-tls"
W0512 09:35:49.055268       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.
W0512 09:35:52.389186       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.
192.168.64.1 - - [12/May/2024:09:35:59 +0000] "GET / HTTP/2.0" 200 59 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" 483 0.002 [dummy-app-dummy-app-service-80] [] 10.1.83.33:80 59 0.003 200 1c070c2c32ec963c73e1362e53810ddd
W0512 09:36:30.943402       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.
W0512 09:38:22.094224       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.
192.168.64.1 - - [12/May/2024:09:40:56 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/8.4.0" 89 0.000 [dummy-app-dummy-app-service-80] [] - - - - 5be2245d5981b9751d9513f72fbfa931
192.168.64.1 - - [12/May/2024:09:41:29 +0000] "GET / HTTP/2.0" 200 59 "-" "curl/8.4.0" 39 0.005 [dummy-app-dummy-app-service-80] [] 10.1.83.33:80 59 0.005 200 3e5f8b7522997b11e6bfa8e3c028216d
192.168.64.1 - - [12/May/2024:09:42:07 +0000] "GET / HTTP/2.0" 200 59 "-" "curl/8.4.0" 39 0.010 [dummy-app-dummy-app-service-80] [] 10.1.83.33:80 59 0.010 200 0cad01a07265c613a413bad4212e2213
W0512 09:43:40.925780       7 controller.go:1112] Service "monitoring/promlens" does not have any active Endpoint.

Log Details and possible steps to fix issues:

Secret Addition and Parsing:

"Secret was added and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"

A TLS/SSL secret named dummy-app/devoriales-local-tls was added to the Kubernetes cluster and is referenced in Ingress annotations. The Ingress controller is parsing this information to configure TLS termination.

The process is working as expected, setting up secure communications for your Ingress resources.

Secret Addition to Local Store:

"Adding secret to local store" name="dummy-app/devoriales-local-tls"

The newly added secret is being stored locally within the Ingress controller's cache or configuration.

This is part of the normal handling and caching mechanism that speeds up SSL operations within the cluster.

Secret Deletion and Parsing:

"secret was deleted and it is used in ingress annotations. Parsing" secret="dummy-app/devoriales-local-tls"

The referenced secret was deleted from the cluster, and the Ingress controller is updating its configuration to reflect this change.

This might lead to issues if not handled properly, as the SSL configuration could be left without a valid certificate temporarily.

Service without Active Endpoints:

Service "monitoring/promlens" does not have any active Endpoint.

The Kubernetes service named monitoring/promlens has no pods available to serve traffic, likely due to pods not running or a mismatch in selector labels.

Traffic directed to this service will fail, leading to potential downtime or unavailability of the service.

We could verify it pretty simply:

 kubectl get pods -n monitoring | grep promlens
promlens-8b5cc9f98-525k8                             0/1     ImagePullBackOff         0               47h

As we can see, there is an issue with the pod which never gets registred as the active endpoint.

SSL Certificate Not Found:

Error getting SSL certificate "dummy-app/devoriales-local-tls": local SSL certificate dummy-app/devoriales-local-tls was not found. Using default certificate

After the deletion of the SSL secret, the specific certificate dummy-app/devoriales-local-tls could not be found, and as a fallback, a default certificate is being used.

This could lead to security warnings for clients or failed TLS handshakes if the default certificate is not trusted by clients.

Successful and Redirected Requests:

Logs:

"GET / HTTP/2.0" 200 59
"GET / HTTP/1.1" 308 164

These logs from the Nginx Ingress show successful HTTP requests (200 status code) and HTTP redirects (308 status code). The requests are served over HTTP/2.0 and HTTP/1.1 respectively.

Our application is responding to requests, but some requests are being redirected, possibly due to HTTPS enforcement or other configured redirection rules.

Wrapping Up

Congratulations on successfully completing this tutorial, which was specifically designed for setting up Ingress-Nginx within a local Kubernetes environment. This setup not only streamlines a development process but also mirrors real-world production settings, providing a practical learning experience.

We've...

learned the use of Ingress to expose services in a local Kubernetes setup, moving beyond traditional node ports and port-forwarding.
implemented TLS security, enhancing the security of our dummy-app application by using locally trusted, self-signed certificates.

See you in the next article!

About the Author

Aleksandro Matejic, a Cloud Architect, began working in the IT industry over 21 years ago as a technical specialist, right after his studies. Since then, he has worked in various companies and industries in various system engineer and IT architect roles. He currently works on designing Cloud solutions, Kubernetes, and other DevOps technologies.

You can contact Aleksandro by visiting his LinkedIn Profile

Join Devoriales Today!