Introduction

Consider the following secret in Kubernetes:

apiVersion: v1
kind: Secret
metadata:
  name: my-database-secret
type: Opaque
data:
  username: dXNlcg==  # base64 encoded value for 'user'
  password: cGFzc3dvcmQ=  # base64 encoded value for 'password'

It is important to note that the username and password values are base64 encoded; this does not provide encryption. Base64 encoding is simply a way to represent binary data in text form and does not offer any security. To protect sensitive information, it is crucial to use additional security measures such as encryption. We shall never commit secret manifests to the git repositories:

The best practice is to manage access through a strengthened RBAC (Role-Based Access Control) model. Additionally, storing Secret manifests in a Git repository is not a good (secure) practise, as it poses a security risk. If the repository is accessible to others, they could easily decode the base64-encoded secrets, exposing sensitive information.

In this blog post, we will walk through a use case where we need to synchronize secrets from AWS Secrets Manager into Kubernetes secrets using Secrets Store CSI driver.

This could be useful when you have applications running in your Kubernetes cluster that need to access sensitive data stored in AWS Secrets Manager.

AWS Secrets Manager is a secrets management service that helps you protect access to your applications, services, and IT resources. On the other hand, Kubernetes secrets are a Kubernetes-native way to store sensitive information, such as passwords, OAuth tokens, and ssh keys.

One could directly use AWS SDK to fetch secrets from AWS Secrets Manager, but this approach has its own drawbacks.

It tightly couples your application with AWS as a cloud provider, and increases the complexity of the application code.

By using this tool, you can either mount the secret from AWS Secrets Manager directly into a pod, or sync it with a Kubernetes secret.

Even though this tutorial is focused on AWS EKS, you can apply the same concept to integrate any Kubernetes cluster with any Secrets Manager like Hashicorp Vault.

Diagram

The following diagram illustrates what we will cover in this tutorial:

The diagram illustrates the architecture of a system that involves the synchronization of secrets between AWS Secrets Manager and Kubernetes.

Here is a description of each component:

1. EKS Cluster: Represents an Amazon Elastic Kubernetes Service (EKS) cluster, which is a managed Kubernetes service provided by AWS.

2. Kubernetes Namespace: Refers to a logical boundary within the Kubernetes cluster, allowing for the segregation and organization of resources.

3. Kubernetes Service Account: Represents an identity that is associated with a Kubernetes namespace. It is used to authenticate and authorize access to Kubernetes resources.

4. IAM Role: Denotes an AWS Identity and Access Management (IAM) role that is annotated with the Kubernetes Service Account. This association enables the Kubernetes Service Account to assume the IAM role and access AWS resources. 

5. CronJob: Represents a Kubernetes CronJob. In this context, the CronJob is responsible for periodically triggering the synchronization process. In the tutorial, we will also include a Deployment in addition to CronJob.

6. AWS Secrets Manager: Refers to the AWS service that securely stores and manages secrets, such as API keys, database credentials, and other sensitive information.

7. Secrets Store CSI Driver: Represents the Secrets Store Container Storage Interface (CSI) Driver, which is a Kubernetes extension that enables the retrieval and mounting of secrets from external secret providers. In this case, it interacts with AWS Secrets Manager to fetch the secrets.

8. Secrets Store CSI Driver Provider: Responsible for integrating AWS Secrets Manager with the Secrets Store CSI Driver. It facilitates the communication and retrieval of secrets from AWS Secrets Manager.

9. Kubernetes Secret: Denotes a Kubernetes resource used to store sensitive data, such as passwords or tokens. The Secrets Store CSI Driver syncs the secrets from AWS Secrets Manager and makes them available as Kubernetes Secrets through mounting.

The diagram visualizes the flow of secrets synchronization from AWS Secrets Manager to Kubernetes using the Secrets Store CSI Driver. The CronJob triggers the synchronization process, and the Secrets Store CSI Driver Provider interacts with AWS Secrets Manager to fetch the secrets and expose them as Kubernetes Secrets. The Secrets Store CSI Driver mounts the secret as a volume to a pod. It also supports dynamic update of the mounted volume in case of secret rotation. 

Prerequisites

• A running EKS cluster. 
• Permissions to create AWS resources
• Permissions to create EKS (Kubernetes) objects

Out Of Scope

This tutorial does not cover the automation of the components involved in implementing this solution.

The focus is on understanding the architecture, concepts and implementation of each component. 

However, in a future tutorial, we plan to explore automation using AWS CDK and Terraform.

Overview

The overall approach is to use the Secrets Store CSI driver and Secrets Store CSI drive Provider to fetch secrets from AWS Secrets Manager and mount them into the pods and sync them with a Kubernetes secret. Then, we will configure a CronJob to periodically sync the secret into Kubernetes secret.

The synchronization is necessary because the sync with Kubernetes secret only happens at the time of pod creation. However, we will be able to sync the secret volume dynamically without a new pod creation. You could of course run a script inside the pod that dynamically updates the Kubernetes secret, but will keep to the core functionality of the Secrets Store CSI driver. 

In addition, we will create a Kubernetes service account that will be used by the CronJob, and associate it with an AWS IAM role that has permissions (the associated policy to the role to be precise) to access the secrets in AWS Secrets Manager.

Step 1 : Install Secrets Store CSI driver And Secrets Store AWS provider

The purpose of the Secrets Store CSI driver and AWS provider is to allow Kubernetes to securely access secrets stored in AWS Secrets Manager. By using the Secrets Store CSI driver, Kubernetes can mount secrets from AWS Secrets Manager as volumes in pods, making them available to applications running in the Kubernetes cluster. This approach simplifies application code and makes it more cloud-agnostic.

The AWS provider is used to authenticate with AWS Secrets Manager and retrieve the secrets.

 Secrets Store CSI driver installation

The following shows how to add secrets-store-csi-driver to your cluster.

1. Add the secrets-store-csi-driver chart repository:
   

   helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   

2. Update your local Helm chart repository cache:
   

   helm repo update

3. Install the secrets-store-csi-driver chart into the kube-system namespace:
   

   helm install secrets-store-csi-driver secrets-store-csi-driver/secrets-store-csi-driver --version 1.3.3 --namespace kube-system --set syncSecret.enabled=true --set enableSecretRotation=true
   

Please note! It's very important to add the flag --enableSecretRotation=true, otherwise any value changes, either manually or via key rotation, will not be reflected after the initial creation of the secret.

Secrets Store AWS provider installation

The following shows how to add secrets-store-csi-driver to your cluster.

1. Add the secrets-store-csi-driver-provider-aws chart repository:
   

   helm repo add secrets-store-csi-driver-provider-aws https://aws.github.io/secrets-store-csi-driver-provider-aws

2. Update your local Helm chart repository cache:
   

   helm repo update

3. Install the secrets-store-csi-driver-provider-aws chart into the kube-system namespace:
   

   helm install secrets-store-csi-driver-provider-aws secrets-store-csi-driver-provider-aws/secrets-store-csi-driver-provider-aws --version 0.3.3 --namespace kube-system
   

Add ClusterRole and ClusterRoleBinding for Secrets Store CSI driver 

The ClusterRole named secrets-store-csi-driver below defines the permissions that are being granted. It specifies that the holder of this role can get, watch, list, create, update, and patch secrets in the Kubernetes API.

The ClusterRoleBinding then assigns this ClusterRole to the secrets-store-csi-driver service account in the kube-system namespace. This service account is typically used to run the Secrets Store CSI Driver pods. By assigning the ClusterRole to this service account, you are granting these permissions to the CSI Driver.

It's important to note that these permissions are cluster-wide. This means that the Secrets Store CSI Driver can manage secrets in all namespaces, not just the kube-system namespace. If you want to restrict the CSI driver to only manage secrets in a specific namespace, you can use a Role and RoleBinding instead of a ClusterRole and ClusterRoleBinding.

This RBAC configuration is crucial for the Secrets Store CSI Driver to operate correctly. Without it, the driver would not be able to create or update Kubernetes secrets, and thus could not make the secrets from the external system available to your pods.

Add the ClusterRole and ClusterRoleBinding to your cluster:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secrets-store-csi-driver
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secrets-store-csi-driver
subjects:
- kind: ServiceAccount
  name: secrets-store-csi-driver
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: secrets-store-csi-driver
  apiGroup: rbac.authorization.k8s.io

Verification

We can verify the daemonset related to secrets-store-csi-driver and secrets-store-csi-driver-provider-aws

kubectl get daemonset -n kube-system -o wide | grep secret

NAME                                                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
eksclusterclusterchartsecretsstorecsidriver555b2363-secrets-sto   4         4         4       4            4           kubernetes.io/os=linux   44m     node-driver-registrar,secrets-store,liveness-probe   registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0,registry.k8s.io/csi-secrets-store/driver:v1.3.3,registry.k8s.io/sig-storage/livenessprobe:v2.9.0   app=secrets-store-csi-driver
rclusterchartsecretsstorecsidriverprovideraws793894e4-secrets-s   4         4         4       4            4           kubernetes.io/os=linux   16m     provider-aws-installer                               public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-50-g5b4aca1-2023.06.09.21.19

Step 2: Create SecretProviderClass

The SecretProviderClass is a Kubernetes custom resource that defines how to fetch the secrets from AWS Secrets Manager. Here's an example:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets-scp
  namespace: your-namespace
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "my-secret"
        objectType: "secretsmanager"
  secretObjects:
    - secretName: my-k8s-secret
      type: Opaque
      data:
        - objectName: my-secret
          key: secret

In this example, replace "my-secret" with the name of your secret in AWS Secrets Manager, and my-k8s-secret with the name you want for the Kubernetes secret.

Step 3: Create IAM Role

We need to create an AWS IAM role that will be assumed by a Kubernetes service account and which provides permissions to retrieve secrets from AWS Secrets Manager. The following is an example of an IAM policy that grants read access to a particular secret:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:<secret-name>"
    }
  ]
}

Replace <region>, <account-id>, and <secret-name> with the appropriate values for your environment.

Also we need to add a Trust Policy to the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<aws-account-id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<cluster-oidc-id>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<region>.amazonaws.com/id/<cluster-oidc-id>:sub": "system:serviceaccount:<namespace>:<service-account-name>"
        }
      }
    }
  ]
}

This trust policy allows a specific Kubernetes service account in a specific namespace to assume the IAM role when the OIDC provider for the EKS cluster authenticates the service account.

If you are interested to learn how this works, you can read the article series about OIDC: https://devoriales.com/post/208

Once you have created the IAM role, note down its Amazon Resource Name (ARN). We will annotate the Service Account with the AWS IAM Role ARN. 

❗With IAM Roles for Service Accounts (IRSA), you can associate an IAM role with a Kubernetes service account. This allows your applications to directly assume an IAM role instead of using the EC2 instance's role which is less secure and which I don't recommend.

Step 4: Create ServiceAccount, Role and RoleBinding

Service Account

Next, we will create a ServiceAccount.

Replace <aws-account-id> with your AWS account ID and <role-name> with the name of the IAM role that has permissions to access the secrets in AWS Secrets Manager.

Here's how you can define these resources:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secret-manager-sa
  namespace: your-namespace
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/<role-name>

Here's an example of how to create a Role and RoleBinding for the ServiceAccount to be able to create and update Kubernetes secrets:

Role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: secret-manager-role
  namespace: your-namespace
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create", "update"]

RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: secret-manager-role-binding
  namespace: your-namespace
subjects:
- kind: ServiceAccount
  name: secret-manager-sa
  namespace: your-namespace
roleRef:
  kind: Role
  name: secret-manager-role
  apiGroup: rbac.authorization.k8s.io

Replace your-namespace with the name of the namespace where the ServiceAccount was created.

Once you have created the Role and RoleBinding, the ServiceAccount will have the necessary permissions to create and update Kubernetes secrets.

Step 5: Create CronJob

Now, we can create a CronJob that runs on a schedule to synchronize the secrets from AWS Secrets Manager into Kubernetes secrets.

Here's an example of a CronJob manifest that you can apply 

# job to update secrets
kind: CronJob
metadata:
  name: secretmgt-cronjob
  namespace: devoriales-demo
spec:
  schedule: "*/1 * * * *"
  concurrencyPolicy: Forbid  # Forbid: If it is time for a new job run and the previous job run hasn't finished yet, skip the new job run.
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: secretmgt-sa
          volumes:
            - name: secret-store-inline
              csi:
                driver: "secrets-store.csi.k8s.io"
                readOnly: true
                volumeAttributes:
                  secretProviderClass: "aws-secrets-spc"
                  syncSecret: "true"
          containers:
            - name: secret-sync
              image: registry.k8s.io/e2e-test-images/busybox:1.29
              command: ["/bin/sh"]
              args: ["-c", "echo $(date) Updating secrets from AWS Secrets Manager; sleep 30"]
              volumeMounts:
                - name: secret-store-inline
                  mountPath: "/mnt/secrets-store"
                  readOnly: true
          restartPolicy: OnFailure

In this example, replace <aws-region> with the region where your AWS Secrets Manager secrets are stored.

You can adjust the schedule of the cronjob.

Kubernetes Manifests - Full Code

Here are all Kubernetes manifests that were written for this tutorial:

apiVersion: v1
kind: Namespace
metadata:
  name: devoriales-demo
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets-spc
  namespace: devoriales-demo
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "/devoriales/testdatabasesecret"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: username
          - path: password
            objectAlias: password
  secretObjects:
    - secretName: db-secret
      type: Opaque
      data:
        - objectName: username
          key: username
        - objectName: password
          key: password
---
# service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: secretmgt-sa
  namespace: devoriales-demo
  annotations:
    eks.amazonaws.com/role-arn: <ROLE_ARN>
---
# role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secretmgt-role
  namespace: devoriales-demo
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "watch", "list", "create", "update", "patch" ]
---
# role binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secretmgt-role-binding
  namespace: devoriales-demo
subjects:
  - kind: ServiceAccount
    name: secretmgt-sa
    namespace: devoriales-demo
roleRef:
  kind: Role
  name: secretmgt-role
  apiGroup: rbac.authorization.k8s.io
---
# job to update secrets
apiVersion: batch/v1
kind: CronJob
metadata:
  name: secretmgt-cronjob
  namespace: devoriales-demo
spec:
  schedule: "*/1 * * * *"
  concurrencyPolicy: Forbid  # Forbid: If it is time for a new job run and the previous job run hasn't finished yet, skip the new job run.
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: secretmgt-sa
          volumes:
            - name: secret-store-inline
              csi:
                driver: "secrets-store.csi.k8s.io"
                readOnly: true
                volumeAttributes:
                  secretProviderClass: "aws-secrets-spc"
                  syncSecret: "true"
          containers:
            - name: secret-sync
              image: registry.k8s.io/e2e-test-images/busybox:1.29
              command: ["/bin/sh"]
              args: ["-c", "echo $(date) Updating secrets from AWS Secrets Manager; sleep 30"]
              volumeMounts:
                - name: secret-store-inline
                  mountPath: "/mnt/secrets-store"
                  readOnly: true
          restartPolicy: OnFailure
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secrets-store-csi-driver
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secrets-store-csi-driver
subjects:
- kind: ServiceAccount
  name: secrets-store-csi-driver
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: secrets-store-csi-driver
  apiGroup: rbac.authorization.k8s.io

Synchronization Verification

In this section we will check the synchronization. We'll begin with checking the pod that was created by the job:

kubectl get pods


NAME                               READY   STATUS      RESTARTS   AGE
secretmgt-cronjob-28155647-rs9h6   1/1     Running     0          3s

Exec the pod and check the secrets mounted

Now we'll enter the pod.

Once inside the container, the commands cat /mnt/secrets-store/ and cat /mnt/secrets-store/password can be used to list the contents of the /mnt/secrets-store/ directory and display the mounted secrets (technically those are volumes)

The /mnt/secrets-store/ directory is where we configured the Secrets Store CSI driver mounts the secrets fetched from the AWS Secrets Manager. Each secret is represented as a file, with the filename being the secret's name and the file's content being the secret's value.In this case, the directory listing shows that two keys exist. Let's have a look at the content of: /mnt/secrets-store/password

The cat /mnt/secrets-store/password command prints the contents of the file, revealing its value:

kubectl exec -it secretmgt-cronjob-28155647-rs9h6 -- sh


/ # cat /mnt/secrets-store/password
supersecret

Great, we got our secret mounted. Now we can exit the pod.

Get Secret Value

We'll verify the Kubernetes secret. Let's check if the Kubernetes secret has been created according to what we specified in SecretProviderClass manifest:

kubectl get secret

NAME        TYPE     DATA   AGE
db-secret   Opaque   2      38m

Check the secret data

Since we verified that the Kubernetes secret has been created with two items, let's now check the base64 decoded values:

kubectl get secret db-secret -o json | jq -r '.metadata.name as $name | .data | to_entries[] | "\($name) \(.key): \((.value|@base64d))"'

Output:
db-secret password: supersecret
db-secret username: dbadmin

Really great, we got our Kubernetes secret created.

Secret Rotation - Let's change a value

In this section, we will change the password value to extremlydifferentandsecure

We can exec into the container and check if the new secret has been updated:

kubectl exec -it secretmgt-cronjob-28155647-rs9h6 -- sh

/ # cat /mnt/secrets-store/password
extremlydifferentandsecure

As we can see, the password key reflects a new value.

With the following we can also check if the Kubernetes secret has been updated:

kubectl get secret db-secret -o json | jq -r '.metadata.name as $name | .data | to_entries[] | "\($name) \(.key): \((.value|@base64d))"'

Output:
db-secret password: extremlydifferentandsecure
db-secret username: dbadmin

Deployment instead of CronJob

In this tutorial, we've been using a CronJob as an example so far. But you can also mount secrets as volume to a Deployment too.

Assume that you have an application in Kubernetes and you want to provide a secure way to deliver secret content to applications.

In the following example, we have an NGINX application running as a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secretmgt-deployment
  namespace: devoriales-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secretmgt
  template:
    metadata:
      labels:
        app: secretmgt
    spec:
      serviceAccountName: secretmgt-sa
      volumes:
        - name: secret-store-inline
          csi:
            driver: "secrets-store.csi.k8s.io"
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets-spc"
              syncSecret: "true"
      containers:
        - name: secret-sync
          image: nginx
          volumeMounts:
            - name: secret-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true

After we have deployed our Nginx application, we can exec into the container:

kubectl exec -it secretmgt-deployment-5c9699d6dc-w6wws -- bash


root@secretmgt-deployment-5c9699d6dc-w6wws:/# cat /mnt/secrets-store/
_devoriales_testdatabasesecret  password                        username

root@secretmgt-deployment-5c9699d6dc-w6wws:/#cat /mnt/secrets-store/password
extremlydifferentandsecure

What's really nice, since we have enabled support for the secret rotation, you don't have to re-instantiate your application, If the value get's changed in AWS Secrets Manager, it will automatically update the same in your mounted secret.

Conclusion

This blog post explained how to synchronize secrets from AWS Secrets Manager into Kubernetes secrets using the Secrets Store CSI driver and AWS provider.

By using this approach, applications running in the Kubernetes cluster can access sensitive data stored in AWS Secrets Manager which is a great way of keeping data secure.

The post provided step-by-step instructions for setting up the necessary resources, including creating a SecretProviderClass, IAM role, ServiceAccount, Role, RoleBinding, CronJob and Deployment. The blog post also outlines the benefits of using this approach.

We were also able to achieve secret rotation by configuring the secrets-store-csi-driver to suport the rotation by adding the flag --enableSecretRotation