Only registred users can make comments

AWS EKS Secret Encryption: Securing Your EKS Secrets At Rest with AWS KMS

Secrets encryption with AWS KMS


By default, Kubernetes secrets are encoded but not encrypted at rest, meaning that if an unauthorized person gains access to the underlying storage layer, they could potentially decode these secrets. Encrypting these secrets with AWS KMS mitigates this risk by ensuring that the secrets are not only encoded but also encrypted with a strong cryptographic key managed by AWS KMS. This adds an additional layer of security, protecting sensitive information in compliance with your organization's security policies and various regulatory requirements.

When integrating AWS KMS with Amazon EKS for enhanced security, you can either create a new Customer Master Key (CMK) or use an existing one to encrypt and decrypt Kubernetes secrets. These secrets often contain sensitive information, such as passwords, tokens, or certificates, crucial for the operation of applications deployed on EKS. This is something we will learn how to perform later in this article.

In essence, AWS KMS provides the mechanism for robust encryption management in EKS, enabling secure storage and controlled access to Kubernetes secrets, thereby significantly enhancing the overall security posture of your Kubernetes environment on AWS.

It's important to clarify that the focus of this article on using AWS KMS for encrypting Kubernetes secrets in Amazon EKS is specifically about securing secrets at rest. This distinction is crucial for understanding the scope and limitations of the encryption process.

Clarification on Secrets Management in EKS

This article does not go into the specifics of defining secrets in Kubernetes or the access controls surrounding them. Kubernetes employs a role-based access control (RBAC) system that allows fine-grained permission settings for who can access Kubernetes resources, including secrets. A user or process with the necessary permissions, as defined by RBAC roles, can still retrieve and decode secrets stored in EKS.

The Focus on Encryption at Rest

The objective here is enhancing the security of secrets at rest. "At rest" refers to data that is stored, as opposed to "in transit," which refers to data being transmitted from one place to another. By integrating AWS KMS for encryption, the article addresses the risk of unauthorized access to the underlying storage layer where secrets are persisted.

  • Encryption at Rest: This involves encrypting secret data before it is stored in the etcd database of an EKS cluster. AWS KMS provides the encryption keys (Customer Master Keys, or CMKs) for this purpose. Even if an attacker gains physical access to the storage or somehow bypasses other security controls to access the stored data, they would not be able to decrypt the secrets without access to the corresponding CMK in AWS KMS.

  • RBAC and Access Control: While encryption at rest significantly improves the security posture of your Kubernetes environment, managing access to secrets via Kubernetes' RBAC is equally important. Proper configuration of RBAC roles and bindings is necessary to ensure that only authorized users and applications can retrieve and decode secrets, regardless of the encryption at rest.

It's crucial to recognize that encryption at rest, as facilitated by AWS KMS, is part of a broader security strategy that includes network security, access control, and the principle of least privilege. While AWS KMS effectively protects secrets at rest, securing access to those secrets in use and transit involves additional layers of security, including proper RBAC configuration and adherence to best practices in Kubernetes secrets management. This article focuses on the encryption of secrets at rest, highlighting the role of AWS KMS in enhancing the security of stored secrets in an Amazon EKS environment.

Base64 Encode and Decode Secrets

By default, secrets in Kubernetes are just base64 encoded. This means, the data stored is not encrypted and if someone has access to the secret object, the data can easily be decoded locally on the person's machine. 

The following diagram shows the process of encoding and decoding the data stored in a secret:

base64 encoding and decoding

The main purpose of Base64 is to transform data to keep the integrity of it, but at the same time the data itself can be viewed by the one that has acess to the secret , since the algorithm to decode the secret data is publically known.

In the following example, we'll create  a secret from literal:

kubectl create secret generic my-credentials --from-literal=password=supersecretpassword
With the following command, we will retrieve our password from the secret:
kubectl get secrets/my-credentials --template={{.data.password}}


It's not  shown in plain text, instead it's base64 encoded. Since the base64 algorithm is publicly known, and part of most MacOS and Linux distos, we can easily retrieve the password:
kubectl get secrets/my-credentials --template={{.data.password}} | base64 --decode



❗To be able to decode a secret data, the user has to have permissions to get the secret object in Kubernetes.

 It's important that the RBAC is properly configured in your cluster so only those that needs to work with secrets least privileged access to the secret object.


In the realm of digital security, encrypting data is a cornerstone practice, securing information through a method akin to locking it in a vault. This process relies on sophisticated algorithms, which transform readable data into a secured format that can only be deciphered with the correct encryption key. Imagine a complex puzzle that can only be solved with a unique key; similarly, encryption algorithms such as AES, RSA, and Blowfish serve as the puzzle creators, intricately coding data to protect its confidentiality.

Only entities possessing the corresponding decryption key can unlock this data, ensuring that sensitive information remains accessible exclusively to authorized users. This selective accessibility is crucial in safeguarding data across various digital landscapes, from cloud storage to encrypted messaging services.

To visualize this concept, consider the diagram below, which mirrors the encryption journey we've outlined. It depicts not just the locked vault (encrypted data) but also the key's dual role in sealing and unlocking the data's secrecy:



From the official doc:

The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd. The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption. The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS. The KMS provider uses gRPC to communicate with a specific KMS plugin. The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s) as the Kubernetes control plane, is responsible for all communication with the remote KMS.

AWS Key Management Service (KMS) offers a fully managed encryption service that seamlessly integrates with Amazon Elastic Kubernetes Service (EKS) and other AWS services. Its primary function within the EKS ecosystem is to facilitate the envelope encryption of Kubernetes secrets. These secrets, which might include passwords, Docker registry credentials, and TLS keys, are securely managed using the Kubernetes API and stored within the etcd key-value store.

Envelop encryption is a method where data is encrypted with a Data Encryption Key (DEK), and then the DEK itself is encrypted with a Customer Master Key (CMK). AWS KMS leverages this approach to ensure the highest security standards for your secrets. Moreover, AWS KMS employs Hardware Security Modules (HSMs), dedicated cryptographic devices designed to handle key management and encryption tasks. These devices ensure that the CMKs created are securely managed and remain non-exportable, enhancing the security posture of your Kubernetes applications.

In the context of Amazon EKS, etcd volumes, where Kubernetes secrets are stored, benefit from an additional layer of security. AWS KMS encrypts these volumes at the disk level, ensuring that all data at rest is protected. This disk-level encryption, managed by AWS, adds an essential security layer, safeguarding your sensitive information from unauthorized access.

By integrating AWS KMS with Amazon EKS, you unlock robust security capabilities for your Kubernetes secrets, ensuring they are encrypted, securely managed, and protected across their lifecycle. This integration not only strengthens your overall security strategy but also aligns with best practices for cloud-native applications

Secret Flow Encryption in EKS cluster

The following diagram shows the flow when the secret encryption has been enabled:



  1. User or Workflow Creates a Secret: A user or automated workflow initiates the creation of a Kubernetes secret in the EKS cluster.

  2. kube-api Generates a Data Encryption Key (DEK): The Kubernetes API server (kube-api) generates a new DEK for encrypting the secret's data. This DEK is generated for each secret at the time of creation or update, ensuring that the encryption is unique for each secret. The DEK is temporarily held in memory and not stored on disk.

  3. kube-api Calls KMS to Encrypt the DEK: The kube-api server uses AWS KMS to encrypt the DEK, calling the KMS Encrypt API, passing the plaintext DEK and specifying the pre-existing Customer Master Key (CMK) for encryption.

  4. Pre-existing CMK Utilized for Encryption: The Customer Master Key (CMK) is not created during this process; it is pre-selected or created by an administrator before this encryption process starts and is used by AWS KMS to encrypt the DEK. The CMK itself is not generated anew for each encryption operation; instead, it's the DEK that's newly generated and then encrypted with the already existing CMK.

  5. Encrypted DEK and Secret Stored in etcd: Once the DEK is encrypted with the CMK, the encrypted DEK is stored alongside the encrypted secret data in etcd, securing both the secret and the means to decrypt it.

  6. Accessing and Decrypting the Secret: When a pod needs to access the secret, the Kubernetes API server retrieves the encrypted secret and the encrypted DEK from etcd, uses AWS KMS to decrypt the DEK with the CMK, and then decrypts the secret's data to pass it to the pod, allowing it to use the secret.

Please note! All etcd volumes used by AWS EKS are encrypted at the disk level by using AWS managed encryption keys. This is turned on by default, you don't need to enable anything to get this behaviour.

❗It is extremly important that the access to the CMKs are limited and that you use least-priviliged model. The calls to KMS are logged in cloudtrail logs.

What is a Customer Master Key

In the context of AWS Key Management Service (KMS), a Customer Master Key (CMK) plays a central role in data encryption and decryption processes. The CMK is a logical representation of a master key that includes metadata, such as the creation date, description, and the key policy. It is the primary resource in AWS KMS and serves several vital functions in the management and control of data encryption across AWS services.

Here's a simplified representation to help illustrate some key aspects of the Customer Master Key:

DEK Encryption

  • AWS KMS: Central service for key management.
  • Customer Master Key (CMK): Primary encryption key for DEKs, managed within AWS KMS.
  • Data Encryption Key (DEK): Used directly for data encryption, then encrypted and managed by the CMK.
  • Key Policy & Access Control: Defines permissions for CMK usage.
  • AWS Services Integration: Shows how encrypted DEKs are utilized across AWS services like S3, EBS, and RDS for data encryption.
  • AWS CloudTrail: Provides an audit trail for key usage, enhancing security and compliance.

Characteristics of a CMK

  • Unique Identifier: Each CMK is uniquely identified by an Amazon Resource Name (ARN), ensuring that it can be precisely referenced in policies and API calls.
  • Region-Specific: CMKs are region-specific, meaning they are created and used within the confines of a specific AWS region. This ensures data locality and compliance with geographical regulations.
  • Non-Exportable: To maintain a high security standard, CMKs cannot be exported outside of the AWS KMS. This restriction ensures that the keys used for encryption and decryption remain under AWS's managed security infrastructure.
  • Hardware Security Module (HSM): CMKs can be generated in AWS-managed HSMs, providing a hardware layer of security that is resistant to tampering and physical theft.

Key Hierarchy and Encryption Process:

  • Envelope Encryption: AWS KMS uses an envelope encryption strategy where data is encrypted with a data encryption key (DEK), which, in turn, is encrypted with a CMK. This method provides both security and efficiency for encrypting large amounts of data.
  • Data Encryption Key (DEK): For each encryption request, AWS KMS generates a DEK. This key directly encrypts the data, while the CMK encrypts the DEK. The encrypted DEK is stored alongside the encrypted data, ensuring that decrypting the data always involves the CMK.
  • Role in AWS Services: CMKs are integrated with other AWS services, such as Amazon S3 for bucket encryption, Amazon EBS for volume encryption, and Amazon RDS for database encryption. This integration allows for a seamless and secure encryption experience across the AWS ecosystem.

Managing Access and Policies:

  • Key Policies: Each CMK has an associated key policy that specifies who can use the key and for what actions. These policies enable fine-grained access control, ensuring that only authorized entities can use the key for encryption and decryption.
  • Audit and Compliance: Integration with AWS CloudTrail enables logging of all API calls that use a CMK, providing a comprehensive audit trail for compliance and security monitoring.

How To enable KMS encryption in your cluster

AWS Console

You can install your EKS cluster via the AWS console. This is something that I've never done though.


Secret Key Encryption Examples using IaC


Enhancing the security of your Amazon EKS cluster involves encrypting secrets at rest using a custom AWS KMS key. This process ensures that sensitive information, such as passwords and API keys, is protected.

Below, we outline the steps for both AWS Cloud Development Kit (CDK) and Terraform.

Creating a Custom KMS Key

This establishes a custom KMS key dedicated to securing your EKS cluster secrets. By enabling key rotation, you ensure that the key remains secure over time.


import * as kms from 'aws-cdk-lib/aws-kms';

const eksSecretsEncryptionKey = new kms.Key(this, 'EksSecretsEncryptionKey', {
    description: 'EKS Secrets Encryption',
    enableKeyRotation: true,


resource "aws_kms_key" "eks_secrets" {
  description             = "Custom KMS Key for EKS secrets encryption"
  enable_key_rotation     = true

This step establishes a custom KMS key dedicated to securing your EKS cluster secrets. By enabling key rotation, you ensure that the key remains secure over time.

Using the Custom KMS Key for EKS Secrets Encryption

With the custom KMS key in place, the next step is to configure your EKS cluster to utilize this key for encrypting secrets.


import * as eks from 'aws-cdk-lib/aws-eks';

const cluster = new eks.Cluster(this, 'MyEksCluster', {
    version: eks.KubernetesVersion.V1_28,
    secretsEncryptionKey: eksSecretsEncryptionKey,


module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "my-secure-cluster"
  cluster_version = "1.28" 
  subnets         = ["subnet-xxxxxx", "subnet-yyyyyy"] # subnet IDs

  vpc_id = "vpc-xxxxxx"

  cluster_encryption_config = [{
    provider_key_arn = aws_kms_key.eks_secrets.arn
    resources        = ["secrets"]

By integrating the custom KMS key into your EKS cluster configuration, you ensure that all secrets stored within the cluster are encrypted, safeguarding sensitive information and complying with best security practices.

Automatic Rotation for AWS KMS CMKs

For CMKs managed by AWS KMS, you can opt to enable automatic rotation. AWS KMS will automatically rotate the keys once every year without manual intervention.
To enable automatic rotation for a CMK, navigate to the AWS KMS section in the AWS Management Console, select the key, and then choose “Key rotation” options. You can enable rotation by checking the corresponding option.

In AWS CDK, we could add the enableKeyRotation property to our code:

 const eksSecretsEncryptionKey = new kms.Key(this, 'EksSecretsEncryptionKey', {
    description: 'EKS Secrets',
    enableKeyRotation: true,
    removalPolicy: cdk.RemovalPolicy.RETAIN,

Implications of Enabling KMS Encryption for Secrets in Amazon EKS

Using AWS Key Management Service (KMS) to encrypt Kubernetes secrets in Amazon EKS significantly enhances your containerized environment's security. This setup encrypts critical operational data, including passwords, tokens, and certificates, ensuring their protection while at rest on EKS. This encryption process provides substantial security benefits, but also introduces several implications that should be carefully considered. This discussion covers both the security improvements and the challenges associated with implementing KMS encryption for secrets in EKS.

Positive Enhancements Using AWS KMS Encryption

  1. Robust Security: AWS KMS encryption adds a robust layer of security, ensuring that secrets stored in EKS are encrypted using strong cryptographic keys. This significantly mitigates the risk of unauthorized access to sensitive information.

  2. Compliance Assurance: Many organizations are subject to stringent regulatory requirements that mandate the encryption of sensitive data at rest. By leveraging KMS encryption, you align with best practices for data protection and comply with various regulatory standards, thereby safeguarding your organization's reputation and avoiding potential fines.

  3. Centralized Key Management: AWS KMS provides a centralized platform for managing cryptographic keys, including creation, rotation, and policy enforcement. This simplifies the key management process and enhances the overall security infrastructure.

  4. Seamless Integration: AWS KMS integrates seamlessly with other AWS services, facilitating a cohesive and secure environment for your Kubernetes clusters and the applications running within them.

Considerations and Management Implications

  1. Operational Complexity: Implementing KMS encryption introduces additional steps in the secret management process, from key creation to policy configuration. Organizations must invest time in understanding and managing these complexities to ensure seamless operation.
  2. Performance Impact: The process of encrypting and decrypting secrets can introduce latency, especially in high-throughput environments. While typically minimal, this performance impact should be evaluated and mitigated through proper architectural decisions.
  3. Cost Implications: AWS KMS incurs costs based on the number of cryptographic operations and the management of CMKs. Organizations should monitor usage to manage costs effectively, especially in large-scale deployments.
  4. Access Management: Proper configuration of access policies and roles is critical to secure key management. Misconfigurations can lead to security vulnerabilities, underscoring the need for meticulous policy management and regular audits.
  5. Dependency on AWS: Utilizing AWS KMS ties your encryption strategy to the AWS ecosystem. While this integration offers numerous benefits, it also introduces a dependency on AWS services, which may be a consideration for organizations pursuing a multi-cloud strategy.


Exploring the security of Kubernetes secrets within Amazon EKS clusters highlights the critical role of encryption. The basic approach of encoding secrets in Base64, which preserves data integrity, is insufficient for protecting sensitive information against unauthorized access. AWS Key Management Service (KMS) addresses this gap by encrypting secrets using robust cryptographic keys, enhancing data security in line with organizational policies and regulatory demands.

Integrating AWS KMS with Amazon EKS is a key step in bolstering the security framework of your Kubernetes setup. You can choose to establish a new Customer Master Key (CMK) or use an existing one, facilitating a seamless encryption and decryption process for Kubernetes secrets. This approach not only protects essential data such as passwords, tokens, and certificates but also adheres to the highest security standards for application operations.

However, it's crucial to note that this article focuses on securing secrets at rest. While AWS KMS is very effective at encrypting stored data, a comprehensive security strategy includes more than encryption alone. It extends to network security, access control, and applying the principle of least privilege. Properly configuring Kubernetes' role-based access control (RBAC) is essential to ensure that only authorized entities have access to these secrets, enhancing the security measures introduced by AWS KMS.

In summary, integrating AWS KMS with Amazon EKS is a significant advancement in securing Kubernetes secrets. This integration not only strengthens protections for sensitive data at rest but also adheres to cloud-native application security best practices. However, this technology implementation brings challenges such as increased operational complexity, potential impacts on performance, cost considerations, and reliance on AWS services. Effective navigation of these factors through strategic planning is vital to fully benefit from AWS KMS in securing your Amazon EKS environment and maintaining a strong defense against threats and vulnerabilities.