Introduction

By default, Kubernetes secrets are encoded but not encrypted at rest, meaning that if an unauthorized person gains access to the underlying storage layer, they could potentially decode these secrets. Encrypting these secrets with AWS KMS mitigates this risk by ensuring that the secrets are not only encoded but also encrypted with a strong cryptographic key managed by AWS KMS. This adds an additional layer of security, protecting sensitive information in compliance with your organization's security policies and various regulatory requirements.

When integrating AWS KMS with Amazon EKS for enhanced security, you can either create a new Customer Master Key (CMK) or use an existing one to encrypt and decrypt Kubernetes secrets. These secrets often contain sensitive information, such as passwords, tokens, or certificates, crucial for the operation of applications deployed on EKS. This is something we will learn how to perform later in this article.

In essence, AWS KMS provides the mechanism for robust encryption management in EKS, enabling secure storage and controlled access to Kubernetes secrets, thereby significantly enhancing the overall security posture of your Kubernetes environment on AWS.

It's important to clarify that the focus of this article on using AWS KMS for encrypting Kubernetes secrets in Amazon EKS is specifically about securing secrets at rest. This distinction is crucial for understanding the scope and limitations of the encryption process.

Clarification on Secrets Management in EKS

This article does not go into the specifics of defining secrets in Kubernetes or the access controls surrounding them. Kubernetes employs a role-based access control (RBAC) system that allows fine-grained permission settings for who can access Kubernetes resources, including secrets. A user or process with the necessary permissions, as defined by RBAC roles, can still retrieve and decode secrets stored in EKS.

The Focus on Encryption at Rest

The objective here is enhancing the security of secrets at rest. "At rest" refers to data that is stored, as opposed to "in transit," which refers to data being transmitted from one place to another. By integrating AWS KMS for encryption, the article addresses the risk of unauthorized access to the underlying storage layer where secrets are persisted.

• Encryption at Rest: This involves encrypting secret data before it is stored in the etcd database of an EKS cluster. AWS KMS provides the encryption keys (Customer Master Keys, or CMKs) for this purpose. Even if an attacker gains physical access to the storage or somehow bypasses other security controls to access the stored data, they would not be able to decrypt the secrets without access to the corresponding CMK in AWS KMS.

• RBAC and Access Control: While encryption at rest significantly improves the security posture of your Kubernetes environment, managing access to secrets via Kubernetes' RBAC is equally important. Proper configuration of RBAC roles and bindings is necessary to ensure that only authorized users and applications can retrieve and decode secrets, regardless of the encryption at rest.

It's crucial to recognize that encryption at rest, as facilitated by AWS KMS, is part of a broader security strategy that includes network security, access control, and the principle of least privilege. While AWS KMS effectively protects secrets at rest, securing access to those secrets in use and transit involves additional layers of security, including proper RBAC configuration and adherence to best practices in Kubernetes secrets management. This article focuses on the encryption of secrets at rest, highlighting the role of AWS KMS in enhancing the security of stored secrets in an Amazon EKS environment.

Base64 Encode and Decode Secrets

By default, secrets in Kubernetes are just base64 encoded. This means, the data stored is not encrypted and if someone has access to the secret object, the data can easily be decoded locally on the person's machine. 

The following diagram shows the process of encoding and decoding the data stored in a secret:

The main purpose of Base64 is to transform data to keep the integrity of it, but at the same time the data itself can be viewed by the one that has acess to the secret , since the algorithm to decode the secret data is publically known.

In the following example, we'll create  a secret from literal:

kubectl create secret generic my-credentials --from-literal=password=supersecretpassword

kubectl get secrets/my-credentials --template={{.data.password}}

c3VwZXJzZWNyZXRwYXNzd29yZA==

kubectl get secrets/my-credentials --template={{.data.password}} | base64 --decode

supersecretpassword

❗To be able to decode a secret data, the user has to have permissions to get the secret object in Kubernetes.

 It's important that the RBAC is properly configured in your cluster so only those that needs to work with secrets least privileged access to the secret object.

Encryption

In the realm of digital security, encrypting data is a cornerstone practice, securing information through a method akin to locking it in a vault. This process relies on sophisticated algorithms, which transform readable data into a secured format that can only be deciphered with the correct encryption key. Imagine a complex puzzle that can only be solved with a unique key; similarly, encryption algorithms such as AES, RSA, and Blowfish serve as the puzzle creators, intricately coding data to protect its confidentiality.

Only entities possessing the corresponding decryption key can unlock this data, ensuring that sensitive information remains accessible exclusively to authorized users. This selective accessibility is crucial in safeguarding data across various digital landscapes, from cloud storage to encrypted messaging services.

To visualize this concept, consider the diagram below, which mirrors the encryption journey we've outlined. It depicts not just the locked vault (encrypted data) but also the key's dual role in sealing and unlocking the data's secrecy:

AWS KMS

From the official doc:

The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd. The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption. The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS. The KMS provider uses gRPC to communicate with a specific KMS plugin. The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s) as the Kubernetes control plane, is responsible for all communication with the remote KMS.

AWS Key Management Service (KMS) offers a fully managed encryption service that seamlessly integrates with Amazon Elastic Kubernetes Service (EKS) and other AWS services. Its primary function within the EKS ecosystem is to facilitate the envelope encryption of Kubernetes secrets. These secrets, which might include passwords, Docker registry credentials, and TLS keys, are securely managed using the Kubernetes API and stored within the etcd key-value store.

Envelop encryption is a method where data is encrypted with a Data Encryption Key (DEK), and then the DEK itself is encrypted with a Customer Master Key (CMK). AWS KMS leverages this approach to ensure the highest security standards for your secrets. Moreover, AWS KMS employs Hardware Security Modules (HSMs), dedicated cryptographic devices designed to handle key management and encryption tasks. These devices ensure that the CMKs created are securely managed and remain non-exportable, enhancing the security posture of your Kubernetes applications.

In the context of Amazon EKS, etcd volumes, where Kubernetes secrets are stored, benefit from an additional layer of security. AWS KMS encrypts these volumes at the disk level, ensuring that all data at rest is protected. This disk-level encryption, managed by AWS, adds an essential security layer, safeguarding your sensitive information from unauthorized access.

By integrating AWS KMS with Amazon EKS, you unlock robust security capabilities for your Kubernetes secrets, ensuring they are encrypted, securely managed, and protected across their lifecycle. This integration not only strengthens your overall security strategy but also aligns with best practices for cloud-native applications

Secret Flow Encryption in EKS cluster

The following diagram shows the flow when the secret encryption has been enabled:

1. User or Workflow Creates a Secret: A user or automated workflow initiates the creation of a Kubernetes secret in the EKS cluster.

2. kube-api Generates a Data Encryption Key (DEK): The Kubernetes API server (kube-api) generates a new DEK for encrypting the secret's data. This DEK is generated for each secret at the time of creation or update, ensuring that the encryption is unique for each secret. The DEK is temporarily held in memory and not stored on disk.

3. kube-api Calls KMS to Encrypt the DEK: The kube-api server uses AWS KMS to encrypt the DEK, calling the KMS Encrypt API, passing the plaintext DEK and specifying the pre-existing Customer Master Key (CMK) for encryption.

4. Pre-existing CMK Utilized for Encryption: The Customer Master Key (CMK) is not created during this process; it is pre-selected or created by an administrator before this encryption process starts and is used by AWS KMS to encrypt the DEK. The CMK itself is not generated anew for each encryption operation; instead, it's the DEK that's newly generated and then encrypted with the already existing CMK.

5. Encrypted DEK and Secret Stored in etcd: Once the DEK is encrypted with the CMK, the encrypted DEK is stored alongside the encrypted secret data in etcd, securing both the secret and the means to decrypt it.

6. Accessing and Decrypting the Secret: When a pod needs to access the secret, the Kubernetes API server retrieves the encrypted secret and the encrypted DEK from etcd, uses AWS KMS to decrypt the DEK with the CMK, and then decrypts the secret's data to pass it to the pod, allowing it to use the secret.

Please note! All etcd volumes used by AWS EKS are encrypted at the disk level by using AWS managed encryption keys. This is turned on by default, you don't need to enable anything to get this behaviour.

❗It is extremly important that the access to the CMKs are limited and that you use least-priviliged model. The calls to KMS are logged in cloudtrail logs.

What is a Customer Master Key

In the context of AWS Key Management Service (KMS), a Customer Master Key (CMK) plays a central role in data encryption and decryption processes. The CMK is a logical representation of a master key that includes metadata, such as the creation date, description, and the key policy. It is the primary resource in AWS KMS and serves several vital functions in the management and control of data encryption across AWS services.

Here's a simplified representation to help illustrate some key aspects of the Customer Master Key:

• AWS KMS: Central service for key management.
• Customer Master Key (CMK): Primary encryption key for DEKs, managed within AWS KMS.
• Data Encryption Key (DEK): Used directly for data encryption, then encrypted and managed by the CMK.
• Key Policy & Access Control: Defines permissions for CMK usage.
• AWS Services Integration: Shows how encrypted DEKs are utilized across AWS services like S3, EBS, and RDS for data encryption.
• AWS CloudTrail: Provides an audit trail for key usage, enhancing security and compliance.

Characteristics of a CMK

• Unique Identifier: Each CMK is uniquely identified by an Amazon Resource Name (ARN), ensuring that it can be precisely referenced in policies and API calls.
• Region-Specific: CMKs are region-specific, meaning they are created and used within the confines of a specific AWS region. This ensures data locality and compliance with geographical regulations.
• Non-Exportable: To maintain a high security standard, CMKs cannot be exported outside of the AWS KMS. This restriction ensures that the keys used for encryption and decryption remain under AWS's managed security infrastructure.
• Hardware Security Module (HSM): CMKs can be generated in AWS-managed HSMs, providing a hardware layer of security that is resistant to tampering and physical theft.

Key Hierarchy and Encryption Process:

• Envelope Encryption: AWS KMS uses an envelope encryption strategy where data is encrypted with a data encryption key (DEK), which, in turn, is encrypted with a CMK. This method provides both security and efficiency for encrypting large amounts of data.
• Data Encryption Key (DEK): For each encryption request, AWS KMS generates a DEK. This key directly encrypts the data, while the CMK encrypts the DEK. The encrypted DEK is stored alongside the encrypted data, ensuring that decrypting the data always involves the CMK.
• Role in AWS Services: CMKs are integrated with other AWS services, such as Amazon S3 for bucket encryption, Amazon EBS for volume encryption, and Amazon RDS for database encryption. This integration allows for a seamless and secure encryption experience across the AWS ecosystem.

Managing Access and Policies:

• Key Policies: Each CMK has an associated key policy that specifies who can use the key and for what actions. These policies enable fine-grained access control, ensuring that only authorized entities can use the key for encryption and decryption.
• Audit and Compliance: Integration with AWS CloudTrail enables logging of all API calls that use a CMK, providing a comprehensive audit trail for compliance and security monitoring.

How To enable KMS encryption in your cluster

AWS Console

You can install your EKS cluster via the AWS console. This is something that I've never done though.

Secret Key Encryption Examples using IaC

AWS CDK

Enhancing the security of your Amazon EKS cluster involves encrypting secrets at rest using a custom AWS KMS key. This process ensures that sensitive information, such as passwords and API keys, is protected.

Below, we outline the steps for both AWS Cloud Development Kit (CDK) and Terraform.

Creating a Custom KMS Key

This establishes a custom KMS key dedicated to securing your EKS cluster secrets. By enabling key rotation, you ensure that the key remains secure over time.

AWS CDK

import * as kms from 'aws-cdk-lib/aws-kms';

const eksSecretsEncryptionKey = new kms.Key(this, 'EksSecretsEncryptionKey', {
    description: 'EKS Secrets Encryption',
    enableKeyRotation: true,
});

Terraform

resource "aws_kms_key" "eks_secrets" {
  description             = "Custom KMS Key for EKS secrets encryption"
  enable_key_rotation     = true
}

import * as eks from 'aws-cdk-lib/aws-eks';

const cluster = new eks.Cluster(this, 'MyEksCluster', {
    version: eks.KubernetesVersion.V1_28,
    secretsEncryptionKey: eksSecretsEncryptionKey,
});

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "my-secure-cluster"
  cluster_version = "1.28" 
  subnets         = ["subnet-xxxxxx", "subnet-yyyyyy"] # subnet IDs

  vpc_id = "vpc-xxxxxx"

  cluster_encryption_config = [{
    provider_key_arn = aws_kms_key.eks_secrets.arn
    resources        = ["secrets"]
  }]
}

 const eksSecretsEncryptionKey = new kms.Key(this, 'EksSecretsEncryptionKey', {
    description: 'EKS Secrets',
    enableKeyRotation: true,
    removalPolicy: cdk.RemovalPolicy.RETAIN,
  });