Skip to content

Kubernetes Secrets Management Implementations

Security Comparisons

The below table is providing a holistic overview of the different Secret Management implementations and what attack risks they mitigate.

Legend

  • used when does not mitigate the related component
  • used when mitigate has caveat(s)
  • used when mitigation has no caveat
  • Disk () ; if control plane node disk is compromised online (access to the node) or offline (access to a copy/backup)
  • RAM () ; if the memory allocation is compromised online (via a process like the API Server) or offline (memory dump from backup)
  • etcd (); if the etcd data store is compromised (access via etcdctl) or offline (access to a copy/backup)
  • KMS server (); if Key Management Service server is compromised
Implementation
File System Encryption
Native Kubernetes Secrets
External Secrets Operator
Encryption at rest [1]
KMS Provider with Plugin for external KMS [2] [3]

Notes

  • [1] including offline attack if encryption key is not recovered
  • [2] only if KMS Provider Plugin is configured with no caching otherwise DEKs will be in memory and in clear text
  • [3] if the KMS is compromised, the DEKs can be decrypted allowing to decrypt the Secrets. Mitigation would be required to lockdown the platform.

Implementations

File System Encryption

This is referring to encrypt the file system on which the etcd database would deployed and running. Please refer to your Linux and Kubernetes distribution for specifics and how-to (e.g. Linux Encrypted Filesystem with dm-crypt or How To Linux Hard Disk Encryption With LUKS).

Native Kubernetes Secrets

Kubernetes is using a distributed key-value data store to record all API Objects definition along with their state and version. To ensure proper processing, data is encoded in base64 removing challenges with special characters.

Same concept applies to Secret, especially leveraging special characters for extra security, for example:

mysecret.yml
--- 
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: myapp
type: Opaque
data:
  USER_NAME: YWRtaW4K
  PASSWORD: cEBzc3cwcmQkCg==

The above base64 encoded values are admin and p@ssw0rd$. When creating the Secret with kubectl apply -f mysecret.yml, the following flow will be triggered: 

sequenceDiagram
participant User
box control plane
participant etcd
participant kube-apiserver
end
autonumber
  User->>kube-apiserver: create Secret
  Note right of User: base64 encoded sensitive data
  kube-apiserver->>etcd: store Secret

External Secrets Operator

External Secrets Operator (or ESO) project allows to recover secrets from a KMS server and inject them as native Secrets within Kubernetes, or in other words, base64 encoded and not encrypted.

ESO uses two custom resource API objects: SecretStore and ExternalSecret.

The SecretStore provides access to a remote KMS server:

secretstore.yml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

The ExternalSecret identifies the secrets to fetch from an existing SecretStore and inject them as native Kubernetes Secrets:

secretstore.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: secret-to-be-created
    creationPolicy: Owner
  data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: provider-key
      version: provider-key-version
      property: provider-key-property
  dataFrom:
  - extract:
      key: remote-key-in-the-provider

sequenceDiagram
participant User or App
participant etcd
participant API Server
participant ESO
participant KMS Server
autonumber
  User or App->>API Server: request Secret with ExternalSecret
  API Server->>ESO: hand off the ExternalSecret 
  Note left of ESO: verify if SecretStore exist 
  ESO->>KMS Server: recover Secret
  ESO->>API Server: create Secret
  API Server->>etcd: store Secret

Encryption at rest

The Kubernetes API Server can encrypt the sensitive data from Secrets using the KMS Provider. In this scenario, the API server is set up through the "EncryptionConfiguration" definition that will include a cipher as provider and an encryption key (being encoded in base64).

encryptionconfiguration.yml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE 64 ENCODED SECRET>
      - identity: {}

This will require a restart of the API Server to take effect and the following flow will be triggered:

sequenceDiagram
participant User or App
participant etcd
participant API Server
participant KMS Provider
autonumber
  User or App->>API Server: create Secret
  API Server->>KMS Provider: request to encrypt Secret
  Note left of KMS Provider: encrypt Secret with encryption key
  KMS Provider->>API Server: return encrypted Secret
  API Server->>etcd: store encrypted Secrets

References:

KMS Provider with Plugin for external KMS

The Kubernetes API Server can encrypt the sensitve data from Secrets using the KMS Provider. In this scenario, an external KMS is used to encrypt in-flight the Data Encryption Key using to encrypt the sensitive data. This process is called an encryption envelop scheme.

When creating the Secret with kubectl apply -f mysecret.yml, the following flow will be triggered: 

sequenceDiagram
participant User or App
participant etcd
participant API Server
participant KMS Provider
participant KMS Plugin
participant KMS Server
autonumber
  User or App->>API Server: create Secret
  API Server->>KMS Provider: request to encrypt Secret
  Note left of KMS Provider: generate a DEK
  Note left of KMS Provider: encrypt Secret
  KMS Provider->>KMS Plugin: hand off the DEK for encryption
  KMS Plugin->>KMS Server: encrypt DEK with KEK from KMS
  KMS Plugin->>KMS Provider: return encrypted DEK with KID
  KMS Provider->>API Server: return encrypted DEK with KID
  API Server->>etcd: store Secrets and DEK both encrypted