Skip to main content

Secrets Encryption

Secrets Encryption Config

RKE2 supports encrypting secrets at rest, and will do the following automatically:

  • Generate an AES-CBC key
  • Generate an encryption config file with the generated key:
{
"kind": "EncryptionConfiguration",
"apiVersion": "apiserver.config.k8s.io/v1",
"resources": [
{
"resources": [
"secrets"
],
"providers": [
{
"aescbc": {
"keys": [
{
"name": "aescbckey",
"secret": "xxxxxxxxxxxxxxxxxxx"
}
]
}
},
{
"identity": {}
}
]
}
]
}
  • Pass the config to the Kubernetes APIServer as encryption-provider-config

Once enabled any created secret will be encrypted with this key. Note that if you disable encryption then any encrypted secrets will not be readable until you enable encryption again using the same key.

Secrets Encryption Tool

Version Gate

Available as of v1.21.8+rke2r1

RKE2 contains a subcommand secrets-encrypt, which allows administrators to perform the following tasks:

  • Adding new encryption keys
  • Rotating and deleting encryption keys
  • Reencrypting secrets
danger

Failure to follow proper procedure when rotating secrets encryption keys can cause permanent data loss. Creating a snapshot before rotating is recommended. Proceed with caution.

New Encryption Key Rotation

Expiremental Version Gate

Available as of v1.28.1+rke2r1. This new version of the tool utilizes K8s automatic config reloading which is currently in beta. GA is expected in v1.29.0

For older releases, see Encryption Key Rotation Classic

To rotate secrets encryption keys on a single-server cluster:

  1. Rotate secrets encryption keys

    rke2 secrets-encrypt rotate-keys
  2. Wait for reencryption to finish. Watch the server logs, or wait for:

    $ rke2 secrets-encrypt status
    Encryption Status: Enabled
    Current Rotation Stage: reencrypt_finished

Encryption Key Rotation Classic

To rotate secrets encryption keys on a single-node cluster:

  1. Prepare:

    rke2 secrets-encrypt prepare
  2. Restart the kube-apiserver pod:

    # Get the kube-apiserver container ID
    export CONTAINER_RUNTIME_ENDPOINT="unix:///var/run/k3s/containerd/containerd.sock"
    crictl ps --name kube-apiserver
    # Stop the pod
    crictl stop <CONTAINER_ID>
  3. Rotate:

    rke2 secrets-encrypt rotate
  4. Restart the kube-apiserver pod again

  5. Reencrypt:

    rke2 secrets-encrypt reencrypt

Secrets Encryption Status

The secrets-encrypt status subcommand displays information about the current status of secrets encryption on the node.

An example of the command on a single-server node:

$ rke2 secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match

Active Key Type Name
------ -------- ----
* AES-CBC aescbckey

Another example on HA cluster, after rotating the keys, but before restarting the servers:

$ rke2 secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: hash does not match between node-1 and node-2

Active Key Type Name
------ -------- ----
* AES-CBC aescbckey-2021-12-10T22:54:38Z
AES-CBC aescbckey

Details on each section are as follows:

  • Encryption Status: Displayed whether secrets encryption is disabled or enabled on the node
  • Current Rotation Stage: Indicates the current rotation stage on the node.
    Stages are: start, prepare, rotate, reencrypt_request, reencrypt_active, reencrypt_finished
  • Server Encryption Hashes: Useful for HA clusters, this indicates whether all servers are on the same stage with their local files. This can be used to identify whether a restart of servers is required before proceeding to the next stage. In the HA example above, node-1 and node-2 have different hashes, indicating that they currently do not have the same encryption configuration. Restarting the servers will sync up their configuration.
Key TableDescription
ActiveThe * indicates which, if any, of the keys are currently used for secrets encryption. The active key is used by Kubernetes to encrypt any new secrets.
Key TypeRKE2 only supports the AES-CBC key type. Find more info here.
NameName of the encryption key. Default is aescbckey-<DATE_AND_TIME>.