mirror of
https://github.com/kemko/nomad.git
synced 2026-01-04 17:35:43 +03:00
c12ad24de0
* seo operation section * other specifications section * Update website/content/docs/other-specifications/variables.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --------- Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
100 lines
4.7 KiB
Plaintext
100 lines
4.7 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Key management
|
|
description: |-
|
|
Learn how Nomad manages its keyring, which Nomad uses to encrypt variables, sign task workload identities, and sign OpenID Connect (OIDC) client assertions. Review key rotation, key decryption, and key redaction in Raft snapshots. Learn how Nomad v1.9+ can replicate keys from older Nomad versions.
|
|
---
|
|
|
|
# Key management
|
|
|
|
Nomad servers maintain an encryption keyring used to encrypt [Variables][],
|
|
sign task [workload identities][], and sign OIDC [client assertion JWTs][].
|
|
The servers encrypt these data encryption keys (DEK) and store the wrapped keys
|
|
in Raft.
|
|
|
|
The key encryption key (KEK) used to encrypt the DEK is controlled by the
|
|
[`keyring`][] provider. When using an external KMS or Vault transit encryption
|
|
provider, the KEK is securely stored outside of Nomad. For the default AEAD
|
|
provider, the KEK is stored in cleartext in Raft.
|
|
|
|
<Note>
|
|
|
|
Default AEAD provider users should be aware that storing the Key Encryption Key
|
|
(KEK) in cleartext in Raft may expose data in case of a breach.
|
|
|
|
</Note>
|
|
|
|
Under normal operations the keyring is entirely managed by Nomad, but this
|
|
section provides administrators additional context around key replication and
|
|
recovery.
|
|
|
|
## Key rotation
|
|
|
|
Only one key in the keyring is "active" at any given time, and all encryption
|
|
and signing operations happen on the leader. Nomad automatically rotates the
|
|
active encryption key every 30 days. When a key is rotated, the existing keys
|
|
are marked as "inactive" but not deleted, so they can be used for decrypting
|
|
previously encrypted variables and verifying workload identities for existing
|
|
allocations.
|
|
|
|
If you believe key material has been compromised, you can execute [`nomad
|
|
operator root keyring rotate -full`][]. A new "active" key will be created and
|
|
"inactive" keys will be marked "rekeying". Nomad will asynchronously decrypt and
|
|
re-encrypt all variables with the new key. As each key's variables are encrypted
|
|
with the new key, the old key will marked as "deprecated".
|
|
|
|
## Key decryption
|
|
|
|
When a leader is elected, the leader creates the keyring if it does not already
|
|
exist. When a key is added, the new wrapped key material is replicated via
|
|
Raft. As each server replicates the new key, the server starts a task to decrypt
|
|
the key material. Until this task completes, the server is not able to serve
|
|
requests that require this key.
|
|
|
|
## Key redaction in Raft scenario snapshots
|
|
|
|
The default AEAD `keyring` configuration stores the KEK in Raft. Raft snapshots
|
|
contain the cleartext KEK. The `nomad operator snapshot save` command has a
|
|
`-redact` option that removes the key material when creating a snapshot. The
|
|
`nomad operator snapshot redact` command removes key material from an
|
|
existing snapshot.
|
|
|
|
Redacting key material is not required when using an external KMS.
|
|
|
|
## Legacy keystore
|
|
|
|
Versions of Nomad prior to 1.9.0 stored only key metadata in Raft, but the
|
|
encryption key material was stored in a separate file in the `keystore`
|
|
subdirectory of the Nomad [data directory][]. These files have the extension
|
|
`.nks.json`. The key material in each file is wrapped in a unique key encryption
|
|
key (KEK) that is not shared between servers.
|
|
|
|
Each server runs a key replication process that watches for changes to the state
|
|
store and fetches the key material from the leader asynchronously, falling
|
|
back to retrieving from other servers in the case where a key is written
|
|
immediately before a leader election. Nomad 1.9.0 and above can replicate keys
|
|
from older servers.
|
|
|
|
However, replicating keys from older servers means that to restore an older cluster from snapshot you need to
|
|
also provide the keystore directory with the `.nks.json` key files on at least
|
|
one server. The `.nks.json` key files are unique per server, but only one
|
|
server's key files are needed to recover the cluster. Operators should continue
|
|
to include these files as part of your organization's backup and recovery
|
|
strategy for the cluster until the cluster is fully upgraded to Nomad 1.9.0 and
|
|
at least one [`root_key_gc_interval`][] has passed.
|
|
|
|
If you are recovering an older Raft snapshot onto a new cluster without running
|
|
workloads, you can skip restoring the keyring and run [`nomad operator root
|
|
keyring rotate`][] once the servers have joined.
|
|
|
|
|
|
|
|
[variables]: /nomad/docs/concepts/variables
|
|
[workload identities]: /nomad/docs/concepts/workload-identity
|
|
[client assertion JWTs]: /nomad/docs/concepts/acl/auth-methods/oidc#client-assertions
|
|
[data directory]: /nomad/docs/configuration#data_dir
|
|
[`keyring`]: /nomad/docs/configuration/keyring
|
|
[`nomad operator root keyring rotate -full`]: /nomad/docs/commands/operator/root/keyring-rotate
|
|
[`nomad operator root keyring rotate`]: /nomad/docs/commands/operator/root/keyring-rotate
|
|
[`root_key_gc_interval`]: /nomad/docs/configuration/server#root_key_gc_interval
|