Files
nomad/website/content/docs/integrations/vault/acl.mdx
Luiz Aoqui 5267eec3ad vault: fix token revocation during workflow migration (#19689)
When transitioning from the legacy token-based workflow to the new JWT
workflow for Vault the previous code would instantiate a no-op Vault if
the server configuration had a `default_identity` block.

This no-op client returned an error for some of its operations were
called, such as `LookupToken` and `RevokeTokens`. The original intention
was that, in the new JWT workflow, none of these methods should be
called, so returning an error could help surface potential bugs.

But the `RevokeTokens` and `MarkForRevocation` methods _are_ called even
in the JWT flow. When a leadership transition happens, the new server
looks for unused Vault accessors from state and tries to revoke them.
Similarly, the `RevokeTokens` method is called every time the
`Node.UpdataStatus` and `Node.UpdateAlloc` RPCs are made by clients, as
the Nomad server tries to find unused Vault tokens for the node/alloc.

Since the new JWT flow does not require Nomad servers to contact Vault,
calling `RevokeTokens` and `MarkForRevocation` is not able to complete
without a Vault token, so this commit changes the logic to use the no-op
Vault client when no token is configured. It also updates the client
itself to not error if these methods are called, but to rather just log
so operators can be made aware that there are Vault tokens created by
Nomad that have not been force-expired.

When migrating an existing cluster to the new workload identity based
flow, Nomad operators must first upgrade the Nomad version without
removing any of the existing Vault configuration. Doing so can prevent
Nomad servers from managing and cleaning-up existing Vault tokens during
a leadership transition and node or alloc updates.

Operators must also resubmit all jobs with a `vault` block so they are
updated with an `identity` for Vault. Skipping this step may cause
allocations to fail if their Vault token expires (if, for example, the
Nomad client stops running for TTL/2) or if they are rescheduled, since
the new client will try to follow the legacy flow which will fail if the
Nomad server configuration for Vault has already been updated to remove
the Vault address and token.
2024-01-10 13:28:46 -05:00

819 lines
30 KiB
Plaintext

---
layout: docs
page_title: Vault ACL
description: Learn how to integrate Nomad with Vault when ACL is enabled.
---
# Vault ACL
The Vault ACL system protects the cluster from unauthorized access. It must be
properly configured in order for the Vault and Nomad integrations to work.
## Nomad Workload Identities
Starting in Nomad 1.7, Nomad clients can use a task's [Workload Identity][] to
authenticate to Vault and obtain a Vault ACL token specific to the task. When
using Nomad workload identities, you no longer need to pass in a Vault ACL
token to submit a job.
By default, Nomad only generates a workload identity for tasks that can be used
to access Nomad itself, such as for reading [Variables][] from a [`template`][]
block. To access Vault, jobs must have additional workload identities defined
as [`identity`][] blocks.
To avoid having to add these additional identities to every job, you can
configure the Nomad servers with the [`vault.default_identity`][] agent
configuration. Upon job registration, the Nomad servers update tasks that have
a [`vault`][] block with this default identity.
You can also specify identities for Vault directly in the job. When provided,
they override the Nomad server configuration. Refer to the [Workload Identities
for Vault][jobspec_identity_vault] section of the `identity` block
documentation for more information.
### Configuring Vault Authentication
Vault must be configured to receive, validate, and trust these Nomad workload
identities. Since they are encoded as [JSON Web Tokens (JWTs)][jwt], you must
create a [JWT ACL auth method][vault_jwt_auth_method]. The auth method is an
endpoint that Nomad can use to exchange workload identities for Vault ACL
tokens.
Refer to Vault's [Authentication][vault_auth] documentation for more
information.
#### Vault Auth Method
The auth method configuration points to Nomad's [JSON Web Key Set (JWKS)
URL][nomad_jwks_url]. Vault servers call this URL to retrieve the public keys
Nomad uses to sign workload identities. With these keys, Vault is able to
validate their origin and confirm that they were actually created by Nomad.
<CodeBlockConfig highlight="2" filename="auth-method.json">
```json
{
"jwks_url": "https://nomad.example.com:4646/.well-known/jwks.json",
"jwt_supported_algs": ["RS256", "EdDSA"],
"default_role": "nomad-workloads"
}
```
</CodeBlockConfig>
The `jwks_url` address must be reachable by all Vault servers and should
resolve to multiple Nomad agents to avoid a single point of failure. Both Nomad
servers and clients are able to handle this request.
Refer to the [Important Considerations About the JWKS
URL](#important-considerations-about-the-jwks-url) section for additional
information on how to configure the `jwks_url` value.
[![Vault Auth Method][img_vault_auth_method]][img_vault_auth_method]
When an allocation that needs access access to Vault starts, the Nomad client
running it exchanges the Nomad workload identities for tasks for Vault ACL
tokens.
[![Vault JWT Login][img_vault_wid_acl_token]][img_vault_wid_acl_token]
#### Vault ACL Role
A Vault [ACL role][vault_role] groups multiple ACL policies to apply to a token
and determine the permissions it receives.
The auth method may define a default ACL role that is applied to the ACL tokens
it generates. If no default role is set, the role must be provided in the job
using the [`vault.role`][] parameter or in the Nomad client configuration
[`vault.create_from_role`][].
<CodeBlockConfig highlight="4" filename="auth-method.json">
```json
{
"jwks_url": "https://nomad.example.com:4646/.well-known/jwks.json",
"jwt_supported_algs": ["RS256", "EdDSA"],
"default_role": "nomad-workloads"
}
```
</CodeBlockConfig>
The ACL role specifies the list of authorized audience values using the
[`bound_audiences`][], which must have at least one match with the values
defined in the Nomad workload identity [`aud`][] parameter. For security
reasons, it is recommended to only define a single audience value.
<CodeBlockConfig highlight="3" filename="acl-role.json">
```json
{
"role_type": "jwt",
"bound_audiences": ["vault.io"],
"bound_claims": {
"nomad_namespace": "default",
"nomad_job_id": "mongo"
},
"user_claim": "/nomad_job_id",
"user_claim_json_pointer": true,
"claim_mappings": {
"nomad_namespace": "nomad_namespace",
"nomad_job_id": "nomad_job_id",
"nomad_task": "nomad_task"
},
"token_type": "service",
"token_policies": ["nomad-workloads"],
"token_period": "30m",
"token_explicit_max_ttl": 0
}
```
</CodeBlockConfig>
Nomad workload identities have a set of [claims][nomad_wid_claims] that can be
referenced in Vault ACL configuration. The ACL role uses the
[`claim_mappings`][] parameter to determine which of these claims are made
available to the rest of the configuration.
The [`bound_claims`][] parameter restricts which workload identities are able
to use the role based on their claims. Refer to Vault's [Bound
Claims][vault_bound_claims] documentation for more information.
<CodeBlockConfig highlight="4-7,10-14" filename="acl-role.json">
```json
{
"role_type": "jwt",
"bound_audiences": ["vault.io"],
"bound_claims": {
"nomad_namespace": "default",
"nomad_job_id": "mongo"
},
"user_claim": "/nomad_job_id",
"user_claim_json_pointer": true,
"claim_mappings": {
"nomad_namespace": "nomad_namespace",
"nomad_job_id": "nomad_job_id",
"nomad_task": "nomad_task"
},
"token_type": "service",
"token_policies": ["nomad-workloads"],
"token_period": "30m",
"token_explicit_max_ttl": 0
}
```
</CodeBlockConfig>
Vault has different [types of ACL tokens][vault_token_types]. Nomad uses tokens
of type `service` since they can be renewed for as long as the workload is
active. Nomad automatically renews the Vault ACL tokens it generates before
they expire. To ensure the tokens can be renewed for as long as necessary,
[`token_explicit_max_ttl`][] must be set to 0.
<CodeBlockConfig highlight="16-18" filename="acl-role.json">
```json
{
"role_type": "jwt",
"bound_audiences": ["vault.io"],
"bound_claims": {
"nomad_namespace": "default",
"nomad_job_id": "mongo"
},
"user_claim": "/nomad_job_id",
"user_claim_json_pointer": true,
"claim_mappings": {
"nomad_namespace": "nomad_namespace",
"nomad_job_id": "nomad_job_id",
"nomad_task": "nomad_task"
},
"token_policies": ["nomad-workloads"],
"token_type": "service",
"token_period": "30m",
"token_explicit_max_ttl": 0
}
```
</CodeBlockConfig>
#### Vault ACL Policy
A Vault ACL role may have one or more ACL policies attached. Vault [ACL
policies][vault_policies] define the permissions granted to an ACL
token.
<CodeBlockConfig highlight="15" filename="acl-role.json">
```json
{
"role_type": "jwt",
"bound_audiences": ["vault.io"],
"bound_claims": {
"nomad_namespace": "default",
"nomad_job_id": "mongo"
},
"user_claim": "/nomad_job_id",
"user_claim_json_pointer": true,
"claim_mappings": {
"nomad_namespace": "nomad_namespace",
"nomad_job_id": "nomad_job_id",
"nomad_task": "nomad_task"
},
"token_policies": ["nomad-workloads"],
"token_type": "service",
"token_period": "30m",
"token_explicit_max_ttl": 0
}
```
</CodeBlockConfig>
ACL policies can reference dynamic values from Nomad workload identities claims
exposed from the ACL role in [templated policies][vault_templated_policies].
The exact ACL policy rules will depend on the level of access required by
tasks.
The following example ACL policy automatically grants `read` permissions to
secrets in the path `secret/data/<job namespace>/<job name>/*`, where `<job
namespace>` and `<job name>` are read from the workload identity claims
`nomad_namespace` and `nomad_job_id`.
<CodeBlockConfig highlight="1,5,9" filename="acl-policy.hcl">
```hcl
path "secret/data/{{identity.entity.aliases.auth_jwt_d34481ad.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_d34481ad.metadata.nomad_job_id}}/*" {
capabilities = ["read"]
}
path "secret/data/{{identity.entity.aliases.auth_jwt_d34481ad.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_d34481ad.metadata.nomad_job_id}}" {
capabilities = ["read"]
}
path "secret/metadata/{{identity.entity.aliases.auth_jwt_d34481ad.metadata.nomad_namespace}}/*" {
capabilities = ["list"]
}
path "secret/metadata/*" {
capabilities = ["list"]
}
```
</CodeBlockConfig>
The overall configuration structure is illustrated in the following diagram.
[![Vault Auth Overview][img_vault_auth_overview]][img_vault_auth_overview]
#### Vault Namespaces <EnterpriseAlert inline />
Vault Enterprise supports multiple namespaces and jobs in Nomad Enterprise can
use the [`vault.namespace`][] parameter to specify which namespace to use. In a
multi-namespace environment, the authentication setup described must be applied
to each Vault namespace used by jobs.
### Important Considerations About the JWKS URL
The recommended configuration assumes Vault servers are able to connect to
Nomad agents (either client or servers) to retrieve the JSON Web Key Set
information.
This section covers additional aspects you should consider depending on how
your Vault and Nomad clusters are configured and deployed.
#### Mutual TLS in Nomad
It is highly recommended to use [mutual TLS][tutorial_mtls] in production
deployments of Nomad. With mTLS enabled, the [`tls.verify_https_client`][]
configuration must be set to `false` since it is not possible to provide client
certificates to the Vault auth method.
Alternatively, you may expose Nomad's JWKS URL from a proxy or a load balancer
that handles the mutual TLS connection to Nomad and exposes the JWKS URL
endpoint over standard TLS.
#### Vault Servers Not Able to Connect to Nomad
If the Vault servers are not able to reach Nomad's JWKS URL, you may read the
public keys from Nomad's [`/.well-known/jwks.json`][nomad_jwks_url] endpoint
and provide them to the auth method directly using the
[`jwt_validation_pubkeys`][] parameter. The keys must be converted from JWKS to
PEM format.
You may also host the JWKS JSON response from Nomad in an external location
that is reachable by the Vault servers, and use that address as the value for
`jwks_url`.
It is important to remember that the Nomad keys **are rotated periodically**,
so both approaches should be automated and done continually. The rotation
frequency is controlled by the [`server.root_key_rotation_threshold`][]
configuration of the Nomad servers.
### Additional References
The [Vault ACL with Nomad Workload Identities][vault_tutorial_wid] tutorial
provides guided instructions on how to configure Vault and Nomad for workload
identities.
The [`nomad setup vault`][nomad_cli_setup_vault] command and the
[`hashicorp-modules/nomad-setup/vault`][tf_nomad_setup_vault] Terraform
module can help you automate the process of applying configuration to a Vault
cluster.
## Authentication Without Workload Identity (Legacy)
To use the legacy Vault integration, Nomad servers must be provided a Vault
token. This token can either be a root token or a periodic token with
permissions to create from a token role. The root token is the easiest way to
get started, but we recommend a token role based token for production
installations. Nomad servers will renew the token automatically. **Note that the
Nomad clients do not need to be provided with a Vault token.**
<Note title="Deprecation Warning">
This legacy workflow will be removed in Nomad 1.9. Before upgrading to Nomad 1.9
you will need to have configured authentication with Vault as described in
[Nomad Workload Identities](#nomad-workload-identities).
</Note>
See the [Enterprise specific section][ent] for configuring Vault Enterprise.
### Root Token Integration
If Nomad is given a [root token](/vault/docs/concepts/tokens#root-tokens), no
further configuration is needed as Nomad can derive a token for jobs using any
Vault policies. Best practices recommend using a periodic token with the minimal
permissions necessary instead of providing Nomad the root vault token.
### Token Role based Integration
Vault's [Token Authentication Backend][auth] supports a concept called "roles".
Token roles allow policies to be grouped together and token creation to be
delegated to a trusted service such as Nomad. By creating a token role, the set
of policies that tasks managed by Nomad can access may be limited compared to
giving Nomad a root token. Token roles allow both allowlist and denylist
management of policies accessible to the role.
To configure Nomad and Vault to create tokens against a role, the following must
occur:
1. Create a "nomad-server" policy used by Nomad to create and manage tokens.
2. Create a Vault token role with the configuration described below.
3. Configure Nomad to use the created token role.
4. Give Nomad servers a periodic token with the "nomad-server" policy created
above.
#### Required Vault Policies
The token Nomad receives must have the capabilities listed below. An explanation
for the use of each capability is given.
```hcl
# Allow creating tokens under "nomad-cluster" token role. The token role name
# should be updated if "nomad-cluster" is not used.
path "auth/token/create/nomad-cluster" {
capabilities = ["update"]
}
# Allow looking up "nomad-cluster" token role. The token role name should be
# updated if "nomad-cluster" is not used.
path "auth/token/roles/nomad-cluster" {
capabilities = ["read"]
}
# Allow looking up the token passed to Nomad to validate # the token has the
# proper capabilities. This is provided by the "default" policy.
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow looking up incoming tokens to validate they have permissions to access
# the tokens they are requesting. This is only required if
# `allow_unauthenticated` is set to false.
path "auth/token/lookup" {
capabilities = ["update"]
}
# Allow revoking tokens that should no longer exist. This allows revoking
# tokens for dead tasks.
path "auth/token/revoke-accessor" {
capabilities = ["update"]
}
# Allow checking the capabilities of our own token. This is used to validate the
# token upon startup. Note this requires update permissions because the Vault API
# is a POST
path "sys/capabilities-self" {
capabilities = ["update"]
}
# Allow our own token to be renewed.
path "auth/token/renew-self" {
capabilities = ["update"]
}
```
The above [`nomad-server` policy](https://nomadproject.io/data/vault/nomad-server-policy.hcl) is
available for download. Below is an example of writing this policy to Vault:
```shell-session
# Download the policy
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
```
#### Vault Token Role Configuration
A Vault token role must be created for use by Nomad. The token role can be used
to manage what Vault policies are accessible by jobs submitted to Nomad. The
policies can be managed as a allowlist by using `allowed_policies` in the token
role definition or as a denylist by using `disallowed_policies`.
If using `allowed_policies`, tasks may only request Vault policies that are in
the list. If `disallowed_policies` is used, task may request any policy that is
not in the `disallowed_policies` list. There are trade-offs to both approaches
but generally it is easier to use the denylist approach and add policies that
you would not like tasks to have access to into the `disallowed_policies` list.
An example token role definition is given below:
```json
{
"disallowed_policies": "nomad-server",
"token_explicit_max_ttl": 0,
"name": "nomad-cluster",
"orphan": true,
"token_period": 259200,
"renewable": true
}
```
##### Token Role Requirements
Nomad checks that token role has an appropriate configuration for use by the
cluster. Fields that are checked are documented below as well as descriptions of
the important fields. See Vault's [Token Authentication Backend][auth]
documentation for all possible fields and more complete documentation.
- `allowed_policies` - Specifies the list of allowed policies as a
comma-separated string. This list should contain all policies that jobs running
under Nomad should have access to.
- `disallowed_policies` - Specifies the list of disallowed policies as a
comma-separated string. This list should contain all policies that jobs running
under Nomad should **not** have access to. The policy created above that
grants Nomad the ability to generate tokens from the token role should be
included in list of disallowed policies. This prevents tokens created by
Nomad from generating new tokens with different policies than those granted
by Nomad.
A regression occurred in Vault 0.6.4 when validating token creation using a
token role with `disallowed_policies` such that it is not usable with
Nomad. This was remedied in 0.6.5 and does not effect earlier versions
of Vault.
- `token_explicit_max_ttl` - Specifies the max TTL of a token. **Must be set to `0`** to
allow periodic tokens.
- `name` - Specifies the name of the policy. We recommend using the name
`nomad-cluster`. If a different name is chosen, replace the token role in the
above policy.
- `orphan` - Specifies whether tokens created against this token role will be
orphaned and have no parents. Nomad does not enforce the value of this field
but understanding the implications of each value is important.
If set to false, all tokens will be revoked when the Vault token given to
Nomad expires. This makes it easy to revoke all tokens generated by Nomad but
forces all Nomad servers to use the same Vault token, even through upgrades of
Nomad servers. If the Vault token that was given to Nomad and used to generate
a tasks token expires, the token used by the task will also be revoked which
is not ideal.
When set to true, the tokens generated for tasks will not be revoked when
Nomad's token is revoked. However Nomad will still revoke tokens when the
allocation is no longer running, minimizing the lifetime of any task's token.
With orphaned enabled, each Nomad server may also use a unique Vault token,
making bootstrapping and upgrading simpler. As such, **setting `orphan = true`
is the recommended setting**.
- `token_period` - Specifies the length the TTL is extended by each renewal in
seconds. It is suggested to set this value on the order of magnitude of 3 days
(259200 seconds) to avoid a large renewal request rate to Vault. **Must be set
to a positive value**.
- `renewable` - Specifies whether created tokens are renewable. **Must be set to
`true`**. This allows Nomad to renew tokens for tasks.
The above [`nomad-cluster` token role](https://nomadproject.io/data/vault/nomad-cluster-role.json) is
available for download. Below is an example of writing this role to Vault:
```shell-session
# Download the token role
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
#### Example Configuration
To make getting started easy, the basic [`nomad-server`
policy](https://nomadproject.io/data/vault/nomad-server-policy.hcl) and
[`nomad-cluster` role](https://nomadproject.io/data/vault/nomad-cluster-role.json) described above are
available for download.
The below example assumes Vault is accessible, unsealed and the operator has
appropriate permissions.
```shell-session
# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
#### Retrieving the Token Role based Token
After the token role is created, a token suitable for the Nomad servers may be
retrieved by issuing the following Vault command:
```shell-session
$ vault token create -policy nomad-server -period 72h -orphan
Key Value
--- -----
token f02f01c2-c0d1-7cb7-6b88-8a14fada58c0
token_accessor 8cb7fcb3-9a4f-6fbf-0efc-83092bb0cb1c
token_duration 259200s
token_renewable true
token_policies [default nomad-server]
```
The `-orphan` flag is included when generating the Nomad server token above to
prevent revocation of the token when its parent expires. Vault typically
creates tokens with a parent-child relationship. When an ancestor token is
revoked, all of its descendant tokens and their associated leases are revoked
as well.
When generating Nomad's Vault token, we need to ensure that revocation of the
parent token does not revoke Nomad's token. To prevent this behavior we
specify the `-orphan` flag when we create the Nomad's Vault token. All
other tokens generated by Nomad for jobs will be generated using the policy
default of `orphan = false`.
More information about creating orphan tokens can be found in
[Vault's Token Hierarchies and Orphan Tokens documentation][tokenhierarchy].
The [`-period` flag](/vault/docs/commands/token/create#period) is required to allow the automatic renewal of the token. If this is left out, a [`vault token renew` command](/vault/docs/commands/token/renew) will need to be run manually to renew the token.
The token can then be set in the server configuration's
[`vault` block][config], as a command-line flag, or via an environment
variable.
```shell-session
$ VAULT_TOKEN=f02f01c2-c0d1-7cb7-6b88-8a14fada58c0 nomad agent -config /path/to/config
```
An example of what may be contained in the configuration is shown below. For
complete documentation please see the [Nomad agent Vault integration][config]
configuration.
```hcl
vault {
enabled = true
ca_path = "/etc/certs/ca"
cert_file = "/var/certs/vault.crt"
key_file = "/var/certs/vault.key"
address = "https://vault.service.consul:8200"
create_from_role = "nomad-cluster"
}
```
### Troubleshooting Legacy Authentication
#### Invalid Vault token
Upon startup, Nomad will attempt to connect to the specified Vault server. Nomad
will lookup the passed token and if the token is from a token role, the token
role will be validated. Nomad will not shutdown if given an invalid Vault token,
but will log the reasons the token is invalid and disable Vault integration.
#### No Secret Exists
Vault has two APIs for secrets, [`v1` and `v2`][vault-secrets-version]. Each version
has different paths, and Nomad does not abstract this for you. As such you will
need to specify the path as reflected by Vault's HTTP API, rather than the path
used in the `vault kv` command.
You can see examples of `v1` and `v2` syntax in the
[template documentation][vault-kv-templates].
## Enterprise Configuration
<EnterpriseAlert />
Nomad Enterprise allows jobs to use multiple [Vault Namespaces][]. There are a
few configuration settings to consider when using this functionality.
### Example Configuration
Below is an example for creating two Namespaces within Vault.
```shell-session
# Create a namespace "engineering" within Vault
$ vault namespace create engineering
# Create a child namespace "frontend" under "engineering"
$ vault namespace create -namespace=engineering frontend
```
### Required Vault Policies
Policies are configured per Vault namespace. We will apply the policy in the
example above to each namespace—engineering and engineering/frontend.
```shell-session
# Create the "nomad-server" policy in the "engineering" namespace
$ vault policy write -namespace=engineering nomad-server nomad-server-policy.hcl
# Create the "nomad-server" policy in the "engineering/frontend" namespace
$ vault policy write -namespace=engineering/frontend nomad-server nomad-server-policy.hcl
```
We will also configure the previously configured `nomad-workloads` role with each
Namespace
```shell-session
# Create the "nomad-cluster" token role in the "engineering" namespace
$ vault write -namespace=engineering /auth/token/roles/nomad-workloads @nomad-workloads-role.json
# Create the "nomad-cluster" token role in the "engineering/frontend" namespace
$ vault write -namespace=engineering/frontend /auth/token/roles/nomad-workloads @nomad-workloads-role.json
```
The [Nomad agent Vault integration][config] configuration supports specifying a
Vault Namespace, but since we will be using multiple it can be left blank. By
default Nomad will interact with Vault's root Namespace, but individual jobs may
specify other Vault Namespaces to use.
```hcl
vault {
enabled = true
ca_path = "/etc/certs/ca"
cert_file = "/var/certs/vault.crt"
key_file = "/var/certs/vault.key"
address = "https://vault.service.consul:8200"
default_identity {
aud = ["vault.io"]
}
}
```
For legacy authentication, the same steps can be taken to inject a Vault token
from the [Retrieving the Token Role based
Token](#retrieving-the-token-role-based-token) steps.
### Submitting a job with a Vault Namespace
The example job file below specifies to use the `engineering` Namespace in
Vault. It will authenticate to Vault using its workload identity with the
`nomad-workloads` Vault role, then read the value at secret/foo and fetch the
value for key `bar`.
```hcl
job "vault" {
group "demo" {
task "task" {
vault {
namespace = "engineering"
role = "nomad-workloads"
}
driver = "raw_exec"
config {
command = "/usr/bin/cat"
args = ["secrets/config.txt"]
}
template {
data = <<EOF
{{ with secret "secret/foo" }}
SOME_VAL={{.Data.bar}}
{{ end }}
EOF
destination = "secrets/config.txt"
}
}
}
}
```
### Submitting a job with a Vault Namespace with Legacy Authentication
For the legacy authentication, because [`allow_unauthenticated`][allow_unauth]
is set to `false` job submitters will need to provide a sufficiently privileged
token when submitting a job. A token that has access to an appropriate policy in
`engineering` namespace is needed:
```shell-session
$ vault token create -policy access-kv -namespace=engineering -period 72h -orphan
Key Value
--- -----
token s.H39hfS7eHSbb1GpkdzOQLTmz.fvuLy
token_accessor VsKtJwaShwtTo1r9nWV9Rlad.fvuLy
token_duration 72h
token_renewable true
token_policies ["access-kv" "default"]
identity_policies []
policies ["access-kv" "default"]
```
The token can then be submitted with our job
```shell-session
$ VAULT_TOKEN=s.H39hfS7eHSbb1GpkdzOQLTmz.fvuLy nomad job run vault.nomad
```
## Migrating to Using Workload Identity with Vault
Migrating from the legacy (pre-1.7) workflow where workloads use the agent's
Vault token requires configuration on your Vault cluster and your Nomad server
agents. It does not require updating your running Nomad jobs unless you wish to
specify a non-default role. To migrate:
* Create the Vault auth method, default role, and policies on your Vault
cluster.
* Enable [`vault.default_identity`][] blocks in your Nomad server agent
configurations, but **do not modify any of the existing Vault
configuration**.
* Upgrade your cluster following the documented [Upgrade
Process][docs_upgrade].
* Resubmit Nomad jobs that need access to Vault to redeploy them with a new
workload identity for Vault.
* (Optionally) Add [`vault.role`][] fields to any Nomad jobs that will not
use the default role.
* (Optionally) add [`identity`][] blocks to your jobs if you want to use a
different identity because of how your auth method and roles are
configured.
* Once all jobs have been resubmitted, you may remove parameters no longer used
by the Nomad server agents from the [`vault`][config] configuration block.
[Variables]: /nomad/docs/concepts/variables
[Vault Namespaces]: /vault/docs/enterprise/namespaces
[Workload Identity]: /nomad/docs/concepts/workload-identity
[`aud`]: /nomad/docs/job-specification/identity#aud
[`bound_audiences`]: /vault/api-docs/auth/jwt#bound_audiences
[`bound_claims`]: /vault/api-docs/auth/jwt#bound_claims
[`claim_mappings`]: /vault/api-docs/auth/jwt#claim_mappings
[`identity`]: /nomad/docs/job-specification/identity
[`jwt_validation_pubkeys`]: /vault/api-docs/auth/jwt#jwt_validation_pubkeys
[`server.root_key_rotation_threshold`]: /nomad/docs/configuration/server#root_key_rotation_threshold
[`template`]: /nomad/docs/job-specification/template
[`tls.verify_https_client`]: /nomad/docs/configuration/tls#verify_https_client
[`token_explicit_max_ttl`]: /vault/api-docs/auth/jwt#token_explicit_max_ttl
[`vault.create_from_role`]: /nomad/docs/configuration/vault#create_from_role
[`vault.default_identity`]: /nomad/docs/configuration/vault#default_identity
[`vault.namespace`]: /nomad/docs/job-specification/vault#namespace
[`vault.role`]: /nomad/docs/job-specification/vault#role
[`vault`]: /nomad/docs/job-specification/vault
[allow_unauth]: /nomad/docs/configuration/vault#allow_unauthenticated
[auth]: /vault/docs/auth/token 'Vault Authentication Backend'
[config]: /nomad/docs/configuration/vault 'Nomad Vault Configuration Block'
[docs_upgrade]: /nomad/docs/upgrade#upgrade-process
[ent]: #enterprise-configuration
[img_vault_auth_method]: /img/vault-integration-auth-method.png
[img_vault_auth_overview]: /img/vault-integration-auth-overview.png
[img_vault_wid_acl_token]: /img/vault-integration-wid-acl-token.png
[jobspec_identity_vault]: /nomad/docs/job-specification/identity#workload-identities-for-vault
[jwt]: https://jwt.io/
[nomad_cli_setup_vault]: /nomad/docs/commands/setup/vault
[nomad_jwks_url]: /nomad/api-docs/operator/keyring#list-active-public-keys
[nomad_wid_claims]: /nomad/docs/concepts/workload-identity#workload-identity-claims
[tf_nomad_setup_vault]: https://registry.terraform.io/modules/hashicorp-modules/nomad-setup/vault
[tokenhierarchy]: /vault/docs/concepts/tokens#token-hierarchies-and-orphan-tokens 'Vault Tokens - Token Hierarchies and Orphan Tokens'
[tutorial_mtls]: /nomad/tutorials/transport-security/security-enable-tls
[vault-kv-templates]: /nomad/docs/job-specification/template#vault-kv-api-v1 'Vault KV API v1'
[vault-secrets-version]: /vault/docs/secrets/kv 'KV Secrets Engine'
[vault_auth]: /vault/docs/concepts/auth
[vault_bound_claims]: /vault/docs/auth/jwt#bound-claims
[vault_jwt_auth_method]: /vault/api-docs/auth/jwt
[vault_policies]: /vault/docs/concepts/policies
[vault_role]: /vault/api-docs/auth/jwt#create-update-role
[vault_templated_policies]: /vault/tutorials/policies/policy-templating
[vault_token_types]: /vault/tutorials/tokens/tokens#token-types
[vault_tutorial_wid]: /nomad/tutorials/integrate-vault/vault-acl