nomad/website/content/docs/secure/acl/consul.mdx

---
layout: docs
page_title: Integrate Consul ACL
description: |-
  Learn how to integrate Consul access control list (ACL) with Nomad workload identity. Learn how to migrate to using Nomad workload identity with Consul. Configure Consul so it can receive, validate, and trust workload identities from Nomad. Create binding and namespace rules. Review considerations for using the JSON Web Key Set (JWKS) URL.
---

# Integrate Consul ACL

The [Consul ACL][consul_acls] system protects the cluster from unauthorized
access. When enabled, both Consul and Nomad must be properly configured in
order for their integrations to work.

Nomad agents must be configured with their own Consul ACL token, and Consul
must be configured to accept workload identities from tasks and services
running in Nomad.

Refer to the following for details on integrating Consul with Nomad:

- [`consul` block in Agent configuration](/nomad/docs/configuration/consul) for
  configuring Nomad and Consul.
- [Consul networking integration guide](/nomad/docs/networking/consul) for
  details on integrating Consul service discovery and service mesh.

## Nomad Agents

Nomad agents need access to Consul in order to register themselves in the
service catalog and discover other Nomad agents via service discovery for
automatic clustering. Nomad clients use Consul tokens from Workload Identity to
register services and checks but need permissions on their own token to
deregister. Nomad servers also create [configuration
entries][consul_config_entry] for Consul Service Mesh, so the specific
permissions vary slightly between Nomad servers and clients. The following
Consul ACL policies represent the minimal permissions Nomad servers and clients
need.

<Tabs>
<Tab heading="Nomad Servers">

```hcl
agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}

acl  = "write"
mesh = "write"
```

</Tab>
<Tab heading="Nomad Clients">

```hcl
agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}
```

</Tab>
</Tabs>

## Nomad Workload Identities

Starting in Nomad 1.7, Nomad clients can use a task or service's [Workload
Identity][nomad_wid] to authenticate to Consul and obtain an ACL token specific
to the service or task. When using Nomad workload identities, you no longer
need to pass in a Consul ACL token to submit a job.

By default, Nomad does not generate workload identities for services, and tasks
only receive an identity that can be used to access data from Nomad itself,
such as for reading [Variables][] from a [`template`][] block. To access
Consul, jobs must have additional workload identities defined as [`identity`][]
blocks.

To avoid having to add these additional identities to every job, you can
configure Nomad servers with the [`consul.service_identity`][] and
[`consul.task_identity`][] agent configuration. Upon job registration, the
Nomad servers update tasks that have a [`consul`][] block and services that use
the Consul service provider with these default identities.

Job specifications that include [`template`][] blocks are not provided with
default identities because Nomad is unable to decipher the contents of the
template data. You must specify the identities required for Consul in the job
specification. Refer to the [Workload Identities for Consul][jobspec_identity_consul]
section of the `identity` block documentation for more information.

You can also specify identities for Consul directly in the job. When provided,
they override the Nomad server configuration. Refer to the [Workload Identities
for Consul][jobspec_identity_consul] section of the `identity` block
documentation for more information.

### Configuring Consul Authentication

You must configure Consul so it can receive, validate, and trust workload
identities from Nomad. Since they are encoded as [JSON Web Tokens (JWTs)][jwt],
you must create a [JWT ACL auth method][consul_jwt_auth_method]. The auth
method is an endpoint that Nomad can use to exchange workload identities for
Consul ACL tokens.

Refer to Consul's [Auth Methods Overview][consul_auth_methods] documentation
for more information.

#### Consul Auth Method

The auth method configuration points to Nomad's [JSON Web Key Set (JWKS)
URL][nomad_jwks_url]. Consul servers call this URL to retrieve the public keys
Nomad uses to sign workload identities. With these keys, Consul is able to
validate their origin and confirm that they were actually created by Nomad.

Nomad cannot recreate Consul tokens that have been deleted. The auth method
configuration should never set the `MaxTokenTTL` field. Consul tokens are
local to the Consul datacenter unless you set `TokenLocality: "global"` in the
auth method. We recommend using local tokens, which is the default. Global tokens
require that the primary Consul datacenter is available when allocations start.

<CodeBlockConfig highlight="2" filename="auth-method.json">

```json
{
  "JWKSURL": "https://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
```

</CodeBlockConfig>

The `JWKSURL` address must be reachable by all Consul servers and should
resolve to multiple Nomad agents to avoid a single point of failure. Both Nomad
servers and clients are able to handle this request.

Refer to the [Important Considerations About the JWKS
URL](#important-considerations-about-the-jwks-url) section for additional
information on how to configure the `JWKSURL` value.

[![Consul Auth Method][img_consul_auth_method]][img_consul_auth_method]

When an allocation that needs access to Consul starts, the Nomad client running
it exchanges the Nomad workload identities for tasks and services for Consul
ACL tokens.

[![Consul JWT Login][img_consul_wid_acl_token]][img_consul_wid_acl_token]

The auth method also defines the list of authorized audience values, which must
have at least one match with the values defined in the Nomad workload identity
[`aud`][] parameter. For security reasons, it is recommended to only define a
single audience value.

<CodeBlockConfig highlight="4" filename="auth-method.json">

```json
{
  "JWKSURL": "http://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
```

</CodeBlockConfig>

Nomad workload identities have a set of [claims][nomad_wid_claims] that can be
referenced as dynamic values in Consul ACL configuration. The auth method
determines which of these claims are made available to the rest of the
configuration.

<CodeBlockConfig highlight="6-9" filename="auth-method.json">

```json
{
  "JWKSURL": "http://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
```

</CodeBlockConfig>

#### Consul Binding Rules

Consul auth methods use [binding rules][consul_binding_rules] to determine the
set of policies applied to the generated ACL token. Nomad workload identities
can be used in Consul for two main purpose: registering services and retrieving
configuration values and service addresses from Consul using `template` blocks
in tasks. Each purpose requires tokens with different permissions so Nomad
requires two binding rules.

The first binding rule associates the Consul ACL token with a [service
identity][consul_si], allowing the token to register and manage the lifecyle of
a given service. This binding rule is only applied to Nomad workload identities
for services because they are the only ones that have a `nomad_service` claim.

```shell-session
$ consul acl binding-rule create \
    -method 'nomad-workloads' \
    -bind-type 'service' \
    -bind-name '${value.nomad_service}' \
    -selector '"nomad_service" in value'
```

The [`-bind-name`][] flag restricts the token to only be able to modify
services with the same name as the one defined in the Nomad workload identity
claim. The [`-selector`][] flag ensures this binding rule only applies to
workload identities for services.

The second binding rule associates the Consul ACL token with an [ACL
role][consul_acl_role], which is a collection of [ACL
policies][consul_acl_policy] that define what the token is authorized to do.
This binding rule is applied to Nomad workload identities for tasks to access
information from Consul. The exact ACL policy rules will depend on the level of
access required by the tasks (typically, accessing service addresses and KV with
[`template`][] blocks).

```shell-session
$ consul acl binding-rule create \
  -method 'nomad-workloads' \
  -bind-type 'role' \
  -bind-name 'nomad-tasks-${value.nomad_namespace}' \
  -selector '"nomad_service" not in value'
```

The `-bind-name` flag defines which role is used for the token. It may
reference claim values from the Nomad workload identity to apply different
roles to different tasks. Similarly to the binding rule for services, the
`-selector` flag ensures this binding rule only applies to workload identities
for tasks since they don't have the `nomad_service` claim.

The overall configuration structure is illustrated in the following diagram.

[![Consul Auth Overview][img_consul_auth]][img_consul_auth]

The [`consul.service_auth_method`][] and [`consul.task_auth_method`][]
configuration define the auth method used by Nomad to retrieve Consul ACL
tokens for services and tasks.

By default, they follow the structure described above and use a single auth
method for both, but it is possible to use two different auth methods, provided
the resulting Consul ACL token from each have the expected service identity and
role applied.

#### Consul Namespace Rules <EnterpriseAlert inline />

Consul Enterprise supports multiple namespaces and Nomad Enterprise allows jobs
to use the [`consul.namespace`][] parameter to register services and read KV
data from different Consul namespaces.

In a multi-namespace environment, you should create the auth method and binding
rules in the `default` namespace and configure the auth method with a set of
[`NamespaceRules`][].

```shell-session
$ consul acl auth-method create \
  -name 'nomad-workloads' \
  -type 'jwt' \
  -config '@auth-method.json' \
  -namespace-rule-selector '"consul_namespace" in value' \
  -namespace-rule-bind-namespace '${value.consul_namespace}'
```

Similarly to binding rules, namespace rules have a [`Selector`][] expression to
determine when the rule should be applied and a [`BindNamespace`][] value that
defines the namespace used.

Auth methods with a namespace rule create Consul tokens in that Consul
namespace. Binding rules with `-bind-type role` also target a role and
associated policies in that same Consul namespace. So you should create the
auth method and binding rules in the default namespace, and the role and
policies in the target namespaces.

In Nomad Enterprise, workload identities for tasks and services placed within
the scope of a `consul` block with a `namespace` value, have an additional
claim called `consul_namespace` that represents the Consul namespace defined
in Nomad for the workload.

<CodeBlockConfig highlight="9-11" filename="example.nomad.hcl">

```hcl
job "example" {
  group "cache" {
    network {
      port "db" {
        to = 6379
      }
    }

    consul {
      namespace = "prod"
    }

    service {
      port     = "db"
      name     = "redis"
      provider = "consul"
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        ports = ["db"]
      }
    }
  }
}
```

</CodeBlockConfig>

If a `consul` block is not defined, then the workload identity will not have
the `consul_namespace` claim, since Nomad is not able to determine which Consul
namespace will be used.

Refer to the [Consul Namespaces][int_consul_ns] section for more information.

### Important Considerations About the JWKS URL

The recommended configuration assumes Consul servers are able to connect to
Nomad agents (either client or servers) to retrieve the JSON Web Key Set
information.

This section covers additional aspects you should consider depending on how
your Consul and Nomad clusters are configured and deployed.

#### Mutual TLS in Nomad

It is highly recommended to use [mutual TLS][tutorial_mtls] in production
deployments of Nomad. With mTLS enabled, the [`tls.verify_https_client`][]
configuration must be set to `false` since it is not possible to provide client
certificates to the Consul auth method.

Alternatively, you may expose Nomad's JWKS URL from a proxy or a load balancer
that handles the mutual TLS connection to Nomad and exposes the JWKS URL
endpoint over standard TLS.

#### Consul Servers Not Able to Connect to Nomad

If the Consul servers are not able to reach Nomad's JWKS URL, you may read the
public keys from Nomad's [`/.well-known/jwks.json`][nomad_jwks_url] endpoint
and provide them to the auth method directly using the
[`JWTValidationPubKeys`][] parameter. The keys must be converted from JWKS to
PEM format.

You may also host the JWKS JSON response from Nomad in an external location
that is reachable by the Consul servers, and use that address as the value for
`JWKSURL`.

It is important to remember that the Nomad keys **are rotated periodically**, so
both approaches should be automated and done continually. The rotation frequency
is controlled by the [`server.root_key_rotation_threshold`][] configuration of
the Nomad servers. Keys will be prepublished at half the rotation threshold.

### Additional References

The [Consul ACL with Nomad Workload Identities][consul_tutorial_wid] tutorial
provides guided instructions on how to configure Consul and Nomad for workload
identities.

The [`nomad setup consul`][nomad_cli_setup_consul] command and the
[`hashicorp-modules/nomad-setup/consul`][tf_nomad_setup_consul] Terraform
module can help you automate the process of applying configuration to a Consul
cluster.

## Migrating to Using Workload Identity with Consul

Nomad version 1.7.0 introduced Workload Identity authentication, and deprecated
token authentication.  Token authentication was then removed in Nomad 1.10.0.
Migrating from the legacy (pre-1.7) workflow where workload use the agent's
Consul token requires configuation on your Consul cluster and your Nomad server
agents. It does not require updating your running Nomad jobs. To migrate:

* Create the Consul auth method and binding rules on your Consul cluster.
* Enable [`consul.service_identity`][] blocks in your Nomad server agent configurations.
* Enable [`consul.task_identity`][] blocks in your Nomad server agent configurations.
* (Optionally) add [`identity`][] blocks to your jobs if you want to use a
  different identity because of how your auth method and binding rules are
  configured.

Note that when using Workload Identity you will no longer need to pass in a
Consul token to submit a job.

[Variables]: /nomad/docs/concepts/variables
[`-bind-name`]: /consul/commands/acl/binding-rule/create#bind-name
[`-selector`]: /consul/commands/acl/binding-rule/create#selector
[`BindNamespace`]: /consul/api-docs/acl/auth-methods#bindnamespace
[`JWTValidationPubKeys`]: /consul/docs/security/acl/auth-methods/jwt#jwtvalidationpubkeys
[`NamespaceRules`]: /consul/api-docs/acl/auth-methods#namespacerules
[`Selector`]: /consul/api-docs/acl/auth-methods#selector
[`aud`]: /nomad/docs/job-specification/identity#aud
[`consul.namespace`]: /nomad/docs/job-specification/consul#namespace
[`consul.service_auth_method`]: /nomad/docs/configuration/consul#service_auth_method
[`consul.service_identity`]: /nomad/docs/configuration/consul#service_identity
[`consul.task_auth_method`]: /nomad/docs/configuration/consul#task_auth_method
[`consul.task_identity`]: /nomad/docs/configuration/consul#task_identity
[`consul`]: /nomad/docs/job-specification/consul
[`identity`]: /nomad/docs/job-specification/identity
[`server.root_key_rotation_threshold`]: /nomad/docs/configuration/server#root_key_rotation_threshold
[`template`]: /nomad/docs/job-specification/template
[`tls.verify_https_client`]: /nomad/docs/configuration/tls#verify_https_client
[consul_acl_policy]: /consul/docs/security/acl/acl-policies
[consul_acl_role]: /consul/docs/security/acl/acl-roles
[consul_acls]: /consul/docs/security/acl
[consul_auth_methods]: /consul/docs/security/acl/auth-methods
[consul_binding_rules]: /consul/api-docs/acl/binding-rules
[consul_config_entry]: /consul/docs/connect/config-entries
[consul_jwt_auth_method]: /consul/docs/security/acl/auth-methods/jwt
[consul_si]: /consul/docs/security/acl/acl-roles#service-identities
[consul_tutorial_wid]: /nomad/tutorials/integrate-consul/consul-acl
[img_consul_auth]: /img/consul-integration-wi.png
[img_consul_auth_method]: /img/consul-integration-auth-method.png
[img_consul_wid_acl_token]: /img/consul-integration-wid-acl-token.png
[int_consul_ns]: /nomad/docs/networking/consul#consul-namespaces
[jobspec_identity_consul]: /nomad/docs/job-specification/identity#workload-identities-for-consul
[jwt]: https://jwt.io/
[nomad_cli_setup_consul]: /nomad/commands/setup/consul
[nomad_jwks_url]: /nomad/api-docs/operator/keyring#list-active-public-keys
[nomad_wid]: /nomad/docs/concepts/workload-identity
[nomad_wid_claims]: /nomad/docs/concepts/workload-identity#workload-identity-claims
[tf_nomad_setup_consul]: https://registry.terraform.io/modules/hashicorp-modules/nomad-setup/consul/
[tutorial_mtls]: /nomad/docs/secure/traffic/tls