nomad/website/content/docs/integrations/consul/acl.mdx

---
layout: docs
page_title: Consul ACL
description: Learn how to integrate Nomad with Consul when ACL is enabled.
---

# Consul ACL

The [Consul ACL][consul_acls] system protects the cluster from unauthorized
access. When enabled, both Consul and Nomad must be properly configured in
order for their integrations to work.

Nomad agents must be configured with their own Consul ACL token, and Consul
must be configured to accept workload identities from tasks and services
running in Nomad.

## Nomad Agents

Nomad agents need access to Consul in order to register themselves in the
service catalog and discover other Nomad agents via service discovery for
automatic clustering. Nomad clients use Consul tokens from Workload Identity to
register services and checks but need permissions on their own token to
deregister. Nomad servers also create [configuration
entries][consul_config_entry] for Consul Service Mesh, so the specific
permissions vary slightly between Nomad servers and clients. The following
Consul ACL policies represent the minimal permissions Nomad servers and clients
need.

<Tabs>
<Tab heading="Nomad Servers">

```hcl
agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}

acl  = "write"
mesh = "write"
```

</Tab>
<Tab heading="Nomad Clients">

```hcl
agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}
```

</Tab>
</Tabs>

## Nomad Workload Identities

Starting in Nomad 1.7, Nomad clients can use a task or service's [Workload
Identity][nomad_wid] to authenticate to Consul and obtain an ACL token specific
to the service or task. When using Nomad workload identities, you no longer
need to pass in a Consul ACL token to submit a job.

By default, Nomad does not generate workload identities for services, and tasks
only receive an identity that can be used to access data from Nomad itself,
such as for reading [Variables][] from a [`template`][] block. To access
Consul, jobs must have additional workload identities defined as [`identity`][]
blocks.

To avoid having to add these additional identities to every job, you can
configure Nomad servers with the [`consul.service_identity`][] and
[`consul.task_identity`][] agent configuration. Upon job registration, the
Nomad servers update tasks that have a [`consul`][] or [`template`][] block and
services that use the Consul service provider with these default identities.

You can also specify identities for Consul directly in the job. When provided,
they override the Nomad server configuration. Refer to the [Workload Identities
for Consul][jobspec_identity_consul] section of the `identity` block
documentation for more information.

### Configuring Consul Authentication

You must configure Consul so it can receive, validate, and trust workload
identities from Nomad. Since they are encoded as [JSON Web Tokens (JWTs)][jwt],
you must create a [JWT ACL auth method][consul_jwt_auth_method]. The auth
method is an endpoint that Nomad can use to exchange workload identities for
Consul ACL tokens.

Refer to Consul's [Auth Methods Overview][consul_auth_methods] documentation
for more information.

#### Consul Auth Method

The auth method configuration points to Nomad's [JSON Web Key Set (JWKS)
URL][nomad_jwks_url]. Consul servers call this URL to retrieve the public keys
Nomad uses to sign workload identities. With these keys, Consul is able to
validate their origin and confirm that they were actually created by Nomad.

Nomad cannot recreate Consul tokens that have been deleted. The auth method
configuration should never set the `MaxTokenTTL` field. Consul tokens are
local to the Consul datacenter unless you set `TokenLocality: "global"` in the
auth method. We recommend using local tokens, which is the default. Global tokens
require that the primary Consul datacenter is available when allocations start.

<CodeBlockConfig highlight="2" filename="auth-method.json">

```json
{
  "JWKSURL": "https://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
```

</CodeBlockConfig>

The `JWKSURL` address must be reachable by all Consul servers and should
resolve to multiple Nomad agents to avoid a single point of failure. Both Nomad
servers and clients are able to handle this request.

Refer to the [Important Considerations About the JWKS
URL](#important-considerations-about-the-jwks-url) section for additional
information on how to configure the `JWKSURL` value.

[![Consul Auth Method][img_consul_auth_method]][img_consul_auth_method]

When an allocation that needs access to Consul starts, the Nomad client running
it exchanges the Nomad workload identities for tasks and services for Consul
ACL tokens.

[![Consul JWT Login][img_consul_wid_acl_token]][img_consul_wid_acl_token]

The auth method also defines the list of authorized audience values, which must
have at least one match with the values defined in the Nomad workload identity
[`aud`][] parameter. For security reasons, it is recommended to only define a
single audience value.

<CodeBlockConfig highlight="4" filename="auth-method.json">

```json
{
  "JWKSURL": "http://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
```

</CodeBlockConfig>

Nomad workload identities have a set of [claims][nomad_wid_claims] that can be
referenced as dynamic values in Consul ACL configuration. The auth method
determines which of these claims are made available to the rest of the
configuration.

<CodeBlockConfig highlight="6-9" filename="auth-method.json">

```json
{
  "JWKSURL": "http://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
```

</CodeBlockConfig>

#### Consul Binding Rules

Consul auth methods use [binding rules][consul_binding_rules] to determine the
set of policies applied to the generated ACL token. Nomad workload identities
can be used in Consul for two main purpose: registering services and retrieving
configuration values and service addresses from Consul using `template` blocks
in tasks. Each purpose requires tokens with different permissions so Nomad
requires two binding rules.

The first binding rule associates the Consul ACL token with a [service
identity][consul_si], allowing the token to register and manage the lifecyle of
a given service. This binding rule is only applied to Nomad workload identities
for services because they are the only ones that have a `nomad_service` claim.

```shell-session
$ consul acl binding-rule create \
    -method 'nomad-workloads' \
    -bind-type 'service' \
    -bind-name '${value.nomad_service}' \
    -selector '"nomad_service" in value'
```

The [`-bind-name`][] flag restricts the token to only be able to modify
services with the same name as the one defined in the Nomad workload identity
claim. The [`-selector`][] flag ensures this binding rule only applies to
workload identities for services.

The second binding rule associates the Consul ACL token with an [ACL
role][consul_acl_role], which is a collection of [ACL
policies][consul_acl_policy] that define what the token is authorized to do.
This binding rule is applied to Nomad workload identities for tasks to access
information from Consul. The exact ACL policy rules will depend on the level of
access required by the tasks (typically, accessing service addresses and KV with
[`template`][] blocks).

```shell-session
$ consul acl binding-rule create \
  -method 'nomad-workloads' \
  -bind-type 'role' \
  -bind-name 'nomad-tasks-${value.nomad_namespace}' \
  -selector '"nomad_service" not in value'
```

The `-bind-name` flag defines which role is used for the token. It may
reference claim values from the Nomad workload identity to apply different
roles to different tasks. Similarly to the binding rule for services, the
`-selector` flag ensures this binding rule only applies to workload identities
for tasks since they don't have the `nomad_service` claim.

The overall configuration structure is illustrated in the following diagram.

[![Consul Auth Overview][img_consul_auth]][img_consul_auth]

The [`consul.service_auth_method`][] and [`consul.task_auth_method`][]
configuration define the auth method used by Nomad to retrieve Consul ACL
tokens for services and tasks.

By default, they follow the structure described above and use a single auth
method for both, but it is possible to use two different auth methods, provided
the resulting Consul ACL token from each have the expected service identity and
role applied.

#### Consul Namespace Rules <EnterpriseAlert inline />

Consul Enterprise supports multiple namespaces and Nomad Enterprise allows jobs
to use the [`consul.namespace`][] parameter to register services and read KV
data from different Consul namespaces.

In a multi-namespace environment, you should create the auth method and binding
rules in the `default` namespace and configure the auth method with a set of
[`NamespaceRules`][].

```shell-session
$ consul acl auth-method create \
  -name 'nomad-workloads' \
  -type 'jwt' \
  -config '@auth-method.json' \
  -namespace-rule-selector '"consul_namespace" in value' \
  -namespace-rule-bind-namespace '${value.consul_namespace}'
```

Similarly to binding rules, namespace rules have a [`Selector`][] expression to
determine when the rule should be applied and a [`BindNamespace`][] value that
defines the namespace used.

In Nomad Enterprise, workload identities for tasks and services placed within
the scope of a `consul` block with a `namespace` value, have an additional
claim called `consul_namespace` that represents the Consul namespace defined
in Nomad for the workload.

<CodeBlockConfig highlight="9-11" filename="example.nomad.hcl">

```hcl
job "example" {
  group "cache" {
    network {
      port "db" {
        to = 6379
      }
    }

    consul {
      namespace = "prod"
    }

    service {
      port     = "db"
      name     = "redis"
      provider = "consul"
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7"
        ports = ["db"]
      }
    }
  }
}
```

</CodeBlockConfig>

If a `consul` block is not defined, then the workload identity will not have
the `consul_namespace` claim, since Nomad is not able to determine which Consul
namespace will be used.

Refer to the [Consul Namespaces][int_consul_ns] section for more information.

### Important Considerations About the JWKS URL

The recommended configuration assumes Consul servers are able to connect to
Nomad agents (either client or servers) to retrieve the JSON Web Key Set
information.

This section covers additional aspects you should consider depending on how
your Consul and Nomad clusters are configured and deployed.

#### Mutual TLS in Nomad

It is highly recommended to use [mutual TLS][tutorial_mtls] in production
deployments of Nomad. With mTLS enabled, the [`tls.verify_https_client`][]
configuration must be set to `false` since it is not possible to provide client
certificates to the Consul auth method.

Alternatively, you may expose Nomad's JWKS URL from a proxy or a load balancer
that handles the mutual TLS connection to Nomad and exposes the JWKS URL
endpoint over standard TLS.

#### Consul Servers Not Able to Connect to Nomad

If the Consul servers are not able to reach Nomad's JWKS URL, you may read the
public keys from Nomad's [`/.well-known/jwks.json`][nomad_jwks_url] endpoint
and provide them to the auth method directly using the
[`JWTValidationPubKeys`][] parameter. The keys must be converted from JWKS to
PEM format.

You may also host the JWKS JSON response from Nomad in an external location
that is reachable by the Consul servers, and use that address as the value for
`JWKSURL`.

It is important to remember that the Nomad keys **are rotated periodically**, so
both approaches should be automated and done continually. The rotation frequency
is controlled by the [`server.root_key_rotation_threshold`][] configuration of
the Nomad servers. Keys will be prepublished at half the rotation threshold.

### Additional References

The [Consul ACL with Nomad Workload Identities][consul_tutorial_wid] tutorial
provides guided instructions on how to configure Consul and Nomad for workload
identities.

The [`nomad setup consul`][nomad_cli_setup_consul] command and the
[`hashicorp-modules/nomad-setup/consul`][tf_nomad_setup_consul] Terraform
module can help you automate the process of applying configuration to a Consul
cluster.

## Authenticating Without Workload Identity (Legacy)

If [Consul ACLs][consul_acls] are enabled, the [`allow_unauthenticated`][]
configuration parameter will control whether a Consul token will be required
when submitting a job with Consul namespace configured. The provided Consul
token must belong to the correct namespace, and must be backed by a Consul ACL
Policy with sufficient `service:write` and `kv:read` permissions. An example
policy might look like the following.

```hcl
key_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "write"
}
```

<Note title="Deprecation Warning">

This legacy workflow will be removed in Nomad 1.10. Before upgrading to Nomad 1.10,
you need to configure authentication with Consul as described in
[Nomad Workload Identities](#nomad-workload-identities).

</Note>

## Migrating to Using Workload Identity with Consul

Migrating from the legacy (pre-1.7) workflow where workload use the agent's
Consul token requires configuation on your Consul cluster and your Nomad server
agents. It does not require updating your running Nomad jobs. To migrate:

* Create the Consul auth method and binding rules on your Consul cluster.
* Enable [`consul.service_identity`][] blocks in your Nomad server agent configurations.
* Enable [`consul.task_identity`][] blocks in your Nomad server agent configurations.
* (Optionally) add [`identity`][] blocks to your jobs if you want to use a
  different identity because of how your auth method and binding rules are
  configured.

Note that when using Workload Identity you will no longer need to pass in a
Consul token to submit a job.

[Variables]: /nomad/docs/concepts/variables
[`-bind-name`]: /consul/commands/acl/binding-rule/create#bind-name
[`-selector`]: /consul/commands/acl/binding-rule/create#selector
[`BindNamespace`]: /consul/api-docs/acl/auth-methods#bindnamespace
[`JWTValidationPubKeys`]: /consul/docs/security/acl/auth-methods/jwt#jwtvalidationpubkeys
[`NamespaceRules`]: /consul/api-docs/acl/auth-methods#namespacerules
[`Selector`]: /consul/api-docs/acl/auth-methods#selector
[`allow_unauthenticated`]: /nomad/docs/configuration/consul#allow_unauthenticated
[`aud`]: /nomad/docs/job-specification/identity#aud
[`consul.namespace`]: /nomad/docs/job-specification/consul#namespace
[`consul.service_auth_method`]: /nomad/docs/configuration/consul#service_auth_method
[`consul.service_identity`]: /nomad/docs/configuration/consul#service_identity
[`consul.task_auth_method`]: /nomad/docs/configuration/consul#task_auth_method
[`consul.task_identity`]: /nomad/docs/configuration/consul#task_identity
[`consul`]: /nomad/docs/job-specification/consul
[`identity`]: /nomad/docs/job-specification/identity
[`server.root_key_rotation_threshold`]: /nomad/docs/configuration/server#root_key_rotation_threshold
[`template`]: /nomad/docs/job-specification/template
[`tls.verify_https_client`]: /nomad/docs/configuration/tls#verify_https_client
[consul_acl_policy]: /consul/docs/security/acl/acl-policies
[consul_acl_role]: /consul/docs/security/acl/acl-roles
[consul_acls]: /consul/docs/security/acl
[consul_auth_methods]: /consul/docs/security/acl/auth-methods
[consul_binding_rules]: /consul/api-docs/acl/binding-rules
[consul_config_entry]: /consul/docs/connect/config-entries
[consul_jwt_auth_method]: /consul/docs/security/acl/auth-methods/jwt
[consul_si]: /consul/docs/security/acl/acl-roles#service-identities
[consul_tutorial_wid]: /nomad/tutorials/integrate-consul/consul-acl
[img_consul_auth]: /img/consul-integration-wi.png
[img_consul_auth_method]: /img/consul-integration-auth-method.png
[img_consul_wid_acl_token]: /img/consul-integration-wid-acl-token.png
[int_consul_ns]: /nomad/docs/integrations/consul#consul-namespaces
[jobspec_identity_consul]: /nomad/docs/job-specification/identity#workload-identities-for-consul
[jwt]: https://jwt.io/
[nomad_cli_setup_consul]: /nomad/docs/commands/setup/consul
[nomad_jwks_url]: /nomad/api-docs/operator/keyring#list-active-public-keys
[nomad_wid]: /nomad/docs/concepts/workload-identity
[nomad_wid_claims]: /nomad/docs/concepts/workload-identity#workload-identity-claims
[tf_nomad_setup_consul]: https://registry.terraform.io/modules/hashicorp-modules/nomad-setup/consul/
[tutorial_mtls]: /nomad/tutorials/transport-security/security-enable-tls