docs: update service mesh integration docs for transparent proxy (#20251)

Update the service mesh integration docs to explain how Consul needs to be configured for transparent proxy. Update the walkthrough to assume that `transparent_proxy` mode is the best approach, and move the manually-configured `upstreams` to a separate section for users who don't want to use Consul DNS. Ref: https://github.com/hashicorp/nomad/pull/20175 Ref: https://github.com/hashicorp/nomad/pull/20241
2026-01-01 16:05:42 +03:00 · 2024-04-03 16:22:11 -04:00
parent 76009d89af
commit bb062deadc
2 changed files with 139 additions and 17 deletions
--- a/website/content/docs/integrations/consul/service-mesh.mdx
+++ b/website/content/docs/integrations/consul/service-mesh.mdx
@@ -134,6 +134,43 @@ service_prefix "" { policy = "read" }
 node_prefix    "" { policy = "read" }
 ```

+#### Transparent Proxy
+
+Using Nomad's support for [transparent proxy][] configures the network namespace
+so that traffic flows through the Envoy proxy. When the [`transparent_proxy`][]
+block is enabled:
+
+* Nomad will invoke the [`consul-cni`][] CNI plugin to configure `iptables` rules
+  in the network namespace to force outbound traffic from an allocation to flow
+  through the proxy.
+* If the local Consul agent is serving DNS, Nomad will set the IP address of the
+  Consul agent as the nameserver in the task's `/etc/resolv.conf`.
+* Consul will provide a [virtual IP][] for any upstream service the workload
+  has access to, based on the service intentions.
+
+Using transparent proxy has several important requirements:
+
+* You must have the [`consul-cni`][] CNI plugin installed on the client host
+  along with the usual [required CNI plugins][cni_plugins].
+* To use Consul DNS and virtual IPs, you'll need to configure Consul's DNS
+  listener to be exposed to the workload network namespace. You can do this
+  without exposing the Consul agent on a public IP by setting the Consul
+  `bind_addr` to bind on a private IP address (the default is to use the
+  `client_addr`).
+* The Consul agent must be configured with [`recursors`][] if you want
+  allocations to make DNS queries for applications outside the service mesh.
+* You cannot set a [`network.dns`][] block on the allocation (unless you set
+  [`no_dns`][tproxy_no_dns], see below).
+
+For example, a HCL configuration with a [go-sockaddr/template][] binding to the
+subnet `10.37.105.0/20`, with recursive DNS set to OpenDNS nameservers:
+
+```hcl
+bind_addr   = "{{ GetPrivateInterfaces | include \"network\" \"10.37.105.0/20\" | limit 1 | attr \"address\" }}"
+
+recursors = ["208.67.222.222", "208.67.220.220"]
+```
+
 ### Nomad

 Nomad must schedule onto a routable interface in order for the proxies to
@@ -150,10 +187,14 @@ Nomad uses CNI reference plugins to configure the network namespace used to secu
 Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces
 must have these CNI plugins [installed][cni_install].

+To use [`transparent_proxy`][] mode, Nomad client nodes will also need the
+[`consul-cni`][] plugin installed.
+
 ## Run the Service Mesh-enabled Services

-Once Nomad and Consul are running, submit the following service mesh-enabled services
-to Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running:
+Once Nomad and Consul are running, with Consul DNS enabled for transparent proxy
+mode as described above, submit the following service mesh-enabled services to
+Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running:
 `nomad job run servicemesh.nomad.hcl`

 ```hcl
@@ -170,7 +211,11 @@ job "countdash" {
      port = "9001"

      connect {
-        sidecar_service {}
+        sidecar_service {
+          proxy {
+            transparent_proxy {}
+          }
+        }
      }
    }

@@ -200,10 +245,7 @@ job "countdash" {
      connect {
        sidecar_service {
          proxy {
-            upstreams {
-              destination_name = "count-api"
-              local_bind_port  = 8080
-            }
+            transparent_proxy {}
          }
        }
      }
@@ -213,7 +255,7 @@ job "countdash" {
      driver = "docker"

      env {
-        COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
+        COUNTING_SERVICE_URL = "http://count-api.virtual.consul"
      }

      config {
@@ -240,8 +282,10 @@ The API service is defined as a task group with a bridge network:
  }
 ```

-Since the API service is only accessible via Consul service mesh, it does not define
-any ports in its network. The service block enables service mesh.
+Since the API service is only accessible via Consul service mesh, it does not
+define any ports in its network. The `connect` block enables the service mesh
+and the `transparent_proxy` block ensures that the service will be reachable via
+a virtual IP address when used with Consul DNS.

 ```hcl
  group "api" {
@@ -253,7 +297,11 @@ any ports in its network. The service block enables service mesh.
      port = "9001"

      connect {
-        sidecar_service {}
+        sidecar_service {
+          proxy {
+            transparent_proxy {}
+          }
+        }
      }
    }

@@ -299,6 +347,66 @@ This allows you to connect to the web frontend in a browser by visiting

 The web frontend connects to the API service via Consul service mesh.

+```hcl
+    service {
+      name = "count-dashboard"
+      port = "http"
+
+      connect {
+        sidecar_service {
+          proxy {
+            transparent_proxy {}
+          }
+        }
+      }
+    }
+```
+
+The `connect` block with `transparent_proxy` configures the web frontend's
+network namespace to route all access to the `count-api` service through the
+Envoy proxy.
+
+The web frontend is configured to communicate with the API service with an
+environment variable `$COUNTING_SERVICE_URL`:
+
+```hcl
+      env {
+        COUNTING_SERVICE_URL = "http://count-api.virtual.consul"
+      }
+```
+
+The `transparent_proxy` block ensures that DNS queries are made to Consul so
+that the `count-api.virtual.consul` name resolves to a virtual IP address. Note
+that you don't need to specify a port number because the virtual IP will only be
+directed to the correct service port.
+
+### Manually Configured Upstreams
+
+If you don't want to use Consul DNS and `transparent_proxy` mode, you can add
+`upstream` blocks to the job spec. In that case, you don't need the
+`transparent_proxy` block for the `count-api` service:
+
+```hcl
+  group "api" {
+
+    # ...
+
+    service {
+      name = "count-api"
+      port = "9001"
+
+      connect {
+        sidecar_service {}
+      }
+    }
+
+    # ...
+
+  }
+```
+
+But you'll need to add an `upstreams` block to the `count-dashboard` service:
+
 ```hcl
    service {
      name = "count-dashboard"
@@ -320,8 +428,8 @@ The web frontend connects to the API service via Consul service mesh.
 The `upstreams` block defines the remote service to access (`count-api`) and
 what port to expose that service on inside the network namespace (`8080`).

-The web frontend is configured to communicate with the API service with an
-environment variable:
+The web frontend will also need to use an environment variable to communicate
+with the API service:

 ```hcl
      env {
@@ -329,10 +437,9 @@ environment variable:
      }
 ```

-The web frontend is configured via the `$COUNTING_SERVICE_URL`, so you must
-interpolate the upstream's address into that environment variable. Note that
-dashes (`-`) are converted to underscores (`_`) in environment variables so
-`count-api` becomes `count_api`.
+This environment variable value gets interpolated with the upstream's
+address. Note that dashes (`-`) are converted to underscores (`_`) in
+environment variables so `count-api` becomes `count_api`.

 ## Limitations

@@ -377,3 +484,13 @@ filesystem.
 [consul_ports]: /consul/docs/agent/config/config-files#ports
 [consul_grpc_tls]: /consul/docs/upgrading/upgrade-specific#changes-to-grpc-tls-configuration
 [cni_install]: /nomad/docs/install#post-installation-steps
+[transparent proxy]: /consul/docs/k8s/connect/transparent-proxy
+[go-sockaddr/template]: https://godoc.org/github.com/hashicorp/go-sockaddr/template
+[`recursors`]: /consul/docs/agent/config/config-files#recursors
+[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy
+[tproxy_no_dns]: /nomad/docs/job-specification/transparent_proxy#no_dns
+[`consul-cni`]: https://releases.hashicorp.com/consul-cni
+[virtual IP]: /consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups
+[cni_plugins]: /nomad/docs/networking/cni#cni-reference-plugins
+[consul_dns_port]: /consul/docs/agent/config/config-files#dns_port
+[`network.dns`]: /nomad/docs/job-specification/network#dns-parameters
--- a/website/content/docs/networking/cni.mdx
+++ b/website/content/docs/networking/cni.mdx
@@ -31,6 +31,9 @@ with Consul service mesh.

 See the Linux [post-install steps][cni_install] for installing CNI reference plugins.

+To use Nomad's [`transparent_proxy`][] feature, you will also need the
+[`consul-cni`][] plugin.
+
 ## CNI plugins

 Spec-compliant plugins should work with Nomad, however, it's possible a plugin
@@ -225,3 +228,5 @@ a unique value for your configuration.
 [loopback]: https://github.com/containernetworking/plugins#main-interface-creating
 [nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps
 [portmap]: https://www.cni.dev/plugins/current/meta/portmap/
+[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy
+[`consul-cni`]: https://releases.hashicorp.com/consul-cni