kemko/nomad
1
0
Fork 0
mirror of https://github.com/kemko/nomad.git synced 2026-01-04 09:25:46 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
0367b60ca9b06d2f5801ac22e209d7fecd929d37
nomad/website/content/docs/networking/cni.mdx
Daniel Bennett fdd46e6fd3 docs: cni: add tproxy conflist example (#26532)
2025-08-18 12:04:34 -04:00

388 lines
13 KiB
Plaintext
Raw Blame History

---
layout: docs
page_title: Container Network Interface plugins and custom bridge networking
description: |-
Install Container Network Interface (CNI) reference plugins. Learn how to configure Nomad's bridge networking. Use CNI networks with Nomad jobs.
---
# Container Network Interface plugins and custom bridge networking
This page describes the process to install the [Container Network Interface (CNI)
reference plugins][cni-plugin-docs] on your Linux distribution and configure
custom CNI bridge networking on your Nomad clients.
You may apply this guide's workflow to installing any network plugin that
complies with the [CNI Specification][cni-spec], but you should verify plugin
compatibility with Nomad before deploying in production.
## Background
The CNI specification defines a network configuration format for administrators.
The configuration contains directives for both the orchestrator and the plugins
to consume. Nomad implements custom networking through a combination of CNI
reference plugin binaries and CNI configuration files. Networking features, like
bridge network mode and Consul service mesh, leverage the CNI reference plugins
to provide an operating-system agnostic interface to configure workload
networking.
Nomad reads the following extensions from the [`cni_config_dir`][] (`/opt/cni/config` by default):
- `.conflist` files load as [network
configurations][cni_spec_net_config] that contain a list of plugin
configurations.
- `.conf` and `.json` files load as individual [plugin
configurations][cni_spec_plugin_config] for a specific network.
## Requirements
- You are familiar with [CNI reference plugins][cni-plugin-docs].
- You are familiar with [how Nomad uses Container Network Interface (CNI) plugins for bridge networking](/nomad/docs/networking#bridge-networking).
- You are running Nomad on Linux.
## CNI plugins and bridge networking workflow
Install and configure CNI plugins on a per-client basis. Nomad consults
the path given in the client's [`cni_path`][] to find CNI plugin executables.
Perform the following on each Nomad client:
1. [Install CNI reference plugins](#install-cni-reference-plugins).
1. [Configure bridge module to route traffic through iptables](#configure-bridge-module-to-route-traffic-through-iptables).
1. [Create a bridge mode configuration](#create-a-cni-bridge-mode-configuration).
1. [Configure Nomad clients](#configure-nomad-clients).
After you configure and restart your Nomad clients, refer to [Use a CNI network
with a job][] for job configuration.
### Install CNI reference plugins
@include 'install/install-cni-plugins.mdx'
### Configure bridge module to route traffic through iptables
@include 'install/bridge-iptables.mdx'
## Create a custom bridge mode configuration with CNI plugins
Nomad itself uses CNI plugins and configuration as the underlying implementation
for the `bridge` network mode, using the loopback, [bridge][], [firewall][], and
[portmap][] CNI reference plugins configured together to create Nomad's bridge
network.
[comment-source-image]:
https://www.figma.com/file/Ne2qaPUlBTmTYer9biCfK9/Networking?node-id=0%3A1&t=BepgOoQ0kb76GwIr-1
[![Visual example of Nomad bridge
network](/img/nomad-bridge-network.png)](/img/nomad-bridge-network.png)
When setting up a bridge network, Nomad uses a configuration template based on
the CNI Specification's [example
configuration](https://www.cni.dev/docs/spec/#example-configuration).
Refer to that documentation for a complete explanation of the fields.
You can use the following templates as a basis for your own CNI-based bridge
network configuration in cases where you need access to an unsupported option
in the default configuration.
### Configuration files
<Tabs>
<Tab heading="Default">
This example uses two default values from Nomad client configuration.
- The default value for
[`bridge_network_name`](/nomad/docs/configuration/client#bridge_network_name)
is the value for the bridge plugin name.
- The default value for bridge subnet [`bridge_network_subnet`](
/nomad/docs/configuration/client#bridge_network_subnet) is the bridge plugin
subnet.
The `NOMAD-ADMIN` internal constant provides the value for
`iptablesAdminChainName`. In your own configuration, ensure that you change the
`iptablesAdminChainName` to a unique value.
<CodeBlockConfig highlight="10,18,29">
```json
{
"cniVersion": "1.0.0",
"name": "nomad",
"plugins": [
{
"type": "loopback"
},
{
"type": "bridge",
"bridge": "nomad",
"ipMasq": true,
"isGateway": true,
"forceAddress": true,
"hairpinMode": false,
"ipam": {
"type": "host-local",
"ranges": [
[{"subnet": "172.26.64.0/20"}]
],
"routes": [
{"dst": "0.0.0.0/0"}
],
"dataDir": "/var/run/cni"
}
},
{
"type": "firewall",
"backend": "iptables",
"iptablesAdminChainName": "NOMAD-ADMIN"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
}
]
}
```
</CodeBlockConfig>
</Tab>
<Tab heading="IPv6">
If you [configure IPv6][] on the Nomad client agent using the example range
`2001:db8::/112`, Nomad adds two more lines to the configuration.
<CodeBlockConfig highlight="19,23">
```json
{
"cniVersion": "1.0.0",
"name": "nomad",
"plugins": [
{
"type": "loopback"
},
{
"type": "bridge",
"bridge": "nomad",
"ipMasq": true,
"isGateway": true,
"forceAddress": true,
"hairpinMode": false,
"ipam": {
"type": "host-local",
"ranges": [
[{"subnet": "172.26.64.0/20"}],
[{"subnet": "2001:db8::/112"}]
],
"routes": [
{"dst": "0.0.0.0/0"},
{"dst": "::/0"}
],
"dataDir": "/var/run/cni"
}
},
{
"type": "firewall",
"backend": "iptables",
"iptablesAdminChainName": "NOMAD-ADMIN"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
}
]
}
```
</CodeBlockConfig>
</Tab>
<Tab heading="Transparent Proxy">
Jobs that use the
[`transparent_proxy`](/nomad/docs/job-specification/transparent_proxy)
block add a `consul-cni` plugin to the bottom of the CNI configuration.
Refer to the
[transparent proxy](/nomad/docs/networking/consul/service-mesh#transparent-proxy)
section of Nomad's Consul service mesh documentation to ensure that
the node is configured correctly for this feature.
<CodeBlockConfig highlight="36-39">
```json
{
"cniVersion": "1.0.0",
"name": "nomad",
"plugins": [
{
"type": "loopback"
},
{
"type": "bridge",
"bridge": "nomad",
"ipMasq": true,
"isGateway": true,
"forceAddress": true,
"hairpinMode": false,
"ipam": {
"type": "host-local",
"ranges": [
[{"subnet": "172.26.64.0/20"}]
],
"routes": [
{"dst": "0.0.0.0/0"}
],
"dataDir": "/var/run/cni"
}
},
{
"type": "firewall",
"backend": "iptables",
"iptablesAdminChainName": "NOMAD-ADMIN"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
},
{
"type": "consul-cni",
"log_level": "debug"
}
]
}
```
</CodeBlockConfig>
</Tab>
</Tabs>
This configuration uses the following CNI reference plugins:
- loopback: The loopback plugin sets the default local interface, lo0, created
inside the bridge network's network namespace to UP. This allows workload
running inside the namespace to bind to a namespace-specific loopback
interface.
- bridge: The [bridge][] plugin creates a bridge (virtual switch) named `nomad`
that resides in the host network namespace. Because Nomad intends this bridge
to provide network connectivity to allocations, we configured it to be a
gateway by setting `isGateway` to `true`. This tells the plugin to assign an
IP address to the bridge interface.
The bridge plugin connects allocations on the same host into a virtual switch
bridge that resides in the host network namespace. By default, Nomad creates
a single bridge for each client.
Since Nomad's bridge network is designed to provide network connectivity to
the allocations, Nomad configures the bridge interface to be a gateway for
outgoing traffic by providing it with an address using an `ipam`
configuration. The default configuration creates a host-local address for the
host side of the bridge in the `172.26.64.0/20` subnet at `172.26.64.1`.
When associating allocations to the bridge, Nomad creates addresses for the
allocations from that same subnet using the host-local plugin. The
configuration also specifies a default route for the allocations of the
host-side bridge address.
- firewall: The [firewall][] plugin creates firewall rules to allow traffic to
and from the allocation's IP address via the host network. Nomad uses the
iptables backend for the firewall plugin.
This configuration creates two new iptables chains, `CNI-FORWARD` and
`NOMAD-ADMIN`, in the filter table and adds rules that allow the given
interface to send and receive traffic.
The firewall creates an admin chain using the `iptablesAdminChainName` value,
which is `NOMAD-ADMIN` in this example. The admin chain is a user-controlled
chain for custom rules that run before rules managed by the firewall plugin.
The firewall plugin does not add, delete, or modify rules in the admin chain.
Nomad adds a new chain called `CNI-FORWARD` to to the `FORWARD` chain.
`CNI-FORWARD` is the chain where Nomad adds rules when it creates allocations
and removes the rules when those allocations stop. The `CNI-FORWARD` chain
first sends all traffic to the `NOMAD-ADMIN` chain.
Use the `iptables` command to list the iptables rules present in each chain.
```shell-session
$ sudo iptables -L
```
- portmap: Nomad needs to be able to map specific ports from the host to tasks
running in the allocation namespace. The [portmap][] plugin forwards traffic
from one or more ports on the host to the allocation using network address
translation (NAT) rules.
The plugin sets up two sequences of chains and rules:
- One primary destination NAT (DNAT) sequence to rewrite the destination.
- One source NAT (SNAT) sequence to masquerade traffic as needed.
Use the `iptables` command to list the iptables rules in the NAT table.
```shell-session
$ sudo iptables -t nat -L
```
Save your bridge network configuration file to a Nomad-accessible directory. By
default, Nomad loads configuration files from the `/opt/cni/config` directory.
However, you may configure a different location using the
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter.
Refer to the [Configure Nomad clients](#configure-nomad-clients) section for an
example.
## Configure Nomad clients
At plugin execution time, Nomad interprets your CNI network configuration and
transforms it into arguments for the plugins.
Nomad reads the following files from the
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter —
`/opt/cni/config` by default:
- `.conflist`: Nomad loads these files as [network
configurations](https://www.cni.dev/docs/spec/#configuration-format) that
contain a list of plugin configurations.
- `.conf` and `.json`: Nomad loads these files as individual [plugin
configurations](https://www.cni.dev/docs/spec/#plugin-configuration-objects)
for a specific network.
Add the [`cni_path`](/nomad/docs/configuration/client#cni_path) and
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) attributes
to each client's `client.hcl` file.
This example uses the default values for both attributes.
<CodeBlockConfig filename="/etc/nomad.d/client.hcl">
```hcl
client {
enabled = true
cni_path = "opt/cni/bin"
cni_config_dir = "opt/cni/config"
}
```
</CodeBlockConfig>
## Next steps
Refer to [Use a CNI network with a job][] for job configuration details.
[cni-spec]: https://www.cni.dev/docs/spec/
[cni-plugin-docs]: https://www.cni.dev/plugins/current/
[bridge]: https://www.cni.dev/plugins/current/main/bridge/
[firewall]: https://www.cni.dev/plugins/current/meta/firewall/
[portmap]: https://www.cni.dev/plugins/current/meta/portmap/
[Use a CNI network with a job]: /nomad/docs/job-networking/cni
[`cni_config_dir`]: /nomad/docs/configuration/client#cni_config_dir
[`cni_path`]: /nomad/docs/configuration/client#cni_path
[cni_spec_net_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#configuration-format
[cni_spec_plugin_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#plugin-configuration-objects
[configure IPv6]: /nomad/docs/configuration/client#bridge_network_subnet_ipv6
Reference in New Issue View Git Blame Copy Permalink