mirror of
https://github.com/kemko/nomad.git
synced 2026-01-04 09:25:46 +03:00
266 lines
10 KiB
Plaintext
266 lines
10 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: CNI plugins and bridge networking
|
|
description: |-
|
|
Learn how to install Container Network Interface (CNI) reference plugins, configure Nomad's bridge networking, and how to use CNI networks with Nomad jobs.
|
|
---
|
|
|
|
# Container Network Interface (CNI)
|
|
|
|
This page show you how to install the [CNI reference plugins][cni-plugin-docs]
|
|
on your Linux distribution and configure bridge networking on your Nomad
|
|
clients. You can apply this guide's workflow to installing any plugin that
|
|
complies with the [Container Network Interface (CNI) Specification][cni-spec],
|
|
but you should verify plugin compatibility with Nomad before deploying in
|
|
production.
|
|
|
|
## Introduction
|
|
|
|
The CNI specification defines a network configuration format for administrators.
|
|
The configuration contains directives for both the orchestrator and the plugins
|
|
to consume. Nomad implements custom networking through a combination of CNI
|
|
reference plugin binaries and CNI configuration files. Networking features, like
|
|
bridge network mode and Consul service mesh, leverage the CNI reference plugins
|
|
to provide an operating-system agnostic interface to configure workload
|
|
networking.
|
|
|
|
## Requirements
|
|
|
|
- You are familiar with [CNI reference plugins][cni-plugin-docs].
|
|
- You are familiar with [how Nomad uses Container Network Interface (CNI) plugins for bridge networking](/nomad/docs/networking#bridge-networking).
|
|
- You are running Nomad on Linux.
|
|
|
|
## CNI plugins and bridge networking workflow
|
|
|
|
Perform the following on each Nomad client:
|
|
|
|
1. [Install CNI reference plugins](#install-cni-reference-plugins).
|
|
1. [Configure bridge module to route traffic through iptables](#configure-bridge-module-to-route-traffic-through-iptables).
|
|
1. [Create a bridge mode configuration](#create-a-cni-bridge-mode-configuration).
|
|
1. [Configure Nomad clients](#configure-nomad-clients).
|
|
|
|
After you configure and restart your Nomad clients, [use a CNI network with a
|
|
job](#use-a-cni-network-with-a-job).
|
|
|
|
### Install CNI reference plugins
|
|
|
|
@include 'install/install-cni-plugins.mdx'
|
|
|
|
### Configure bridge module to route traffic through iptables
|
|
|
|
@include 'install/bridge-iptables.mdx'
|
|
|
|
## Create a CNI bridge mode configuration
|
|
|
|
Nomad itself uses CNI plugins and configuration as the underlying implementation
|
|
for the `bridge` network mode, using the loopback, [bridge][], [firewall][], and
|
|
[portmap][] CNI reference plugins configured together to create Nomad's bridge
|
|
network.
|
|
|
|
[comment-source-image]:
|
|
https://www.figma.com/file/Ne2qaPUlBTmTYer9biCfK9/Networking?node-id=0%3A1&t=BepgOoQ0kb76GwIr-1
|
|
|
|
[](/img/nomad-bridge-network.png)
|
|
|
|
When setting up a bridge network, Nomad uses a configuration template based on
|
|
the CNI Specification's [example
|
|
configuration](https://www.cni.dev/docs/spec/#example-configuration). Refer to
|
|
the external [configuration
|
|
format](https://www.cni.dev/docs/spec/#example-configuration) for a complete
|
|
explanation of the fields.
|
|
|
|
You can use this template as a basis for your own CNI-based bridge network
|
|
configuration in cases where you need access to an unsupported option in the
|
|
default configuration, like hairpin mode.
|
|
|
|
This example uses two default values from Nomad client configuration.
|
|
|
|
- The default value for
|
|
[`bridge_network_name`](/nomad/docs/configuration/client#bridge_network_name)
|
|
is the value for the bridge plugin name.
|
|
- The default value for bridge subnet [`bridge_network_subnet`](
|
|
/nomad/docs/configuration/client#bridge_network_subnet) is the bridge plugin
|
|
subnet.
|
|
|
|
The `NOMAD-ADMIN` internal constant provides the value for
|
|
`iptablesAdminChainName`. In your own configuration, ensure that you change the
|
|
`iptablesAdminChainName` to a unique value.
|
|
|
|
<CodeBlockConfig highlight="10,20,32">
|
|
|
|
```json
|
|
{
|
|
"cniVersion": "1.0.0",
|
|
"name": "nomad",
|
|
"plugins": [
|
|
{
|
|
"type": "loopback"
|
|
},
|
|
{
|
|
"type": "bridge",
|
|
"bridge": "nomad",
|
|
"ipMasq": true,
|
|
"isGateway": true,
|
|
"forceAddress": true,
|
|
"hairpinMode": false,
|
|
"ipam": {
|
|
"type": "host-local",
|
|
"ranges": [
|
|
[
|
|
{
|
|
"subnet": "172.26.64.0/20"
|
|
}
|
|
]
|
|
],
|
|
"routes": [
|
|
{ "dst": "0.0.0.0/0" }
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"type": "firewall",
|
|
"backend": "iptables",
|
|
"iptablesAdminChainName": "NOMAD-ADMIN"
|
|
},
|
|
{
|
|
"type": "portmap",
|
|
"capabilities": {"portMappings": true},
|
|
"snat": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
This configuration uses the following CNI reference plugins:
|
|
|
|
- loopback: The loopback plugin sets the default local interface, lo0, created
|
|
inside the bridge network's network namespace to UP. This allows workload
|
|
running inside the namespace to bind to a namespace-specific loopback
|
|
interface.
|
|
- bridge: The [bridge][] plugin creates a bridge (virtual switch) named `nomad`
|
|
that resides in the host network namespace. Because Nomad intends this bridge
|
|
to provide network connectivity to allocations, we configured it to be a
|
|
gateway by setting `isGateway` to `true`. This tells the plugin to assign an
|
|
IP address to the bridge interface.
|
|
|
|
The bridge plugin connects allocations on the same host into a virtual switch
|
|
bridge that resides in the host network namespace. By default, Nomad creates
|
|
a single bridge for each client.
|
|
|
|
Since Nomad's bridge network is designed to provide network connectivity to
|
|
the allocations, Nomad configures the bridge interface to be a gateway for
|
|
outgoing traffic by providing it with an address using an `ipam`
|
|
configuration. The default configuration creates a host-local address for the
|
|
host side of the bridge in the `172.26.64.0/20` subnet at `172.26.64.1`.
|
|
|
|
When associating allocations to the bridge, Nomad creates addresses for the
|
|
allocations from that same subnet using the host-local plugin. The
|
|
configuration also specifies a default route for the allocations of the
|
|
host-side bridge address.
|
|
- firewall: The [firewall][] plugin creates firewall rules to allow traffic to
|
|
and from the allocation's IP address via the host network. Nomad uses the
|
|
iptables backend for the firewall plugin.
|
|
|
|
This configuration creates two new iptables chains, `CNI-FORWARD` and
|
|
`NOMAD-ADMIN`, in the filter table and adds rules that allow the given
|
|
interface to send and receive traffic.
|
|
|
|
The firewall creates an admin chain using the `iptablesAdminChainName` value,
|
|
which is `NOMAD-ADMIN` in this example. The admin chain is a user-controlled
|
|
chain for custom rules that run before rules managed by the firewall plugin.
|
|
The firewall plugin does not add, delete, or modify rules in the admin chain.
|
|
|
|
Nomad adds a new chain called `CNI-FORWARD` to to the `FORWARD` chain.
|
|
`CNI-FORWARD` is the chain where Nomad adds rules when it creates allocations
|
|
and removes the rules when those allocations stop. The `CNI-FORWARD` chain
|
|
first sends all traffic to the `NOMAD-ADMIN` chain.
|
|
|
|
Use the `iptables` command to list the iptables rules present in each chain.
|
|
|
|
```shell-session
|
|
$ sudo iptables -L
|
|
```
|
|
|
|
- portmap: Nomad needs to be able to map specific ports from the host to tasks
|
|
running in the allocation namespace. The [portmap][] plugin forwards traffic
|
|
from one or more ports on the host to the allocation using network address
|
|
translation (NAT) rules.
|
|
|
|
The plugin sets up two sequences of chains and rules:
|
|
|
|
- One primary destination NAT (DNAT) sequence to rewrite the destination.
|
|
- One source NAT (SNAT) sequence to masquerade traffic as needed.
|
|
|
|
Use the `iptables` command to list the iptables rules in the NAT table.
|
|
|
|
```shell-session
|
|
$ sudo iptables -t nat -L
|
|
```
|
|
|
|
Save your bridge network configuration file to a Nomad-accessible directory. By
|
|
default, Nomad loads configuration files from the `/opt/cni/config` directory.
|
|
However, you may configure a different location using the
|
|
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter.
|
|
Refer to the [Configure Nomad clients](#configure-nomad-clients) section for an
|
|
example.
|
|
|
|
## Configure Nomad clients
|
|
|
|
At plugin execution time, Nomad interprets your CNI network configuration and
|
|
transforms it into arguments for the plugins.
|
|
|
|
Nomad reads the following files from the
|
|
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter —
|
|
`/opt/cni/config` by default:
|
|
|
|
- `.conflist`: Nomad loads these files as [network
|
|
configurations](https://www.cni.dev/docs/spec/#configuration-format) that
|
|
contain a list of plugin configurations.
|
|
|
|
- `.conf` and `.json`: Nomad loads these files as individual [plugin
|
|
configurations](https://www.cni.dev/docs/spec/#plugin-configuration-objects)
|
|
for a specific network.
|
|
|
|
Add the [`cni_path`](/nomad/docs/configuration/client#cni_path) and
|
|
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) attributes
|
|
to each client's `client.hcl` file.
|
|
|
|
This example uses the default values for both attributes.
|
|
|
|
<CodeBlockConfig filename="/etc/nomad.d/client.hcl">
|
|
|
|
```hcl
|
|
client {
|
|
enabled = true
|
|
cni_path = "opt/cni/bin"
|
|
cni_config_dir = "opt/cni/config"
|
|
}
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
## Use a CNI network with a job
|
|
|
|
To specify that a job should use a CNI network, set the task group's network
|
|
[`mode`](/nomad/docs/job-specification/network#mode) attribute to the value
|
|
`cni/<your_cni_config_name>`. Nomad then schedules the workload on client nodes
|
|
that have fingerprinted a CNI configuration with the given name. For example, to
|
|
use the configuration named `mynet`, you should set the task group's network
|
|
mode to `cni/mynet`. Nodes that have a network configuration defining a network
|
|
named `mynet` in their `cni_config_dir` are eligible to run the workload. Nomad
|
|
additionally supplies the following arguments via `CNI_ARGS` to the CNI network:
|
|
`NOMAD_REGION`, `NOMAD_NAMESPACE`, `NOMAD_JOB_ID`, `NOMAD_GROUP_NAME`, and
|
|
`NOMAD_ALLOC_ID`. Since the `CNI_ARGS` do not allow values to contain a semicolon
|
|
Nomad will not set keys where the value contains a semicolon (this could happen
|
|
with the job ID). CNI plugins utilizing `NOMAD_*` CNI arguments are advised to
|
|
apply a defensive policy or simply error out.
|
|
|
|
[cni-spec]: https://www.cni.dev/docs/spec/
|
|
[cni-plugin-docs]: https://www.cni.dev/plugins/current/
|
|
[bridge]: https://www.cni.dev/plugins/current/main/bridge/
|
|
[firewall]: https://www.cni.dev/plugins/current/meta/firewall/
|
|
[portmap]: https://www.cni.dev/plugins/current/meta/portmap/
|