nomad/website/content/docs/networking/cni.mdx

---
layout: docs
page_title: CNI plugins and bridge networking
description: |-
  Learn how to install Container Network Interface (CNI) reference plugins, configure Nomad's bridge networking, and use CNI networks with Nomad jobs.
---

# Container Network Interface (CNI)

This page shows you how to install the [CNI reference plugins][cni-plugin-docs]
on your Linux distribution and configure bridge networking on your Nomad
clients. You can apply this guide's workflow to installing any plugin that
complies with the [Container Network Interface (CNI) Specification][cni-spec],
but you should verify plugin compatibility with Nomad before deploying in
production.

## Background

The CNI specification defines a network configuration format for administrators.
The configuration contains directives for both the orchestrator and the plugins
to consume. Nomad implements custom networking through a combination of CNI
reference plugin binaries and CNI configuration files. Networking features, like
bridge network mode and Consul service mesh, leverage the CNI reference plugins
to provide an operating-system agnostic interface to configure workload
networking.

## Requirements

- You are familiar with [CNI reference plugins][cni-plugin-docs].
- You are familiar with [how Nomad uses Container Network Interface (CNI) plugins for bridge networking](/nomad/docs/networking#bridge-networking).
- You are running Nomad on Linux.

## CNI plugins and bridge networking workflow

Perform the following on each Nomad client:

1. [Install CNI reference plugins](#install-cni-reference-plugins).
1. [Configure bridge module to route traffic through iptables](#configure-bridge-module-to-route-traffic-through-iptables).
1. [Create a bridge mode configuration](#create-a-cni-bridge-mode-configuration).
1. [Configure Nomad clients](#configure-nomad-clients).

After you configure and restart your Nomad clients, refer to [Use a CNI network
with a job][] for job configuration.

### Install CNI reference plugins

@include 'install/install-cni-plugins.mdx'

### Configure bridge module to route traffic through iptables

@include 'install/bridge-iptables.mdx'

## Create a CNI bridge mode configuration

Nomad itself uses CNI plugins and configuration as the underlying implementation
for the `bridge` network mode, using the loopback, [bridge][], [firewall][], and
[portmap][] CNI reference plugins configured together to create Nomad's bridge
network.

[comment-source-image]:
    https://www.figma.com/file/Ne2qaPUlBTmTYer9biCfK9/Networking?node-id=0%3A1&t=BepgOoQ0kb76GwIr-1

[![Visual example of Nomad bridge
network](/img/nomad-bridge-network.png)](/img/nomad-bridge-network.png)

When setting up a bridge network, Nomad uses a configuration template based on
the CNI Specification's [example
configuration](https://www.cni.dev/docs/spec/#example-configuration). Refer to
the external [configuration
format](https://www.cni.dev/docs/spec/#example-configuration) for a complete
explanation of the fields.

You can use this template as a basis for your own CNI-based bridge network
configuration in cases where you need access to an unsupported option in the
default configuration, like hairpin mode.

This example uses two default values from Nomad client configuration.

- The default value for
  [`bridge_network_name`](/nomad/docs/configuration/client#bridge_network_name)
  is the value for the bridge plugin name.
- The default value for bridge subnet [`bridge_network_subnet`](
  /nomad/docs/configuration/client#bridge_network_subnet) is the bridge plugin
  subnet.

The `NOMAD-ADMIN` internal constant provides the value for
`iptablesAdminChainName`. In your own configuration, ensure that you change the
`iptablesAdminChainName` to a unique value.

<CodeBlockConfig highlight="10,20,32">

```json
{
  "cniVersion": "1.0.0",
  "name": "nomad",
  "plugins": [
    {
      "type": "loopback"
    },
    {
      "type": "bridge",
      "bridge": "nomad",
      "ipMasq": true,
      "isGateway": true,
      "forceAddress": true,
      "hairpinMode": false,
      "ipam": {
        "type": "host-local",
        "ranges": [
          [
            {
              "subnet": "172.26.64.0/20"
            }
          ]
        ],
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "firewall",
      "backend": "iptables",
      "iptablesAdminChainName": "NOMAD-ADMIN"
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true},
      "snat": true
    }
  ]
}
```

</CodeBlockConfig>

This configuration uses the following CNI reference plugins:

- loopback: The loopback plugin sets the default local interface, lo0, created
  inside the bridge network's network namespace to UP. This allows workload
  running inside the namespace to bind to a namespace-specific loopback
  interface.
- bridge: The [bridge][] plugin creates a bridge (virtual switch) named `nomad`
  that resides in the host network namespace. Because Nomad intends this bridge
  to provide network connectivity to allocations, we configured it to be a
  gateway by setting `isGateway` to `true`. This tells the plugin to assign an
  IP address to the bridge interface.

   The bridge plugin connects allocations on the same host into a virtual switch
   bridge that resides in the host network namespace. By default, Nomad creates
   a single bridge for each client.

   Since Nomad's bridge network is designed to provide network connectivity to
   the allocations, Nomad configures the bridge interface to be a gateway for
   outgoing traffic by providing it with an address using an `ipam`
   configuration. The default configuration creates a host-local address for the
   host side of the bridge in the `172.26.64.0/20` subnet at `172.26.64.1`.

   When associating allocations to the bridge, Nomad creates addresses for the
   allocations from that same subnet using the host-local plugin. The
   configuration also specifies a default route for the allocations of the
   host-side bridge address.
- firewall: The [firewall][] plugin creates firewall rules to allow traffic to
  and from the allocation's IP address via the host network. Nomad uses the
  iptables backend for the firewall plugin.

   This configuration creates two new iptables chains, `CNI-FORWARD` and
   `NOMAD-ADMIN`, in the filter table and adds rules that allow the given
   interface to send and receive traffic.

   The firewall creates an admin chain using the `iptablesAdminChainName` value,
   which is `NOMAD-ADMIN` in this example. The admin chain is a user-controlled
   chain for custom rules that run before rules managed by the firewall plugin.
   The firewall plugin does not add, delete, or modify rules in the admin chain.

   Nomad adds a new chain called `CNI-FORWARD` to to the `FORWARD` chain.
   `CNI-FORWARD` is the chain where Nomad adds rules when it creates allocations
   and removes the rules when those allocations stop. The `CNI-FORWARD` chain
   first sends all traffic to the `NOMAD-ADMIN` chain.

   Use the `iptables` command to list the iptables rules present in each chain.

   ```shell-session
   $ sudo iptables -L
   ```

- portmap: Nomad needs to be able to map specific ports from the host to tasks
  running in the allocation namespace. The [portmap][] plugin forwards traffic
  from one or more ports on the host to the allocation using network address
  translation (NAT) rules.

   The plugin sets up two sequences of chains and rules:

   - One primary destination NAT (DNAT) sequence to rewrite the destination.
   - One source NAT (SNAT) sequence to masquerade traffic as needed.

   Use the `iptables` command to list the iptables rules in the NAT table.

   ```shell-session
   $ sudo iptables -t nat -L
   ```

Save your bridge network configuration file to a Nomad-accessible directory. By
default, Nomad loads configuration files from the `/opt/cni/config` directory.
However, you may configure a different location using the
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter.
Refer to the [Configure Nomad clients](#configure-nomad-clients) section for an
example.

## Configure Nomad clients

At plugin execution time, Nomad interprets your CNI network configuration and
transforms it into arguments for the plugins.

Nomad reads the following files from the
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter —
`/opt/cni/config` by default:

- `.conflist`: Nomad loads these files as [network
  configurations](https://www.cni.dev/docs/spec/#configuration-format) that
  contain a list of plugin configurations.

- `.conf` and `.json`: Nomad loads these files as individual [plugin
  configurations](https://www.cni.dev/docs/spec/#plugin-configuration-objects)
  for a specific network.

Add the [`cni_path`](/nomad/docs/configuration/client#cni_path) and
[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) attributes
to each client's `client.hcl` file.

This example uses the default values for both attributes.

<CodeBlockConfig filename="/etc/nomad.d/client.hcl">

```hcl
client {
  enabled = true
  cni_path = "opt/cni/bin"
  cni_config_dir = "opt/cni/config"
}
```

</CodeBlockConfig>

## Next steps

 Refer to [Use a CNI network with a job][] for job configuration details.

[cni-spec]: https://www.cni.dev/docs/spec/
[cni-plugin-docs]: https://www.cni.dev/plugins/current/
[bridge]: https://www.cni.dev/plugins/current/main/bridge/
[firewall]: https://www.cni.dev/plugins/current/meta/firewall/
[portmap]: https://www.cni.dev/plugins/current/meta/portmap/
[Use a CNI network with a job]: /nomad/docs/job-networking/cni