kemko/nomad
1
0
Fork 0
mirror of https://github.com/kemko/nomad.git synced 2026-01-01 16:05:42 +03:00
Code Issues Packages Projects Releases Wiki Activity

Dynamic host volume reference documentation (#24797)

Browse Source
This commit is contained in:
Tim Gross
2025-02-13 12:25:58 -05:00
committed by GitHub
parent 25426f0777 35365bc1fb
commit c2298e0999
25 changed files with 2153 additions and 770 deletions
Download Patch File Download Diff File Expand all files Collapse all files

523
website/content/api-docs/volumes.mdx
View File

@@ -1,13 +1,14 @@
---
layout: api
page_title: Volumes - HTTP API
description: The `/volume` endpoints are used to query for and interact with volumes.
description: |-
The Nomad `/volume` and `/volumes` endpoints query for and interact with Container Storage Interface (CSI) volumes and dynamic host volumes.
---
# Volumes HTTP API
The `/volume` and `/volumes` endpoints are used to query for and interact with
volumes.
The `/volume` and `/volumes` endpoints query for and interact with
Container Storage Interface (CSI) volumes and dynamic host volumes.
## List Volumes
@@ -17,29 +18,28 @@ This endpoint lists all volumes.
| ------ | ------------- | ------------------ |
| `GET` | `/v1/volumes` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | --------------------------- |
| `YES` | `namespace:csi-list-volume` |
| Blocking Queries | ACL Required |
|------------------|--------------------------------------------------------------|
| `YES` | `namespace:csi-list-volume`<br/>`namespace:host-volume-read` |
### Parameters
- `type` `(string: "")` - Specifies the type of volume to
query. Currently only supports `csi`. This is specified as a query
string parameter. Returns an empty list if omitted.
- `type` `(string: <required>)` - Specifies the type of volume to query. One of
`csi` or `host`. The `host` value queries dynamic host volumes. Specify this
as a query string parameter.
- `node_id` `(string: "")` - Specifies a string to filter volumes
based on an Node ID prefix. Because the value is decoded to bytes,
the prefix must have an even number of hexadecimal characters
(0-9a-f). This is specified as a query string parameter.
(0-9a-f). Specify this as a query string parameter.
- `plugin_id` `(string: "")` - Specifies a string to filter volumes
based on a plugin ID prefix. Because the value is decoded to bytes,
the prefix must have an even number of hexadecimal characters
(0-9a-f). This is specified as a query string parameter.
(0-9a-f). Specify this as a query string parameter.
- `next_token` `(string: "")` - This endpoint supports paging. The `next_token`
parameter accepts a string which identifies the next expected volume. This
@@ -62,7 +62,7 @@ $ curl \
https://localhost:4646/v1/volumes?type=csi&node_id=foo&plugin_id=plugin-id1
```
### Sample Response
### Sample Response for CSI Volumes
```json
[
@@ -95,17 +95,37 @@ $ curl \
]
```
## Read Volume
### Sample Response for dynamic host volumes
This endpoint reads information about a specific volume.
```json
[
{
"CapacityBytes": 1048576000,
"CreateIndex": 42,
"CreateTime": 1736191825846395400,
"ID": "3735cc2c-cc64-11ef-89ed-bfb5b3bc38ea",
"ModifyIndex": 64,
"ModifyTime": 1736191825846395400,
"Name": "example",
"Namespace": "default",
"NodeID": "5c5830d0-cc64-11ef-a293-4f03e55969ea",
"NodePool": "default",
"PluginID": "plugin-id1",
"State": "ready"
}
]
```
## Read CSI Volume
This endpoint reads information about a specific CSI volume.
| Method | Path | Produces |
| ------ | --------------------------- | ------------------ |
| `GET` | `/v1/volume/csi/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | --------------------------- |
@@ -114,7 +134,7 @@ The table below shows this endpoint's support for
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
volume. This must be the full ID. Specify this as part of the
path.
### Sample Request
@@ -275,10 +295,11 @@ $ curl \
}
```
## Register Volume
## Register CSI Volume
This endpoint registers an external volume with Nomad. The volume must exist
in the external storage provider (see [Create Volume] below).
This endpoint registers an external CSI volume with Nomad. The volume must exist
in the external storage provider. Refer to the [Create CSI Volume][] section for
details.
Making the same request again with a higher `RequestedCapacityMin` value
may trigger a [Volume Expansion][].
@@ -287,9 +308,8 @@ may trigger a [Volume Expansion][].
| ------ | --------------------------- | ------------------ |
| `PUT` | `/v1/volume/csi/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -298,13 +318,13 @@ The table below shows this endpoint's support for
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
volume. This must be the full ID. Specify this as part of the
path.
### Sample Payload
The payload must include a JSON document that describes the volume's
parameters.
parameters. Note that the `NodeID` field is required for the register API.
```json
{
@@ -343,9 +363,9 @@ $ curl \
https://localhost:4646/v1/volume/csi/volume-id1
```
## Create Volume
## Create CSI Volume
This endpoint creates a volume in an external storage provider and registers
This endpoint creates a CSI volume in an external storage provider and registers
it with Nomad. Only CSI plugins that implement the
[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_VOLUME`
capability support this endpoint.
@@ -357,9 +377,8 @@ may trigger a [Volume Expansion][].
| ------ | ---------------------------------- | ------------------ |
| `PUT` | `/v1/volume/csi/:volume_id/create` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -368,7 +387,7 @@ The table below shows this endpoint's support for
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
volume. This must be the full ID. Specify this as part of the
path.
### Sample Payload
@@ -420,18 +439,17 @@ $ curl \
```
## Deregister Volume
## Deregister CSI Volume
This endpoint deregisters an external volume with Nomad. It is an error to
This endpoint deregisters an external CSI volume from Nomad. It is an error to
deregister a volume that is in use.
| Method | Path | Produces |
| -------- | --------------------------- | ------------------ |
| `DELETE` | `/v1/volume/csi/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -440,13 +458,13 @@ The table below shows this endpoint's support for
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
volume. This must be the full ID. Specify this as part of the
path.
- `force` `(bool: false)` - Force deregistration of the volume and immediately
drop claims for terminal allocations. Returns an error if the volume has
running allocations. This does not detach the volume from client nodes.
This is specified as a query string parameter.
Specify this as a query string parameter.
### Sample Request
@@ -456,9 +474,9 @@ $ curl \
https://localhost:4646/v1/volume/csi/volume-id1?force=false
```
## Delete Volume
## Delete CSI Volume
This endpoint deletes an external volume from the storage provider, and
This endpoint deletes an external CSI volume from the storage provider, and
deregisters it from Nomad. It is an error to delete a volume that is in
use. Only CSI plugins that implement the [Controller][csi_plugins_internals]
interface with the `CREATE_DELETE_VOLUME` capability support this endpoint.
@@ -467,9 +485,8 @@ interface with the `CREATE_DELETE_VOLUME` capability support this endpoint.
| -------- | ---------------------------------- | ------------------ |
| `DELETE` | `/v1/volume/csi/:volume_id/delete` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -483,7 +500,7 @@ stored when the CSI volume was created.
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
volume. This must be the full ID. Specify this as part of the
path.
### Sample Request
@@ -495,18 +512,17 @@ $ curl \
https://localhost:4646/v1/volume/csi/volume-id1/delete
```
## Detach Volume
## Detach CSI Volume
This endpoint detaches an external volume from a Nomad client node. It is an
This endpoint detaches an external CSI volume from a Nomad client node. It is an
error to detach a volume that is in use.
| Method | Path | Produces |
| -------- | ---------------------------------- | ------------------ |
| `DELETE` | `/v1/volume/csi/:volume_id/detach` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -515,11 +531,11 @@ The table below shows this endpoint's support for
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
volume. This must be the full ID. Specify this as part of the
path.
- `node` `(string: <required>)` - The node to detach the volume from.
This is specified as a query string parameter.
Specify this as a query string parameter.
### Sample Request
@@ -529,20 +545,19 @@ $ curl \
https://localhost:4646/v1/volume/csi/volume-id/detach?node=00000000-0000-0000-0000-000000000000
```
## List External Volumes
## List External CSI Volumes
This endpoint lists storage volumes that are known to the external storage
provider but may not be registered with Nomad. Only CSI plugins that
implement the [Controller][csi_plugins_internals] interface with the
`LIST_VOLUMES` capability support this endpoint.
This endpoint lists storage CSI volumes that are known to the external storage
provider but may not be registered with Nomad. Only CSI plugins that implement
the [Controller][csi_plugins_internals] interface with the `LIST_VOLUMES`
capability support this endpoint.
| Method | Path | Produces |
|--------|------------------------|--------------------|
| `GET` | `/v1/volumes/external` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -553,14 +568,14 @@ The table below shows this endpoint's support for
- `plugin_id` `(string: "")` - Specifies a string to filter volumes
based on a plugin ID prefix. Because the value is decoded to bytes,
the prefix must have an even number of hexadecimal characters
(0-9a-f). This is specified as a query string parameter.
(0-9a-f). Specify this as a query string parameter.
- `next_token` `(string: "")` - This endpoint supports paging. The
`next_token` parameter accepts a string returned in a previous response's
`NextToken` field to request the next page of results.
- `per_page` `(int: <required>)` - Specifies a maximum number of snapshots to
return for this request. The response will include a `NextToken` field that
return for this request. The response includes a `NextToken` field that
can be passed to the next request to fetch additional pages.
### Sample Request
@@ -596,20 +611,19 @@ $ curl \
}
```
## Create Snapshot
## Create CSI Volume Snapshot
This endpoint creates a snapshot of a volume on the external storage
This endpoint creates a snapshot of a CSI volume on the external storage
provider. Only CSI plugins that implement the
[Controller][csi_plugins_internals] interface with the
`CREATE_DELETE_SNAPSHOT` capability support this endpoint.
[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_SNAPSHOT`
capability support this endpoint.
| Method | Path | Produces |
| -------- | --------------------------------- | ------------------ |
| `POST` | `/v1/volumes/snapshot` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -664,20 +678,19 @@ $ curl \
}
```
## Delete Snapshot
## Delete CSI Volume Snapshot
This endpoint deletes a volume snapshot from the external storage
This endpoint deletes a CSI volume snapshot from the external storage
provider. Only CSI plugins that implement the
[Controller][csi_plugins_internals] interface with the
`CREATE_DELETE_SNAPSHOT` capability support this endpoint.
[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_SNAPSHOT`
capability support this endpoint.
| Method | Path | Produces |
| -------- | ---------------------- | ------------------ |
| `DELETE` | `/v1/volumes/snapshot` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
@@ -691,12 +704,12 @@ stored when the CSI snapshot was created.
### Parameters
- `plugin_id` `(string: <required>)` - Specifies the prefix of a CSI plugin ID
to perform the delete. Because the value is decoded to bytes, the prefix
must have an even number of hexadecimal characters (0-9a-f). This is
specified as a query string parameter.
to perform the delete. Because the value is decoded to bytes, the prefix must
have an even number of hexadecimal characters (0-9a-f). Specify this as a
query string parameter.
- `snapshot_id` `(string: <required>)` - Specifies the snapshot ID to
delete. This is specified as a query parameter.
delete. Specify this as a query parameter.
### Sample Request
@@ -707,9 +720,9 @@ $ curl \
https://localhost:4646/v1/volumes/snapshot
```
## List Snapshots
## List CSI Volume Snapshots
This endpoint lists volume snapshots on the external storage provider. Only
This endpoint lists CSI volume snapshots on the external storage provider. Only
CSI plugins that implement the [Controller][csi_plugins_internals] interface
with the `LIST_SNAPSHOTS` capability support this endpoint.
@@ -718,9 +731,8 @@ with the `LIST_SNAPSHOTS` capability support this endpoint.
| ------ | ---------------------- | ------------------ |
| `GET` | `/v1/volumes/snapshot` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
| ---------------- | --------------------------- |
@@ -735,7 +747,7 @@ stored when the CSI snapshot was created.
- `plugin_id` `(string: <required>)` - Specifies the prefix of a CSI plugin ID
to perform the list. Because the value is decoded to bytes, the prefix must
have an even number of hexadecimal characters (0-9a-f). This is specified as
have an even number of hexadecimal characters (0-9a-f). Specify this as
a query string parameter.
- `next_token` `(string: "")` - This endpoint supports paging. The
@@ -743,7 +755,7 @@ stored when the CSI snapshot was created.
`NextToken` field to request the next page of results.
- `per_page` `(int: <required>)` - Specifies a maximum number of snapshots to
return for this request. The response will include a `NextToken` field that
return for this request. The response includes a `NextToken` field that
can be passed to the next request to fetch additional pages.
### Sample Request
@@ -782,8 +794,335 @@ $ curl \
}
```
## Read Dynamic Host Volume
This endpoint reads information about a specific dynamic host volume.
| Method | Path | Produces |
|--------|------------------------------|--------------------|
| `GET` | `/v1/volume/host/:volume_id` | `application/json` |
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
|------------------|------------------------------|
| `YES` | `namespace:host-volume-read` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. Specify this as part of the
path.
### Sample Request
```shell-session
$ curl \
https://localhost:4646/v1/volume/host/c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f
```
### Sample Response
```json
{
"Allocations": [],
"CapacityBytes": 49283072,
"CreateIndex": 11,
"CreateTime": 1736191993011594200,
"HostPath": "/run/nomad/dev/alloc_mounts/c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f",
"ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f",
"ModifyIndex": 12,
"ModifyTime": 1736191993011594200,
"Name": "external-plugin",
"Namespace": "default",
"NodeID": "670cb259-bc26-653b-e316-655af04ad260",
"NodePool": "default",
"Parameters": {
"hello": "world"
},
"PluginID": "example-plugin-mkfs",
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
}
],
"RequestedCapacityMaxBytes": 50000000,
"RequestedCapacityMinBytes": 50000000,
"State": "ready"
}
```
## Create Dynamic Host Volume
This endpoint creates a dynamic host volume and registers it with Nomad.
The response body includes the volume definition with the `ID` and `NodeID`
fields populated. Making the same request without the ID creates a new volume on
a different node. Making a request with the ID set updates the volume to the new
parameters, if possible.
| Method | Path | Produces |
|--------|--------------------------|--------------------|
| `PUT` | `/v1/volume/host/create` | `application/json` |
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
|------------------|--------------------------------------------------------------------------------------------------------------------------------|
| `NO` | `namespace:host-volume-create` or<br/>`namespace:host-volume-write`.<br/>`namespace:sentinel-override` if `PolicyOverride` set |
### Parameters
- `Volume` `(Volume: <required>)` - Specifies the JSON definition of the host
volume. You should include the ID field if you are updating an existing
volume.
- `PolicyOverride` `(bool: false)` - If set, Nomad overrides any soft mandatory
Sentinel policies. This field allows creating a volume when it would be denied
by policy.
### Sample Payload
```json
{
"PolicyOverride": false,
"Volume": {
"Name": "example",
"Namespace": "default",
"NodePool": "prod",
"PluginID": "mkdir",
"RequestedCapacityMinBytes": 10737418240,
"RequestedCapacityMaxBytes": 21474836480,
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"Constraints": [
{
"LTarget": "${attr.kernel.name}",
"RTarget": "linux",
"Operand": "="
}
],
"Parameters": {
"foo": "bar"
}
}
}
```
### Sample Request
```shell-session
$ curl \
--request PUT \
--data @payload.json \
https://localhost:4646/v1/volume/host/create
```
### Sample Response
```json
{
"Volume": {
"Allocations": [],
"CapacityBytes": 21474836480,
"Constraints": [
{
"LTarget": "${attr.kernel.name}",
"RTarget": "linux",
"Operand": "="
}
],
"CreateIndex": 11,
"CreateTime": 1736191993011594200,
"ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f",
"ModifyIndex": 11,
"ModifyTime": 1736191993011594200,
"Name": "example",
"Namespace": "default",
"NodeID": "45460554-cc67-11ef-84b7-33d383a55487",
"NodePool": "prod",
"Parameters": {
"foo": "bar"
},
"PluginID": "mkdir",
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"RequestedCapacityMaxBytes": 21474836480,
"RequestedCapacityMinBytes": 10737418240,
"State": "pending"
},
"Warnings": null
}
```
## Register Dynamic Host Volume
This endpoint registers an existing dynamic host volume with Nomad.
| Method | Path | Produces |
|--------|----------------------------|--------------------|
| `PUT` | `/v1/volume/host/register` | `application/json` |
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
|------------------|----------------------------------------------------------------------------------------------------------------------------------|
| `NO` | `namespace:host-volume-register` or<br/>`namespace:host-volume-write`.<br/>`namespace:sentinel-override` if `PolicyOverride` set |
### Parameters
- `Volume` `(Volume: <required>)` - Specifies the JSON definition of the host
volume. You should include the ID field if you are updating an existing
volume.
- `PolicyOverride` `(bool: false)` - If set, Nomad overrides any soft mandatory
Sentinel policies. This field allows registering a volume when it would be denied
by policy.
### Sample Payload
```json
{
"PolicyOverride": false,
"Volume": {
"Name": "example",
"Namespace": "default",
"NodePool": "prod",
"PluginID": "mkdir",
"RequestedCapacityMinBytes": 10737418240,
"RequestedCapacityMaxBytes": 21474836480,
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"Constraints": [
{
"LTarget": "${attr.kernel.name}",
"RTarget": "linux",
"Operand": "="
}
],
"Parameters": {
"foo": "bar"
}
}
}
```
### Sample Request
```shell-session
$ curl \
--request PUT \
--data @payload.json \
https://localhost:4646/v1/volume/host/register
```
### Sample Response
```json
{
"Volume": {
"Allocations": [],
"CapacityBytes": 21474836480,
"Constraints": [
{
"LTarget": "${attr.kernel.name}",
"RTarget": "linux",
"Operand": "="
}
],
"CreateIndex": 11,
"CreateTime": 1736191993011594200,
"ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f",
"ModifyIndex": 31,
"ModifyTime": 1736191993721594200,
"Name": "example",
"Namespace": "default",
"NodeID": "45460554-cc67-11ef-84b7-33d383a55487",
"NodePool": "prod",
"Parameters": {
"foo": "bar"
},
"PluginID": "mkdir",
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"RequestedCapacityMaxBytes": 21474836480,
"RequestedCapacityMinBytes": 10737418240,
"State": "ready"
},
"Warnings": null
}
```
## Delete Dynamic Host Volume
This endpoint deletes a dynamic host volume, and deregisters it from Nomad. It
is an error to delete a volume that is in use.
| Method | Path | Produces |
|----------|-------------------------------------|--------------------|
| `DELETE` | `/v1/volume/host/:volume_id/delete` | `application/json` |
The following table shows this endpoint's support for [blocking queries][] and
[required ACLs][].
| Blocking Queries | ACL Required |
|------------------|---------------------------------------------------------------------|
| `NO` | `namespace:host-volume-write` or<br/>`namespace:host-volume-delete` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. Specify this as part of the
path.
### Sample Request
```shell-session
$ curl \
--request DELETE \
https://localhost:4646/v1/volume/host/ba97ef42-cc68-11ef-a2e7-ffddaecbdb89
```
[blocking queries]: /nomad/api-docs#blocking-queries
[required ACLs]: /nomad/api-docs#acls
[csi]: https://github.com/container-storage-interface/spec
[csi_plugin]: /nomad/docs/job-specification/csi_plugin
[csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins
[Create Volume]: #create-volume
[Volume Expansion]: /nomad/docs/other-specifications/volume#volume-expansion
[Create CSI Volume]: #create-csi-volume
[Volume Expansion]: /nomad/docs/other-specifications/volume/csi#volume-expansion

51
website/content/docs/commands/volume/create.mdx
View File

@@ -2,15 +2,19 @@
layout: docs
page_title: 'Commands: volume create'
description: |
Create volumes with CSI plugins.
The `nomad volume create` command creates storage volumes that are either
Container Storage Interface (CSI) volumes or dynamic host volumes.
---
# Command: volume create
The `volume create` command creates external storage volumes with Nomad's
[Container Storage Interface (CSI)][csi] support. Only CSI plugins that
implement the [Controller][csi_plugins_internals] interface support this
command. The volume will also be [registered] when it is successfully created.
The `volume create` command creates storage volumes as either [Container Storage
Interface (CSI)][csi] volumes or dynamic host volumes.
The [`volume create`][] command can create dynamic host volumes if host volume
plugins are installed on the node, or CSI volumes if the CSI plugins implement
the [Controller][csi_plugins_internals] interface. The `volume create` command
automatically [registers][] the volume as well.
## Usage
@@ -18,21 +22,41 @@ command. The volume will also be [registered] when it is successfully created.
nomad volume create [options] [file]
```
The `volume create` command requires a single argument, specifying the path to
a file containing a valid [volume specification][volume_specification]. This
file will be read and the volume will be submitted to Nomad for scheduling. If
the supplied path is "-", the volume file is read from STDIN. Otherwise it is
The `volume create` command requires a single argument, specifying the path to a
file containing a valid [volume specification][volume_specification]. Nomad
reads the file and submits the volume to the server for placement. If the
supplied path is "-", the volume file is read from STDIN. Otherwise the file is
read from the file at the supplied path.
When ACLs are enabled, this command requires a token with the
`csi-write-volume` capability for the volume's namespace.
When ACLs are enabled, this command requires a token with the appropriate
capability in the volume's namespace: the `csi-write-volume` capability for CSI
volumes or `host-volume-create` for dynamic host volumes.
## General Options
@include 'general_options.mdx'
## Volume Create Options
- `-detach`: Return immediately instead of entering monitor mode for dynamic
host volumes. After creating a volume, Nomad prints the volume ID to the
screen, which you can use to examine the volume using the `volume status`
command. If `-detach` is omitted or false, the command monitors the state of
the volume until it has been fingerprinted by the client and is ready to be
scheduled. Not valid for CSI volumes.
- `-verbose`: Display full information when monitoring volume state. Used for
dynamic host volumes only. Not valid for CSI volumes.
- `-policy-override`: Sets the flag to force override any soft mandatory
Sentinel policies. Used for dynamic host volumes only. Not valid for CSI
volumes.
## Volume Specification
Refer to the [CSI Volume Specification][csi_vol_spec] and the [Dynamic Host
Volume Specification][host_vol_spec] for further information.
<!--
Redirection rules are applied server-side, so we can't redirect these sections
that used to be located in this page since they use URL fragments. Creating
@@ -63,10 +87,11 @@ the exact section.
<span id="segments" />
<span id="unused-fields" />
The volume specification is documented in the [Volume
Specification][volume_specification] page.
[csi]: https://github.com/container-storage-interface/spec
[csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins
[registers]: /nomad/docs/commands/volume/register
[registered]: /nomad/docs/commands/volume/register
[volume_specification]: /nomad/docs/other-specifications/volume
[csi_vol_spec]: /nomad/docs/other-specifications/volume/csi
[host_vol_spec]: /nomad/docs/other-specifications/volume/host

41
website/content/docs/commands/volume/delete.mdx
View File

@@ -2,16 +2,20 @@
layout: docs
page_title: 'Commands: volume delete'
description: |
Delete volumes with CSI plugins.
The `nomad volume delete` command deletes Container Storage Interface (CSI) and dynamic host storage volumes.
---
# Command: volume delete
The `volume delete` command deletes external storage volumes with Nomad's
[Container Storage Interface (CSI)][csi] support. Only CSI plugins that
implement the [Controller][csi_plugins_internals] interface support this
command. The volume will also be [deregistered] when it is successfully
deleted.
The `volume delete` command deletes storage volumes that are either [Container
Storage Interface (CSI)][csi] volumes or dynamic host volumes.
For CSI volumes, only CSI plugins that implement the
[Controller][csi_plugins_internals] interface support this command. The volume
is [deregistered][] when it is successfully deleted.
For dynamic host volumes, only volumes with plugins are deleted. Volumes without
plugins are deregistered without deleting them.
## Usage
@@ -20,24 +24,29 @@ nomad volume delete [options] [volume]
```
The `volume delete` command requires a single argument, specifying the ID of
volume to be deleted. The volume must still be [registered] with Nomad in
order to be deleted. Deleting will fail if the volume is still in use by an
volume to be deleted. The volume must still be [registered][] with Nomad in
order to be deleted. Deleting fails if the volume is still in use by an
allocation or in the process of being unpublished. If the volume no longer
exists, this command will silently return without an error.
exists, this command silently returns without an error.
When ACLs are enabled, this command requires a token with the
`csi-write-volume` capability for the volume's namespace.
When ACLs are enabled, this command requires a token with the appropriate
capability in the volume's namespace: the `csi-write-volume` capability for CSI
volumes or `host-volume-delete` for dynamic host volumes.
## General Options
@include 'general_options.mdx'
## Delete Options
- `-secret`: Secrets to pass to the plugin to delete the snapshot. Accepts
multiple flags in the form `-secret key=value`. Only available for CSI
volumes.
- `-type`: Type of volume to delete. Must be one of "csi" or "host". Defaults to
"csi".
[csi]: https://github.com/container-storage-interface/spec
[csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins
[deregistered]: /nomad/docs/commands/volume/deregister
[registered]: /nomad/docs/commands/volume/register
## Delete Options
- `-secret`: Secrets to pass to the plugin to delete the
snapshot. Accepts multiple flags in the form `-secret key=value`

5
website/content/docs/commands/volume/deregister.mdx
View File

@@ -2,7 +2,7 @@
layout: docs
page_title: 'Commands: volume deregister'
description: |
Deregister volumes with CSI plugins.
The `nomad volume deregister` command deregisters Container Storage Interface (CSI) storage volumes.
---
# Command: volume deregister
@@ -13,6 +13,9 @@ removed from Nomad's state store but not deleted from the external storage
provider. Note that deregistering a volume prevents Nomad from deleting it via
[`volume delete`] at a later time.
For dynamic host volumes, use the [`volume delete`][] command.
## Usage
```plaintext

5
website/content/docs/commands/volume/init.mdx
View File

@@ -2,7 +2,7 @@
layout: docs
page_title: 'Commands: volume init'
description: |
Generate an example volume specification.
The `nomad volume init` command generates an example storage volume specification.
---
# Command: volume init
@@ -20,6 +20,9 @@ nomad volume init
- `-json`: Create an example JSON volume specification.
- `-type`: Create an example for a specific type of volume. Use either "csi" or
"host", defaults to "csi".
## Examples
Create an example volume specification:

90
website/content/docs/commands/volume/register.mdx
View File

@@ -2,18 +2,19 @@
layout: docs
page_title: 'Commands: volume register'
description: |
Register volumes with CSI plugins.
The `nomad volume register` command registers existing Container Storage Interface (CSI) and dynamic host storage volumes.
---
# Command: volume register
The `volume register` command registers external storage volumes with Nomad's
[Container Storage Interface (CSI)][csi] support. The volume must exist on the
remote storage provider before it can be registered and used by a task.
The `volume register` command registers storage volumes as either [Container
Storage Interface (CSI)][csi] volumes or dynamic host volumes. The volume must
already exist on a node or remote storage provider before you can register
the volume for use in job task.
CSI plugins that implement the [Controller][csi_plugins_internals] interface
can be created via the [`volume create`] command, which will automatically
register the volume as well.
To instead create a volume that does not already exist, use the [`volume
create`][] command. The `volume create` command automatically registers the
volume as well.
## Usage
@@ -21,21 +22,65 @@ register the volume as well.
nomad volume register [options] [file]
```
The `volume register` command requires a single argument, specifying the path
to a file containing a valid [volume specification][volume_specification]. This
file will be read and the job will be submitted to Nomad for scheduling. If the
supplied path is "-", the job file is read from STDIN. Otherwise it is read
from the file at the supplied path.
The `volume register` command requires a single argument, specifying the path to
a file containing a valid [volume specification][volume_specification]. Nomad
reads the file and submits the volume to the server for placement. If the
supplied path is "-", Nomad reads the volume file from STDIN. Otherwise Nomad
reads the file from the supplied path.
When ACLs are enabled, this command requires a token with the
`csi-write-volume` capability for the volume's namespace.
When ACLs are enabled, this command requires a token with the appropriate
capability in the volume's namespace: the `csi-write-volume` capability for CSI
volumes or `host-volume-register` for dynamic host volumes.
## General Options
@include 'general_options.mdx'
## Volume Register Options
- `-policy-override`: Sets the flag to force override any soft mandatory
Sentinel policies. Used for dynamic host volumes only. Not valid for CSI
volumes.
## Volume Specification
Refer to the [CSI Volume Specification][csi_vol_spec] and the [Dynamic Host
Volume Specification][host_vol_spec] for further information.
The `volume register` command allows updating a volume definition. However, you
can only update some fields after the volume is registered.
For CSI volumes:
- Update the `capacity_min` and `capacity_max` fields and increase
the volume size if the CSI plugin supports it. Expansion may or may not be
possible while the volume is in use, again depending on the plugin.
Reducing volume capacity is not allowed per the CSI spec.
- Add or remove the `capability` blocks, but only if the capability is
not currently in use by a mounted volume.
- Update the `mount_options` block if the volume is not in use.
- Update the `secrets` block.
- Update the `context` block. The values for this field are typically provided
by the CSI plugin, and you should not update them unless recommended by the
CSI plugin's documentation.
For dynamic host volumes:
- Update the `capacity_min` and `capacity_max` fields. The fields are ignored
but must be consistent with the `capacity` value if set.
- Update the `capacity` field to record that you've increased the size of the
volume. Reducing volume capacity is not allowed.
- Update the `constraint` fields on `volume create`, but not after you create the volume.
- Add or remove the `capability` blocks but only if a mounted volume is not
using currently using the capability.
- Change the `node_pool` field only from empty to the node pool that matches the
`node_id` field.
You cannot update the following:
- The `name`, `type`, and `plugin_id` fields.
- The `node_id` field.
<!--
Redirection rules are applied server-side, so we can't redirect these sections
that used to be located in this page since they use URL fragments. Creating
@@ -64,23 +109,6 @@ the exact section.
<span id="segments" />
<span id="unused-fields" />
The volume specification is documented in the [Volume
Specification][volume_specification] page.
Not all fields can be updated after the volume is registered:
* The `capacity_min` and `capacity_max` fields can be updated, and may increase
the volume size if the CSI plugin supports it. Expansion may or may not be
possible while the volume is in use, again depending on the plugin.
Reducing volume capacity is not allowed per the CSI spec.
* The `capability` blocks can be added or removed, but only if the capability is
not currently in use by a mounted volume.
* The `mount_options` block can be updated if the volume is not in use.
* The `secrets` block can be updated.
* The `context` block can be updated. The values for this field are typically
provided by the CSI plugin, and should not be updated unless recommended by
the CSI plugin's documentation.
[csi]: https://github.com/container-storage-interface/spec
[csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins
[volume_specification]: /nomad/docs/other-specifications/volume

43
website/content/docs/concepts/plugins/csi.mdx → website/content/docs/concepts/plugins/storage/csi.mdx
View File

@@ -1,38 +1,29 @@
---
layout: docs
page_title: Storage Plugins
page_title: Container Storage Interface (CSI) Plugins
description: |-
Nomad's storage plugin support enables scheduling tasks with external storage volumes that conform to the Container Storage Interface (CSI), including AWS Elastic Block Storage (EBS) volumes, Google Cloud Platform (GCP) persistent disks, Ceph, and Portworx. Learn about controller, node, and monolith storage plugins, as well as storage volume lifecyle.
Nomad's Container Storage Interface (CSI) plugin support enables scheduling tasks with external storage volumes that conform to the CSI specification, including AWS Elastic Block Storage (EBS) volumes, Google Cloud Platform (GCP) persistent disks, Ceph, and Portworx. Learn about controller, node, and monolith storage plugins, as well as storage volume lifecycle and state.
---
# Storage Plugins
# Container Storage Interface (CSI) Plugins
This page provides conceptual information on Nomad's storage plugin support, which enables scheduling tasks with external storage volumes that conform to the Container Storage Interface (CSI), including AWS Elastic Block Storage (EBS) volumes, Google Cloud Platform (GCP) persistent disks, Ceph, and Portworx. Learn about controller, node, and monolith storage plugins, as well as storage volume lifecyle.
This page provides conceptual information on Nomad's storage plugin support,
which enables scheduling tasks with external storage volumes that conform to the
Container Storage Interface (CSI), including AWS Elastic Block Storage (EBS)
volumes, Google Cloud Platform (GCP) persistent disks, Ceph, and Portworx. Learn
about controller, node, and monolith storage plugins, as well as storage volume
lifecycle and state.
Nomad has built-in support for scheduling compute resources such as
CPU, memory, and networking. Nomad's storage plugin support extends
this to allow scheduling tasks with externally created storage
volumes. Storage plugins are third-party plugins that conform to the
[Container Storage Interface (CSI)][csi-spec] specification.
Storage plugins are created dynamically as Nomad jobs, unlike device
and task driver plugins that need to be installed and configured on
each client. Each dynamic plugin type has its own type-specific job
spec block; currently there is only the `csi_plugin` type. Nomad
tracks which clients have instances of a given plugin, and
communicates with plugins over a Unix domain socket that it creates
inside the plugin's tasks.
## CSI Plugins
## Introduction
Every storage vendor has its own APIs and workflows, and the industry-standard
Container Storage Interface specification unifies these APIs in a way that's
agnostic to both the storage vendor and the container orchestrator. Each storage
provider can build its own CSI plugin. Jobs can claim storage volumes from AWS
Elastic Block Storage (EBS) volumes, GCP persistent disks, Ceph, Portworx,
etc. The Nomad scheduler will be aware of volumes created by CSI plugins and
schedule workloads based on the availability of volumes on a given Nomad client
node.
[Container Storage Interface][csi-spec] specification unifies these APIs in a
way that's agnostic to both the storage vendor and the container orchestrator.
Each storage provider can build its own CSI plugin. Jobs can claim storage
volumes from AWS Elastic Block Storage (EBS) volumes, GCP persistent disks,
Ceph, Portworx, etc. The Nomad scheduler will be aware of volumes created by
CSI plugins and schedule workloads based on the availability of volumes on a
given Nomad client node.
A list of available CSI plugins can be found in the [Kubernetes CSI
documentation][csi-drivers-list]. Spec-compliant plugins should work with Nomad.

400
website/content/docs/concepts/plugins/storage/host-volumes.mdx Normal file
View File

@@ -0,0 +1,400 @@
---
layout: docs
page_title: Host volume plugins
description: |-
Learn how to implement Nomad's dynamic host volume plugins specification so that you can dynamically configure persistent storage on your client nodes. Review examples that create a directory and a Linux filesystem.
Reference plugin arguments and environment variables.
---
# Host volume plugins
This page describes the plugin specification for
[dynamic host volumes][stateful-workloads], with examples, so you can write
your own plugin to dynamically configure persistent storage on your Nomad
client nodes.
Review the examples first to get a sense of the specification in action.
A plugin can be as basic as a short shell script.
The full [specification](#specification) is after the examples,
followed by a list of [general considerations](#considerations).
## Examples
The specification is lean enough to be readily fulfilled in any language.
The following examples are written in `bash`.
The `custom-mkdir` plugin creates a directory, and `mkfs-ext4` creates a Linux
filesystem. You may imagine others, such as an NFS mount, perhaps, or various
storage infrastructure APIs.
Each example includes a minimal [volume specification][], which assumes that
you have placed the plugin in the [host volume plugin directory][plugin_dir]
and made it executable.
<Tabs>
<Tab heading="custom-mkdir">
`custom-mkdir` only creates a directory. There is a plugin built into Nomad that
does this called "mkdir", but this serves as a minimal example.
Volume specification:
```hcl
type = "host"
name = "mkdir-vol"
plugin_id = "custom-mkdir" # plugin filename
```
Plugin code:
```bash
#!/usr/bin/env bash
# set -u to error by name if an env var is not set
set -eu
# echo to stderr, because Nomad expects stdout to match the spec.
# this (and stdout) show up in Nomad agent debug logs.
stderr() {
1>&2 echo "$@"
}
get_path() {
# since delete runs `rm -rf` (frightening), check to make sure
# the path is something sensible.
if [ -z "$DHV_VOLUMES_DIR" ]; then
stderr "DHV_VOLUMES_DIR must not be empty"
exit 1
fi
if [ -z "$DHV_VOLUME_NAME" ]; then
stderr "DHV_VOLUME_NAME must not be empty"
exit 1
fi
# create and delete assign this echo output to a variable
echo "$DHV_VOLUMES_DIR/$DHV_VOLUME_NAME"
}
case "$DHV_OPERATION" in
"fingerprint")
echo '{"version": "0.0.1"}'
;;
"create")
path="$(get_path)"
stderr "creating directory: $path"
# `mkdir -p` may be run repeatedly with the same result;
# it is idempotent.
mkdir -p "$path"
# 0 bytes because plain directories are not any particular size.
printf '{"path": "%s", "bytes": 0}' "$path"
;;
"delete")
path="$(get_path)"
stderr "deleting directory: $path"
rm -rf "$path" # `rm -f` is also idempotent.
;;
*)
echo "unknown operation: '$DHV_OPERATION'"
exit 1
;;
esac
```
</Tab>
<Tab heading="mkfs-ext4">
`mkfs-ext4` creates a Linux ext4 filesystem with the `mkfs.ext4` command and
mounts it as a loopback device. Unlike `mkdir`, `mkfs` can restrict the size
of the volume.
Volume specification:
```hcl
type = "host"
name = "mkfs-vol"
plugin_id = "mkfs-ext4" # plugin filename
capacity_min = "50MB" # capacity values are required by this plugin
capacity_max = "50MB"
```
Plugin code:
```bash
#!/usr/bin/env bash
set -euo pipefail
version='0.0.1'
stderr() {
1>&2 echo "$@"
}
get_path() {
if [ -z "$DHV_VOLUMES_DIR" ]; then
stderr "DHV_VOLUMES_DIR must not be empty"
exit 1
fi
if [ -z "$DHV_VOLUME_NAME" ]; then
stderr "DHV_VOLUME_NAME must not be empty"
exit 1
fi
echo "$DHV_VOLUMES_DIR/$DHV_VOLUME_NAME"
}
is_mounted() {
mount | grep -q " $1 "
}
create_volume() {
local path="$1"
local bytes="$2"
# translate to mb for dd block size
local megs=$((bytes / 1024 / 1024)) # lazy, approximate
if [ $megs -le 0 ]; then
stderr "minimum capacity must be greater than zero."
exit 2
fi
mkdir -p "$path"
# the if statements ensure idempotency
if [ ! -f "$path.ext4" ]; then
# dd only writes to stderr, so safe to run without redirection
dd if=/dev/zero of="$path.ext4" bs=1M count="$megs"
# mkfs includes stdout, so we need to redirect that to stderr
mkfs.ext4 "$path.ext4" 1>&2
fi
if ! is_mounted "$path"; then
mount "$path.ext4" "$path"
fi
}
delete_volume() {
local path="$1"
is_mounted "$path" && umount "$path"
rm -rf "$path"
rm -f "$path.ext4"
}
case "$1" in
"fingerprint")
printf '{"version": "%s"}' "$version"
;;
"create")
path="$(get_path)"
stderr "creating volume at $path"
create_volume "$path" "$DHV_CAPACITY_MIN_BYTES"
# output what Nomad expects
bytes="$(stat --format='%s' "$path.ext4")"
printf '{"path": "%s", "bytes": %s}' "$path" "$bytes"
;;
"delete")
path="$(get_path)"
stderr "deleting volume at $path"
delete_volume "$path" ;;
*)
echo "unknown operation: $1"
exit 1 ;;
esac
```
</Tab>
</Tabs>
## Specification
A host volume plugin is registered with Nomad if it:
* Is an executable file (a script or binary)
* Is located in the [`client.host_volume_plugin_dir`][plugin_dir] directory
on Nomad client nodes
* Responds appropriately to a `fingerprint` call
### Operations
To fully manage the lifecycle of a volume, plugins must fulfill all of the
following operations:
* [fingerprint](#fingerprint)
* [create](#create)
* [delete](#delete)
Nomad passes the operation as the first positional argument to the plugin.
That and other information are passed as environment variables. Environment
variables are prefixed with `"DHV_"` (i.e. Dynamic Host Volume).
Some variables may be required for certain plugins (namely, `DHV_CAPACITY_*`
for plugins that can restrict size). Most are for plugin author convenience.
#### fingerprint
<blockquote style={{borderLeft: "solid 1px #00ca8e"}}>
Nomad calls `fingerprint` when the client agent starts (or is reloaded with a
SIGHUP) to discover valid plugins. The returned "version" is used to register
the plugin on the Nomad node for volume scheduling.
**CLI arguments:** `$1=fingerprint`
**Environment variables:**
```
DHV_OPERATION=fingerprint
```
**Expected stdout:**
```
{"version": "0.0.1"}
```
**Requirements:**
* Must complete within 5 seconds, or Nomad will kill it. It should be much
faster, as no actual work should be done.
* "version" value must be valid per the [hashicorp/go-version][go-version]
golang package.
</blockquote>
#### create
<blockquote style={{borderLeft: "solid 1px #00ca8e"}}>
Nomad calls `create` when you run [`nomad volume create`][cli-create] CLI or
use the [create API][api-create]).
You can run `create` again for the same volume if you include the volume
`id` in the volume specification.
Nomad also calls `create` when the agent starts, for each volume that was
previously created on the node, so plugins can ensure the volumes are available
after an agent restart or host reboot.
**CLI Arguments:** `$1=create`
**Environment variables:**
```
DHV_OPERATION=create
DHV_VOLUMES_DIR={directory to put the volume in}
DHV_PLUGIN_DIR={path to directory containing plugins}
DHV_NAMESPACE={volume namespace}
DHV_VOLUME_NAME={name from the volume specification}
DHV_VOLUME_ID={volume ID generated by Nomad}
DHV_NODE_ID={Nomad node ID}
DHV_NODE_POOL={Nomad node pool}
DHV_CAPACITY_MIN_BYTES={capacity_min from the volume spec, expressed in bytes}
DHV_CAPACITY_MAX_BYTES={capacity_max from the volume spec, expressed in bytes}
DHV_PARAMETERS={stringified json of parameters from the volume spec}
```
**Expected stdout:**
```
{"path": "/path/to/created/volume", "bytes": 50000000}
```
**Requirements:**
* Must complete within 60 seconds, or Nomad will kill it.
* Must create a path on disk, within `DHV_VOLUMES_DIR` or otherwise, and return
it as `"path"`. Nomad will mount this path into workloads that request the
volume, and it will also be sent to `delete` later as `DHV_CREATED_PATH`.
* Must be idempotent - running create with the same inputs should produce the
same result.
* Must be safe to run concurrently, per volume name per node.
* If the plugin fails partway through create, it must clean up after itself
and exit non-0. Nomad will not attempt to delete partial creates.
* However, if during an _initial_ create, Nomad fails to save the volume in its
own state, it will issue `delete` automatically to avoid leaving any
stray volumes on disk.
</blockquote>
#### delete
<blockquote style={{borderLeft: "solid 1px #00ca8e"}}>
Nomad calls `delete` when you run [`nomad volume delete`][cli-delete] CLI or
use the [delete API][api-delete].
**CLI Arguments:** `$1=delete`
**Environment variables:**
```
DHV_OPERATION=delete
DHV_CREATED_PATH={path that `create` returned}
DHV_VOLUMES_DIR={directory that volumes should be put in}
DHV_PLUGIN_DIR={path to directory containing plugins}
DHV_NAMESPACE={volume namespace}
DHV_VOLUME_NAME={name from the volume specification}
DHV_VOLUME_ID={volume ID generated by Nomad}
DHV_NODE_ID={Nomad node ID}
DHV_NODE_POOL={Nomad node pool}
DHV_PARAMETERS={stringified json of parameters from the volume spec}
```
**Expected stdout:** none (stdout is discarded)
**Requirements:**
* Must complete within 60 seconds, or Nomad will kill it.
* Must remove the `DHV_CREATED_PATH` that was returned by `create`,
or derive the path from other variables the same way that `create` did.
* Must be idempotent - calling delete on an already deleted volume must not
return an error.
* Must be safe to run concurrently, per volume name per node.
* Will be run after a failed `create` operation during initial volume creation.
</blockquote>
## Considerations
Plugin authors should consider these details when writing plugins.
### Execution
* The plugin is executed as the same user as the `nomad` agent (likely root).
* Plugin `stdout` and `stderr` are exposed as client agent debug logs,
so plugins should not output sensitive information.
* Nomad does not retry automatically on error. The caller of create/delete
must retry manually. The plugin may do so internally with its own retry
logic, provided it still completes within the deadline.
* Errors from `create` while restoring a volume during Nomad agent start
do not halt the client. The error will be in client logs, and the volume
is not registered as available on the node.
### Uniqueness
* Volume `name` is unique per _node_, and volume `ID` is unique per _region_.
* Only one create/delete operation at a time is executed per volume `name`
per node, and similarly by `id` on Nomad servers, but many create/delete
operations for different volume IDs may run concurrently, even on the same
node.
* We suggest placing volumes in `DHV_VOLUMES_DIR` for consistency, but it is not
required. Often `$DHV_VOLUMES_DIR/$DVH_VOLUME_NAME` will suffice, as the
volume `name` is unique per node, or `$DHV_VOLUMES_DIR/$DHV_VOLUME_ID` for
uniqueness per Nomad region.
* Plugins that write into network storage need to take care not to delete
remote/shared state by `name`, unless they know that there are no other
volumes with workloads using that name.
### Configuration
* Per-_volume_ configuration should be set in the volume specification file's
`parameters`. Per-_node_ configuration should be put in config file(s) as
described next.
* There is no mechanism built into Nomad for plugin configuration. As a
convention, we suggest placing any necessary configuration file(s) next to
the executable plugin in the plugin directory. You may use `DHV_PLUGIN_DIR`
to refer to the directory.
* If a plugin needs to retain state across operations (e.g. delete needs
some value that was generated during create), then you may store that on
the host filesystem, or some external data store of your choosing, perhaps
even Nomad variables.
[stateful-workloads]: /nomad/docs/operations/stateful-workloads#host-volumes
[plugin_dir]: /nomad/docs/configuration/client#host_volume_plugin_dir
[volume specification]: /nomad/docs/other-specifications/volume/host
[go-version]: https://pkg.go.dev/github.com/hashicorp/go-version#pkg-constants
[cli-create]: /nomad/docs/commands/volume/create
[api-create]: /nomad/api-docs/volumes#create-dynamic-host-volume
[cli-delete]: /nomad/docs/commands/volume/delete
[api-delete]: /nomad/api-docs/volumes#delete-dynamic-host-volume

32
website/content/docs/concepts/plugins/storage/index.mdx Normal file
View File

@@ -0,0 +1,32 @@
---
layout: docs
page_title: Storage Plugins
description: |-
Storage plugins help you schedule job tasks with dynamically managed storage volumes. Dynamic host volume plugins provision host volumes on client nodes. Container Storage Interface (CSI) plugins provision volumes in
third-party systems, such as AWS Elastic Block Storage (EBS) volumes, Google Cloud Platform (GCP) persistent disks, Ceph, and Portworx.
---
# Storage Plugins
Nomad has built-in support for scheduling compute resources such as
CPU, memory, and networking. Nomad's storage plugin support extends
this to allow scheduling tasks with dynamically managed storage volumes.
Choose between two types of storage plugins:
* [Dynamic host volume (DHV) plugins][dhv] provision host volumes on client
nodes, and jobs request placement on any node that has a particular host
volume by name.
* DHV brings allocations to volumes.
* Install DHV plugins on Nomad nodes, similar to task drivers. Creating a
host volume ties it to the node until the volume is deleted.
* [Container Storage Interface (CSI) plugins][csi] provision volumes in
third-party systems, such as cloud vendors. Nomad calls on them to make
the volume available on any node that Nomad chooses to place an allocation.
* CSI brings volumes to allocations.
* Deploy CSI plugins as Nomad jobs, often two jobs per plugin. Nomad tracks
which clients have instances of a given plugin, so it can call on the
plugin to mount the volume.
[dhv]: /nomad/docs/concepts/plugins/storage/host-volumes
[csi]: /nomad/docs/concepts/plugins/storage/csi

12
website/content/docs/configuration/client.mdx
View File

@@ -633,11 +633,14 @@ refer to the [drivers documentation](/nomad/docs/drivers).
### `host_volume` Block
The `host_volume` block is used to make volumes available to jobs.
The `host_volume` block is used to make volumes available to jobs. You can also
configure [dynamic host volumes][] via the [`volume create`][] or [`volume
register`][] commands.
The key of the block corresponds to the name of the volume for use in the
`source` parameter of a `"host"` type [`volume`](/nomad/docs/job-specification/volume)
and ACLs.
`source` parameter of a `"host"` type
[`volume`](/nomad/docs/job-specification/volume) and ACLs. A host volume in the
configuration cannot have the same name as a dynamic host volume on the same node.
```hcl
client {
@@ -827,3 +830,6 @@ client {
[`TimeoutStopSec`]: https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStopSec=
[top_level_data_dir]: /nomad/docs/configuration#data_dir
[unveil]: /nomad/docs/concepts/plugins/task-drivers#fsisolation-unveil
[dynamic host volumes]: /nomad/docs/other-specifications/volume/host
[`volume create`]: /nomad/docs/commands/volume/create
[`volume register`]: /nomad/docs/commands/volume/register

72
website/content/docs/enterprise/sentinel.mdx
View File

@@ -34,7 +34,10 @@ taken to avoid adversely impacting performance.
In each scope, there are different objects made available for introspection,
such a job being submitted. Policies can inspect these objects to apply
fine-grained policies.
fine-grained policies. Nomad supports a `submit-job` scope for registering jobs,
and a `submit-host-volume` scope for creating or updating dynamic host volumes.
### submit-job scope
The following top-level objects are available to policies in the `submit-job`
scope automatically, without an explicit import.
@@ -42,7 +45,7 @@ scope automatically, without an explicit import.
* `job`: the submitted job. This is a [Sentinel Job
Object](#sentinel-job-objects).
* `existing_job`: the previous version of the job. If `job_exists` is true, this
will always be non-nil. This is also a [Sentinel Job
is always non-nil. This is also a [Sentinel Job
Object](#sentinel-job-objects).
* `job_exists`: a boolean field that indicates that a previous version of the
job exists.
@@ -51,11 +54,31 @@ scope automatically, without an explicit import.
* `namespace`: the namespace the job is in. This is a [Sentinel Nomad Namespace
Object](#sentinel-namespace-objects).
### submit-host-volume scope
The following top-level objects are available to policies in the
`submit-host-volume` scope automatically, without an explicit import.
* `volume`: the submitted volume. This is a [Sentinel Dynamic Host Volume
Object](#sentinel-dynamic-host-volume-objects).
* `existing_volume`: the previous version of the volume. If `volume_exists` is
true, this is always non-nil. This is also a [Sentinel Dynamic Host
Volume Object](#sentinel-dynamic-host-volume-objects). `volume_exists`: a
boolean field that indicates that a previous version of the volume exists.
* `nomad_acl_token`: the ACL token the job was submitted with. This is a
[Sentinel Nomad ACL Token Object](#sentinel-acl-token-objects).
* `namespace`: the namespace the job is in. This is a [Sentinel Nomad Namespace
Object](#sentinel-namespace-objects).
* `node`: the node the volume has been placed on. This is a [Sentinel Node
Object](#sentinel-node-objects).
* `node_pool`: the node pool of the node the volume has been placed on. This is
a [Sentinel Node Pool Object](#sentinel-node-pool-objects).
Sentinel convention for identifiers is lower case and separated by underscores.
All fields on an object are accessed by the same name, converted to lower case
and separating camel case to underscores.
### Sentinel Job Objects
## Sentinel Job Objects
The `job` and `existing_job` objects map to the [JSON job specification][], with
the fields converted to the Sentinel convention. Here are some examples:
@@ -68,8 +91,7 @@ the fields converted to the Sentinel convention. Here are some examples:
| `job.TaskGroups` | `job.task_groups` |
| `job.TaskGroups[0].EphemeralDisk.SizeMB` | `job.task_groups[0].ephemeral_disk.size_mb` |
### Sentinel ACL Token Objects
## Sentinel ACL Token Objects
The `nomad_acl_token` object maps to the [ACL token][], with the fields
converted to the Sentinel convention. Here are some examples:
@@ -81,7 +103,7 @@ converted to the Sentinel convention. Here are some examples:
Note that the `SecretID` field is always redacted to prevent credential leaks.
### Sentinel Namespace Objects
## Sentinel Namespace Objects
The `namespace` object maps to the [Namespace][], with the fields converted to
the Sentinel convention. Here are some examples:
@@ -91,9 +113,43 @@ the Sentinel convention. Here are some examples:
| `namespace.Description` | `namespace.description` |
| `namespace.NodePoolConfiguration.Allowed[0]` | `namespace.node_pool_configuration.allowed[0]` |
## Sentinel Node Objects
The `node` object maps to the [Node][], with the fields converted to the
Sentinel convention. Here are some examples:
| Node Field | Sentinel Accessor |
|--------------------|--------------------|
| `node.Class` | `node.class` |
| `node.Meta["foo"]` | `node.meta["foo"]` |
## Sentinel Node Pool Objects
The `node_pool` object maps to the [Node Pool][], with the fields converted to
the Sentinel convention. Here are some examples:
| Node Pool Field | Sentinel Accessor |
|-------------------------------------------------------|---------------------------------------------------------|
| `node_pool.Description` | `node_pool.description` |
| `node_pool.SchedulerConfiguration.SchedulerAlgorithm` | `node_pool.scheduler_configuration.scheduler_algorithm` |
## Sentinel Dynamic Host Volume Objects
The `volume` object maps to the [Dynamic Host Volume][], with the fields
converted to the Sentinel convention. Here are some examples:
| Dynamic Host Volume Field | Sentinel Accessor |
|----------------------------------------------|------------------------------------------------|
| `volume.Name` | `volume.name` |
| `volume.RequestedCapabilities[0].AccessMode` | `volume.requested_capabilities[0].access_mode` |
[Nomad Sentinel Tutorial]: /nomad/tutorials/governance-and-policy/sentinel
[`nomad sentinel`]: /nomad/docs/commands/sentinel
[sentinel]: https://docs.hashicorp.com/sentinel
[JSON job specification]: /nomad/api-docs/json-jobs
[ACL token]: https://github.com/hashicorp/nomad/blob/v1.7.0-rc.1/nomad/structs/structs.go#L12991-L13020
[Namespace]: https://github.com/hashicorp/nomad/blob/v1.7.0-rc.1/nomad/structs/structs.go#L5306-L5338
[ACL token]: https://github.com/hashicorp/nomad/blob/v1.9.4/nomad/structs/structs.go#L13502-L13531
[Namespace]: https://github.com/hashicorp/nomad/blob/v1.9.4/nomad/structs/structs.go#L5578-L5610
[Node]: https://github.com/hashicorp/nomad/blob/v1.9.4/nomad/structs/structs.go#L2086-L2210
[Node Pool]: https://github.com/hashicorp/nomad/blob/v1.9.4/nomad/structs/node_pool.go#L46-L68
[Dynamic Host Volume]: https://github.com/hashicorp/nomad/blob/main/nomad/structs/host_volumes.go#L18-L87

78
website/content/docs/job-specification/volume.mdx
View File

@@ -2,9 +2,7 @@
layout: docs
page_title: volume Block - Job Specification
description: >-
The "volume" block allows the group to specify that it requires a given
volume from the cluster. Nomad will automatically handle ensuring that the
volume is available and mounted into the task.
Configure storage volumes in the "volume" block of a Nomad job specification. Specify dynamic host or Container Storage Interface (CSI) volume type, node access mode, filesystem or block device attachment mode, and mount options. Enable read only access and mounting a unique volume per node. Learn about volume interpolation.
---
# `volume` Block
@@ -50,19 +48,22 @@ job "docs" {
```
The Nomad server will ensure that the allocations are only scheduled
on hosts that have a set of volumes that meet the criteria specified
in the `volume` blocks. These may be [host volumes][host_volume]
configured on the client, or [CSI volumes][csi_volume] dynamically
mounted by [CSI plugins][csi_plugin].
The Nomad server ensures that the allocations are only scheduled on hosts
that have a set of volumes that meet the criteria specified in the `volume`
blocks. These may be [static host volumes][host_volume] configured on the
client, [dynamic host volumes][] created via [`volume create`][] or [`volume
register`][], or [CSI volumes][csi_volume] dynamically mounted by [CSI
plugins][csi_plugin].
The Nomad client will make the volumes available to tasks according to
The Nomad client makes the volumes available to tasks according to
the [volume_mount][volume_mount] block in the `task` configuration.
## `volume` Parameters
- `type` `(string: "")` - Specifies the type of a given volume. The
valid volume types are `"host"` and `"csi"`.
- `type` `(string: "")` - Specifies the type of a given volume. The valid volume
types are `"host"` and `"csi"`. Setting the `"host"` value can request either
statically configured host volumes or dynamic host volumes, depending on what
is available on a given node.
- `source` `(string: <required>)` - The name of the volume to
request. When using `host_volume`'s this should match the published
@@ -85,22 +86,48 @@ the [volume_mount][volume_mount] block in the `task` configuration.
The `per_alloc` field cannot be true for system jobs, sysbatch jobs, or jobs
that use canaries.
The following fields are only valid for volumes with `type = "csi"`:
The following fields are only valid for volumes with `type = "csi"` or dynamic
host volumes with `type = "host"`:
- `access_mode` `(string: <required>)` - Defines whether a volume should be
- `access_mode` `(string)` - Defines whether a volume should be
available concurrently. The `access_mode` and `attachment_mode` together
must exactly match one of the volume's `capability` blocks. Can be one of
`"single-node-reader-only"`, `"single-node-writer"`,
`"multi-node-reader-only"`, `"multi-node-single-writer"`, or
`"multi-node-multi-writer"`. Most CSI plugins support only single-node
modes. Consult the documentation of the storage provider and CSI plugin.
must exactly match one of the volume's `capability` blocks.
- `attachment_mode` `(string: <required>)` - The storage API that will be used
by the volume. The `access_mode` and `attachment_mode` together must exactly
match one of the volume's `capability` blocks. Most storage providers will
support `"file-system"`, to mount volumes using the CSI filesystem API. Some
storage providers will support `"block-device"`, which will mount the volume
with the CSI block device API within the container.
- For CSI volumes the `access_mode` is required. Can be one of the following:
- `"single-node-reader-only"`
- `"single-node-writer"`
- `"multi-node-reader-only"`
- `"multi-node-single-writer"`
- `"multi-node-multi-writer"`
Most CSI plugins support only single-node modes.
Consult the documentation of the storage provider and CSI plugin.
- For dynamic host volumes the `access_mode` is optional. Can be one of the following:
- `"single-node-writer"`
- `"single-node-reader-only"`
- `"single-node-single-writer"`
- `"single-node-multi-writer"`
Defaults to `single-node-writer` unless `read_only = true`, in which case it
defaults to `single-node-reader-only`.
- `attachment_mode` `(string)` - The storage API used by the
volume. One of `"file-system"` or `"block-device"`. The `access_mode` and
`attachment_mode` together must exactly match one of the volume's `capability`
blocks.
- For CSI volumes the `attachment_mode` field is required. Most storage
providers support `"file-system"`, to mount volumes using the CSI
filesystem API. Some storage providers support `"block-device"`, which
mounts the volume with the CSI block device API within the container.
- For dynamic host volumes the `attachment_mode` field is optional and
defaults to `"file-system"`.
The following fields are only valid for volumes with `type = "csi"`:
- `mount_options` - Options for mounting CSI volumes that have the
`file-system` [attachment mode]. These options override the `mount_options`
@@ -207,3 +234,6 @@ ID Node ID Task Group Version Desired Status Created Modified
[csi_volume]: /nomad/docs/commands/volume/register
[attachment mode]: /nomad/docs/commands/volume/register#attachment_mode
[volume registration]: /nomad/docs/commands/volume/register#mount_options
[dynamic host volumes]: /nomad/docs/other-specifications/volume/host
[`volume create`]: /nomad/docs/commands/volume/create
[`volume register`]: /nomad/docs/commands/volume/register

472
website/content/docs/operations/metrics-reference.mdx
View File

@@ -263,241 +263,243 @@ Job status metrics are emitted by the Nomad leader server.
The following table includes metrics for overall cluster health in addition to
those listed in [Key Metrics](#key-metrics) above.
| Metric | Description | Unit | Type | Labels |
|------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|---------|---------------------------------------------------------|
| `nomad.memberlist.gossip` | Time elapsed to broadcast gossip messages | Milliseconds | Timer | host |
| `nomad.nomad.acl.bootstrap` | Time elapsed for `ACL.Bootstrap` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.delete_policies` | Time elapsed for `ACL.DeletePolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.delete_tokens` | Time elapsed for `ACL.DeleteTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_policies` | Time elapsed for `ACL.GetPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_policy` | Time elapsed for `ACL.GetPolicy` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_token` | Time elapsed for `ACL.GetToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_tokens` | Time elapsed for `ACL.GetTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.list_policies` | Time elapsed for `ACL.ListPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.list_tokens` | Time elapsed for `ACL.ListTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.resolve_token` | Time elapsed for `ACL.ResolveToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.upsert_policies` | Time elapsed for `ACL.UpsertPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.upsert_tokens` | Time elapsed for `ACL.UpsertTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.exec` | Time elapsed to establish alloc exec | Milliseconds | Timer | host |
| `nomad.nomad.alloc.get_alloc` | Time elapsed for `Alloc.GetAlloc` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.get_allocs` | Time elapsed for `Alloc.GetAllocs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.list` | Time elapsed for `Alloc.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.stop` | Time elapsed for `Alloc.Stop` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.update_desired_transition` | Time elapsed for `Alloc.UpdateDesiredTransition` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.blocked_evals.cpu` | Amount of CPU shares requested by blocked evals | Integer | Gauge | datacenter, host, node_class |
| `nomad.nomad.blocked_evals.memory` | Amount of memory requested by blocked evals | Integer | Gauge | datacenter, host, node_class |
| `nomad.nomad.blocked_evals.job.cpu` | Amount of CPU shares requested by blocked evals of a job | Integer | Gauge | host, job, namespace |
| `nomad.nomad.blocked_evals.job.memory` | Amount of memory requested by blocked evals of a job | Integer | Gauge | host, job, namespace |
| `nomad.nomad.blocked_evals.total_blocked` | Count of evals in the blocked state for any reason (cluster resource exhaustion or quota limits) | Integer | Gauge | host |
| `nomad.nomad.blocked_evals.total_escaped` | Count of evals that have escaped computed node classes. This indicates a scheduler optimization was skipped and is not usually a source of concern. | Integer | Gauge | host |
| `nomad.nomad.blocked_evals.total_quota_limit` | Count of blocked evals due to quota limits (the resources for these jobs are *not* counted in other blocked_evals metrics, except for `total_blocked`) | Integer | Gauge | host |
| `nomad.nomad.broker.batch_ready` | Count of batch evals ready to be scheduled | Integer | Gauge | host |
| `nomad.nomad.broker.batch_unacked` | Count of unacknowledged batch evals | Integer | Gauge | host |
| `nomad.nomad.broker.eval_waiting` | Time elapsed with evaluation waiting to be enqueued | Milliseconds | Gauge | eval_id, job, namespace |
| `nomad.nomad.broker.process_time` | Time elapsed while the evaluation was dequeued and finished processing. This metric is only valid within a single term | ms / Evaluation Process | Timer | host, job, namespace, eval_type, triggered_by |
| `nomad.nomad.broker.response_time` | Time elapsed from when the evaluation was last enqueued and finished processing. This metric is only valid within a single term | ms / Evaluation Response | Timer | host, job, namespace, eval_type, triggered_by |
| `nomad.nomad.broker.service_ready` | Count of service evals ready to be scheduled | Integer | Gauge | host |
| `nomad.nomad.broker.service_unacked` | Count of unacknowledged service evals | Integer | Gauge | host |
| `nomad.nomad.broker.system_ready` | Count of system evals ready to be scheduled | Integer | Gauge | host |
| `nomad.nomad.broker.system_unacked` | Count of unacknowledged system evals | Integer | Gauge | host |
| `nomad.nomad.broker.total_ready` | Count of evals in the ready state | Integer | Gauge | host |
| `nomad.nomad.broker.total_waiting` | Count of evals waiting to be enqueued | Integer | Gauge | host |
| `nomad.nomad.broker.wait_time` | Time elapsed while the evaluation was ready to be processed and waiting to be dequeued. This metric is only valid within a single term | ms / Evaluation Wait | Timer | host, job, namespace, eval_type, triggered_by |
| `nomad.nomad.client.batch_deregister` | Time elapsed for `Node.BatchDeregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.deregister` | Time elapsed for `Node.Deregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.derive_si_token` | Time elapsed for `Node.DeriveSIToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.derive_vault_token` | Time elapsed for `Node.DeriveVaultToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.emit_events` | Time elapsed for `Node.EmitEvents` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.evaluate` | Time elapsed for `Node.Evaluate` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.get_allocs` | Time elapsed for `Node.GetAllocs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.get_client_allocs` | Time elapsed for `Node.GetClientAllocs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.get_node` | Time elapsed for `Node.GetNode` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.list` | Time elapsed for `Node.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.register` | Time elapsed for `Node.Register` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.stats` | Time elapsed for `Client.Stats` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_alloc` | Time elapsed for `Node.UpdateAlloc` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_drain` | Time elapsed for `Node.UpdateDrain` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_eligibility` | Time elapsed for `Node.UpdateEligibility` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_status` | Time elapsed for `Node.UpdateStatus` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.garbage_collect_all` | Time elapsed for `ClientAllocations.GarbageCollectAll` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.garbage_collect` | Time elapsed for `ClientAllocations.GarbageCollect` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.restart` | Time elapsed for `ClientAllocations.Restart` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.signal` | Time elapsed for `ClientAllocations.Signal` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.stats` | Time elapsed for `ClientAllocations.Stats` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_controller.attach_volume` | Time elapsed for `Controller.AttachVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_controller.detach_volume` | Time elapsed for `Controller.DetachVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_controller.validate_volume` | Time elapsed for `Controller.ValidateVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_node.detach_volume` | Time elapsed for `Node.DetachVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.allocations` | Time elapsed for `Deployment.Allocations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.cancel` | Time elapsed for `Deployment.Cancel` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.fail` | Time elapsed for `Deployment.Fail` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.get_deployment` | Time elapsed for `Deployment.GetDeployment` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.list` | Time elapsed for `Deployment.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.pause` | Time elapsed for `Deployment.Pause` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.promote` | Time elapsed for `Deployment.Promote` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.reap` | Time elapsed for `Deployment.Reap` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.run` | Time elapsed for `Deployment.Run` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.set_alloc_health` | Time elapsed for `Deployment.SetAllocHealth` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.unblock` | Time elapsed for `Deployment.Unblock` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.ack` | Time elapsed for `Eval.Ack` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.allocations` | Time elapsed for `Eval.Allocations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.create` | Time elapsed for `Eval.Create` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.dequeue` | Time elapsed for `Eval.Dequeue` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.get_eval` | Time elapsed for `Eval.GetEval` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.list` | Time elapsed for `Eval.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.nack` | Time elapsed for `Eval.Nack` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.reap` | Time elapsed for `Eval.Reap` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.reblock` | Time elapsed for `Eval.Reblock` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.update` | Time elapsed for `Eval.Update` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.file_system.list` | Time elapsed for `FileSystem.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.file_system.logs` | Time elapsed to establish `FileSystem.Logs` RPC | Milliseconds | Timer | host |
| `nomad.nomad.file_system.stat` | Time elapsed for `FileSystem.Stat` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.file_system.stream` | Time elapsed to establish `FileSystem.Stream` RPC | Milliseconds | Timer | host |
| `nomad.nomad.fsm.alloc_client_update` | Time elapsed to apply `AllocClientUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.alloc_update_desired_transition` | Time elapsed to apply `AllocUpdateDesiredTransition` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_policy_delete` | Time elapsed to apply `ApplyACLPolicyDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_policy_upsert` | Time elapsed to apply `ApplyACLPolicyUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_token_bootstrap` | Time elapsed to apply `ApplyACLTokenBootstrap` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_token_delete` | Time elapsed to apply `ApplyACLTokenDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_token_upsert` | Time elapsed to apply `ApplyACLTokenUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_plugin_delete` | Time elapsed to apply `ApplyCSIPluginDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_batch_claim` | Time elapsed to apply `ApplyCSIVolumeBatchClaim` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_claim` | Time elapsed to apply `ApplyCSIVolumeClaim` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_deregister` | Time elapsed to apply `ApplyCSIVolumeDeregister` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_register` | Time elapsed to apply `ApplyCSIVolumeRegister` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_alloc_health` | Time elapsed to apply `ApplyDeploymentAllocHealth` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_delete` | Time elapsed to apply `ApplyDeploymentDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_promotion` | Time elapsed to apply `ApplyDeploymentPromotion` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_status_update` | Time elapsed to apply `ApplyDeploymentStatusUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_job_stability` | Time elapsed to apply `ApplyJobStability` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_job_version_tag` | Time elapsed to apply `ApplyJobVersionTag` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_namespace_delete` | Time elapsed to apply `ApplyNamespaceDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_namespace_upsert` | Time elapsed to apply `ApplyNamespaceUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_node_pool_upsert` | Time elapsed to apply `ApplyNodePoolUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_node_pool_delete` | Time elapsed to apply `ApplyNodePoolDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_plan_results` | Time elapsed to apply `ApplyPlanResults` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_scheduler_config` | Time elapsed to apply `ApplySchedulerConfig` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.autopilot` | Time elapsed to apply `Autopilot` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.batch_deregister_job` | Time elapsed to apply `BatchDeregisterJob` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.batch_deregister_node` | Time elapsed to apply `BatchDeregisterNode` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.batch_node_drain_update` | Time elapsed to apply `BatchNodeDrainUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.cluster_meta` | Time elapsed to apply `ClusterMeta` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.delete_eval` | Time elapsed to apply `DeleteEval` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_job` | Time elapsed to apply `DeregisterJob` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_node` | Time elapsed to apply `DeregisterNode` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_si_accessor` | Time elapsed to apply `DeregisterSITokenAccessor` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_vault_accessor` | Time elapsed to apply `DeregisterVaultAccessor` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.node_drain_update` | Time elapsed to apply `NodeDrainUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.node_eligibility_update` | Time elapsed to apply `NodeEligibilityUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.node_status_update` | Time elapsed to apply `NodeStatusUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.persist` | Time elapsed to apply `Persist` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.register_job` | Time elapsed to apply `RegisterJob` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.register_node` | Time elapsed to apply `RegisterNode` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.update_eval` | Time elapsed to apply `UpdateEval` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_node_events` | Time elapsed to apply `UpsertNodeEvents` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_scaling_event` | Time elapsed to apply `UpsertScalingEvent` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_si_accessor` | Time elapsed to apply `UpsertSITokenAccessors` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_vault_accessor` | Time elapsed to apply `UpsertVaultAccessor` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.job.allocations` | Time elapsed for `Job.Allocations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.batch_deregister` | Time elapsed for `Job.BatchDeregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.deployments` | Time elapsed for `Job.Deployments` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.deregister` | Time elapsed for `Job.Deregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.dispatch` | Time elapsed for `Job.Dispatch` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.evaluate` | Time elapsed for `Job.Evaluate` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.evaluations` | Time elapsed for `Job.Evaluations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.get_job_versions` | Time elapsed for `Job.GetJobVersions` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.get_job` | Time elapsed for `Job.GetJob` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.latest_deployment` | Time elapsed for `Job.LatestDeployment` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.list` | Time elapsed for `Job.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.plan` | Time elapsed for `Job.Plan` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.register` | Time elapsed for `Job.Register` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.revert` | Time elapsed for `Job.Revert` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.scale_status` | Time elapsed for `Job.ScaleStatus` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.scale` | Time elapsed for `Job.Scale` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.stable` | Time elapsed for `Job.Stable` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.validate` | Time elapsed for `Job.Validate` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job_summary.get_job_summary` | Time elapsed for `Job.Timer` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.leader.barrier` | Time elapsed to establish a raft barrier during leader transition | Milliseconds | Timer | host |
| `nomad.nomad.leader.reconcileMember` | Time elapsed to reconcile a serf peer with state store | Milliseconds | Timer | host |
| `nomad.nomad.leader.reconcile` | Time elapsed to reconcile all serf peers with state store | Milliseconds | Timer | host |
| `nomad.nomad.namespace.delete_namespaces` | Time elapsed for `Namespace.DeleteNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.get_namespace` | Time elapsed for `Namespace.GetNamespace` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.get_namespaces` | Time elapsed for `Namespace.GetNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.list_namespace` | Time elapsed for `Namespace.ListNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.upsert_namespaces` | Time elapsed for `Namespace.UpsertNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.list` | Time elapsed for `NodePool.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.list_jobs` | Time elapsed for `NodePool.ListJobs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.list_nodes` | Time elapsed for `NodePool.ListNodes` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.get_node_pool` | Time elapsed for `NodePool.GetNodePool` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.upsert_node_pools` | Time elapsed for `NodePool.UpsertNodePools` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.delete_node_pools` | Time elapsed for `NodePool.DeleteNodePools` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.periodic.force` | Time elapsed for `Periodic.Force` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plan.apply` | Time elapsed to apply a plan | Milliseconds | Timer | host |
| `nomad.nomad.plan.evaluate` | Time elapsed to evaluate a plan | Milliseconds | Timer | host |
| `nomad.nomad.plan.node_rejected` | Number of times a node has had a plan rejected | Integer | Counter | host, node_id |
| `nomad.nomad.plan.rejection_tracker.node_score` | Number of times a node has had a plan rejected within the tracker window | Integer | Gauge | host, node_id |
| `nomad.nomad.plan.queue_depth` | Count of evals in the plan queue | Integer | Gauge | host |
| `nomad.nomad.plan.submit` | Time elapsed for `Plan.Submit` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plan.wait_for_index` | Time elapsed that planner waits for the raft index of the plan to be processed | Milliseconds | Timer | host |
| `nomad.nomad.plugin.delete` | Time elapsed for `CSIPlugin.Delete` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plugin.get` | Time elapsed for `CSIPlugin.Get` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plugin.list` | Time elapsed for `CSIPlugin.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.quota.utilization.cpu` | Utilization of the CPU quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.quota.utilization.cores` | Utilization of the CPU Cores quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.quota.utilization.memory_mb` | Utilization of the Memory MB quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.scaling.get_policy` | Time elapsed for `Scaling.GetPolicy` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.scaling.list_policies` | Time elapsed for `Scaling.ListPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.search.prefix_search` | Time elapsed for `Search.PrefixSearch` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.vault.create_token` | Time elapsed to create Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.distributed_tokens_revoked` | Count of revoked tokens | Integer | Gauge | host |
| `nomad.nomad.vault.lookup_token` | Time elapsed to lookup Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.renew_failed` | Count of failed attempts to renew Vault token | Integer | Gauge | host |
| `nomad.nomad.vault.renew` | Time elapsed to renew Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.revoke_tokens` | Time elapsed to revoke Vault tokens | Milliseconds | Timer | host |
| `nomad.nomad.vault.token_last_renewal` | Time since last successful Vault token renewal | Milliseconds | Timer | host |
| `nomad.nomad.vault.token_next_renewal` | Time until next Vault token renewal attempt | Milliseconds | Timer | host |
| `nomad.nomad.vault.token_ttl` | Time to live for Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.undistributed_tokens_abandoned` | Count of abandoned tokens | Integer | Gauge | host |
| `nomad.nomad.volume.claim` | Time elapsed for `CSIVolume.Claim` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.deregister` | Time elapsed for `CSIVolume.Deregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.get` | Time elapsed for `CSIVolume.Get` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.list` | Time elapsed for `CSIVolume.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.register` | Time elapsed for `CSIVolume.Register` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.unpublish` | Time elapsed for `CSIVolume.Unpublish` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.worker.create_eval` | Time elapsed for worker to create an eval | Milliseconds | Timer | host |
| `nomad.nomad.worker.dequeue_eval` | Time elapsed for worker to dequeue an eval | Milliseconds | Timer | host |
| `nomad.nomad.worker.invoke_scheduler.<type>` | Time elapsed for worker to invoke the scheduler of type `<type>` | Milliseconds | Timer | host |
| `nomad.nomad.worker.send_ack` | Time elapsed for worker to send acknowledgement | Milliseconds | Timer | host |
| `nomad.nomad.worker.submit_plan` | Time elapsed for worker to submit plan | Milliseconds | Timer | host |
| `nomad.nomad.worker.update_eval` | Time elapsed for worker to submit updated eval | Milliseconds | Timer | host |
| `nomad.nomad.worker.wait_for_index` | Time elapsed that worker waits for the raft index of the eval to be processed | Milliseconds | Timer | host |
| `nomad.raft.appliedIndex` | Current index applied to FSM | Integer | Gauge | host |
| `nomad.raft.barrier` | Count of blocking raft API calls | Integer | Counter | host |
| `nomad.raft.commitNumLogs` | Count of logs enqueued | Integer | Gauge | host |
| `nomad.raft.commitTime` | Time elapsed to commit writes | Milliseconds | Timer | host |
| `nomad.raft.fsm.apply` | Time elapsed to apply write to FSM | Milliseconds | Timer | host |
| `nomad.raft.fsm.enqueue` | Time elapsed to enqueue write to FSM | Milliseconds | Timer | host |
| `nomad.raft.lastIndex` | Most recent index seen | Integer | Gauge | host |
| `nomad.raft.leader.dispatchLog` | Time elapsed to write log, mark in flight, and start replication | Milliseconds | Timer | host |
| `nomad.raft.leader.dispatchNumLogs` | Count of logs dispatched | Integer | Gauge | host |
| `nomad.raft.replication.appendEntries` | Raft transaction commit time | ms / Raft Log Append | Timer | |
| `nomad.raft.state.candidate` | Count of entering candidate state | Integer | Gauge | host |
| `nomad.raft.state.follower` | Count of entering follower state | Integer | Gauge | host |
| `nomad.raft.state.leader` | Count of entering leader state | Integer | Gauge | host |
| `nomad.raft.transition.heartbeat_timeout` | Count of failing to heartbeat and starting election | Integer | Gauge | host |
| `nomad.raft.transition.leader_lease_timeout` | Count of stepping down as leader after losing quorum | Integer | Gauge | host |
| `nomad.runtime.free_count` | Count of objects freed from heap by go runtime GC | Integer | Gauge | host |
| `nomad.runtime.gc_pause_ns` | Go runtime GC pause times | Nanoseconds | Summary | host |
| `nomad.runtime.sys_bytes` | Go runtime GC metadata size | # of bytes | Gauge | host |
| `nomad.runtime.total_gc_pause_ns` | Total elapsed go runtime GC pause times | Nanoseconds | Gauge | host |
| `nomad.runtime.total_gc_runs` | Count of go runtime GC runs | Integer | Gauge | host |
| `nomad.serf.queue.Event` | Count of memberlist events received | Integer | Summary | host |
| `nomad.serf.queue.Intent` | Count of memberlist changes | Integer | Summary | host |
| `nomad.serf.queue.Query` | Count of memberlist queries | Integer | Summary | host |
| `nomad.scheduler.allocs.rescheduled.attempted` | Count of attempts to reschedule an allocation | Integer | Count | alloc_id, job, namespace, task_group |
| `nomad.scheduler.allocs.rescheduled.limit` | Maximum number of attempts to reschedule an allocation | Integer | Count | alloc_id, job, namespace, task_group |
| `nomad.scheduler.allocs.rescheduled.wait_until` | Time that a rescheduled allocation will be delayed | Float | Gauge | alloc_id, job, namespace, task_group, follow_up_eval_id |
| `nomad.state.snapshotIndex` | Current snapshot index | Integer | Gauge | host |
| Metric | Description | Unit | Type | Labels |
|---------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|---------|---------------------------------------------------------|
| `nomad.memberlist.gossip` | Time elapsed to broadcast gossip messages | Milliseconds | Timer | host |
| `nomad.nomad.acl.bootstrap` | Time elapsed for `ACL.Bootstrap` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.delete_policies` | Time elapsed for `ACL.DeletePolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.delete_tokens` | Time elapsed for `ACL.DeleteTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_policies` | Time elapsed for `ACL.GetPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_policy` | Time elapsed for `ACL.GetPolicy` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_token` | Time elapsed for `ACL.GetToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.get_tokens` | Time elapsed for `ACL.GetTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.list_policies` | Time elapsed for `ACL.ListPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.list_tokens` | Time elapsed for `ACL.ListTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.resolve_token` | Time elapsed for `ACL.ResolveToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.upsert_policies` | Time elapsed for `ACL.UpsertPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.acl.upsert_tokens` | Time elapsed for `ACL.UpsertTokens` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.exec` | Time elapsed to establish alloc exec | Milliseconds | Timer | host |
| `nomad.nomad.alloc.get_alloc` | Time elapsed for `Alloc.GetAlloc` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.get_allocs` | Time elapsed for `Alloc.GetAllocs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.list` | Time elapsed for `Alloc.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.stop` | Time elapsed for `Alloc.Stop` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.alloc.update_desired_transition` | Time elapsed for `Alloc.UpdateDesiredTransition` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.blocked_evals.cpu` | Amount of CPU shares requested by blocked evals | Integer | Gauge | datacenter, host, node_class |
| `nomad.nomad.blocked_evals.memory` | Amount of memory requested by blocked evals | Integer | Gauge | datacenter, host, node_class |
| `nomad.nomad.blocked_evals.job.cpu` | Amount of CPU shares requested by blocked evals of a job | Integer | Gauge | host, job, namespace |
| `nomad.nomad.blocked_evals.job.memory` | Amount of memory requested by blocked evals of a job | Integer | Gauge | host, job, namespace |
| `nomad.nomad.blocked_evals.total_blocked` | Count of evals in the blocked state for any reason (cluster resource exhaustion or quota limits) | Integer | Gauge | host |
| `nomad.nomad.blocked_evals.total_escaped` | Count of evals that have escaped computed node classes. This indicates a scheduler optimization was skipped and is not usually a source of concern. | Integer | Gauge | host |
| `nomad.nomad.blocked_evals.total_quota_limit` | Count of blocked evals due to quota limits (the resources for these jobs are *not* counted in other blocked_evals metrics, except for `total_blocked`) | Integer | Gauge | host |
| `nomad.nomad.broker.batch_ready` | Count of batch evals ready to be scheduled | Integer | Gauge | host |
| `nomad.nomad.broker.batch_unacked` | Count of unacknowledged batch evals | Integer | Gauge | host |
| `nomad.nomad.broker.eval_waiting` | Time elapsed with evaluation waiting to be enqueued | Milliseconds | Gauge | eval_id, job, namespace |
| `nomad.nomad.broker.process_time` | Time elapsed while the evaluation was dequeued and finished processing. This metric is only valid within a single term | ms / Evaluation Process | Timer | host, job, namespace, eval_type, triggered_by |
| `nomad.nomad.broker.response_time` | Time elapsed from when the evaluation was last enqueued and finished processing. This metric is only valid within a single term | ms / Evaluation Response | Timer | host, job, namespace, eval_type, triggered_by |
| `nomad.nomad.broker.service_ready` | Count of service evals ready to be scheduled | Integer | Gauge | host |
| `nomad.nomad.broker.service_unacked` | Count of unacknowledged service evals | Integer | Gauge | host |
| `nomad.nomad.broker.system_ready` | Count of system evals ready to be scheduled | Integer | Gauge | host |
| `nomad.nomad.broker.system_unacked` | Count of unacknowledged system evals | Integer | Gauge | host |
| `nomad.nomad.broker.total_ready` | Count of evals in the ready state | Integer | Gauge | host |
| `nomad.nomad.broker.total_waiting` | Count of evals waiting to be enqueued | Integer | Gauge | host |
| `nomad.nomad.broker.wait_time` | Time elapsed while the evaluation was ready to be processed and waiting to be dequeued. This metric is only valid within a single term | ms / Evaluation Wait | Timer | host, job, namespace, eval_type, triggered_by |
| `nomad.nomad.client.batch_deregister` | Time elapsed for `Node.BatchDeregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.deregister` | Time elapsed for `Node.Deregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.derive_si_token` | Time elapsed for `Node.DeriveSIToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.derive_vault_token` | Time elapsed for `Node.DeriveVaultToken` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.emit_events` | Time elapsed for `Node.EmitEvents` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.evaluate` | Time elapsed for `Node.Evaluate` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.get_allocs` | Time elapsed for `Node.GetAllocs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.get_client_allocs` | Time elapsed for `Node.GetClientAllocs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.get_node` | Time elapsed for `Node.GetNode` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.list` | Time elapsed for `Node.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.register` | Time elapsed for `Node.Register` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.stats` | Time elapsed for `Client.Stats` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_alloc` | Time elapsed for `Node.UpdateAlloc` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_drain` | Time elapsed for `Node.UpdateDrain` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_eligibility` | Time elapsed for `Node.UpdateEligibility` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client.update_status` | Time elapsed for `Node.UpdateStatus` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.garbage_collect_all` | Time elapsed for `ClientAllocations.GarbageCollectAll` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.garbage_collect` | Time elapsed for `ClientAllocations.GarbageCollect` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.restart` | Time elapsed for `ClientAllocations.Restart` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.signal` | Time elapsed for `ClientAllocations.Signal` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_allocations.stats` | Time elapsed for `ClientAllocations.Stats` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_controller.attach_volume` | Time elapsed for `Controller.AttachVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_controller.detach_volume` | Time elapsed for `Controller.DetachVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_controller.validate_volume` | Time elapsed for `Controller.ValidateVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.client_csi_node.detach_volume` | Time elapsed for `Node.DetachVolume` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.allocations` | Time elapsed for `Deployment.Allocations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.cancel` | Time elapsed for `Deployment.Cancel` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.fail` | Time elapsed for `Deployment.Fail` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.get_deployment` | Time elapsed for `Deployment.GetDeployment` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.list` | Time elapsed for `Deployment.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.pause` | Time elapsed for `Deployment.Pause` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.promote` | Time elapsed for `Deployment.Promote` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.reap` | Time elapsed for `Deployment.Reap` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.run` | Time elapsed for `Deployment.Run` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.set_alloc_health` | Time elapsed for `Deployment.SetAllocHealth` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.deployment.unblock` | Time elapsed for `Deployment.Unblock` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.ack` | Time elapsed for `Eval.Ack` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.allocations` | Time elapsed for `Eval.Allocations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.create` | Time elapsed for `Eval.Create` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.dequeue` | Time elapsed for `Eval.Dequeue` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.get_eval` | Time elapsed for `Eval.GetEval` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.list` | Time elapsed for `Eval.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.nack` | Time elapsed for `Eval.Nack` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.reap` | Time elapsed for `Eval.Reap` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.reblock` | Time elapsed for `Eval.Reblock` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.eval.update` | Time elapsed for `Eval.Update` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.file_system.list` | Time elapsed for `FileSystem.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.file_system.logs` | Time elapsed to establish `FileSystem.Logs` RPC | Milliseconds | Timer | host |
| `nomad.nomad.file_system.stat` | Time elapsed for `FileSystem.Stat` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.file_system.stream` | Time elapsed to establish `FileSystem.Stream` RPC | Milliseconds | Timer | host |
| `nomad.nomad.fsm.alloc_client_update` | Time elapsed to apply `AllocClientUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.alloc_update_desired_transition` | Time elapsed to apply `AllocUpdateDesiredTransition` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_policy_delete` | Time elapsed to apply `ApplyACLPolicyDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_policy_upsert` | Time elapsed to apply `ApplyACLPolicyUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_token_bootstrap` | Time elapsed to apply `ApplyACLTokenBootstrap` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_token_delete` | Time elapsed to apply `ApplyACLTokenDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_acl_token_upsert` | Time elapsed to apply `ApplyACLTokenUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_plugin_delete` | Time elapsed to apply `ApplyCSIPluginDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_batch_claim` | Time elapsed to apply `ApplyCSIVolumeBatchClaim` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_claim` | Time elapsed to apply `ApplyCSIVolumeClaim` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_deregister` | Time elapsed to apply `ApplyCSIVolumeDeregister` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_csi_volume_register` | Time elapsed to apply `ApplyCSIVolumeRegister` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_alloc_health` | Time elapsed to apply `ApplyDeploymentAllocHealth` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_delete` | Time elapsed to apply `ApplyDeploymentDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_promotion` | Time elapsed to apply `ApplyDeploymentPromotion` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_deployment_status_update` | Time elapsed to apply `ApplyDeploymentStatusUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_job_stability` | Time elapsed to apply `ApplyJobStability` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_job_version_tag` | Time elapsed to apply `ApplyJobVersionTag` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_namespace_delete` | Time elapsed to apply `ApplyNamespaceDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_namespace_upsert` | Time elapsed to apply `ApplyNamespaceUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_node_pool_upsert` | Time elapsed to apply `ApplyNodePoolUpsert` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_node_pool_delete` | Time elapsed to apply `ApplyNodePoolDelete` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_plan_results` | Time elapsed to apply `ApplyPlanResults` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.apply_scheduler_config` | Time elapsed to apply `ApplySchedulerConfig` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.autopilot` | Time elapsed to apply `Autopilot` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.batch_deregister_job` | Time elapsed to apply `BatchDeregisterJob` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.batch_deregister_node` | Time elapsed to apply `BatchDeregisterNode` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.batch_node_drain_update` | Time elapsed to apply `BatchNodeDrainUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.cluster_meta` | Time elapsed to apply `ClusterMeta` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.delete_eval` | Time elapsed to apply `DeleteEval` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_job` | Time elapsed to apply `DeregisterJob` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_node` | Time elapsed to apply `DeregisterNode` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_si_accessor` | Time elapsed to apply `DeregisterSITokenAccessor` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.deregister_vault_accessor` | Time elapsed to apply `DeregisterVaultAccessor` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.node_drain_update` | Time elapsed to apply `NodeDrainUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.node_eligibility_update` | Time elapsed to apply `NodeEligibilityUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.node_status_update` | Time elapsed to apply `NodeStatusUpdate` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.persist` | Time elapsed to apply `Persist` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.register_job` | Time elapsed to apply `RegisterJob` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.register_node` | Time elapsed to apply `RegisterNode` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.update_eval` | Time elapsed to apply `UpdateEval` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_node_events` | Time elapsed to apply `UpsertNodeEvents` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_scaling_event` | Time elapsed to apply `UpsertScalingEvent` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_si_accessor` | Time elapsed to apply `UpsertSITokenAccessors` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.fsm.upsert_vault_accessor` | Time elapsed to apply `UpsertVaultAccessor` raft entry | Milliseconds | Timer | host |
| `nomad.nomad.job.allocations` | Time elapsed for `Job.Allocations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.batch_deregister` | Time elapsed for `Job.BatchDeregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.deployments` | Time elapsed for `Job.Deployments` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.deregister` | Time elapsed for `Job.Deregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.dispatch` | Time elapsed for `Job.Dispatch` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.evaluate` | Time elapsed for `Job.Evaluate` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.evaluations` | Time elapsed for `Job.Evaluations` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.get_job_versions` | Time elapsed for `Job.GetJobVersions` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.get_job` | Time elapsed for `Job.GetJob` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.latest_deployment` | Time elapsed for `Job.LatestDeployment` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.list` | Time elapsed for `Job.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.plan` | Time elapsed for `Job.Plan` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.register` | Time elapsed for `Job.Register` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.revert` | Time elapsed for `Job.Revert` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.scale_status` | Time elapsed for `Job.ScaleStatus` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.scale` | Time elapsed for `Job.Scale` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.stable` | Time elapsed for `Job.Stable` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job.validate` | Time elapsed for `Job.Validate` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.job_summary.get_job_summary` | Time elapsed for `Job.Timer` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.leader.barrier` | Time elapsed to establish a raft barrier during leader transition | Milliseconds | Timer | host |
| `nomad.nomad.leader.reconcileMember` | Time elapsed to reconcile a serf peer with state store | Milliseconds | Timer | host |
| `nomad.nomad.leader.reconcile` | Time elapsed to reconcile all serf peers with state store | Milliseconds | Timer | host |
| `nomad.nomad.namespace.delete_namespaces` | Time elapsed for `Namespace.DeleteNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.get_namespace` | Time elapsed for `Namespace.GetNamespace` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.get_namespaces` | Time elapsed for `Namespace.GetNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.list_namespace` | Time elapsed for `Namespace.ListNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.namespace.upsert_namespaces` | Time elapsed for `Namespace.UpsertNamespaces` | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.list` | Time elapsed for `NodePool.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.list_jobs` | Time elapsed for `NodePool.ListJobs` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.list_nodes` | Time elapsed for `NodePool.ListNodes` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.get_node_pool` | Time elapsed for `NodePool.GetNodePool` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.upsert_node_pools` | Time elapsed for `NodePool.UpsertNodePools` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.node_pool.delete_node_pools` | Time elapsed for `NodePool.DeleteNodePools` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.periodic.force` | Time elapsed for `Periodic.Force` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plan.apply` | Time elapsed to apply a plan | Milliseconds | Timer | host |
| `nomad.nomad.plan.evaluate` | Time elapsed to evaluate a plan | Milliseconds | Timer | host |
| `nomad.nomad.plan.node_rejected` | Number of times a node has had a plan rejected | Integer | Counter | host, node_id |
| `nomad.nomad.plan.rejection_tracker.node_score` | Number of times a node has had a plan rejected within the tracker window | Integer | Gauge | host, node_id |
| `nomad.nomad.plan.queue_depth` | Count of evals in the plan queue | Integer | Gauge | host |
| `nomad.nomad.plan.submit` | Time elapsed for `Plan.Submit` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plan.wait_for_index` | Time elapsed that planner waits for the raft index of the plan to be processed | Milliseconds | Timer | host |
| `nomad.nomad.plugin.delete` | Time elapsed for `CSIPlugin.Delete` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plugin.get` | Time elapsed for `CSIPlugin.Get` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.plugin.list` | Time elapsed for `CSIPlugin.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.quota.utilization.cpu` | Utilization of the CPU quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.quota.utilization.cores` | Utilization of the CPU Cores quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.quota.utilization.memory_mb` | Utilization of the Memory MB quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.quota.utilization.storage.host_volumes_mb` | Utilization of the Host Volumes MB quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.quota.utilization.storage.variables_mb` | Utilization of the Variables MB quota | Integer | Gauge | quota_name, namespace, region |
| `nomad.nomad.scaling.get_policy` | Time elapsed for `Scaling.GetPolicy` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.scaling.list_policies` | Time elapsed for `Scaling.ListPolicies` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.search.prefix_search` | Time elapsed for `Search.PrefixSearch` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.vault.create_token` | Time elapsed to create Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.distributed_tokens_revoked` | Count of revoked tokens | Integer | Gauge | host |
| `nomad.nomad.vault.lookup_token` | Time elapsed to lookup Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.renew_failed` | Count of failed attempts to renew Vault token | Integer | Gauge | host |
| `nomad.nomad.vault.renew` | Time elapsed to renew Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.revoke_tokens` | Time elapsed to revoke Vault tokens | Milliseconds | Timer | host |
| `nomad.nomad.vault.token_last_renewal` | Time since last successful Vault token renewal | Milliseconds | Timer | host |
| `nomad.nomad.vault.token_next_renewal` | Time until next Vault token renewal attempt | Milliseconds | Timer | host |
| `nomad.nomad.vault.token_ttl` | Time to live for Vault token | Milliseconds | Timer | host |
| `nomad.nomad.vault.undistributed_tokens_abandoned` | Count of abandoned tokens | Integer | Gauge | host |
| `nomad.nomad.volume.claim` | Time elapsed for `CSIVolume.Claim` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.deregister` | Time elapsed for `CSIVolume.Deregister` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.get` | Time elapsed for `CSIVolume.Get` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.list` | Time elapsed for `CSIVolume.List` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.register` | Time elapsed for `CSIVolume.Register` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.volume.unpublish` | Time elapsed for `CSIVolume.Unpublish` RPC call | Milliseconds | Timer | host |
| `nomad.nomad.worker.create_eval` | Time elapsed for worker to create an eval | Milliseconds | Timer | host |
| `nomad.nomad.worker.dequeue_eval` | Time elapsed for worker to dequeue an eval | Milliseconds | Timer | host |
| `nomad.nomad.worker.invoke_scheduler.<type>` | Time elapsed for worker to invoke the scheduler of type `<type>` | Milliseconds | Timer | host |
| `nomad.nomad.worker.send_ack` | Time elapsed for worker to send acknowledgement | Milliseconds | Timer | host |
| `nomad.nomad.worker.submit_plan` | Time elapsed for worker to submit plan | Milliseconds | Timer | host |
| `nomad.nomad.worker.update_eval` | Time elapsed for worker to submit updated eval | Milliseconds | Timer | host |
| `nomad.nomad.worker.wait_for_index` | Time elapsed that worker waits for the raft index of the eval to be processed | Milliseconds | Timer | host |
| `nomad.raft.appliedIndex` | Current index applied to FSM | Integer | Gauge | host |
| `nomad.raft.barrier` | Count of blocking raft API calls | Integer | Counter | host |
| `nomad.raft.commitNumLogs` | Count of logs enqueued | Integer | Gauge | host |
| `nomad.raft.commitTime` | Time elapsed to commit writes | Milliseconds | Timer | host |
| `nomad.raft.fsm.apply` | Time elapsed to apply write to FSM | Milliseconds | Timer | host |
| `nomad.raft.fsm.enqueue` | Time elapsed to enqueue write to FSM | Milliseconds | Timer | host |
| `nomad.raft.lastIndex` | Most recent index seen | Integer | Gauge | host |
| `nomad.raft.leader.dispatchLog` | Time elapsed to write log, mark in flight, and start replication | Milliseconds | Timer | host |
| `nomad.raft.leader.dispatchNumLogs` | Count of logs dispatched | Integer | Gauge | host |
| `nomad.raft.replication.appendEntries` | Raft transaction commit time | ms / Raft Log Append | Timer | |
| `nomad.raft.state.candidate` | Count of entering candidate state | Integer | Gauge | host |
| `nomad.raft.state.follower` | Count of entering follower state | Integer | Gauge | host |
| `nomad.raft.state.leader` | Count of entering leader state | Integer | Gauge | host |
| `nomad.raft.transition.heartbeat_timeout` | Count of failing to heartbeat and starting election | Integer | Gauge | host |
| `nomad.raft.transition.leader_lease_timeout` | Count of stepping down as leader after losing quorum | Integer | Gauge | host |
| `nomad.runtime.free_count` | Count of objects freed from heap by go runtime GC | Integer | Gauge | host |
| `nomad.runtime.gc_pause_ns` | Go runtime GC pause times | Nanoseconds | Summary | host |
| `nomad.runtime.sys_bytes` | Go runtime GC metadata size | # of bytes | Gauge | host |
| `nomad.runtime.total_gc_pause_ns` | Total elapsed go runtime GC pause times | Nanoseconds | Gauge | host |
| `nomad.runtime.total_gc_runs` | Count of go runtime GC runs | Integer | Gauge | host |
| `nomad.serf.queue.Event` | Count of memberlist events received | Integer | Summary | host |
| `nomad.serf.queue.Intent` | Count of memberlist changes | Integer | Summary | host |
| `nomad.serf.queue.Query` | Count of memberlist queries | Integer | Summary | host |
| `nomad.scheduler.allocs.rescheduled.attempted` | Count of attempts to reschedule an allocation | Integer | Count | alloc_id, job, namespace, task_group |
| `nomad.scheduler.allocs.rescheduled.limit` | Maximum number of attempts to reschedule an allocation | Integer | Count | alloc_id, job, namespace, task_group |
| `nomad.scheduler.allocs.rescheduled.wait_until` | Time that a rescheduled allocation will be delayed | Float | Gauge | alloc_id, job, namespace, task_group, follow_up_eval_id |
| `nomad.state.snapshotIndex` | Current snapshot index | Integer | Gauge | host |
## Raft BoltDB Metrics

55
website/content/docs/operations/stateful-workloads.mdx
View File

@@ -2,7 +2,7 @@
layout: docs
page_title: Considerations for Stateful Workloads
description: |-
Learn about persistent storage options for stateful workloads on Nomad.
Learn about persistent storage options for stateful workloads on Nomad. Compare AWS, Azure, and Google Cloud Platform (GCP) storage services. Review the advantages and disadvantages of using Container Storage Interface (CSI) volumes, dynamic host volumes, static host volumes, and ephemeral disks.
---
# Considerations for Stateful Workloads
@@ -141,22 +141,32 @@ NFS/CIFS volume from a NAS or a public cloud service such as AWS EFS. Therefore
you can use host volumes for both local somewhat persistent storage and for
highly persistent networked storage.
Because you need to declare host volumes in the Nomad agent's configuration
file, you must restart the Nomad client to reconfigure them. This makes host
volumes impractical if you frequently change your storage configuration.
Furthermore, it might require coordination between different
[personas](/nomad/docs/concepts/security#personas) to configure and consume
host volumes. For example, a Nomad Administrator must modify Nomad's
configuration file to add/update/remove host volumes to make them available for
consumption by Nomad Operators. Or, with networked host volumes, a Storage
Administrator will need to provision the volumes and make them available to the
Nomad clients. A System Administrator will then mount them on the Nomad clients.
Host volumes may be dynamic or static. Provision dynamic host volumes
with the [`volume create`](/nomad/docs/commands/volume/create) command or
API. [ACL policies](/nomad/docs/other-specifications/acl-policy#namespace-rules)
allow delegation of control for storage within a namespace to Nomad
Operators. The dynamic host volume [plugin
specification](/nomad/docs/concepts/plugins/storage/host-volumes) allows you to
develop plugins specific to your local storage environment. For example, in an
on-prem cluster you could write a plugin to perform LVM thin-provisioning.
Host volumes backed by local storage help persist data that is not critical, for
example an on-disk cache that can be rebuilt if needed. When backed by networked
storage such as NFS/CIFS-mounted volumes or distributed storage via
GlusterFS/Ceph, host volumes provide a quick option to consume highly available
and reliable storage.
You declare static host volumes in the Nomad agent's configuration file, and
you must restart the Nomad client to reconfigure them. This makes static host
volumes impractical if you frequently change your storage
configuration. Furthermore, it might require coordination between different
[personas](/nomad/docs/concepts/security#personas) to configure and consume host
volumes. For example, a Nomad Administrator must modify Nomad's configuration
file to add, update, and remove host volumes to make them available for consumption by
Nomad Operators. Or, with networked host volumes, a Storage Administrator
needs to provision the volumes and makes them available to the Nomad clients. A
System Administrator then mounts them on the Nomad clients.
Host volumes backed by local storage help persist data that is not critical or
can be readily restored. For example, an on-disk cache that can be rebuilt, or a
clustered application where a single node can rebuild its state from the rest
of the cluster. When backed by networked storage such as NFS/CIFS-mounted
volumes or distributed storage with GlusterFS or Ceph, host volumes provide a quick
option to consume highly available and reliable storage.
Refer to the [Stateful workloads with Nomad host
volumes](/nomad/tutorials/stateful-workloads/stateful-workloads-host-volumes)
@@ -205,8 +215,10 @@ choose the storage options that best addresses your Nomad storage requirements.
| Storage option | Advantages | Disadvantages | Ideal for |
|---|---|---|---|
| CSI volumes | <ul><li>Wide ecosystem with many providers</li><li>Advanced features such as snapshots, cloning, and resizing</li><li>Dynamic, flexible, and self-service (anyone with the correct ACL policies can create volumes on-demand)</li></ul> | <ul><li>Some complexity and ongoing maintenance</li><li>Plugin upgrades have to follow the underlying storage provider's API changes/upgrades</li><li>Not all CSI plugins implement all features</li><li>Not all CSI plugins respect the CSI spec and are Nomad compatible</li><li>Node plugins need to run in privileged mode to be able to mount the volumes in allocations</li></ul> | <ul><li>Environments where Nomad cluster operators and consumers need to easily add/change storage, and where the storage provider of choice has a CSI plugin that respects the CSI spec</li></ul> |
| Host volumes backed by local storage | <ul><li>Readily available</li><li>Fast due to being local</li><li>Doesn't require ongoing maintenance</li></ul> | <ul><li>Requires coordination between multiple personas to configure and consume (operators running the Nomad clients need to configure them statically in the Nomad client's configuration file)</li><li>Not fault tolerant. In case of hardware failure on a single instance, the data will be lost</li></ul> | <ul><li> Environments with low persistent storage requirements that could tolerate some failure but prefer not to or have high performance and low latency needs.</li></ul> |
| Host volumes backed by networked or clustered storage | <ul><li>Readily available</li><li>Require no ongoing maintenance on the Nomad side (but might on the storage provider) </li></ul> | <ul><li>Require coordination between multiple personas to configure and consume (storage admins need to provision volumes, operators running the Nomad clients need to configure them statically in the Nomad client's configuration file)</li><li>The underlying networked storage and its limitations are decoupled from the consumer, but need to be understood. For example, is concurrent access possible</li></ul> | <ul><li>Environments with low amounts or low frequency of change of storage that have an existing storage provider that can be consumed via NFS/CIFS.</li></ul> |
| Dynamic host volumes backed by local storage | <ul><li>Readily available</li><li>Fast due to being local</li><li>Doesn't require ongoing maintenance</li></ul> | <ul><li>Not fault tolerant. In case of hardware failure on a single instance, the data will be lost unless the application can restore the data</li></ul> | <ul><li>Environments with high performance and low latency persistent storage requirements where the application is designed to tolerate node failures.</li></ul> |
| Dynamic host volumes backed by networked or clustered storage | <ul><li>Readily available</li><li>Require no ongoing maintenance on the Nomad side (but might on the storage provider) </li></ul> | <ul><li>The underlying networked storage and its limitations are decoupled from the consumer, but need to be understood. For example, is concurrent access possible</li></ul> | <ul><li>Environments that have an existing storage provider that can be consumed via NFS/CIFS.</li></ul> |
| Static host volumes backed by local storage | <ul><li>Readily available</li><li>Fast due to being local</li><li>Doesn't require ongoing maintenance</li></ul> | <ul><li>Requires coordination between multiple personas to configure and consume (operators running the Nomad clients need to configure them statically in the Nomad client's configuration file)</li><li>Not fault tolerant. In case of hardware failure on a single instance, the data will be lost</li></ul> | <ul><li> Environments with low persistent storage requirements that could tolerate some failure but prefer not to or have high performance and low latency needs.</li></ul> |
| Static host volumes backed by networked or clustered storage | <ul><li>Readily available</li><li>Require no ongoing maintenance on the Nomad side (but might on the storage provider) </li></ul> | <ul><li>Require coordination between multiple personas to configure and consume (storage admins need to provision volumes, operators running the Nomad clients need to configure them statically in the Nomad client's configuration file)</li><li>The underlying networked storage and its limitations are decoupled from the consumer, but need to be understood. For example, is concurrent access possible</li></ul> | <ul><li>Environments with low amounts or low frequency of change of storage that have an existing storage provider that can be consumed via NFS/CIFS.</li></ul> |
| Ephemeral disks | <ul><li>Fast due to being local</li><li>Basic best effort persistence, including optional migration across Nomad clients</li></ul> | <ul><li> Not fault tolerant. In case of hardware failure on a single instance, the data will be lost </li></ul> | <ul><li>Environments that need temporary caches, somewhere to store files undergoing processing, etc. Everything which is ephemeral and can be easily rebuilt.</li></ul> |
## Additional resources
@@ -222,7 +234,7 @@ following resources:
### CSI
- [Nomad CSI plugin concepts](/nomad/docs/concepts/plugins/csi)
- [Nomad CSI plugin concepts](/nomad/docs/concepts/plugins/storage/csi)
- [Nomad CSI
tutorial](/nomad/tutorials/stateful-workloads/stateful-workloads-csi-volumes)
- [Nomad CSI examples](https://github.com/hashicorp/nomad/tree/main/demo/csi)
@@ -232,3 +244,8 @@ following resources:
- [JuiceFS CSI with Nomad](https://juicefs.com/docs/csi/csi-in-nomad/)
- [Hetzner CSI](https://github.com/hetznercloud/csi-driver/blob/main/docs/nomad/README.md)
- [NFS CSI Plugin](https://gitlab.com/rocketduck/csi-plugin-nfs)
### Dynamic Host Volumes
- [Dynamic host volume plugins](/nomad/docs/concepts/plugins/storage/host-volumes)
- [Dynamic host volume tutorial](/nomad/tutorial/stateful-workloads/stateful-workloads-dynamic-host-volumes)

13
website/content/docs/other-specifications/acl-policy.mdx
View File

@@ -77,8 +77,8 @@ Namespace rules control access to APIs in Nomad that are namespaced:
[Jobs][api_jobs], [Allocations][api_allocations],
[Deployments][api_deployments], [Evaluations][api_evaluations],
[Recommendations][api_recommendations], [Scaling
Policies][api_scaling_policies], [Services][api_services], and [CSI
Volumes][api_volumes]. Namespace rules also filter items related to the above
Policies][api_scaling_policies], [Services][api_services], and
[Volumes][api_volumes]. Namespace rules also filter items related to the above
APIs from the [Event Stream][api_events] and [Search][api_search] APIs.
Each namespace rule is labeled with the namespace name it applies to. If no
@@ -149,6 +149,11 @@ of `capabilities`. This includes:
and listing external volumes and snapshots.
- `csi-mount-volume` - Allows jobs to be submitted that claim a CSI volume. This
implicitly grants `csi-read-volume`.
- `host-volume-create` - Allows creating dynamic host volumes. This implicitly grants `host-volume-read`.
- `host-volume-delete` - Allows deleting dynamic host volumes.
- `host-volume-read` - Allows inspecting dynamic host volumes.
- `host-volume-register` - Allows registering dynamic host volumes that have been created without a plugin. This implicitly grants `host-volume-read` and `host-volume-create`.
- `host-volume-write` - Allows all write operations on dynamic host volumes. This implicitly grants `host-volume-read`, `host-volume-create`, `host-volume-register`, and `host-volume-delete`.
- `list-scaling-policies` - Allows listing scaling policies.
- `read-scaling-policy` - Allows inspecting a scaling policy.
- `read-job-scaling` - Allows inspecting the current scaling of a job.
@@ -164,8 +169,8 @@ grained namespace capabilities:
| Policy | Capabilities |
|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `deny` | deny |
| `read` | list-jobs<br />parse-job<br />read-job<br />csi-list-volume<br />csi-read-volume<br />list-scaling-policies<br />read-scaling-policy<br />read-job-scaling |
| `write` | list-jobs<br />parse-job<br />read-job<br />submit-job<br />dispatch-job<br />read-logs<br />read-fs<br />alloc-exec<br />alloc-lifecycle<br />csi-write-volume<br />csi-mount-volume<br />list-scaling-policies<br />read-scaling-policy<br />read-job-scaling<br />scale-job<br />submit-recommendation |
| `read` | list-jobs<br />parse-job<br />read-job<br />csi-list-volume<br />csi-read-volume<br />host-volume-read<br />list-scaling-policies<br />read-scaling-policy<br />read-job-scaling |
| `write` | list-jobs<br />parse-job<br />read-job<br />submit-job<br />dispatch-job<br />read-logs<br />read-fs<br />alloc-exec<br />alloc-lifecycle<br />csi-write-volume<br />csi-mount-volume<br />host-volume-write<br />list-scaling-policies<br />read-scaling-policy<br />read-job-scaling<br />scale-job<br />submit-recommendation |
| `scale` | list-scaling-policies<br />read-scaling-policy<br />read-job-scaling<br />scale-job |
<!-- markdownlint-enable -->

27
website/content/docs/other-specifications/namespace.mdx
View File

@@ -108,20 +108,23 @@ consul {
### `node_pool_config` Parameters <EnterpriseAlert inline />
- `default` `(string: "default")` - Specifies the node pool to use for jobs in
this namespace that don't define a node pool in their specification.
- `default` `(string: "default")` - Specifies the node pool to use for jobs or
dynamic host volumes in this namespace that don't define a node pool in their
specification.
- `allowed` `(array<string>: nil)` - Specifies the node pools that are allowed
to be used by jobs in this namespace. By default, all node pools are allowed.
If an empty list is provided only the namespace's default node pool is
allowed. This field supports wildcard globbing through the use of `*` for
multi-character matching. This field cannot be used with `denied`.
- `allowed` `(array<string>: nil)` - Specifies the node pools that jobs or
dynamic host volumes in this namespace are allowed to use. By default, all
node pools are allowed. If an empty list is provided only the namespace's
default node pool is allowed. This field supports wildcard globbing through
the use of `*` for multi-character matching. This field cannot be used with
`denied`.
- `denied` `(array<string>: nil)` - Specifies the node pools that are not
allowed to be used by jobs in this namespace. This field supports wildcard
globbing through the use of `*` for multi-character matching. If specified,
any node pool is allowed to be used, except for those that match any of these
patterns. This field cannot be used with `allowed`.
- `denied` `(array<string>: nil)` - Specifies the node pools that jobs or
dynamic host volumes in this namespace are not allowed to use. This field
supports wildcard globbing through the use of `*` for multi-character
matching. If specified, jobs and dynamic host volumes are allowed to use any
node pool, except for those that match any of these patterns. This field
cannot be used with `allowed`.
### `vault` Parameters <EnterpriseAlert inline />

28
website/content/docs/other-specifications/quota.mdx
View File

@@ -35,11 +35,16 @@ limit {
cpu = 2500
memory = 1000
memory_max = 1000
device "nvidia/gpu/1080ti" {
count = 1
}
storage {
variables = 1000 # in MB
host_volumes = "1000 GiB"
}
}
variables_limit = 1000
}
```
@@ -56,9 +61,6 @@ in a Nomad region.
- `region` `(string)` - The Nomad `region` that the limit applies to.
- `region_limit` <code>([RegionLimit](#region_limit-parameters))</code> -
Resources to limit.
- `variables_limit` `(int: 0)` - Maximum total size of all Nomad
[`variables`][] in MiB. The default `0` means unlimited, and `-1` means
variables are fully disabled.
### `region_limit` parameters
@@ -75,6 +77,7 @@ all `resources.memory` in the namespace.
- `memory_max` `(int: <optional>)` - The limit on total mount of hard memory
limits in MB from all `resources.memory_max` in the namespace.
- `device` <code>([Device](#device-parameters): nil)</code>
- `storage` <code>([Storage](#storage-parameters): nil)</code>
### `device` parameters
@@ -83,12 +86,27 @@ equivalent to the [`device`][] block in a job specification.
- `count` `(int)` - How many of this device may be used.
### `storage` parameters
The `storage` block defines limits on provisioned storage.
- `host_volumes` `(int: 0)` - Maximum total size of all [dynamic host volumes][]
in MiB. The default `0` means unlimited, and `-1` means variables are fully
disabled. This field accepts human-friendly string inputs such as "100
GiB". The quota for host volumes is enforced at the time the volume is created
via [`volume create`][].
- `variables` `(int: 0)` - Maximum total size of all Nomad [variables][] in
MiB. The default `0` means unlimited, and `-1` means variables are fully
disabled. This field accepts human-friendly string inputs such as "100 GiB".
[Resource Quotas]: /nomad/docs/enterprise#resource-quotas
[`quota`]: /nomad/docs/commands/quota
[Quota HTTP API]: /nomad/api-docs/quotas
[Quotas tutorial]: /nomad/tutorials/governance-and-policy/quotas
[`Namespace`]: /nomad/docs/other-specifications/namespace
[`variables`]: /nomad/docs/concepts/variables
[variables]: /nomad/docs/concepts/variables
[`resources`]: /nomad/docs/job-specification/resources
[CPU concepts]: /nomad/docs/concepts/cpu
[`device`]: /nomad/docs/job-specification/device#device-parameters
[dynamic host volumes]: /nomad/docs/other-specifications/volume/host

50
website/content/docs/other-specifications/volume/capability.mdx
View File

@@ -1,7 +1,8 @@
---
layout: docs
page_title: capability Block - Volume Specification
description: The "capability" block allows for validating the capability of a volume.
description: |-
Configure Container Storage Interface (CSI) and dynamic host storage volume capability in the "capability" block of the Nomad volume specification. Set single node or multiple node access and file system or block device attachment mode.
---
# `capability` Block
@@ -35,18 +36,43 @@ for each capability you intend to use in a job's [`volume`] block.
## `capability` Parameters
- `access_mode` `(string: <required>)` - Defines whether a volume should be
available concurrently. Can be one of `"single-node-reader-only"`,
`"single-node-writer"`, `"multi-node-reader-only"`,
`"multi-node-single-writer"`, or `"multi-node-multi-writer"`. Most CSI plugins
support only single-node modes. Consult the documentation of the storage
provider and CSI plugin.
- `access_mode` `(string)` - Defines whether a volume should be available
concurrently. The `access_mode` and `attachment_mode` from the volume request
must exactly match one of the volume's `capability` blocks.
- `attachment_mode` `(string: <required>)` - The storage API that will be used
by the volume. Most storage providers will support `"file-system"`, to mount
volumes using the CSI filesystem API. Some storage providers will support
`"block-device"`, which will mount the volume with the CSI block device API
within the container.
- For CSI volumes the `access_mode` is required. Can be one of the following:
- `"single-node-reader-only"`
- `"single-node-writer"`
- `"multi-node-reader-only"`
- `"multi-node-single-writer"`
- `"multi-node-multi-writer"`
Most CSI plugins support only single-node modes.
Consult the documentation of the storage provider and CSI plugin.
- For dynamic host volumes the `access_mode` is optional. Can be one of the following:
- `"single-node-writer"`
- `"single-node-reader-only"`
- `"single-node-single-writer"`
- `"single-node-multi-writer"`
In the job specification, the default is `single-node-writer` unless
`read_only = true`, which translates to `single-node-reader-only`.
- `attachment_mode` `(string)` - The storage API used by the volume. One of
`"file-system"` or `"block-device"`. The `access_mode` and `attachment_mode`
from the volume request must exactly match one of the volume's `capability`
blocks.
- For CSI volumes the `attachment_mode` field is required. Most storage
providers support `"file-system"`, to mount volumes using the CSI
filesystem API. Some storage providers support `"block-device"`, which
mounts the volume with the CSI block device API within the container.
- For dynamic host volumes the `attachment_mode` field is optional and
defaults to `"file-system"`.
## `capability` Examples

291
website/content/docs/other-specifications/volume/csi.mdx Normal file
View File

@@ -0,0 +1,291 @@
---
layout: docs
page_title: CSI Volume Specification
description: |-
Learn about the Nomad Container Storage Interface (CSI) volume
specification. Create and register CSI volumes using the `volume create`
and `volume register` commands and the
`PUT /v1/volume/csi/:volume_id/create` and
`PUT /v1/volume/csi/:volume_id` API endpoints. Define capacity, capabilities, mount
options, secrets, topology requests, and a context map of values passed
directly to the CSI plugin to validate the volume. Learn how volume
creation and registration are different. Additionally, learn how to
expand the size of a volume and update a volume definition.
---
# CSI Volume Specification
This page provides reference information for the Nomad Container Storage
Interface (CSI) volume specification. Create and register CSI volumes using the
[`volume create`][] and [`volume register`][] commands and the [`PUT
/v1/volume/csi/:volume_id/create`][api_volume_create] and [`PUT
/v1/volume/csi/:volume_id`][api_volume_register] API endpoints. Define capacity,
capabilities, mount options, secrets, topology requests, and a context map of
values passed directly to the CSI plugin to validate the volume. Learn how
volume creation and registration are different. Additionally, learn how to
expand the size of a volume and update a volume definition.
Some attributes are only supported by a specific operation, while others may
have a different meaning for each action, so read the documentation for each
attribute carefully. The section [Differences Between Create and
Register](#differences-between-create-and-register) provides a summary of the
differences.
Provide the file as either HCL or JSON to the commands and as JSON to the API.
## Volume Specification Parameters
- `capability` <code>([Capability][capability]: &lt;required&gt;)</code> -
Option for validating the capability of a volume.
- `capacity_min` `(string: <optional>)` - Option for requesting a minimum
capacity, in bytes. The capacity of a volume may be the physical size of a
disk, or a quota, depending on the storage provider. The specific size of the
resulting volume is somewhere between `capacity_min` and `capacity_max`; the
exact behavior is up to the storage provider. If you want to specify an exact
size, you should set `capacity_min` and `capacity_max` to the same
value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not
be supported by all storage providers. Increasing this value and reissuing
`volume create` or `volume register` may expand the volume, if the CSI plugin
supports it.
- `capacity_max` `(string: <optional>)` - Option for requesting a maximum
capacity, in bytes. The capacity of a volume may be the physical size of a
disk, or a quota, depending on the storage provider. The specific size of the
resulting volume is somewhere between `capacity_min` and `capacity_max`; the
exact behavior is up to the storage provider. If you want to specify an exact
size, you should set `capacity_min` and `capacity_max` to the same
value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not
be supported by all storage providers.
- `clone_id` `(string: <optional>)` - If the storage provider supports cloning,
the external ID of the volume to clone when creating this volume. If omitted,
the volume is created from scratch. The `clone_id` cannot be set if the
`snapshot_id` field is set. Only allowed on volume creation.
- `context` <code>(map<string|string>:nil)</code> - An optional key-value map of
strings passed directly to the CSI plugin to validate the volume. The details
of these parameters are specific to each storage provider, so consult the
specific plugin documentation for more information. Only allowed on volume
registration. Note that, like the rest of the volume specification, this block
is declarative, and an update replaces it in its entirety, therefore all
parameters need to be specified.
- `external_id` `(string: <required>)` - The ID of the physical volume from the
storage provider. For example, the volume ID of an AWS EBS volume or Digital
Ocean volume. Only allowed on volume registration.
- `id` `(string: <required>)` - The unique ID of the volume. This is how the
[`volume.source`][csi_volume_source] field in a job specification refers to
the volume.
- `mount_options` <code>([MountOptions][mount_options]:
&lt;required&gt;)</code> - Options for mounting `file-system` volumes that
don't already have a pre-formatted file system.
- `name` `(string: <required>)` - The display name of the volume. On volume
creation, this field may be used by the external storage provider to tag the
volume.
- `namespace` `(string: <optional>)` - The namespace of the volume. This field
overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE`
environment variable. Defaults to `"default"` if unset.
- `parameters` <code>(map<string|string>:nil)</code> - An optional key-value map
of strings passed directly to the CSI plugin to configure the volume. The
details of these parameters are specific to each storage provider, so consult
the specific plugin documentation for more information.
- `plugin_id` `(string: <required>)` - The ID of the [CSI plugin][csi_plugin]
that manages this volume.
- `secrets` <code>(map<string|string>:nil)</code> - An optional key-value map of
strings used as credentials for publishing and unpublishing volumes.
- `snapshot_id` `(string: <optional>)` - If the storage provider supports
snapshots, the external ID of the snapshot to restore when creating this
volume. If omitted, the volume is created from scratch. The `snapshot_id`
cannot be set if the `clone_id` field is set. Only allowed on volume creation.
- `topology_request` <code>([TopologyRequest][topology_request]: nil)</code> -
Specify locations such as region, zone, and rack where the provisioned volume
must be accessible from in the case of volume creation, or the locations where
the existing volume is accessible from in the case of volume registration.
- `type` `(string: <required>)` - The type of volume. Must be `"csi"` for CSI
volumes.
## Differences Between Create and Register
Several fields are set automatically by the plugin when `volume create` or
`volume register` commands are successful and you should not set their values if
they are not supported by the operation.
On **volume registration** you should not set the [`snapshot_id`](#snapshot_id),
or [`clone_id`](#clone_id) fields.
On **volume creation** you should not set the [`external_id`](#external_id) or
[`context`](#context) fields.
## Updating a Volume Definition
The `volume register` command allows updating a volume definition. However,
after volume registration, you are only allowed to update the following fields:
* `capacity_min` and `capacity_max`. You may increase the volume size if the CSI
plugin supports it. Expansion may or may not be possible while the volume is
in use, again depending on the plugin. Reducing volume capacity is not
allowed per the CSI spec.
* `mount_options` block if the volume is not in use.
* `secrets` block
* `context` block. The values for this field are typically provided by the CSI
plugin and should not be updated unless recommended by the CSI plugin's
documentation.
Additionally, you may add or remove `capability` blocks, but only if the
capability is not currently in use by a mounted volume.
## Volume Expansion
You may expand CSI Volumes if the CSI controller plugin and node plugin, if
required, has the `EXPAND_VOLUME` capability.
To trigger a volume expansion, increase [`capacity_min`](#capacity_min) above
the current real capacity of the volume, as received from the [`volume
status`][] command, and re-issue either [`volume create`][] or [`volume
register`][].
Nomad reconciles the requested capacity by issuing expand volume requests to the
controller plugin, and if required by the controller, also to the node plugins
for each allocation that has a claim on the volume.
## Examples
### Volume registration
This is an example file used for the [`volume register`][] command.
```hcl
id = "ebs_prod_db1"
name = "database"
type = "csi"
external_id = "vol-23452345"
plugin_id = "ebs-prod"
capability {
access_mode = "single-node-reader-only"
attachment_mode = "file-system"
}
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
mount_options {
fs_type = "ext4"
mount_flags = ["noatime"]
}
topology_request {
required {
topology { segments { "rack" = "R2" } }
topology { segments { "rack" = "R1", "zone" = "us-east-1a"} }
}
}
secrets {
example_secret = "xyzzy"
}
parameters {
skuname = "Premium_LRS"
}
context {
endpoint = "http://192.168.1.101:9425"
}
```
### Example Volume Expansion
Either [`volume create`][] or [`volume register`][] commands can trigger
an expand to occur, after the volume has already been created or registered.
This example shows you how to expand a volume using the `volume create` command.
1. Create a volume configuration file called `volume.hcl` that defines ID, name,
type, plugin ID, and capacity.
```hcl
id = "ebs_prod_db1"
name = "database"
type = "csi"
plugin_id = "ebs-prod"
capacity_min = "50gib"
capacity_max = "50gib"
# ... etc ...
```
1. Create the volume using the [`volume create`] command.
```shell-session
$ nomad volume create volume.hcl
Created external volume vol-asdf1234 with ID ebs_prod_db1
```
1. Review the volume's current capacity using the [`volume status`] command.
```shell-session
$ nomad volume status ebs_prod_db1 | grep Capacity
Capacity = 50 GiB
```
1. Increase volume capacity in the `volume.hcl` file.
Update the `capacity_min` and `capacity_max` fields.
```hcl
id = "ebs_prod_db1"
name = "database"
type = "csi"
plugin_id = "ebs-prod"
# all of the above must remain the same
capacity_min = "100gib" # double
capacity_max = "100gib" # increased to match
# ... etc ...
```
1. Expand the volume using the [`volume create`] command.
```shell-session
$ nomad volume create volume.hcl
Created external volume vol-asdf1234 with ID ebs_prod_db1
```
1. Review the new capacity by running the [`volume status`] command.
```shell-session
$ nomad volume status ebs_prod_db1 | grep Capacity
Capacity = 100 GiB
```
If you encounter errors that are not clear from the command output,
the CSI plugin allocation logs and Nomad leader server logs may be
helpful.
[api_volume_create]: /nomad/api-docs/volumes#create-csi-volume
[api_volume_register]: /nomad/api-docs/volumes#register-volume
[capability]: /nomad/docs/other-specifications/volume/capability
[csi_plugin]: /nomad/docs/job-specification/csi_plugin
[csi_volume_source]: /nomad/docs/job-specification/volume#source
[mount_options]: /nomad/docs/other-specifications/volume/mount_options
[topology_request]: /nomad/docs/other-specifications/volume/topology_request
[`volume create`]: /nomad/docs/commands/volume/create
[`volume register`]: /nomad/docs/commands/volume/register
[`volume status`]: /nomad/docs/commands/volume/status

307
website/content/docs/other-specifications/volume/host.mdx Normal file
View File

@@ -0,0 +1,307 @@
---
layout: docs
page_title: Dynamic Host Volume Specification
description: |-
Learn about the Nomad dynamic host volume specification. Create and
register dynamic host volumes using the `volume create` and `volume
register` commands and the `PUT /v1/volume/host/create` and
`PUT /v1/volume/host/register` API endpoints. Define capacity,
capability, constraint, node, node pool, and parameters passed directly
to the plugin to configure the volume. Learn how volume creation and
registration are different. Additionally, learn how to place a volume on
specific nodes, update a volume, and expand a volume's capacity.
---
# Dynamic Host Volume Specification
This page provides reference information for the Nomad dynamic host volume
specification. Create and register dynamic host volumes using the [`volume
create`][] and [`volume register`][] commands and the [`PUT
/v1/volume/host/create`][api_volume_create] and [`PUT
/v1/volume/host/register`][api_volume_register] API endpoints. Define capacity,
capability, constraint, node, node pool, and parameters passed directly to the
plugin to configure the volume. Learn how volume creation and registration are
different. Additionally, learn how to place a volume on specific nodes, update a
volume, and expand a volume's capacity.
Some attributes are only be supported by specific operation, while others may
have a different meaning for each action, so read the documentation for each
attribute carefully. The section [Differences Between Create and
Register](#differences-between-create-and-register) provides a summary of the
differences.
The file may be provided as either HCL or JSON to the commands and as JSON to
the API.
## Volume Specification Parameters
- `capacity` `(string: <optional>)` - The size of a volume in bytes. Either the
physical size of a disk or a quota, depending on the plugin. This field must
be between the `capacity_min` and `capacity_max` values unless they are
omitted. Accepts human-friendly suffixes such as `"100GiB"`. Only supported
for volume registration.
- `capacity_min` `(string: <optional>)` - Option for requesting a minimum
capacity, in bytes. The capacity of a volume may be the physical size of a
disk, or a quota, depending on the plugin. The specific size of the resulting
volume is somewhere between `capacity_min` and `capacity_max`; the exact
behavior is up to the plugin. If you want to specify an exact size, set
`capacity_min` and `capacity_max` to the same value. Accepts human-friendly
suffixes such as `"100GiB"`. Plugins that cannot restrict the size of volumes
may ignore this field.
- `capacity_max` `(string: <optional>)` - Option for requesting a maximum
capacity, in bytes. The capacity of a volume may be the physical size of a
disk, or a quota, depending on the plugin. The specific size of the resulting
volume is somewhere between `capacity_min` and `capacity_max`; the exact
behavior is up to the plugin. If you want to specify an exact size, set
`capacity_min` and `capacity_max` to the same value. Accepts human-friendly
suffixes such as `"100GiB"`. Plugins that cannot restrict the size of volumes
may ignore this field.
- `capability` <code>([Capability][capability]: &lt;required&gt;)</code> -
Option for validating the capability of a volume.
- `constraint` <code>([Constraint][constraint]: &lt;optional&gt;)</code> - A
restriction on the eligible nodes where a volume can be created. Refer to the
[volume placement](#volume-placement) section for details. You can provide
multiple `constraint` blocks to add more constraints. Optional for volume
creation and ignored for volume registration.
- `id` `(string: <optional>)` - The ID of a previously created volume to update
via `volume create` or `volume register`. You should never set this field when
initially creating or registering a volume, and you should only use the values
returned from the Nomad API for the ID.
- `host_path` `(string)` - The path on disk where the volume exists. You should
set this only for volume registration. It is ignored for volume creation.
- `name` `(string: <required>)` - The name of the volume, which is used as the
[`volume.source`][volume_source] field in job specifications that claim this
volume. Host volume names must be unique per node. Names are visible to any
user with `node:read` ACL, even across namespaces, so they should not be
treated as sensitive values.
- `namespace` `(string: <optional>)` - The namespace of the volume. This field
overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE`
environment variable. Defaults to `"default"` if unset.
- `node_id` `(string)` - A specific node where you would like the volume to be
created. Refer to the [volume placement](#volume-placement) section for
details. Optional for volume creation but required for volume registration.
- `node_pool` `(string: <optional>)` - A specific node pool where you would like
the volume to be created. Refer to the [volume placement](#volume-placement)
section for details. Optional for volume creation or volume registration. If
you also provide `node_id`, the node must be in the provided `node_pool`.
- `parameters` <code>(map<string|string>:nil)</code> - An optional key-value map
of strings passed directly to the plugin to configure the volume. The details
of these parameters are specific to the plugin.
- `plugin_id` `(string)` - The ID of the [dynamic host volume
plugin][dhv_plugin] that manages this volume. Required for volume creation.
- `type` `(string: <required>)` - The type of volume. Must be `"host"` for
dynamic host volumes.
## Differences Between Create and Register
Several fields are set automatically by Nomad or the plugin when `volume create`
or `volume register` commands, or the equivalent APIs, are successful and you
should not set their values if they are not supported by the operation.
In **volume creation** you must set the [`plugin_id`](#plugin_id) field. The
[`capacity`](#capacity) and [`host_path`](#host_path) fields are ignored.
In **volume registration** you must set the [`node_id`](#node_id) and
[`host_path`](#host_path) fields. The [`plugin_id`](#plugin_id) and
[`constraint`](#constraint) fields are ignored. The [`node_pool`](#node_pool),
[`capacity_max`](#capacity_max), and [`capacity_min`](#capacity_min) fields are
ignored but must be consistent if set; the `node_pool` must match the node set
by `node_id`, and the minimum capacity must be less than the `capacity_max` and
`capacity`.
## Volume Placement
The `volume create` command creates the volume on a single node. If `node_id` is
set, the volume is created on that node. The `node_pool` must be unset or match
that node. Otherwise, the Nomad server iterates over the available nodes and
place the volume on the first available node based on the following factors:
* The node cannot already have a host volume with the same name.
* If `node_pool` is set, the selected node must be in that node pool.
* The node must meet any and all constraints defined by the `constraint` fields.
## Updating a Volume Definition
The `volume create` and `volume register` commands allow updating a volume
definition. However, after volume registration, you are only allowed to update
the following fields:
- `plugin_id`
- `capacity_min` and `capacity_max`. You may increase
the volume size if the plugin supports it. Expansion may or may not be
possible while the volume is in use, again depending on the plugin. Reducing
volume capacity is not allowed. Only available for volume creation.
- `capacity`, but only updated for volume registration.
- `constraint` fields. Ignored after the volume is created.
Additionally, you may add or remove `capability` blocks, but only if the
capability is not currently in use by a mounted volume.
You cannot update the `name`, `type`, and `node_id` fields. You may only change
the `node_pool` field from empty to the node pool that matches the `node_id`
field.
## Volume Expansion
Dynamic host volumes may be expanded if the plugin allows. Reducing the size of
a volume is not permitted.
To trigger a volume expansion, increase [`capacity_min`](#capacity_min) above
the current real capacity of the volume (as shown with the [`volume status`][]
command), and re-issue [`volume create`][].
Nomad reconciles the requested capacity by issuing a create request to the
plugin.
## Examples
### Volume creation
This is an example file used for the [`volume create`][] command.
```hcl
name = "database"
type = "host"
plugin_id = "lvm-thin-provisioner"
node_pool = "prod"
capacity_min = "80G"
capacity_max = "100G"
capability {
access_mode = "single-node-reader-only"
attachment_mode = "file-system"
}
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
parameters {
skuname = "Premium_LRS"
}
```
### Volume registration
This is an example file used for the [`volume register`][] command.
```hcl
name = "database"
type = "host"
node_id = "a7b4c0ca-cc78-11ef-8b5a-cb6ea67b844c"
host_path = "/var/srv/example"
capacity = "80G"
capability {
access_mode = "single-node-reader-only"
attachment_mode = "file-system"
}
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
parameters {
skuname = "Premium_LRS"
}
```
### Example Volume Expansion
This example shows how to expand a volume after it has been created or
registered.
1. Create a volume configuration file called `volume.hcl` that defines name,
type, plugin ID, and capacity.
```hcl
name = "database"
type = "host"
plugin_id = "external-plugin"
capacity_min = "30GiB"
capacity_max = "50GiB"
```
1. Create the volume using the [`volume create`] command.
```shell-session
$ nomad volume create ./volume.hcl
==> Created host volume database with ID 0c903229-311d-ba8a-f77e-45c31b83fab3
✓ Host volume "0c903229" ready
2025-01-06T16:56:09-05:00
ID = 0c903229-311d-ba8a-f77e-45c31b83fab3
Name = database
Namespace = default
Plugin ID = external-plugin
Node ID = cfe033a7-50de-2c46-cd18-12be7429eeb3
Node Pool = default
Capacity = 50 GiB
State = ready
Host Path = /run/nomad/dev/alloc_mounts/0c903229-311d-ba8a-f77e-45c31b83fab3
```
1. Review the volume's current capacity using the [`volume status`] command.
```shell-session
$ nomad volume status -type=host 0c903229 | grep Capacity
Capacity = 50 GiB
```
1. Increase volume capacity in the `volume.hcl` file.
Update the `capacity_min` and `capacity_max` fields.
```hcl
id = "0c903229-311d-ba8a-f77e-45c31b83fab3"
name = "database"
type = "host"
plugin_id = "external-plugin"
capacity_min = "100GiB" # double
capacity_max = "100GiB" # increased to match
```
1. Expand the volume using the [`volume create`] command.
The `volume create` command can trigger an expansion to occur, after the
volume has already been created or registered.
```shell-session
$ nomad volume create volume.hcl
Created host volume database with ID 0c903229-311d-ba8a-f77e-45c31b83fab3
```
1. Review the new capacity by running the [`volume status`] command.
```shell-session
$ nomad volume status -type=host 0c903229 | grep Capacity
Capacity = 100 GiB
```
[api_volume_create]: /nomad/api-docs/volumes#create-dynamic-host-volume
[api_volume_register]: /nomad/api-docs/volumes#register-dynamic-host-volume
[volume_source]: /nomad/docs/job-specification/volume#source
[capability]: /nomad/docs/other-specifications/volume/capability
[constraint]: /nomad/docs/job-specification/constraint
[`volume create`]: /nomad/docs/commands/volume/create
[`volume register`]: /nomad/docs/commands/volume/register
[`volume status`]: /nomad/docs/commands/volume/status
[dhv_plugin]: /nomad/docs/concepts/plugins/storage/host-volumes

282
website/content/docs/other-specifications/volume/index.mdx
View File

@@ -1,24 +1,32 @@
---
layout: docs
page_title: Volume Specification
description: Learn about the Volume specification used to create and register volumes to Nomad.
description: |-
Learn about the Nomad volume specification. Create and register Container
Storage Interface (CSI) and dynamic host volumes using the `volume create`
and `volume register` commands and the equivalent API endpoints. Define
capability, mount options, topology requests, secrets, and parameters.
---
# Volume Specification
The Nomad volume specification defines the schema for creating and registering
volumes using the [`volume create`] and [`volume register`] commands and the
[`PUT /v1/volume/csi/:volume_id/create`][api_volume_create] and [`PUT
/v1/volume/csi/:volume_id`][api_volume_register] API endpoints.
This page provides reference information for the Nomad volume
specification. Create and register Container Storage Interface (CSI) and dynamic
host volumes using the [`volume create`][] and [`volume register`][] commands
and the equivalent API endpoints. Define capability, mount options, topology
requests, secrets, and parameters.
Some attributes are only be supported by specific operation, while others may
have a different meaning for each action, so read the documentation for each
attribute carefully. The section [Differences Between Create and
Register](#differences-between-create-and-register) provides a summary of the
differences.
Some attributes are only be supported by one volume type or the other, or a
specific operation, while others may have a different meaning for each action,
so read the documentation for each attribute carefully.
The file may be provided as either HCL or JSON to the commands and as JSON to
the API. An example HCL configuration for a `volume create` command:
* [CSI volume specification][csi_spec]
* [Dynamic host volume specification][dhv_spec]
Provide the file as either HCL or JSON to the commands and as JSON to
the API. An example HCL configuration for a `volume create` command with a CSI
volume:
```hcl
id = "ebs_prod_db1"
@@ -64,255 +72,7 @@ parameters {
}
```
## Volume Specification Parameters
- `id` `(string: <required>)` - The unique ID of the volume. This is how the
[`volume.source`][csi_volume_source] field in a job specification will refer
to the volume.
- `namespace` `(string: <optional>)` - The namespace of the volume. This field
overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE`
environment variable. Defaults to `"default"` if unset.
- `name` `(string: <required>)` - The display name of the volume. On **volume
creation**, this field may be used by the external storage provider to tag
the volume.
- `type` `(string: <required>)` - The type of volume. Currently only `"csi"`
is supported.
- `external_id` `(string: <required>)` - The ID of the physical volume from
the storage provider. For example, the volume ID of an AWS EBS volume or
Digital Ocean volume. Only allowed on **volume registration**.
- `plugin_id` `(string: <required>)` - The ID of the [CSI plugin][csi_plugin]
that manages this volume.
- `snapshot_id` `(string: <optional>)` - If the storage provider supports
snapshots, the external ID of the snapshot to restore when creating this
volume. If omitted, the volume will be created from scratch. The
`snapshot_id` cannot be set if the `clone_id` field is set. Only allowed on
**volume creation**.
- `clone_id` `(string: <optional>)` - If the storage provider supports cloning,
the external ID of the volume to clone when creating this volume. If omitted,
the volume will be created from scratch. The `clone_id` cannot be set if the
`snapshot_id` field is set. Only allowed on **volume creation**.
- `capacity_min` `(string: <optional>)` - Option for requesting a minimum
capacity, in bytes. The capacity of a volume may be the physical size of a
disk, or a quota, depending on the storage provider. The specific size of the
resulting volume will be somewhere between `capacity_min` and `capacity_max`;
the exact behavior is up to the storage provider. If you want to specify an
exact size, you should set `capacity_min` and `capacity_max` to the same
value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not
be supported by all storage providers. Increasing this value and re-issuing
`volume create` or `volume register` may expand the volume, if the CSI plugin
supports it.
- `capacity_max` `(string: <optional>)` - Option for requesting a maximum
capacity, in bytes. The capacity of a volume may be the physical size of a
disk, or a quota, depending on the storage provider. The specific size of the
resulting volume will be somewhere between `capacity_min` and `capacity_max`;
the exact behavior is up to the storage provider. If you want to specify an
exact size, you should set `capacity_min` and `capacity_max` to the same
value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not
be supported by all storage providers.
- `capability` <code>([Capability][capability]: &lt;required&gt;)</code> -
Option for validating the capability of a volume.
- `mount_options` <code>([MountOptions][mount_options]: &lt;required&gt;)</code> -
Options for mounting `file-system` volumes that don't already have a
pre-formatted file system.
- `topology_request` <code>([TopologyRequest][topology_request]: nil)</code> -
Specify locations (region, zone, rack, etc.) where the provisioned volume
must be accessible from in the case of **volume creation** or the locations
where the existing volume is accessible from in the case of **volume
registration**.
- `secrets` <code>(map<string|string>:nil)</code> - An optional key-value map
of strings used as credentials for publishing and unpublishing volumes.
- `parameters` <code>(map<string|string>:nil)</code> - An optional key-value
map of strings passed directly to the CSI plugin to configure the volume. The
details of these parameters are specific to each storage provider, so consult
the specific plugin documentation for more information.
- `context` <code>(map<string|string>:nil)</code> - An optional key-value map
of strings passed directly to the CSI plugin to validate the volume. The
details of these parameters are specific to each storage provider, so consult
the specific plugin documentation for more information. Only allowed on
**volume registration**. Note that, like the rest of the volume specification,
this block is declarative, and an update replaces it in its entirety, therefore
all parameters need to be specified.
## Differences Between Create and Register
Several fields are set automatically by the plugin when `volume create` or
`volume register` commands are successful and you should not set their values
if they are not supported by the operation.
You should not set the [`snapshot_id`](#snapshot_id), or [`clone_id`](#clone_id)
fields on **volume registration**.
And you should not set the [`external_id`](#external_id) or
[`context`](#context) fields on **volume creation**.
## Updating a Volume Definition
The `volume register` command allows updating a volume definition. But not all
fields can be updated after the volume is registered:
* The `capacity_min` and `capacity_max` fields can be updated, and may increase
the volume size if the CSI plugin supports it. Expansion may or may not be
possible while the volume is in use, again depending on the plugin.
Reducing volume capacity is not allowed per the CSI spec.
* The `capability` blocks can be added or removed, but only if the capability is
not currently in use by a mounted volume.
* The `mount_options` block can be updated if the volume is not in use.
* The `secrets` block can be updated.
* The `context` block can be updated. The values for this field are typically
provided by the CSI plugin, and should not be updated unless recommended by
the CSI plugin's documentation.
## Volume Expansion
CSI Volumes may be expanded (increased in size) if the CSI controller plugin
(and node plugin, if required) has the `EXPAND_VOLUME` capability.
To trigger a volume expansion, increase [`capacity_min`](#capacity_min)
above the current real capacity of the volume (as seen with the
[`volume status`][] command), and re-issue either [`volume create`][]
or [`volume register`][].
Nomad will reconcile the requested capacity by issuing expand volume requests
to the controller plugin, and if required by the controller, also to the
node plugins for each allocation that has a claim on the volume.
## Examples
### Volume registration
This is an example file used for the [`volume register`] command.
```hcl
id = "ebs_prod_db1"
name = "database"
type = "csi"
external_id = "vol-23452345"
plugin_id = "ebs-prod"
capability {
access_mode = "single-node-reader-only"
attachment_mode = "file-system"
}
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
mount_options {
fs_type = "ext4"
mount_flags = ["noatime"]
}
topology_request {
required {
topology { segments { "rack" = "R2" } }
topology { segments { "rack" = "R1", "zone" = "us-east-1a"} }
}
}
secrets {
example_secret = "xyzzy"
}
parameters {
skuname = "Premium_LRS"
}
context {
endpoint = "http://192.168.1.101:9425"
}
```
### Example Volume Expansion
Either [`volume create`][] or [`volume register`][] commands can trigger
an expand to occur, after the volume has already been created or registered.
Example using `volume create`, with a `volume.hcl` file:
```hcl
id = "ebs_prod_db1"
name = "database"
type = "csi"
plugin_id = "ebs-prod"
capacity_min = "50gib"
capacity_max = "50gib"
# ... etc ...
```
Create the volume:
```shell-session
$ nomad volume create volume.hcl
Created external volume vol-asdf1234 with ID ebs_prod_db1
```
See its current capacity with [`volume status`][]:
```shell-session
$ nomad volume status ebs_prod_db1 | grep Capacity
Capacity = 50 GiB
```
Update `volume.hcl`:
```hcl
id = "ebs_prod_db1"
name = "database"
type = "csi"
plugin_id = "ebs-prod"
# all of the above must remain the same
capacity_min = "100gib" # double
capacity_max = "100gib" # increased to match
# ... etc ...
```
Run `volume create` again:
```shell-session
$ nomad volume create volume.hcl
Created external volume vol-asdf1234 with ID ebs_prod_db1
```
Check the volume capacity again:
```shell-session
$ nomad volume status ebs_prod_db1 | grep Capacity
Capacity = 100 GiB
```
If you encounter errors that are not clear from the command output,
the CSI plugin allocation logs and/or Nomad leader server logs may be
helpful.
[api_volume_create]: /nomad/api-docs/volumes#create-volume
[api_volume_register]: /nomad/api-docs/volumes#register-volume
[capability]: /nomad/docs/other-specifications/volume/capability
[csi_plugin]: /nomad/docs/job-specification/csi_plugin
[csi_volume_source]: /nomad/docs/job-specification/volume#source
[mount_options]: /nomad/docs/other-specifications/volume/mount_options
[topology_request]: /nomad/docs/other-specifications/volume/topology_request
[`volume create`]: /nomad/docs/commands/volume/create
[`volume register`]: /nomad/docs/commands/volume/register
[`volume status`]: /nomad/docs/commands/volume/status
[csi_spec]: /nomad/docs/other-specifications/volume/csi
[dhv_spec]: /nomad/docs/other-specifications/volume/host

8
website/content/docs/other-specifications/volume/mount_options.mdx
View File

@@ -1,7 +1,8 @@
---
layout: docs
page_title: mount_options Block - Volume Specification
description: The "mount_options" block allows for configuring how a volume is mounted.
description: |-
Configure Container Storage Interface (CSI) storage volume file system type and mount flags in the "mount_options" block of the Nomad volume specification.
---
# `mount_options` Block
@@ -12,8 +13,9 @@ description: The "mount_options" block allows for configuring how a volume is mo
]}
/>
Options for mounting `file-system` volumes that don't already have a
pre-formatted file system.
Options for mounting `file-system` CSI volumes that don't already have a
pre-formatted file system. The `mount_options` block is not supported for
dynamic host volumes.
```hcl
id = "ebs_prod_db1"

9
website/content/docs/other-specifications/volume/topology_request.mdx
View File

@@ -1,7 +1,8 @@
---
layout: docs
page_title: topology_request Block - Volume Specification
description: The "topology_request" block allows specifying locations where the provisioned volume must be accessible from.
description: |-
Configure Container Storage Interface (CSI) storage volume topology in the "topology_request" block of the Nomad volume specification. Specify region, zone, and rack so that Nomad can access a provisioned CSI volume. Review volume creation examples with preferred and required topologies.
---
# `topology_request` Block
@@ -12,8 +13,10 @@ description: The "topology_request" block allows specifying locations where the
]}
/>
Specify locations (region, zone, rack, etc.) where the provisioned volume must
be accessible from or from where an existing volume is accessible from.
Specify locations such as region, zone, and rack, where a provisioned CSI volume must
be accessible, or where an existing volume is accessible. The `topology_request`
block is not supported for dynamic host volumes.
```hcl
id = "ebs_prod_db1"

23
website/data/docs-nav-data.json
View File

@@ -167,7 +167,20 @@
},
{
"title": "Storage",
"path": "concepts/plugins/csi"
"routes": [
{
"title": "Overview",
"path": "concepts/plugins/storage"
},
{
"title": "CSI",
"path": "concepts/plugins/storage/csi"
},
{
"title": "Host Volumes",
"path": "concepts/plugins/storage/host-volumes"
}
]
},
{
"title": "Networking",
@@ -1947,6 +1960,14 @@
"title": "Overview",
"path": "other-specifications/volume"
},
{
"title": "CSI",
"path": "other-specifications/volume/csi"
},
{
"title": "Dynamic Host Volumes",
"path": "other-specifications/volume/host"
},
{
"title": "capability",
"path": "other-specifications/volume/capability"

6
website/redirects.js
View File

@@ -98,6 +98,12 @@ module.exports = [
destination: '/nomad/plugins/drivers/community/',
permanent: true,
},
// CSI plugins moved under new storage path alongside new host volume plugins
{
source: '/nomad/docs/concepts/plugins/csi',
destination: '/nomad/docs/concepts/plugins/storage/csi',
permanent: true,
},
{
source: '/nomad/plugins/drivers/virt/client',
destination: '/nomad/plugins/drivers/virt/install',