diff --git a/website/content/api-docs/volumes.mdx b/website/content/api-docs/volumes.mdx index 9b56c16e8..5995137c9 100644 --- a/website/content/api-docs/volumes.mdx +++ b/website/content/api-docs/volumes.mdx @@ -1,13 +1,14 @@ --- layout: api page_title: Volumes - HTTP API -description: The `/volume` endpoints are used to query for and interact with volumes. +description: |- + The Nomad `/volume` and `/volumes` endpoints query for and interact with Container Storage Interface (CSI) volumes and dynamic host volumes. --- # Volumes HTTP API -The `/volume` and `/volumes` endpoints are used to query for and interact with -volumes. +The `/volume` and `/volumes` endpoints query for and interact with +Container Storage Interface (CSI) volumes and dynamic host volumes. ## List Volumes @@ -17,29 +18,28 @@ This endpoint lists all volumes. | ------ | ------------- | ------------------ | | `GET` | `/v1/volumes` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. -| Blocking Queries | ACL Required | -| ---------------- | --------------------------- | -| `YES` | `namespace:csi-list-volume` | +| Blocking Queries | ACL Required | +|------------------|--------------------------------------------------------------| +| `YES` | `namespace:csi-list-volume`
`namespace:host-volume-read` | ### Parameters -- `type` `(string: "")` - Specifies the type of volume to - query. Currently only supports `csi`. This is specified as a query - string parameter. Returns an empty list if omitted. +- `type` `(string: )` - Specifies the type of volume to query. One of + `csi` or `host`. The `host` value queries dynamic host volumes. Specify this + as a query string parameter. - `node_id` `(string: "")` - Specifies a string to filter volumes based on an Node ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters - (0-9a-f). This is specified as a query string parameter. + (0-9a-f). Specify this as a query string parameter. - `plugin_id` `(string: "")` - Specifies a string to filter volumes based on a plugin ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters - (0-9a-f). This is specified as a query string parameter. + (0-9a-f). Specify this as a query string parameter. - `next_token` `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string which identifies the next expected volume. This @@ -62,7 +62,7 @@ $ curl \ https://localhost:4646/v1/volumes?type=csi&node_id=foo&plugin_id=plugin-id1 ``` -### Sample Response +### Sample Response for CSI Volumes ```json [ @@ -95,17 +95,37 @@ $ curl \ ] ``` -## Read Volume +### Sample Response for dynamic host volumes -This endpoint reads information about a specific volume. +```json +[ + { + "CapacityBytes": 1048576000, + "CreateIndex": 42, + "CreateTime": 1736191825846395400, + "ID": "3735cc2c-cc64-11ef-89ed-bfb5b3bc38ea", + "ModifyIndex": 64, + "ModifyTime": 1736191825846395400, + "Name": "example", + "Namespace": "default", + "NodeID": "5c5830d0-cc64-11ef-a293-4f03e55969ea", + "NodePool": "default", + "PluginID": "plugin-id1", + "State": "ready" + } +] +``` + +## Read CSI Volume + +This endpoint reads information about a specific CSI volume. | Method | Path | Produces | | ------ | --------------------------- | ------------------ | | `GET` | `/v1/volume/csi/:volume_id` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | --------------------------- | @@ -114,7 +134,7 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Request @@ -275,10 +295,11 @@ $ curl \ } ``` -## Register Volume +## Register CSI Volume -This endpoint registers an external volume with Nomad. The volume must exist -in the external storage provider (see [Create Volume] below). +This endpoint registers an external CSI volume with Nomad. The volume must exist +in the external storage provider. Refer to the [Create CSI Volume][] section for +details. Making the same request again with a higher `RequestedCapacityMin` value may trigger a [Volume Expansion][]. @@ -287,9 +308,8 @@ may trigger a [Volume Expansion][]. | ------ | --------------------------- | ------------------ | | `PUT` | `/v1/volume/csi/:volume_id` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -298,13 +318,13 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Payload The payload must include a JSON document that describes the volume's -parameters. +parameters. Note that the `NodeID` field is required for the register API. ```json { @@ -343,9 +363,9 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id1 ``` -## Create Volume +## Create CSI Volume -This endpoint creates a volume in an external storage provider and registers +This endpoint creates a CSI volume in an external storage provider and registers it with Nomad. Only CSI plugins that implement the [Controller][csi_plugins_internals] interface with the `CREATE_DELETE_VOLUME` capability support this endpoint. @@ -357,9 +377,8 @@ may trigger a [Volume Expansion][]. | ------ | ---------------------------------- | ------------------ | | `PUT` | `/v1/volume/csi/:volume_id/create` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -368,7 +387,7 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Payload @@ -420,18 +439,17 @@ $ curl \ ``` -## Deregister Volume +## Deregister CSI Volume -This endpoint deregisters an external volume with Nomad. It is an error to +This endpoint deregisters an external CSI volume from Nomad. It is an error to deregister a volume that is in use. | Method | Path | Produces | | -------- | --------------------------- | ------------------ | | `DELETE` | `/v1/volume/csi/:volume_id` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -440,13 +458,13 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. - `force` `(bool: false)` - Force deregistration of the volume and immediately drop claims for terminal allocations. Returns an error if the volume has running allocations. This does not detach the volume from client nodes. - This is specified as a query string parameter. + Specify this as a query string parameter. ### Sample Request @@ -456,9 +474,9 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id1?force=false ``` -## Delete Volume +## Delete CSI Volume -This endpoint deletes an external volume from the storage provider, and +This endpoint deletes an external CSI volume from the storage provider, and deregisters it from Nomad. It is an error to delete a volume that is in use. Only CSI plugins that implement the [Controller][csi_plugins_internals] interface with the `CREATE_DELETE_VOLUME` capability support this endpoint. @@ -467,9 +485,8 @@ interface with the `CREATE_DELETE_VOLUME` capability support this endpoint. | -------- | ---------------------------------- | ------------------ | | `DELETE` | `/v1/volume/csi/:volume_id/delete` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -483,7 +500,7 @@ stored when the CSI volume was created. ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Request @@ -495,18 +512,17 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id1/delete ``` -## Detach Volume +## Detach CSI Volume -This endpoint detaches an external volume from a Nomad client node. It is an +This endpoint detaches an external CSI volume from a Nomad client node. It is an error to detach a volume that is in use. | Method | Path | Produces | | -------- | ---------------------------------- | ------------------ | | `DELETE` | `/v1/volume/csi/:volume_id/detach` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -515,11 +531,11 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. - `node` `(string: )` - The node to detach the volume from. - This is specified as a query string parameter. + Specify this as a query string parameter. ### Sample Request @@ -529,20 +545,19 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id/detach?node=00000000-0000-0000-0000-000000000000 ``` -## List External Volumes +## List External CSI Volumes -This endpoint lists storage volumes that are known to the external storage -provider but may not be registered with Nomad. Only CSI plugins that -implement the [Controller][csi_plugins_internals] interface with the -`LIST_VOLUMES` capability support this endpoint. +This endpoint lists storage CSI volumes that are known to the external storage +provider but may not be registered with Nomad. Only CSI plugins that implement +the [Controller][csi_plugins_internals] interface with the `LIST_VOLUMES` +capability support this endpoint. | Method | Path | Produces | |--------|------------------------|--------------------| | `GET` | `/v1/volumes/external` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -553,14 +568,14 @@ The table below shows this endpoint's support for - `plugin_id` `(string: "")` - Specifies a string to filter volumes based on a plugin ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters - (0-9a-f). This is specified as a query string parameter. + (0-9a-f). Specify this as a query string parameter. - `next_token` `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string returned in a previous response's `NextToken` field to request the next page of results. - `per_page` `(int: )` - Specifies a maximum number of snapshots to - return for this request. The response will include a `NextToken` field that + return for this request. The response includes a `NextToken` field that can be passed to the next request to fetch additional pages. ### Sample Request @@ -596,20 +611,19 @@ $ curl \ } ``` -## Create Snapshot +## Create CSI Volume Snapshot -This endpoint creates a snapshot of a volume on the external storage +This endpoint creates a snapshot of a CSI volume on the external storage provider. Only CSI plugins that implement the -[Controller][csi_plugins_internals] interface with the -`CREATE_DELETE_SNAPSHOT` capability support this endpoint. +[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_SNAPSHOT` +capability support this endpoint. | Method | Path | Produces | | -------- | --------------------------------- | ------------------ | | `POST` | `/v1/volumes/snapshot` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -664,20 +678,19 @@ $ curl \ } ``` -## Delete Snapshot +## Delete CSI Volume Snapshot -This endpoint deletes a volume snapshot from the external storage +This endpoint deletes a CSI volume snapshot from the external storage provider. Only CSI plugins that implement the -[Controller][csi_plugins_internals] interface with the -`CREATE_DELETE_SNAPSHOT` capability support this endpoint. +[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_SNAPSHOT` +capability support this endpoint. | Method | Path | Produces | | -------- | ---------------------- | ------------------ | | `DELETE` | `/v1/volumes/snapshot` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -691,12 +704,12 @@ stored when the CSI snapshot was created. ### Parameters - `plugin_id` `(string: )` - Specifies the prefix of a CSI plugin ID - to perform the delete. Because the value is decoded to bytes, the prefix - must have an even number of hexadecimal characters (0-9a-f). This is - specified as a query string parameter. + to perform the delete. Because the value is decoded to bytes, the prefix must + have an even number of hexadecimal characters (0-9a-f). Specify this as a + query string parameter. - `snapshot_id` `(string: )` - Specifies the snapshot ID to - delete. This is specified as a query parameter. + delete. Specify this as a query parameter. ### Sample Request @@ -707,9 +720,9 @@ $ curl \ https://localhost:4646/v1/volumes/snapshot ``` -## List Snapshots +## List CSI Volume Snapshots -This endpoint lists volume snapshots on the external storage provider. Only +This endpoint lists CSI volume snapshots on the external storage provider. Only CSI plugins that implement the [Controller][csi_plugins_internals] interface with the `LIST_SNAPSHOTS` capability support this endpoint. @@ -718,9 +731,8 @@ with the `LIST_SNAPSHOTS` capability support this endpoint. | ------ | ---------------------- | ------------------ | | `GET` | `/v1/volumes/snapshot` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | --------------------------- | @@ -735,7 +747,7 @@ stored when the CSI snapshot was created. - `plugin_id` `(string: )` - Specifies the prefix of a CSI plugin ID to perform the list. Because the value is decoded to bytes, the prefix must - have an even number of hexadecimal characters (0-9a-f). This is specified as + have an even number of hexadecimal characters (0-9a-f). Specify this as a query string parameter. - `next_token` `(string: "")` - This endpoint supports paging. The @@ -743,7 +755,7 @@ stored when the CSI snapshot was created. `NextToken` field to request the next page of results. - `per_page` `(int: )` - Specifies a maximum number of snapshots to - return for this request. The response will include a `NextToken` field that + return for this request. The response includes a `NextToken` field that can be passed to the next request to fetch additional pages. ### Sample Request @@ -782,8 +794,335 @@ $ curl \ } ``` +## Read Dynamic Host Volume + +This endpoint reads information about a specific dynamic host volume. + +| Method | Path | Produces | +|--------|------------------------------|--------------------| +| `GET` | `/v1/volume/host/:volume_id` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|------------------------------| +| `YES` | `namespace:host-volume-read` | + +### Parameters + +- `:volume_id` `(string: )` - Specifies the ID of the + volume. This must be the full ID. Specify this as part of the + path. + +### Sample Request + +```shell-session +$ curl \ + https://localhost:4646/v1/volume/host/c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f +``` + +### Sample Response + +```json +{ + "Allocations": [], + "CapacityBytes": 49283072, + "CreateIndex": 11, + "CreateTime": 1736191993011594200, + "HostPath": "/run/nomad/dev/alloc_mounts/c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ModifyIndex": 12, + "ModifyTime": 1736191993011594200, + "Name": "external-plugin", + "Namespace": "default", + "NodeID": "670cb259-bc26-653b-e316-655af04ad260", + "NodePool": "default", + "Parameters": { + "hello": "world" + }, + "PluginID": "example-plugin-mkfs", + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + } + ], + "RequestedCapacityMaxBytes": 50000000, + "RequestedCapacityMinBytes": 50000000, + "State": "ready" +} +``` + +## Create Dynamic Host Volume + +This endpoint creates a dynamic host volume and registers it with Nomad. + +The response body includes the volume definition with the `ID` and `NodeID` +fields populated. Making the same request without the ID creates a new volume on +a different node. Making a request with the ID set updates the volume to the new +parameters, if possible. + +| Method | Path | Produces | +|--------|--------------------------|--------------------| +| `PUT` | `/v1/volume/host/create` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|--------------------------------------------------------------------------------------------------------------------------------| +| `NO` | `namespace:host-volume-create` or
`namespace:host-volume-write`.
`namespace:sentinel-override` if `PolicyOverride` set | + +### Parameters + +- `Volume` `(Volume: )` - Specifies the JSON definition of the host + volume. You should include the ID field if you are updating an existing + volume. + +- `PolicyOverride` `(bool: false)` - If set, Nomad overrides any soft mandatory + Sentinel policies. This field allows creating a volume when it would be denied + by policy. + +### Sample Payload + +```json +{ + "PolicyOverride": false, + "Volume": { + "Name": "example", + "Namespace": "default", + "NodePool": "prod", + "PluginID": "mkdir", + "RequestedCapacityMinBytes": 10737418240, + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "Parameters": { + "foo": "bar" + } + } +} +``` + +### Sample Request + +```shell-session +$ curl \ + --request PUT \ + --data @payload.json \ + https://localhost:4646/v1/volume/host/create +``` + +### Sample Response + +```json +{ + "Volume": { + "Allocations": [], + "CapacityBytes": 21474836480, + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "CreateIndex": 11, + "CreateTime": 1736191993011594200, + "ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ModifyIndex": 11, + "ModifyTime": 1736191993011594200, + "Name": "example", + "Namespace": "default", + "NodeID": "45460554-cc67-11ef-84b7-33d383a55487", + "NodePool": "prod", + "Parameters": { + "foo": "bar" + }, + "PluginID": "mkdir", + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapacityMinBytes": 10737418240, + "State": "pending" + }, + "Warnings": null +} +``` + +## Register Dynamic Host Volume + +This endpoint registers an existing dynamic host volume with Nomad. + +| Method | Path | Produces | +|--------|----------------------------|--------------------| +| `PUT` | `/v1/volume/host/register` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|----------------------------------------------------------------------------------------------------------------------------------| +| `NO` | `namespace:host-volume-register` or
`namespace:host-volume-write`.
`namespace:sentinel-override` if `PolicyOverride` set | + +### Parameters + +- `Volume` `(Volume: )` - Specifies the JSON definition of the host + volume. You should include the ID field if you are updating an existing + volume. + +- `PolicyOverride` `(bool: false)` - If set, Nomad overrides any soft mandatory + Sentinel policies. This field allows registering a volume when it would be denied + by policy. + +### Sample Payload + +```json +{ + "PolicyOverride": false, + "Volume": { + "Name": "example", + "Namespace": "default", + "NodePool": "prod", + "PluginID": "mkdir", + "RequestedCapacityMinBytes": 10737418240, + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "Parameters": { + "foo": "bar" + } + } +} +``` + +### Sample Request + +```shell-session +$ curl \ + --request PUT \ + --data @payload.json \ + https://localhost:4646/v1/volume/host/register +``` + +### Sample Response + +```json +{ + "Volume": { + "Allocations": [], + "CapacityBytes": 21474836480, + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "CreateIndex": 11, + "CreateTime": 1736191993011594200, + "ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ModifyIndex": 31, + "ModifyTime": 1736191993721594200, + "Name": "example", + "Namespace": "default", + "NodeID": "45460554-cc67-11ef-84b7-33d383a55487", + "NodePool": "prod", + "Parameters": { + "foo": "bar" + }, + "PluginID": "mkdir", + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapacityMinBytes": 10737418240, + "State": "ready" + }, + "Warnings": null +} +``` + +## Delete Dynamic Host Volume + +This endpoint deletes a dynamic host volume, and deregisters it from Nomad. It +is an error to delete a volume that is in use. + +| Method | Path | Produces | +|----------|-------------------------------------|--------------------| +| `DELETE` | `/v1/volume/host/:volume_id/delete` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|---------------------------------------------------------------------| +| `NO` | `namespace:host-volume-write` or
`namespace:host-volume-delete` | + +### Parameters + +- `:volume_id` `(string: )` - Specifies the ID of the + volume. This must be the full ID. Specify this as part of the + path. + +### Sample Request + +```shell-session +$ curl \ + --request DELETE \ + https://localhost:4646/v1/volume/host/ba97ef42-cc68-11ef-a2e7-ffddaecbdb89 +``` + +[blocking queries]: /nomad/api-docs#blocking-queries +[required ACLs]: /nomad/api-docs#acls [csi]: https://github.com/container-storage-interface/spec [csi_plugin]: /nomad/docs/job-specification/csi_plugin [csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins -[Create Volume]: #create-volume -[Volume Expansion]: /nomad/docs/other-specifications/volume#volume-expansion +[Create CSI Volume]: #create-csi-volume +[Volume Expansion]: /nomad/docs/other-specifications/volume/csi#volume-expansion diff --git a/website/content/docs/commands/volume/create.mdx b/website/content/docs/commands/volume/create.mdx index f383469f0..798390783 100644 --- a/website/content/docs/commands/volume/create.mdx +++ b/website/content/docs/commands/volume/create.mdx @@ -2,15 +2,19 @@ layout: docs page_title: 'Commands: volume create' description: | - Create volumes with CSI plugins. + The `nomad volume create` command creates storage volumes that are either + Container Storage Interface (CSI) volumes or dynamic host volumes. --- # Command: volume create -The `volume create` command creates external storage volumes with Nomad's -[Container Storage Interface (CSI)][csi] support. Only CSI plugins that -implement the [Controller][csi_plugins_internals] interface support this -command. The volume will also be [registered] when it is successfully created. +The `volume create` command creates storage volumes as either [Container Storage +Interface (CSI)][csi] volumes or dynamic host volumes. + +The [`volume create`][] command can create dynamic host volumes if host volume +plugins are installed on the node, or CSI volumes if the CSI plugins implement +the [Controller][csi_plugins_internals] interface. The `volume create` command +automatically [registers][] the volume as well. ## Usage @@ -18,21 +22,41 @@ command. The volume will also be [registered] when it is successfully created. nomad volume create [options] [file] ``` -The `volume create` command requires a single argument, specifying the path to -a file containing a valid [volume specification][volume_specification]. This -file will be read and the volume will be submitted to Nomad for scheduling. If -the supplied path is "-", the volume file is read from STDIN. Otherwise it is +The `volume create` command requires a single argument, specifying the path to a +file containing a valid [volume specification][volume_specification]. Nomad +reads the file and submits the volume to the server for placement. If the +supplied path is "-", the volume file is read from STDIN. Otherwise the file is read from the file at the supplied path. -When ACLs are enabled, this command requires a token with the -`csi-write-volume` capability for the volume's namespace. +When ACLs are enabled, this command requires a token with the appropriate +capability in the volume's namespace: the `csi-write-volume` capability for CSI +volumes or `host-volume-create` for dynamic host volumes. ## General Options @include 'general_options.mdx' +## Volume Create Options + +- `-detach`: Return immediately instead of entering monitor mode for dynamic + host volumes. After creating a volume, Nomad prints the volume ID to the + screen, which you can use to examine the volume using the `volume status` + command. If `-detach` is omitted or false, the command monitors the state of + the volume until it has been fingerprinted by the client and is ready to be + scheduled. Not valid for CSI volumes. + +- `-verbose`: Display full information when monitoring volume state. Used for + dynamic host volumes only. Not valid for CSI volumes. + +- `-policy-override`: Sets the flag to force override any soft mandatory + Sentinel policies. Used for dynamic host volumes only. Not valid for CSI + volumes. + ## Volume Specification +Refer to the [CSI Volume Specification][csi_vol_spec] and the [Dynamic Host +Volume Specification][host_vol_spec] for further information. + diff --git a/website/content/docs/other-specifications/namespace.mdx b/website/content/docs/other-specifications/namespace.mdx index 20e711ea6..7d091ae2f 100644 --- a/website/content/docs/other-specifications/namespace.mdx +++ b/website/content/docs/other-specifications/namespace.mdx @@ -108,20 +108,23 @@ consul { ### `node_pool_config` Parameters -- `default` `(string: "default")` - Specifies the node pool to use for jobs in - this namespace that don't define a node pool in their specification. +- `default` `(string: "default")` - Specifies the node pool to use for jobs or + dynamic host volumes in this namespace that don't define a node pool in their + specification. -- `allowed` `(array: nil)` - Specifies the node pools that are allowed - to be used by jobs in this namespace. By default, all node pools are allowed. - If an empty list is provided only the namespace's default node pool is - allowed. This field supports wildcard globbing through the use of `*` for - multi-character matching. This field cannot be used with `denied`. +- `allowed` `(array: nil)` - Specifies the node pools that jobs or + dynamic host volumes in this namespace are allowed to use. By default, all + node pools are allowed. If an empty list is provided only the namespace's + default node pool is allowed. This field supports wildcard globbing through + the use of `*` for multi-character matching. This field cannot be used with + `denied`. -- `denied` `(array: nil)` - Specifies the node pools that are not - allowed to be used by jobs in this namespace. This field supports wildcard - globbing through the use of `*` for multi-character matching. If specified, - any node pool is allowed to be used, except for those that match any of these - patterns. This field cannot be used with `allowed`. +- `denied` `(array: nil)` - Specifies the node pools that jobs or + dynamic host volumes in this namespace are not allowed to use. This field + supports wildcard globbing through the use of `*` for multi-character + matching. If specified, jobs and dynamic host volumes are allowed to use any + node pool, except for those that match any of these patterns. This field + cannot be used with `allowed`. ### `vault` Parameters diff --git a/website/content/docs/other-specifications/quota.mdx b/website/content/docs/other-specifications/quota.mdx index 3245fb58e..ebb4d84b9 100644 --- a/website/content/docs/other-specifications/quota.mdx +++ b/website/content/docs/other-specifications/quota.mdx @@ -35,11 +35,16 @@ limit { cpu = 2500 memory = 1000 memory_max = 1000 + device "nvidia/gpu/1080ti" { count = 1 } + + storage { + variables = 1000 # in MB + host_volumes = "1000 GiB" + } } - variables_limit = 1000 } ``` @@ -56,9 +61,6 @@ in a Nomad region. - `region` `(string)` - The Nomad `region` that the limit applies to. - `region_limit` ([RegionLimit](#region_limit-parameters)) - Resources to limit. -- `variables_limit` `(int: 0)` - Maximum total size of all Nomad -[`variables`][] in MiB. The default `0` means unlimited, and `-1` means -variables are fully disabled. ### `region_limit` parameters @@ -75,6 +77,7 @@ all `resources.memory` in the namespace. - `memory_max` `(int: )` - The limit on total mount of hard memory limits in MB from all `resources.memory_max` in the namespace. - `device` ([Device](#device-parameters): nil) +- `storage` ([Storage](#storage-parameters): nil) ### `device` parameters @@ -83,12 +86,27 @@ equivalent to the [`device`][] block in a job specification. - `count` `(int)` - How many of this device may be used. +### `storage` parameters + +The `storage` block defines limits on provisioned storage. + +- `host_volumes` `(int: 0)` - Maximum total size of all [dynamic host volumes][] + in MiB. The default `0` means unlimited, and `-1` means variables are fully + disabled. This field accepts human-friendly string inputs such as "100 + GiB". The quota for host volumes is enforced at the time the volume is created + via [`volume create`][]. + +- `variables` `(int: 0)` - Maximum total size of all Nomad [variables][] in + MiB. The default `0` means unlimited, and `-1` means variables are fully + disabled. This field accepts human-friendly string inputs such as "100 GiB". + [Resource Quotas]: /nomad/docs/enterprise#resource-quotas [`quota`]: /nomad/docs/commands/quota [Quota HTTP API]: /nomad/api-docs/quotas [Quotas tutorial]: /nomad/tutorials/governance-and-policy/quotas [`Namespace`]: /nomad/docs/other-specifications/namespace -[`variables`]: /nomad/docs/concepts/variables +[variables]: /nomad/docs/concepts/variables [`resources`]: /nomad/docs/job-specification/resources [CPU concepts]: /nomad/docs/concepts/cpu [`device`]: /nomad/docs/job-specification/device#device-parameters +[dynamic host volumes]: /nomad/docs/other-specifications/volume/host diff --git a/website/content/docs/other-specifications/volume/capability.mdx b/website/content/docs/other-specifications/volume/capability.mdx index c03e20142..92f4b7364 100644 --- a/website/content/docs/other-specifications/volume/capability.mdx +++ b/website/content/docs/other-specifications/volume/capability.mdx @@ -1,7 +1,8 @@ --- layout: docs page_title: capability Block - Volume Specification -description: The "capability" block allows for validating the capability of a volume. +description: |- + Configure Container Storage Interface (CSI) and dynamic host storage volume capability in the "capability" block of the Nomad volume specification. Set single node or multiple node access and file system or block device attachment mode. --- # `capability` Block @@ -35,18 +36,43 @@ for each capability you intend to use in a job's [`volume`] block. ## `capability` Parameters -- `access_mode` `(string: )` - Defines whether a volume should be -available concurrently. Can be one of `"single-node-reader-only"`, -`"single-node-writer"`, `"multi-node-reader-only"`, -`"multi-node-single-writer"`, or `"multi-node-multi-writer"`. Most CSI plugins -support only single-node modes. Consult the documentation of the storage -provider and CSI plugin. +- `access_mode` `(string)` - Defines whether a volume should be available + concurrently. The `access_mode` and `attachment_mode` from the volume request + must exactly match one of the volume's `capability` blocks. -- `attachment_mode` `(string: )` - The storage API that will be used -by the volume. Most storage providers will support `"file-system"`, to mount -volumes using the CSI filesystem API. Some storage providers will support -`"block-device"`, which will mount the volume with the CSI block device API -within the container. + - For CSI volumes the `access_mode` is required. Can be one of the following: + + - `"single-node-reader-only"` + - `"single-node-writer"` + - `"multi-node-reader-only"` + - `"multi-node-single-writer"` + - `"multi-node-multi-writer"` + + Most CSI plugins support only single-node modes. + Consult the documentation of the storage provider and CSI plugin. + + - For dynamic host volumes the `access_mode` is optional. Can be one of the following: + + - `"single-node-writer"` + - `"single-node-reader-only"` + - `"single-node-single-writer"` + - `"single-node-multi-writer"` + + In the job specification, the default is `single-node-writer` unless + `read_only = true`, which translates to `single-node-reader-only`. + +- `attachment_mode` `(string)` - The storage API used by the volume. One of + `"file-system"` or `"block-device"`. The `access_mode` and `attachment_mode` + from the volume request must exactly match one of the volume's `capability` + blocks. + + - For CSI volumes the `attachment_mode` field is required. Most storage + providers support `"file-system"`, to mount volumes using the CSI + filesystem API. Some storage providers support `"block-device"`, which + mounts the volume with the CSI block device API within the container. + + - For dynamic host volumes the `attachment_mode` field is optional and + defaults to `"file-system"`. ## `capability` Examples diff --git a/website/content/docs/other-specifications/volume/csi.mdx b/website/content/docs/other-specifications/volume/csi.mdx new file mode 100644 index 000000000..dd38c12fe --- /dev/null +++ b/website/content/docs/other-specifications/volume/csi.mdx @@ -0,0 +1,291 @@ +--- +layout: docs +page_title: CSI Volume Specification +description: |- + Learn about the Nomad Container Storage Interface (CSI) volume + specification. Create and register CSI volumes using the `volume create` + and `volume register` commands and the + `PUT /v1/volume/csi/:volume_id/create` and + `PUT /v1/volume/csi/:volume_id` API endpoints. Define capacity, capabilities, mount + options, secrets, topology requests, and a context map of values passed + directly to the CSI plugin to validate the volume. Learn how volume + creation and registration are different. Additionally, learn how to + expand the size of a volume and update a volume definition. +--- + +# CSI Volume Specification + +This page provides reference information for the Nomad Container Storage +Interface (CSI) volume specification. Create and register CSI volumes using the +[`volume create`][] and [`volume register`][] commands and the [`PUT +/v1/volume/csi/:volume_id/create`][api_volume_create] and [`PUT +/v1/volume/csi/:volume_id`][api_volume_register] API endpoints. Define capacity, +capabilities, mount options, secrets, topology requests, and a context map of +values passed directly to the CSI plugin to validate the volume. Learn how +volume creation and registration are different. Additionally, learn how to +expand the size of a volume and update a volume definition. + +Some attributes are only supported by a specific operation, while others may +have a different meaning for each action, so read the documentation for each +attribute carefully. The section [Differences Between Create and +Register](#differences-between-create-and-register) provides a summary of the +differences. + +Provide the file as either HCL or JSON to the commands and as JSON to the API. + +## Volume Specification Parameters + +- `capability` ([Capability][capability]: <required>) - + Option for validating the capability of a volume. + +- `capacity_min` `(string: )` - Option for requesting a minimum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the storage provider. The specific size of the + resulting volume is somewhere between `capacity_min` and `capacity_max`; the + exact behavior is up to the storage provider. If you want to specify an exact + size, you should set `capacity_min` and `capacity_max` to the same + value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not + be supported by all storage providers. Increasing this value and reissuing + `volume create` or `volume register` may expand the volume, if the CSI plugin + supports it. + +- `capacity_max` `(string: )` - Option for requesting a maximum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the storage provider. The specific size of the + resulting volume is somewhere between `capacity_min` and `capacity_max`; the + exact behavior is up to the storage provider. If you want to specify an exact + size, you should set `capacity_min` and `capacity_max` to the same + value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not + be supported by all storage providers. + +- `clone_id` `(string: )` - If the storage provider supports cloning, + the external ID of the volume to clone when creating this volume. If omitted, + the volume is created from scratch. The `clone_id` cannot be set if the + `snapshot_id` field is set. Only allowed on volume creation. + +- `context` (map:nil) - An optional key-value map of + strings passed directly to the CSI plugin to validate the volume. The details + of these parameters are specific to each storage provider, so consult the + specific plugin documentation for more information. Only allowed on volume + registration. Note that, like the rest of the volume specification, this block + is declarative, and an update replaces it in its entirety, therefore all + parameters need to be specified. + +- `external_id` `(string: )` - The ID of the physical volume from the + storage provider. For example, the volume ID of an AWS EBS volume or Digital + Ocean volume. Only allowed on volume registration. + +- `id` `(string: )` - The unique ID of the volume. This is how the + [`volume.source`][csi_volume_source] field in a job specification refers to + the volume. + +- `mount_options` ([MountOptions][mount_options]: + <required>) - Options for mounting `file-system` volumes that + don't already have a pre-formatted file system. + +- `name` `(string: )` - The display name of the volume. On volume + creation, this field may be used by the external storage provider to tag the + volume. + +- `namespace` `(string: )` - The namespace of the volume. This field + overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE` + environment variable. Defaults to `"default"` if unset. + +- `parameters` (map:nil) - An optional key-value map + of strings passed directly to the CSI plugin to configure the volume. The + details of these parameters are specific to each storage provider, so consult + the specific plugin documentation for more information. + +- `plugin_id` `(string: )` - The ID of the [CSI plugin][csi_plugin] + that manages this volume. + +- `secrets` (map:nil) - An optional key-value map of + strings used as credentials for publishing and unpublishing volumes. + +- `snapshot_id` `(string: )` - If the storage provider supports + snapshots, the external ID of the snapshot to restore when creating this + volume. If omitted, the volume is created from scratch. The `snapshot_id` + cannot be set if the `clone_id` field is set. Only allowed on volume creation. + +- `topology_request` ([TopologyRequest][topology_request]: nil) - + Specify locations such as region, zone, and rack where the provisioned volume + must be accessible from in the case of volume creation, or the locations where + the existing volume is accessible from in the case of volume registration. + +- `type` `(string: )` - The type of volume. Must be `"csi"` for CSI + volumes. + +## Differences Between Create and Register + +Several fields are set automatically by the plugin when `volume create` or +`volume register` commands are successful and you should not set their values if +they are not supported by the operation. + +On **volume registration** you should not set the [`snapshot_id`](#snapshot_id), +or [`clone_id`](#clone_id) fields. + +On **volume creation** you should not set the [`external_id`](#external_id) or +[`context`](#context) fields. + +## Updating a Volume Definition + +The `volume register` command allows updating a volume definition. However, +after volume registration, you are only allowed to update the following fields: + +* `capacity_min` and `capacity_max`. You may increase the volume size if the CSI + plugin supports it. Expansion may or may not be possible while the volume is + in use, again depending on the plugin. Reducing volume capacity is not + allowed per the CSI spec. +* `mount_options` block if the volume is not in use. +* `secrets` block +* `context` block. The values for this field are typically provided by the CSI + plugin and should not be updated unless recommended by the CSI plugin's + documentation. + +Additionally, you may add or remove `capability` blocks, but only if the +capability is not currently in use by a mounted volume. + +## Volume Expansion + +You may expand CSI Volumes if the CSI controller plugin and node plugin, if +required, has the `EXPAND_VOLUME` capability. + +To trigger a volume expansion, increase [`capacity_min`](#capacity_min) above +the current real capacity of the volume, as received from the [`volume +status`][] command, and re-issue either [`volume create`][] or [`volume +register`][]. + +Nomad reconciles the requested capacity by issuing expand volume requests to the +controller plugin, and if required by the controller, also to the node plugins +for each allocation that has a claim on the volume. + +## Examples + +### Volume registration + +This is an example file used for the [`volume register`][] command. + +```hcl +id = "ebs_prod_db1" +name = "database" +type = "csi" +external_id = "vol-23452345" +plugin_id = "ebs-prod" + +capability { + access_mode = "single-node-reader-only" + attachment_mode = "file-system" +} + +capability { + access_mode = "single-node-writer" + attachment_mode = "file-system" +} + +mount_options { + fs_type = "ext4" + mount_flags = ["noatime"] +} + +topology_request { + required { + topology { segments { "rack" = "R2" } } + topology { segments { "rack" = "R1", "zone" = "us-east-1a"} } + } +} + +secrets { + example_secret = "xyzzy" +} + +parameters { + skuname = "Premium_LRS" +} + +context { + endpoint = "http://192.168.1.101:9425" +} +``` + + +### Example Volume Expansion + +Either [`volume create`][] or [`volume register`][] commands can trigger +an expand to occur, after the volume has already been created or registered. + +This example shows you how to expand a volume using the `volume create` command. + +1. Create a volume configuration file called `volume.hcl` that defines ID, name, + type, plugin ID, and capacity. + + ```hcl + id = "ebs_prod_db1" + name = "database" + type = "csi" + plugin_id = "ebs-prod" + + capacity_min = "50gib" + capacity_max = "50gib" + + # ... etc ... + ``` + +1. Create the volume using the [`volume create`] command. + + ```shell-session + $ nomad volume create volume.hcl + Created external volume vol-asdf1234 with ID ebs_prod_db1 + ``` + +1. Review the volume's current capacity using the [`volume status`] command. + + ```shell-session + $ nomad volume status ebs_prod_db1 | grep Capacity + Capacity = 50 GiB + ``` + +1. Increase volume capacity in the `volume.hcl` file. + + Update the `capacity_min` and `capacity_max` fields. + + ```hcl + id = "ebs_prod_db1" + name = "database" + type = "csi" + plugin_id = "ebs-prod" + # all of the above must remain the same + + capacity_min = "100gib" # double + capacity_max = "100gib" # increased to match + + # ... etc ... + ``` + +1. Expand the volume using the [`volume create`] command. + + ```shell-session + $ nomad volume create volume.hcl + Created external volume vol-asdf1234 with ID ebs_prod_db1 + ``` + +1. Review the new capacity by running the [`volume status`] command. + + ```shell-session + $ nomad volume status ebs_prod_db1 | grep Capacity + Capacity = 100 GiB + ``` + +If you encounter errors that are not clear from the command output, +the CSI plugin allocation logs and Nomad leader server logs may be +helpful. + +[api_volume_create]: /nomad/api-docs/volumes#create-csi-volume +[api_volume_register]: /nomad/api-docs/volumes#register-volume +[capability]: /nomad/docs/other-specifications/volume/capability +[csi_plugin]: /nomad/docs/job-specification/csi_plugin +[csi_volume_source]: /nomad/docs/job-specification/volume#source +[mount_options]: /nomad/docs/other-specifications/volume/mount_options +[topology_request]: /nomad/docs/other-specifications/volume/topology_request +[`volume create`]: /nomad/docs/commands/volume/create +[`volume register`]: /nomad/docs/commands/volume/register +[`volume status`]: /nomad/docs/commands/volume/status diff --git a/website/content/docs/other-specifications/volume/host.mdx b/website/content/docs/other-specifications/volume/host.mdx new file mode 100644 index 000000000..2247142f9 --- /dev/null +++ b/website/content/docs/other-specifications/volume/host.mdx @@ -0,0 +1,307 @@ +--- +layout: docs +page_title: Dynamic Host Volume Specification +description: |- + Learn about the Nomad dynamic host volume specification. Create and + register dynamic host volumes using the `volume create` and `volume + register` commands and the `PUT /v1/volume/host/create` and + `PUT /v1/volume/host/register` API endpoints. Define capacity, + capability, constraint, node, node pool, and parameters passed directly + to the plugin to configure the volume. Learn how volume creation and + registration are different. Additionally, learn how to place a volume on + specific nodes, update a volume, and expand a volume's capacity. +--- + +# Dynamic Host Volume Specification + +This page provides reference information for the Nomad dynamic host volume +specification. Create and register dynamic host volumes using the [`volume +create`][] and [`volume register`][] commands and the [`PUT +/v1/volume/host/create`][api_volume_create] and [`PUT +/v1/volume/host/register`][api_volume_register] API endpoints. Define capacity, +capability, constraint, node, node pool, and parameters passed directly to the +plugin to configure the volume. Learn how volume creation and registration are +different. Additionally, learn how to place a volume on specific nodes, update a +volume, and expand a volume's capacity. + +Some attributes are only be supported by specific operation, while others may +have a different meaning for each action, so read the documentation for each +attribute carefully. The section [Differences Between Create and +Register](#differences-between-create-and-register) provides a summary of the +differences. + +The file may be provided as either HCL or JSON to the commands and as JSON to +the API. + +## Volume Specification Parameters + +- `capacity` `(string: )` - The size of a volume in bytes. Either the + physical size of a disk or a quota, depending on the plugin. This field must + be between the `capacity_min` and `capacity_max` values unless they are + omitted. Accepts human-friendly suffixes such as `"100GiB"`. Only supported + for volume registration. + +- `capacity_min` `(string: )` - Option for requesting a minimum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the plugin. The specific size of the resulting + volume is somewhere between `capacity_min` and `capacity_max`; the exact + behavior is up to the plugin. If you want to specify an exact size, set + `capacity_min` and `capacity_max` to the same value. Accepts human-friendly + suffixes such as `"100GiB"`. Plugins that cannot restrict the size of volumes + may ignore this field. + +- `capacity_max` `(string: )` - Option for requesting a maximum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the plugin. The specific size of the resulting + volume is somewhere between `capacity_min` and `capacity_max`; the exact + behavior is up to the plugin. If you want to specify an exact size, set + `capacity_min` and `capacity_max` to the same value. Accepts human-friendly + suffixes such as `"100GiB"`. Plugins that cannot restrict the size of volumes + may ignore this field. + +- `capability` ([Capability][capability]: <required>) - + Option for validating the capability of a volume. + +- `constraint` ([Constraint][constraint]: <optional>) - A + restriction on the eligible nodes where a volume can be created. Refer to the + [volume placement](#volume-placement) section for details. You can provide + multiple `constraint` blocks to add more constraints. Optional for volume + creation and ignored for volume registration. + +- `id` `(string: )` - The ID of a previously created volume to update + via `volume create` or `volume register`. You should never set this field when + initially creating or registering a volume, and you should only use the values + returned from the Nomad API for the ID. + +- `host_path` `(string)` - The path on disk where the volume exists. You should + set this only for volume registration. It is ignored for volume creation. + +- `name` `(string: )` - The name of the volume, which is used as the + [`volume.source`][volume_source] field in job specifications that claim this + volume. Host volume names must be unique per node. Names are visible to any + user with `node:read` ACL, even across namespaces, so they should not be + treated as sensitive values. + +- `namespace` `(string: )` - The namespace of the volume. This field + overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE` + environment variable. Defaults to `"default"` if unset. + +- `node_id` `(string)` - A specific node where you would like the volume to be + created. Refer to the [volume placement](#volume-placement) section for + details. Optional for volume creation but required for volume registration. + +- `node_pool` `(string: )` - A specific node pool where you would like + the volume to be created. Refer to the [volume placement](#volume-placement) + section for details. Optional for volume creation or volume registration. If + you also provide `node_id`, the node must be in the provided `node_pool`. + +- `parameters` (map:nil) - An optional key-value map + of strings passed directly to the plugin to configure the volume. The details + of these parameters are specific to the plugin. + +- `plugin_id` `(string)` - The ID of the [dynamic host volume + plugin][dhv_plugin] that manages this volume. Required for volume creation. + +- `type` `(string: )` - The type of volume. Must be `"host"` for + dynamic host volumes. + +## Differences Between Create and Register + +Several fields are set automatically by Nomad or the plugin when `volume create` +or `volume register` commands, or the equivalent APIs, are successful and you +should not set their values if they are not supported by the operation. + +In **volume creation** you must set the [`plugin_id`](#plugin_id) field. The +[`capacity`](#capacity) and [`host_path`](#host_path) fields are ignored. + +In **volume registration** you must set the [`node_id`](#node_id) and +[`host_path`](#host_path) fields. The [`plugin_id`](#plugin_id) and +[`constraint`](#constraint) fields are ignored. The [`node_pool`](#node_pool), +[`capacity_max`](#capacity_max), and [`capacity_min`](#capacity_min) fields are +ignored but must be consistent if set; the `node_pool` must match the node set +by `node_id`, and the minimum capacity must be less than the `capacity_max` and +`capacity`. + +## Volume Placement + +The `volume create` command creates the volume on a single node. If `node_id` is +set, the volume is created on that node. The `node_pool` must be unset or match +that node. Otherwise, the Nomad server iterates over the available nodes and +place the volume on the first available node based on the following factors: + +* The node cannot already have a host volume with the same name. +* If `node_pool` is set, the selected node must be in that node pool. +* The node must meet any and all constraints defined by the `constraint` fields. + +## Updating a Volume Definition + +The `volume create` and `volume register` commands allow updating a volume +definition. However, after volume registration, you are only allowed to update +the following fields: + +- `plugin_id` +- `capacity_min` and `capacity_max`. You may increase + the volume size if the plugin supports it. Expansion may or may not be + possible while the volume is in use, again depending on the plugin. Reducing + volume capacity is not allowed. Only available for volume creation. +- `capacity`, but only updated for volume registration. +- `constraint` fields. Ignored after the volume is created. + +Additionally, you may add or remove `capability` blocks, but only if the +capability is not currently in use by a mounted volume. + +You cannot update the `name`, `type`, and `node_id` fields. You may only change +the `node_pool` field from empty to the node pool that matches the `node_id` +field. + +## Volume Expansion + +Dynamic host volumes may be expanded if the plugin allows. Reducing the size of +a volume is not permitted. + +To trigger a volume expansion, increase [`capacity_min`](#capacity_min) above +the current real capacity of the volume (as shown with the [`volume status`][] +command), and re-issue [`volume create`][]. + +Nomad reconciles the requested capacity by issuing a create request to the +plugin. + +## Examples + +### Volume creation + +This is an example file used for the [`volume create`][] command. + +```hcl +name = "database" +type = "host" +plugin_id = "lvm-thin-provisioner" +node_pool = "prod" + +capacity_min = "80G" +capacity_max = "100G" + +capability { + access_mode = "single-node-reader-only" + attachment_mode = "file-system" +} + +capability { + access_mode = "single-node-writer" + attachment_mode = "file-system" +} + +parameters { + skuname = "Premium_LRS" +} +``` + +### Volume registration + +This is an example file used for the [`volume register`][] command. + +```hcl +name = "database" +type = "host" +node_id = "a7b4c0ca-cc78-11ef-8b5a-cb6ea67b844c" +host_path = "/var/srv/example" +capacity = "80G" + +capability { + access_mode = "single-node-reader-only" + attachment_mode = "file-system" +} + +capability { + access_mode = "single-node-writer" + attachment_mode = "file-system" +} + +parameters { + skuname = "Premium_LRS" +} +``` + +### Example Volume Expansion + +This example shows how to expand a volume after it has been created or +registered. + +1. Create a volume configuration file called `volume.hcl` that defines name, + type, plugin ID, and capacity. + + ```hcl + name = "database" + type = "host" + plugin_id = "external-plugin" + + capacity_min = "30GiB" + capacity_max = "50GiB" + ``` + +1. Create the volume using the [`volume create`] command. + + ```shell-session + $ nomad volume create ./volume.hcl + ==> Created host volume database with ID 0c903229-311d-ba8a-f77e-45c31b83fab3 + ✓ Host volume "0c903229" ready + + 2025-01-06T16:56:09-05:00 + ID = 0c903229-311d-ba8a-f77e-45c31b83fab3 + Name = database + Namespace = default + Plugin ID = external-plugin + Node ID = cfe033a7-50de-2c46-cd18-12be7429eeb3 + Node Pool = default + Capacity = 50 GiB + State = ready + Host Path = /run/nomad/dev/alloc_mounts/0c903229-311d-ba8a-f77e-45c31b83fab3 + ``` + +1. Review the volume's current capacity using the [`volume status`] command. + + ```shell-session + $ nomad volume status -type=host 0c903229 | grep Capacity + Capacity = 50 GiB + ``` + +1. Increase volume capacity in the `volume.hcl` file. + + Update the `capacity_min` and `capacity_max` fields. + + ```hcl + id = "0c903229-311d-ba8a-f77e-45c31b83fab3" + name = "database" + type = "host" + plugin_id = "external-plugin" + + capacity_min = "100GiB" # double + capacity_max = "100GiB" # increased to match + ``` + +1. Expand the volume using the [`volume create`] command. + + The `volume create` command can trigger an expansion to occur, after the + volume has already been created or registered. + + ```shell-session + $ nomad volume create volume.hcl + Created host volume database with ID 0c903229-311d-ba8a-f77e-45c31b83fab3 + ``` + +1. Review the new capacity by running the [`volume status`] command. + + ```shell-session + $ nomad volume status -type=host 0c903229 | grep Capacity + Capacity = 100 GiB + ``` + +[api_volume_create]: /nomad/api-docs/volumes#create-dynamic-host-volume +[api_volume_register]: /nomad/api-docs/volumes#register-dynamic-host-volume +[volume_source]: /nomad/docs/job-specification/volume#source +[capability]: /nomad/docs/other-specifications/volume/capability +[constraint]: /nomad/docs/job-specification/constraint +[`volume create`]: /nomad/docs/commands/volume/create +[`volume register`]: /nomad/docs/commands/volume/register +[`volume status`]: /nomad/docs/commands/volume/status +[dhv_plugin]: /nomad/docs/concepts/plugins/storage/host-volumes diff --git a/website/content/docs/other-specifications/volume/index.mdx b/website/content/docs/other-specifications/volume/index.mdx index 9ba9fc3cc..ee0ba73ee 100644 --- a/website/content/docs/other-specifications/volume/index.mdx +++ b/website/content/docs/other-specifications/volume/index.mdx @@ -1,24 +1,32 @@ --- layout: docs page_title: Volume Specification -description: Learn about the Volume specification used to create and register volumes to Nomad. +description: |- + Learn about the Nomad volume specification. Create and register Container + Storage Interface (CSI) and dynamic host volumes using the `volume create` + and `volume register` commands and the equivalent API endpoints. Define + capability, mount options, topology requests, secrets, and parameters. --- # Volume Specification The Nomad volume specification defines the schema for creating and registering -volumes using the [`volume create`] and [`volume register`] commands and the -[`PUT /v1/volume/csi/:volume_id/create`][api_volume_create] and [`PUT -/v1/volume/csi/:volume_id`][api_volume_register] API endpoints. +This page provides reference information for the Nomad volume +specification. Create and register Container Storage Interface (CSI) and dynamic +host volumes using the [`volume create`][] and [`volume register`][] commands +and the equivalent API endpoints. Define capability, mount options, topology +requests, secrets, and parameters. -Some attributes are only be supported by specific operation, while others may -have a different meaning for each action, so read the documentation for each -attribute carefully. The section [Differences Between Create and -Register](#differences-between-create-and-register) provides a summary of the -differences. +Some attributes are only be supported by one volume type or the other, or a +specific operation, while others may have a different meaning for each action, +so read the documentation for each attribute carefully. -The file may be provided as either HCL or JSON to the commands and as JSON to -the API. An example HCL configuration for a `volume create` command: +* [CSI volume specification][csi_spec] +* [Dynamic host volume specification][dhv_spec] + +Provide the file as either HCL or JSON to the commands and as JSON to +the API. An example HCL configuration for a `volume create` command with a CSI +volume: ```hcl id = "ebs_prod_db1" @@ -64,255 +72,7 @@ parameters { } ``` -## Volume Specification Parameters - -- `id` `(string: )` - The unique ID of the volume. This is how the - [`volume.source`][csi_volume_source] field in a job specification will refer - to the volume. - -- `namespace` `(string: )` - The namespace of the volume. This field - overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE` - environment variable. Defaults to `"default"` if unset. - -- `name` `(string: )` - The display name of the volume. On **volume - creation**, this field may be used by the external storage provider to tag - the volume. - -- `type` `(string: )` - The type of volume. Currently only `"csi"` - is supported. - -- `external_id` `(string: )` - The ID of the physical volume from - the storage provider. For example, the volume ID of an AWS EBS volume or - Digital Ocean volume. Only allowed on **volume registration**. - -- `plugin_id` `(string: )` - The ID of the [CSI plugin][csi_plugin] - that manages this volume. - -- `snapshot_id` `(string: )` - If the storage provider supports - snapshots, the external ID of the snapshot to restore when creating this - volume. If omitted, the volume will be created from scratch. The - `snapshot_id` cannot be set if the `clone_id` field is set. Only allowed on - **volume creation**. - -- `clone_id` `(string: )` - If the storage provider supports cloning, - the external ID of the volume to clone when creating this volume. If omitted, - the volume will be created from scratch. The `clone_id` cannot be set if the - `snapshot_id` field is set. Only allowed on **volume creation**. - -- `capacity_min` `(string: )` - Option for requesting a minimum - capacity, in bytes. The capacity of a volume may be the physical size of a - disk, or a quota, depending on the storage provider. The specific size of the - resulting volume will be somewhere between `capacity_min` and `capacity_max`; - the exact behavior is up to the storage provider. If you want to specify an - exact size, you should set `capacity_min` and `capacity_max` to the same - value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not - be supported by all storage providers. Increasing this value and re-issuing - `volume create` or `volume register` may expand the volume, if the CSI plugin - supports it. - -- `capacity_max` `(string: )` - Option for requesting a maximum - capacity, in bytes. The capacity of a volume may be the physical size of a - disk, or a quota, depending on the storage provider. The specific size of the - resulting volume will be somewhere between `capacity_min` and `capacity_max`; - the exact behavior is up to the storage provider. If you want to specify an - exact size, you should set `capacity_min` and `capacity_max` to the same - value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not - be supported by all storage providers. - -- `capability` ([Capability][capability]: <required>) - - Option for validating the capability of a volume. - -- `mount_options` ([MountOptions][mount_options]: <required>) - - Options for mounting `file-system` volumes that don't already have a - pre-formatted file system. - -- `topology_request` ([TopologyRequest][topology_request]: nil) - - Specify locations (region, zone, rack, etc.) where the provisioned volume - must be accessible from in the case of **volume creation** or the locations - where the existing volume is accessible from in the case of **volume - registration**. - -- `secrets` (map:nil) - An optional key-value map - of strings used as credentials for publishing and unpublishing volumes. - -- `parameters` (map:nil) - An optional key-value - map of strings passed directly to the CSI plugin to configure the volume. The - details of these parameters are specific to each storage provider, so consult - the specific plugin documentation for more information. - -- `context` (map:nil) - An optional key-value map - of strings passed directly to the CSI plugin to validate the volume. The - details of these parameters are specific to each storage provider, so consult - the specific plugin documentation for more information. Only allowed on - **volume registration**. Note that, like the rest of the volume specification, - this block is declarative, and an update replaces it in its entirety, therefore - all parameters need to be specified. - -## Differences Between Create and Register - -Several fields are set automatically by the plugin when `volume create` or -`volume register` commands are successful and you should not set their values -if they are not supported by the operation. - -You should not set the [`snapshot_id`](#snapshot_id), or [`clone_id`](#clone_id) -fields on **volume registration**. - -And you should not set the [`external_id`](#external_id) or -[`context`](#context) fields on **volume creation**. - -## Updating a Volume Definition - -The `volume register` command allows updating a volume definition. But not all -fields can be updated after the volume is registered: - -* The `capacity_min` and `capacity_max` fields can be updated, and may increase - the volume size if the CSI plugin supports it. Expansion may or may not be - possible while the volume is in use, again depending on the plugin. - Reducing volume capacity is not allowed per the CSI spec. -* The `capability` blocks can be added or removed, but only if the capability is - not currently in use by a mounted volume. -* The `mount_options` block can be updated if the volume is not in use. -* The `secrets` block can be updated. -* The `context` block can be updated. The values for this field are typically - provided by the CSI plugin, and should not be updated unless recommended by - the CSI plugin's documentation. - -## Volume Expansion - -CSI Volumes may be expanded (increased in size) if the CSI controller plugin -(and node plugin, if required) has the `EXPAND_VOLUME` capability. - -To trigger a volume expansion, increase [`capacity_min`](#capacity_min) -above the current real capacity of the volume (as seen with the -[`volume status`][] command), and re-issue either [`volume create`][] -or [`volume register`][]. - -Nomad will reconcile the requested capacity by issuing expand volume requests -to the controller plugin, and if required by the controller, also to the -node plugins for each allocation that has a claim on the volume. - -## Examples - -### Volume registration - -This is an example file used for the [`volume register`] command. - -```hcl -id = "ebs_prod_db1" -name = "database" -type = "csi" -external_id = "vol-23452345" -plugin_id = "ebs-prod" - -capability { - access_mode = "single-node-reader-only" - attachment_mode = "file-system" -} - -capability { - access_mode = "single-node-writer" - attachment_mode = "file-system" -} - -mount_options { - fs_type = "ext4" - mount_flags = ["noatime"] -} - -topology_request { - required { - topology { segments { "rack" = "R2" } } - topology { segments { "rack" = "R1", "zone" = "us-east-1a"} } - } -} - -secrets { - example_secret = "xyzzy" -} - -parameters { - skuname = "Premium_LRS" -} - -context { - endpoint = "http://192.168.1.101:9425" -} -``` - - -### Example Volume Expansion - -Either [`volume create`][] or [`volume register`][] commands can trigger -an expand to occur, after the volume has already been created or registered. - -Example using `volume create`, with a `volume.hcl` file: - -```hcl -id = "ebs_prod_db1" -name = "database" -type = "csi" -plugin_id = "ebs-prod" - -capacity_min = "50gib" -capacity_max = "50gib" - -# ... etc ... -``` - -Create the volume: - -```shell-session -$ nomad volume create volume.hcl -Created external volume vol-asdf1234 with ID ebs_prod_db1 -``` - -See its current capacity with [`volume status`][]: - -```shell-session -$ nomad volume status ebs_prod_db1 | grep Capacity -Capacity = 50 GiB -``` - -Update `volume.hcl`: - -```hcl -id = "ebs_prod_db1" -name = "database" -type = "csi" -plugin_id = "ebs-prod" -# all of the above must remain the same - -capacity_min = "100gib" # double -capacity_max = "100gib" # increased to match - -# ... etc ... -``` - -Run `volume create` again: - -```shell-session -$ nomad volume create volume.hcl -Created external volume vol-asdf1234 with ID ebs_prod_db1 -``` - -Check the volume capacity again: - -```shell-session -$ nomad volume status ebs_prod_db1 | grep Capacity -Capacity = 100 GiB -``` - -If you encounter errors that are not clear from the command output, -the CSI plugin allocation logs and/or Nomad leader server logs may be -helpful. - -[api_volume_create]: /nomad/api-docs/volumes#create-volume -[api_volume_register]: /nomad/api-docs/volumes#register-volume -[capability]: /nomad/docs/other-specifications/volume/capability -[csi_plugin]: /nomad/docs/job-specification/csi_plugin -[csi_volume_source]: /nomad/docs/job-specification/volume#source -[mount_options]: /nomad/docs/other-specifications/volume/mount_options -[topology_request]: /nomad/docs/other-specifications/volume/topology_request [`volume create`]: /nomad/docs/commands/volume/create [`volume register`]: /nomad/docs/commands/volume/register -[`volume status`]: /nomad/docs/commands/volume/status +[csi_spec]: /nomad/docs/other-specifications/volume/csi +[dhv_spec]: /nomad/docs/other-specifications/volume/host diff --git a/website/content/docs/other-specifications/volume/mount_options.mdx b/website/content/docs/other-specifications/volume/mount_options.mdx index 6bee03d01..c963df5be 100644 --- a/website/content/docs/other-specifications/volume/mount_options.mdx +++ b/website/content/docs/other-specifications/volume/mount_options.mdx @@ -1,7 +1,8 @@ --- layout: docs page_title: mount_options Block - Volume Specification -description: The "mount_options" block allows for configuring how a volume is mounted. +description: |- + Configure Container Storage Interface (CSI) storage volume file system type and mount flags in the "mount_options" block of the Nomad volume specification. --- # `mount_options` Block @@ -12,8 +13,9 @@ description: The "mount_options" block allows for configuring how a volume is mo ]} /> -Options for mounting `file-system` volumes that don't already have a -pre-formatted file system. +Options for mounting `file-system` CSI volumes that don't already have a +pre-formatted file system. The `mount_options` block is not supported for +dynamic host volumes. ```hcl id = "ebs_prod_db1" diff --git a/website/content/docs/other-specifications/volume/topology_request.mdx b/website/content/docs/other-specifications/volume/topology_request.mdx index 0c1c6a8ae..27cbfda9f 100644 --- a/website/content/docs/other-specifications/volume/topology_request.mdx +++ b/website/content/docs/other-specifications/volume/topology_request.mdx @@ -1,7 +1,8 @@ --- layout: docs page_title: topology_request Block - Volume Specification -description: The "topology_request" block allows specifying locations where the provisioned volume must be accessible from. +description: |- + Configure Container Storage Interface (CSI) storage volume topology in the "topology_request" block of the Nomad volume specification. Specify region, zone, and rack so that Nomad can access a provisioned CSI volume. Review volume creation examples with preferred and required topologies. --- # `topology_request` Block @@ -12,8 +13,10 @@ description: The "topology_request" block allows specifying locations where the ]} /> -Specify locations (region, zone, rack, etc.) where the provisioned volume must -be accessible from or from where an existing volume is accessible from. +Specify locations such as region, zone, and rack, where a provisioned CSI volume must +be accessible, or where an existing volume is accessible. The `topology_request` +block is not supported for dynamic host volumes. + ```hcl id = "ebs_prod_db1" diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 31b867709..0fd8809b1 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -167,7 +167,20 @@ }, { "title": "Storage", - "path": "concepts/plugins/csi" + "routes": [ + { + "title": "Overview", + "path": "concepts/plugins/storage" + }, + { + "title": "CSI", + "path": "concepts/plugins/storage/csi" + }, + { + "title": "Host Volumes", + "path": "concepts/plugins/storage/host-volumes" + } + ] }, { "title": "Networking", @@ -1947,6 +1960,14 @@ "title": "Overview", "path": "other-specifications/volume" }, + { + "title": "CSI", + "path": "other-specifications/volume/csi" + }, + { + "title": "Dynamic Host Volumes", + "path": "other-specifications/volume/host" + }, { "title": "capability", "path": "other-specifications/volume/capability" diff --git a/website/redirects.js b/website/redirects.js index e6b1a5c65..48ec75831 100644 --- a/website/redirects.js +++ b/website/redirects.js @@ -98,6 +98,12 @@ module.exports = [ destination: '/nomad/plugins/drivers/community/', permanent: true, }, + // CSI plugins moved under new storage path alongside new host volume plugins + { + source: '/nomad/docs/concepts/plugins/csi', + destination: '/nomad/docs/concepts/plugins/storage/csi', + permanent: true, + }, { source: '/nomad/plugins/drivers/virt/client', destination: '/nomad/plugins/drivers/virt/install',