From fe2a95f7f65dbed52a5861746075d2fed507c7dc Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Tue, 7 Jan 2025 10:08:01 -0500 Subject: [PATCH 01/14] docs: dynamic host volumes API (#24789) Add API documentation for the dynamic host volumes feature. Ref: https://github.com/hashicorp/nomad/issues/15489 Ref: https://hashicorp.atlassian.net/browse/NET-11482 --- website/content/api-docs/volumes.mdx | 516 +++++++++++++++--- .../other-specifications/volume/index.mdx | 8 +- 2 files changed, 431 insertions(+), 93 deletions(-) diff --git a/website/content/api-docs/volumes.mdx b/website/content/api-docs/volumes.mdx index 9b56c16e8..502e31570 100644 --- a/website/content/api-docs/volumes.mdx +++ b/website/content/api-docs/volumes.mdx @@ -7,7 +7,7 @@ description: The `/volume` endpoints are used to query for and interact with vol # Volumes HTTP API The `/volume` and `/volumes` endpoints are used to query for and interact with -volumes. +Container Storage Interface (CSI) volumes and dynamic host volumes. ## List Volumes @@ -17,29 +17,28 @@ This endpoint lists all volumes. | ------ | ------------- | ------------------ | | `GET` | `/v1/volumes` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. -| Blocking Queries | ACL Required | -| ---------------- | --------------------------- | -| `YES` | `namespace:csi-list-volume` | +| Blocking Queries | ACL Required | +|------------------|--------------------------------------------------------------| +| `YES` | `namespace:csi-list-volume`
`namespace:host-volume-read` | ### Parameters -- `type` `(string: "")` - Specifies the type of volume to - query. Currently only supports `csi`. This is specified as a query - string parameter. Returns an empty list if omitted. +- `type` `(string: )` - Specifies the type of volume to query. One of + `csi` or `host`. The `host` value queries dynamic host volumes. Specify this + as a query string parameter. - `node_id` `(string: "")` - Specifies a string to filter volumes based on an Node ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters - (0-9a-f). This is specified as a query string parameter. + (0-9a-f). Specify this as a query string parameter. - `plugin_id` `(string: "")` - Specifies a string to filter volumes based on a plugin ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters - (0-9a-f). This is specified as a query string parameter. + (0-9a-f). Specify this as a query string parameter. - `next_token` `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string which identifies the next expected volume. This @@ -62,7 +61,7 @@ $ curl \ https://localhost:4646/v1/volumes?type=csi&node_id=foo&plugin_id=plugin-id1 ``` -### Sample Response +### Sample Response for CSI Volumes ```json [ @@ -95,17 +94,37 @@ $ curl \ ] ``` -## Read Volume +### Sample Response for dynamic host volumes -This endpoint reads information about a specific volume. +```json +[ + { + "CapacityBytes": 1048576000, + "CreateIndex": 42, + "CreateTime": 1736191825846395400, + "ID": "3735cc2c-cc64-11ef-89ed-bfb5b3bc38ea", + "ModifyIndex": 64, + "ModifyTime": 1736191825846395400, + "Name": "example", + "Namespace": "default", + "NodeID": "5c5830d0-cc64-11ef-a293-4f03e55969ea", + "NodePool": "default", + "PluginID": "plugin-id1", + "State": "ready" + } +] +``` + +## Read CSI Volume + +This endpoint reads information about a specific CSI volume. | Method | Path | Produces | | ------ | --------------------------- | ------------------ | | `GET` | `/v1/volume/csi/:volume_id` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | --------------------------- | @@ -114,7 +133,7 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Request @@ -275,10 +294,11 @@ $ curl \ } ``` -## Register Volume +## Register CSI Volume -This endpoint registers an external volume with Nomad. The volume must exist -in the external storage provider (see [Create Volume] below). +This endpoint registers an external CSI volume with Nomad. The volume must exist +in the external storage provider. Refer to the [Create CSI Volume][] section for +details. Making the same request again with a higher `RequestedCapacityMin` value may trigger a [Volume Expansion][]. @@ -287,9 +307,8 @@ may trigger a [Volume Expansion][]. | ------ | --------------------------- | ------------------ | | `PUT` | `/v1/volume/csi/:volume_id` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -298,13 +317,13 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Payload The payload must include a JSON document that describes the volume's -parameters. +parameters. Note that the `NodeID` field is required for the register API. ```json { @@ -343,9 +362,9 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id1 ``` -## Create Volume +## Create CSI Volume -This endpoint creates a volume in an external storage provider and registers +This endpoint creates a CSI volume in an external storage provider and registers it with Nomad. Only CSI plugins that implement the [Controller][csi_plugins_internals] interface with the `CREATE_DELETE_VOLUME` capability support this endpoint. @@ -357,9 +376,8 @@ may trigger a [Volume Expansion][]. | ------ | ---------------------------------- | ------------------ | | `PUT` | `/v1/volume/csi/:volume_id/create` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -368,7 +386,7 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Payload @@ -420,18 +438,17 @@ $ curl \ ``` -## Deregister Volume +## Deregister CSI Volume -This endpoint deregisters an external volume with Nomad. It is an error to +This endpoint deregisters an external CSI volume from Nomad. It is an error to deregister a volume that is in use. | Method | Path | Produces | | -------- | --------------------------- | ------------------ | | `DELETE` | `/v1/volume/csi/:volume_id` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -440,13 +457,13 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. - `force` `(bool: false)` - Force deregistration of the volume and immediately drop claims for terminal allocations. Returns an error if the volume has running allocations. This does not detach the volume from client nodes. - This is specified as a query string parameter. + Specify this as a query string parameter. ### Sample Request @@ -456,9 +473,9 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id1?force=false ``` -## Delete Volume +## Delete CSI Volume -This endpoint deletes an external volume from the storage provider, and +This endpoint deletes an external CSI volume from the storage provider, and deregisters it from Nomad. It is an error to delete a volume that is in use. Only CSI plugins that implement the [Controller][csi_plugins_internals] interface with the `CREATE_DELETE_VOLUME` capability support this endpoint. @@ -467,9 +484,8 @@ interface with the `CREATE_DELETE_VOLUME` capability support this endpoint. | -------- | ---------------------------------- | ------------------ | | `DELETE` | `/v1/volume/csi/:volume_id/delete` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -483,7 +499,7 @@ stored when the CSI volume was created. ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. ### Sample Request @@ -495,18 +511,17 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id1/delete ``` -## Detach Volume +## Detach CSI Volume -This endpoint detaches an external volume from a Nomad client node. It is an +This endpoint detaches an external CSI volume from a Nomad client node. It is an error to detach a volume that is in use. | Method | Path | Produces | | -------- | ---------------------------------- | ------------------ | | `DELETE` | `/v1/volume/csi/:volume_id/detach` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -515,11 +530,11 @@ The table below shows this endpoint's support for ### Parameters - `:volume_id` `(string: )` - Specifies the ID of the - volume. This must be the full ID. This is specified as part of the + volume. This must be the full ID. Specify this as part of the path. - `node` `(string: )` - The node to detach the volume from. - This is specified as a query string parameter. + Specify this as a query string parameter. ### Sample Request @@ -529,20 +544,19 @@ $ curl \ https://localhost:4646/v1/volume/csi/volume-id/detach?node=00000000-0000-0000-0000-000000000000 ``` -## List External Volumes +## List External CSI Volumes -This endpoint lists storage volumes that are known to the external storage -provider but may not be registered with Nomad. Only CSI plugins that -implement the [Controller][csi_plugins_internals] interface with the -`LIST_VOLUMES` capability support this endpoint. +This endpoint lists storage CSI volumes that are known to the external storage +provider but may not be registered with Nomad. Only CSI plugins that implement +the [Controller][csi_plugins_internals] interface with the `LIST_VOLUMES` +capability support this endpoint. | Method | Path | Produces | |--------|------------------------|--------------------| | `GET` | `/v1/volumes/external` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -553,14 +567,14 @@ The table below shows this endpoint's support for - `plugin_id` `(string: "")` - Specifies a string to filter volumes based on a plugin ID prefix. Because the value is decoded to bytes, the prefix must have an even number of hexadecimal characters - (0-9a-f). This is specified as a query string parameter. + (0-9a-f). Specify this as a query string parameter. - `next_token` `(string: "")` - This endpoint supports paging. The `next_token` parameter accepts a string returned in a previous response's `NextToken` field to request the next page of results. - `per_page` `(int: )` - Specifies a maximum number of snapshots to - return for this request. The response will include a `NextToken` field that + return for this request. The response includes a `NextToken` field that can be passed to the next request to fetch additional pages. ### Sample Request @@ -596,20 +610,19 @@ $ curl \ } ``` -## Create Snapshot +## Create CSI Volume Snapshot -This endpoint creates a snapshot of a volume on the external storage +This endpoint creates a snapshot of a CSI volume on the external storage provider. Only CSI plugins that implement the -[Controller][csi_plugins_internals] interface with the -`CREATE_DELETE_SNAPSHOT` capability support this endpoint. +[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_SNAPSHOT` +capability support this endpoint. | Method | Path | Produces | | -------- | --------------------------------- | ------------------ | | `POST` | `/v1/volumes/snapshot` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -664,20 +677,19 @@ $ curl \ } ``` -## Delete Snapshot +## Delete CSI Volume Snapshot -This endpoint deletes a volume snapshot from the external storage +This endpoint deletes a CSI volume snapshot from the external storage provider. Only CSI plugins that implement the -[Controller][csi_plugins_internals] interface with the -`CREATE_DELETE_SNAPSHOT` capability support this endpoint. +[Controller][csi_plugins_internals] interface with the `CREATE_DELETE_SNAPSHOT` +capability support this endpoint. | Method | Path | Produces | | -------- | ---------------------- | ------------------ | | `DELETE` | `/v1/volumes/snapshot` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | ---------------------------- | @@ -691,12 +703,12 @@ stored when the CSI snapshot was created. ### Parameters - `plugin_id` `(string: )` - Specifies the prefix of a CSI plugin ID - to perform the delete. Because the value is decoded to bytes, the prefix - must have an even number of hexadecimal characters (0-9a-f). This is - specified as a query string parameter. + to perform the delete. Because the value is decoded to bytes, the prefix must + have an even number of hexadecimal characters (0-9a-f). Specify this as a + query string parameter. - `snapshot_id` `(string: )` - Specifies the snapshot ID to - delete. This is specified as a query parameter. + delete. Specify this as a query parameter. ### Sample Request @@ -707,9 +719,9 @@ $ curl \ https://localhost:4646/v1/volumes/snapshot ``` -## List Snapshots +## List CSI Volume Snapshots -This endpoint lists volume snapshots on the external storage provider. Only +This endpoint lists CSI volume snapshots on the external storage provider. Only CSI plugins that implement the [Controller][csi_plugins_internals] interface with the `LIST_SNAPSHOTS` capability support this endpoint. @@ -718,9 +730,8 @@ with the `LIST_SNAPSHOTS` capability support this endpoint. | ------ | ---------------------- | ------------------ | | `GET` | `/v1/volumes/snapshot` | `application/json` | -The table below shows this endpoint's support for -[blocking queries](/nomad/api-docs#blocking-queries) and -[required ACLs](/nomad/api-docs#acls). +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. | Blocking Queries | ACL Required | | ---------------- | --------------------------- | @@ -735,7 +746,7 @@ stored when the CSI snapshot was created. - `plugin_id` `(string: )` - Specifies the prefix of a CSI plugin ID to perform the list. Because the value is decoded to bytes, the prefix must - have an even number of hexadecimal characters (0-9a-f). This is specified as + have an even number of hexadecimal characters (0-9a-f). Specify this as a query string parameter. - `next_token` `(string: "")` - This endpoint supports paging. The @@ -743,7 +754,7 @@ stored when the CSI snapshot was created. `NextToken` field to request the next page of results. - `per_page` `(int: )` - Specifies a maximum number of snapshots to - return for this request. The response will include a `NextToken` field that + return for this request. The response includes a `NextToken` field that can be passed to the next request to fetch additional pages. ### Sample Request @@ -782,8 +793,335 @@ $ curl \ } ``` +## Read Dynamic Host Volume + +This endpoint reads information about a specific dynamic host volume. + +| Method | Path | Produces | +|--------|------------------------------|--------------------| +| `GET` | `/v1/volume/host/:volume_id` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|------------------------------| +| `YES` | `namespace:host-volume-read` | + +### Parameters + +- `:volume_id` `(string: )` - Specifies the ID of the + volume. This must be the full ID. Specify this as part of the + path. + +### Sample Request + +```shell-session +$ curl \ + https://localhost:4646/v1/volume/host/c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f +``` + +### Sample Response + +```json +{ + "Allocations": [], + "CapacityBytes": 49283072, + "CreateIndex": 11, + "CreateTime": 1736191993011594200, + "HostPath": "/run/nomad/dev/alloc_mounts/c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ModifyIndex": 12, + "ModifyTime": 1736191993011594200, + "Name": "external-plugin", + "Namespace": "default", + "NodeID": "670cb259-bc26-653b-e316-655af04ad260", + "NodePool": "default", + "Parameters": { + "hello": "world" + }, + "PluginID": "example-plugin-mkfs", + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + } + ], + "RequestedCapacityMaxBytes": 50000000, + "RequestedCapacityMinBytes": 50000000, + "State": "ready" +} +``` + +## Create Dynamic Host Volume + +This endpoint creates a dynamic host volume and registers it with Nomad. + +The response body includes the volume definition with the `ID` and `NodeID` +fields populated. Making the same request without the ID creates a new volume on +a different node. Making a request with the ID set updates the volume to the new +parameters, if possible. + +| Method | Path | Produces | +|--------|--------------------------|--------------------| +| `PUT` | `/v1/volume/host/create` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|--------------------------------------------------------------------------------------------------------------------------------| +| `NO` | `namespace:host-volume-create` or
`namespace:host-volume-write`.
`namespace:sentinel-override` if `PolicyOverride` set | + +### Parameters + +- `Volume` `(Volume: )` - Specifies the JSON definition of the host + volume. You should include the ID field if you are updating an existing + volume. + +- `PolicyOverride` `(bool: false)` - If set, Nomad overrides any soft mandatory + Sentinel policies. This field allows creating a volume when it would be denied + by policy. + +### Sample Payload + +```json +{ + "PolicyOverride": false, + "Volume": { + "Name": "example", + "Namespace": "default", + "NodePool": "prod", + "PluginID": "mkdir", + "RequestedCapacityMinBytes": 10737418240, + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "Parameters": { + "foo": "bar" + } + } +} +``` + +### Sample Request + +```shell-session +$ curl \ + --request PUT \ + --data @payload.json \ + https://localhost:4646/v1/volume/host/create +``` + +### Sample Response + +```json +{ + "Volume": { + "Allocations": [], + "CapacityBytes": 21474836480, + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "CreateIndex": 11, + "CreateTime": 1736191993011594200, + "ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ModifyIndex": 11, + "ModifyTime": 1736191993011594200, + "Name": "example", + "Namespace": "default", + "NodeID": "45460554-cc67-11ef-84b7-33d383a55487", + "NodePool": "prod", + "Parameters": { + "foo": "bar" + }, + "PluginID": "mkdir", + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapacityMinBytes": 10737418240, + "State": "pending" + }, + "Warnings": null +} +``` + +## Register Dynamic Host Volume + +This endpoint registers an existing dynamic host volume with Nomad. + +| Method | Path | Produces | +|--------|----------------------------|--------------------| +| `PUT` | `/v1/volume/host/register` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|----------------------------------------------------------------------------------------------------------------------------------| +| `NO` | `namespace:host-volume-register` or
`namespace:host-volume-write`.
`namespace:sentinel-override` if `PolicyOverride` set | + +### Parameters + +- `Volume` `(Volume: )` - Specifies the JSON definition of the host + volume. You should include the ID field if you are updating an existing + volume. + +- `PolicyOverride` `(bool: false)` - If set, Nomad overrides any soft mandatory + Sentinel policies. This field allows registering a volume when it would be denied + by policy. + +### Sample Payload + +```json +{ + "PolicyOverride": false, + "Volume": { + "Name": "example", + "Namespace": "default", + "NodePool": "prod", + "PluginID": "mkdir", + "RequestedCapacityMinBytes": 10737418240, + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "Parameters": { + "foo": "bar" + } + } +} +``` + +### Sample Request + +```shell-session +$ curl \ + --request PUT \ + --data @payload.json \ + https://localhost:4646/v1/volume/host/register +``` + +### Sample Response + +```json +{ + "Volume": { + "Allocations": [], + "CapacityBytes": 21474836480, + "Constraints": [ + { + "LTarget": "${attr.kernel.name}", + "RTarget": "linux", + "Operand": "=" + } + ], + "CreateIndex": 11, + "CreateTime": 1736191993011594200, + "ID": "c0f7ee7d-5cc6-92fd-f2b5-14b79f01979f", + "ModifyIndex": 31, + "ModifyTime": 1736191993721594200, + "Name": "example", + "Namespace": "default", + "NodeID": "45460554-cc67-11ef-84b7-33d383a55487", + "NodePool": "prod", + "Parameters": { + "foo": "bar" + }, + "PluginID": "mkdir", + "RequestedCapabilities": [ + { + "AccessMode": "single-node-writer", + "AttachmentMode": "file-system" + }, + { + "AccessMode": "single-node-writer", + "AttachmentMode": "block-device" + } + ], + "RequestedCapacityMaxBytes": 21474836480, + "RequestedCapacityMinBytes": 10737418240, + "State": "ready" + }, + "Warnings": null +} +``` + +## Delete Dynamic Host Volume + +This endpoint deletes a dynamic host volume, and deregisters it from Nomad. It +is an error to delete a volume that is in use. + +| Method | Path | Produces | +|----------|-------------------------------------|--------------------| +| `DELETE` | `/v1/volume/host/:volume_id/delete` | `application/json` | + +The following table shows this endpoint's support for [blocking queries][] and +[required ACLs][]. + +| Blocking Queries | ACL Required | +|------------------|---------------------------------------------------------------------| +| `NO` | `namespace:host-volume-write` or
`namespace:host-volume-delete` | + +### Parameters + +- `:volume_id` `(string: )` - Specifies the ID of the + volume. This must be the full ID. Specify this as part of the + path. + +### Sample Request + +```shell-session +$ curl \ + --request DELETE \ + https://localhost:4646/v1/volume/host/ba97ef42-cc68-11ef-a2e7-ffddaecbdb89 +``` + +[blocking queries]: /nomad/api-docs#blocking-queries +[required ACLs]: /nomad/api-docs#acls [csi]: https://github.com/container-storage-interface/spec [csi_plugin]: /nomad/docs/job-specification/csi_plugin [csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins -[Create Volume]: #create-volume +[Create CSI Volume]: #create-csi-volume [Volume Expansion]: /nomad/docs/other-specifications/volume#volume-expansion diff --git a/website/content/docs/other-specifications/volume/index.mdx b/website/content/docs/other-specifications/volume/index.mdx index 9ba9fc3cc..49ef3d591 100644 --- a/website/content/docs/other-specifications/volume/index.mdx +++ b/website/content/docs/other-specifications/volume/index.mdx @@ -144,8 +144,8 @@ parameters { of strings passed directly to the CSI plugin to validate the volume. The details of these parameters are specific to each storage provider, so consult the specific plugin documentation for more information. Only allowed on - **volume registration**. Note that, like the rest of the volume specification, - this block is declarative, and an update replaces it in its entirety, therefore + **volume registration**. Note that, like the rest of the volume specification, + this block is declarative, and an update replaces it in its entirety, therefore all parameters need to be specified. ## Differences Between Create and Register @@ -306,8 +306,8 @@ If you encounter errors that are not clear from the command output, the CSI plugin allocation logs and/or Nomad leader server logs may be helpful. -[api_volume_create]: /nomad/api-docs/volumes#create-volume -[api_volume_register]: /nomad/api-docs/volumes#register-volume +[api_volume_create]: /nomad/api-docs/volumes#create-csi-volume +[api_volume_register]: /nomad/api-docs/volumes#register-csi-volume [capability]: /nomad/docs/other-specifications/volume/capability [csi_plugin]: /nomad/docs/job-specification/csi_plugin [csi_volume_source]: /nomad/docs/job-specification/volume#source From b3151a7b0276148874b815496b86c75c727b67ac Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Tue, 7 Jan 2025 11:39:20 -0500 Subject: [PATCH 02/14] docs: dynamic host volumes ACL policies (#24790) Add ACL policy documentation for the dynamic host volumes feature. Ref: https://github.com/hashicorp/nomad/issues/15489 Ref: https://hashicorp.atlassian.net/browse/NET-11482 --- .../docs/other-specifications/acl-policy.mdx | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/website/content/docs/other-specifications/acl-policy.mdx b/website/content/docs/other-specifications/acl-policy.mdx index efbdc8143..1abbe9c4b 100644 --- a/website/content/docs/other-specifications/acl-policy.mdx +++ b/website/content/docs/other-specifications/acl-policy.mdx @@ -77,8 +77,8 @@ Namespace rules control access to APIs in Nomad that are namespaced: [Jobs][api_jobs], [Allocations][api_allocations], [Deployments][api_deployments], [Evaluations][api_evaluations], [Recommendations][api_recommendations], [Scaling -Policies][api_scaling_policies], [Services][api_services], and [CSI -Volumes][api_volumes]. Namespace rules also filter items related to the above +Policies][api_scaling_policies], [Services][api_services], and +[Volumes][api_volumes]. Namespace rules also filter items related to the above APIs from the [Event Stream][api_events] and [Search][api_search] APIs. Each namespace rule is labeled with the namespace name it applies to. If no @@ -149,6 +149,11 @@ of `capabilities`. This includes: and listing external volumes and snapshots. - `csi-mount-volume` - Allows jobs to be submitted that claim a CSI volume. This implicitly grants `csi-read-volume`. +- `host-volume-create` - Allows creating dynamic host volumes. This implicitly grants `host-volume-read`. +- `host-volume-delete` - Allows deleting dynamic host volumes. +- `host-volume-read` - Allows inspecting dynamic host volumes. +- `host-volume-register` - Allows registering dynamic host volumes that have been created without a plugin. This implicitly grants `host-volume-read` and `host-volume-create`. +- `host-volume-write` - Allows all write operations on dynamic host volumes. This implicitly grants `host-volume-read`, `host-volume-create`, `host-volume-register`, and `host-volume-delete`. - `list-scaling-policies` - Allows listing scaling policies. - `read-scaling-policy` - Allows inspecting a scaling policy. - `read-job-scaling` - Allows inspecting the current scaling of a job. @@ -164,8 +169,8 @@ grained namespace capabilities: | Policy | Capabilities | |---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `deny` | deny | -| `read` | list-jobs
parse-job
read-job
csi-list-volume
csi-read-volume
list-scaling-policies
read-scaling-policy
read-job-scaling | -| `write` | list-jobs
parse-job
read-job
submit-job
dispatch-job
read-logs
read-fs
alloc-exec
alloc-lifecycle
csi-write-volume
csi-mount-volume
list-scaling-policies
read-scaling-policy
read-job-scaling
scale-job
submit-recommendation | +| `read` | list-jobs
parse-job
read-job
csi-list-volume
csi-read-volume
host-volume-read
list-scaling-policies
read-scaling-policy
read-job-scaling | +| `write` | list-jobs
parse-job
read-job
submit-job
dispatch-job
read-logs
read-fs
alloc-exec
alloc-lifecycle
csi-write-volume
csi-mount-volume
host-volume-write
list-scaling-policies
read-scaling-policy
read-job-scaling
scale-job
submit-recommendation | | `scale` | list-scaling-policies
read-scaling-policy
read-job-scaling
scale-job | From 590f50c4fe4ff5366a66a0a537c71255913195d4 Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Mon, 13 Jan 2025 08:20:57 -0500 Subject: [PATCH 03/14] docs: jobspec updates for dynamic host volumes (#24828) The job specification's `volume` block can claim either static or dynamic host volumes. Update the documentation to explain this and cover the additional fields this exposes. Ref: https://github.com/hashicorp/nomad/pull/24797 Ref: https://hashicorp.atlassian.net/browse/NET-11482 Co-authored-by: Aimee Ukasick --- .../content/docs/job-specification/volume.mdx | 74 +++++++++++++------ 1 file changed, 53 insertions(+), 21 deletions(-) diff --git a/website/content/docs/job-specification/volume.mdx b/website/content/docs/job-specification/volume.mdx index 58ec2bd35..c99322812 100644 --- a/website/content/docs/job-specification/volume.mdx +++ b/website/content/docs/job-specification/volume.mdx @@ -50,19 +50,22 @@ job "docs" { ``` -The Nomad server will ensure that the allocations are only scheduled -on hosts that have a set of volumes that meet the criteria specified -in the `volume` blocks. These may be [host volumes][host_volume] -configured on the client, or [CSI volumes][csi_volume] dynamically -mounted by [CSI plugins][csi_plugin]. +The Nomad server ensures that the allocations are only scheduled on hosts +that have a set of volumes that meet the criteria specified in the `volume` +blocks. These may be [static host volumes][host_volume] configured on the +client, [dynamic host volumes][] created via [`volume create`][] or [`volume +register`][], or [CSI volumes][csi_volume] dynamically mounted by [CSI +plugins][csi_plugin]. -The Nomad client will make the volumes available to tasks according to +The Nomad client makes the volumes available to tasks according to the [volume_mount][volume_mount] block in the `task` configuration. ## `volume` Parameters -- `type` `(string: "")` - Specifies the type of a given volume. The - valid volume types are `"host"` and `"csi"`. +- `type` `(string: "")` - Specifies the type of a given volume. The valid volume + types are `"host"` and `"csi"`. Setting the `"host"` value can request either + statically configured host volumes or dynamic host volumes, depending on what + is available on a given node. - `source` `(string: )` - The name of the volume to request. When using `host_volume`'s this should match the published @@ -85,22 +88,48 @@ the [volume_mount][volume_mount] block in the `task` configuration. The `per_alloc` field cannot be true for system jobs, sysbatch jobs, or jobs that use canaries. -The following fields are only valid for volumes with `type = "csi"`: +The following fields are only valid for volumes with `type = "csi"` or dynamic +host volumes with `type = "host"`: -- `access_mode` `(string: )` - Defines whether a volume should be +- `access_mode` `(string)` - Defines whether a volume should be available concurrently. The `access_mode` and `attachment_mode` together - must exactly match one of the volume's `capability` blocks. Can be one of - `"single-node-reader-only"`, `"single-node-writer"`, - `"multi-node-reader-only"`, `"multi-node-single-writer"`, or - `"multi-node-multi-writer"`. Most CSI plugins support only single-node - modes. Consult the documentation of the storage provider and CSI plugin. + must exactly match one of the volume's `capability` blocks. -- `attachment_mode` `(string: )` - The storage API that will be used - by the volume. The `access_mode` and `attachment_mode` together must exactly - match one of the volume's `capability` blocks. Most storage providers will - support `"file-system"`, to mount volumes using the CSI filesystem API. Some - storage providers will support `"block-device"`, which will mount the volume - with the CSI block device API within the container. + - For CSI volumes the `access_mode` is required. Can be one of the following: + + - `"single-node-reader-only"` + - `"single-node-writer"` + - `"multi-node-reader-only"` + - `"multi-node-single-writer"` + - `"multi-node-multi-writer"` + + Most CSI plugins support only single-node modes. + Consult the documentation of the storage provider and CSI plugin. + + - For dynamic host volumes the `access_mode` is optional. Can be one of the following: + + - `"single-node-writer"` + - `"single-node-reader-only"` + - `"single-node-single-writer"` + - `"single-node-multi-writer"` + + Defaults to `single-node-writer` unless `read_only = true`, in which case it + defaults to `single-node-reader-only`. + +- `attachment_mode` `(string)` - The storage API used by the + volume. One of `"file-system"` or `"block-device"`. The `access_mode` and + `attachment_mode` together must exactly match one of the volume's `capability` + blocks. + + - For CSI volumes the `attachment_mode` field is required. Most storage + providers support `"file-system"`, to mount volumes using the CSI + filesystem API. Some storage providers support `"block-device"`, which + mounts the volume with the CSI block device API within the container. + + - For dynamic host volumes the `attachment_mode` field is optional and + defaults to `"file-system"`. + +The following fields are only valid for volumes with `type = "csi"`: - `mount_options` - Options for mounting CSI volumes that have the `file-system` [attachment mode]. These options override the `mount_options` @@ -207,3 +236,6 @@ ID Node ID Task Group Version Desired Status Created Modified [csi_volume]: /nomad/docs/commands/volume/register [attachment mode]: /nomad/docs/commands/volume/register#attachment_mode [volume registration]: /nomad/docs/commands/volume/register#mount_options +[dynamic host volumes]: /nomad/docs/other-specifications/volume/host +[`volume create`]: /nomad/docs/commands/volume/create +[`volume register`]: /nomad/docs/commands/volume/register From a12c0f724e5f093d3a6e1c61708b8f5134596b7f Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Mon, 13 Jan 2025 09:24:06 -0500 Subject: [PATCH 04/14] dynamic host volumes: client configuration docs (#24827) Document the client configuration changes needed to support dynamic host volumes. This changeset excludes the plugin specification/concepts, which will be under a separate PR. Ref: https://github.com/hashicorp/nomad/pull/24797 Ref: https://hashicorp.atlassian.net/browse/NET-11482 Co-authored-by: Aimee Ukasick Co-authored-by: Daniel Bennett --- website/content/docs/configuration/client.mdx | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/website/content/docs/configuration/client.mdx b/website/content/docs/configuration/client.mdx index de740eb31..24510e091 100644 --- a/website/content/docs/configuration/client.mdx +++ b/website/content/docs/configuration/client.mdx @@ -630,11 +630,14 @@ refer to the [drivers documentation](/nomad/docs/drivers). ### `host_volume` Block -The `host_volume` block is used to make volumes available to jobs. +The `host_volume` block is used to make volumes available to jobs. You can also +configure [dynamic host volumes][] via the [`volume create`][] or [`volume +register`][] commands. The key of the block corresponds to the name of the volume for use in the -`source` parameter of a `"host"` type [`volume`](/nomad/docs/job-specification/volume) -and ACLs. +`source` parameter of a `"host"` type +[`volume`](/nomad/docs/job-specification/volume) and ACLs. A host volume in the +configuration cannot have the same name as a dynamic host volume on the same node. ```hcl client { @@ -824,3 +827,6 @@ client { [`TimeoutStopSec`]: https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStopSec= [top_level_data_dir]: /nomad/docs/configuration#data_dir [unveil]: /nomad/docs/concepts/plugins/task-drivers#fsisolation-unveil +[dynamic host volumes]: /nomad/docs/other-specifications/volume/host +[`volume create`]: /nomad/docs/commands/volume/create +[`volume register`]: /nomad/docs/commands/volume/register From 0489c35110e0cd3bc2623e92b0d15dbc1ff05793 Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Tue, 14 Jan 2025 13:39:51 -0500 Subject: [PATCH 05/14] docs: dynamic host volume specification (#24810) Dynamic host volumes use the same specification file as CSI volumes but require a different set of parameters and have different semantics. This changeset splits the volume specification page into separate CSI and dynamic host volumes spec pages. While migrating the CSI page, I've also edited it to bring it more in line with the style guide: removed passive voice and future tense, inclusive language, alphabetized the (chaotic!) parameters list, etc. Ref: https://github.com/hashicorp/nomad/pull/24797 Ref: https://hashicorp.atlassian.net/browse/NET-11482 Co-authored-by: Daniel Bennett Co-authored-by: Aimee Ukasick --- website/content/api-docs/volumes.mdx | 2 +- .../volume/capability.mdx | 47 ++- .../docs/other-specifications/volume/csi.mdx | 291 +++++++++++++++++ .../docs/other-specifications/volume/host.mdx | 306 ++++++++++++++++++ .../other-specifications/volume/index.mdx | 282 ++-------------- .../volume/mount_options.mdx | 5 +- .../volume/topology_request.mdx | 6 +- website/data/docs-nav-data.json | 8 + 8 files changed, 670 insertions(+), 277 deletions(-) create mode 100644 website/content/docs/other-specifications/volume/csi.mdx create mode 100644 website/content/docs/other-specifications/volume/host.mdx diff --git a/website/content/api-docs/volumes.mdx b/website/content/api-docs/volumes.mdx index 502e31570..1c1d6793c 100644 --- a/website/content/api-docs/volumes.mdx +++ b/website/content/api-docs/volumes.mdx @@ -1124,4 +1124,4 @@ $ curl \ [csi_plugin]: /nomad/docs/job-specification/csi_plugin [csi_plugins_internals]: /nomad/docs/concepts/plugins/csi#csi-plugins [Create CSI Volume]: #create-csi-volume -[Volume Expansion]: /nomad/docs/other-specifications/volume#volume-expansion +[Volume Expansion]: /nomad/docs/other-specifications/volume/csi#volume-expansion diff --git a/website/content/docs/other-specifications/volume/capability.mdx b/website/content/docs/other-specifications/volume/capability.mdx index c03e20142..1b65b53e4 100644 --- a/website/content/docs/other-specifications/volume/capability.mdx +++ b/website/content/docs/other-specifications/volume/capability.mdx @@ -35,18 +35,43 @@ for each capability you intend to use in a job's [`volume`] block. ## `capability` Parameters -- `access_mode` `(string: )` - Defines whether a volume should be -available concurrently. Can be one of `"single-node-reader-only"`, -`"single-node-writer"`, `"multi-node-reader-only"`, -`"multi-node-single-writer"`, or `"multi-node-multi-writer"`. Most CSI plugins -support only single-node modes. Consult the documentation of the storage -provider and CSI plugin. +- `access_mode` `(string)` - Defines whether a volume should be available + concurrently. The `access_mode` and `attachment_mode` from the volume request + must exactly match one of the volume's `capability` blocks. -- `attachment_mode` `(string: )` - The storage API that will be used -by the volume. Most storage providers will support `"file-system"`, to mount -volumes using the CSI filesystem API. Some storage providers will support -`"block-device"`, which will mount the volume with the CSI block device API -within the container. + - For CSI volumes the `access_mode` is required. Can be one of the following: + + - `"single-node-reader-only"` + - `"single-node-writer"` + - `"multi-node-reader-only"` + - `"multi-node-single-writer"` + - `"multi-node-multi-writer"` + + Most CSI plugins support only single-node modes. + Consult the documentation of the storage provider and CSI plugin. + + - For dynamic host volumes the `access_mode` is optional. Can be one of the following: + + - `"single-node-writer"` + - `"single-node-reader-only"` + - `"single-node-single-writer"` + - `"single-node-multi-writer"` + + In the job specification, the default is `single-node-writer` unless + `read_only = true`, which translates to `single-node-reader-only`. + +- `attachment_mode` `(string)` - The storage API used by the volume. One of + `"file-system"` or `"block-device"`. The `access_mode` and `attachment_mode` + from the volume request must exactly match one of the volume's `capability` + blocks. + + - For CSI volumes the `attachment_mode` field is required. Most storage + providers support `"file-system"`, to mount volumes using the CSI + filesystem API. Some storage providers support `"block-device"`, which + mounts the volume with the CSI block device API within the container. + + - For dynamic host volumes the `attachment_mode` field is optional and + defaults to `"file-system"`. ## `capability` Examples diff --git a/website/content/docs/other-specifications/volume/csi.mdx b/website/content/docs/other-specifications/volume/csi.mdx new file mode 100644 index 000000000..dd38c12fe --- /dev/null +++ b/website/content/docs/other-specifications/volume/csi.mdx @@ -0,0 +1,291 @@ +--- +layout: docs +page_title: CSI Volume Specification +description: |- + Learn about the Nomad Container Storage Interface (CSI) volume + specification. Create and register CSI volumes using the `volume create` + and `volume register` commands and the + `PUT /v1/volume/csi/:volume_id/create` and + `PUT /v1/volume/csi/:volume_id` API endpoints. Define capacity, capabilities, mount + options, secrets, topology requests, and a context map of values passed + directly to the CSI plugin to validate the volume. Learn how volume + creation and registration are different. Additionally, learn how to + expand the size of a volume and update a volume definition. +--- + +# CSI Volume Specification + +This page provides reference information for the Nomad Container Storage +Interface (CSI) volume specification. Create and register CSI volumes using the +[`volume create`][] and [`volume register`][] commands and the [`PUT +/v1/volume/csi/:volume_id/create`][api_volume_create] and [`PUT +/v1/volume/csi/:volume_id`][api_volume_register] API endpoints. Define capacity, +capabilities, mount options, secrets, topology requests, and a context map of +values passed directly to the CSI plugin to validate the volume. Learn how +volume creation and registration are different. Additionally, learn how to +expand the size of a volume and update a volume definition. + +Some attributes are only supported by a specific operation, while others may +have a different meaning for each action, so read the documentation for each +attribute carefully. The section [Differences Between Create and +Register](#differences-between-create-and-register) provides a summary of the +differences. + +Provide the file as either HCL or JSON to the commands and as JSON to the API. + +## Volume Specification Parameters + +- `capability` ([Capability][capability]: <required>) - + Option for validating the capability of a volume. + +- `capacity_min` `(string: )` - Option for requesting a minimum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the storage provider. The specific size of the + resulting volume is somewhere between `capacity_min` and `capacity_max`; the + exact behavior is up to the storage provider. If you want to specify an exact + size, you should set `capacity_min` and `capacity_max` to the same + value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not + be supported by all storage providers. Increasing this value and reissuing + `volume create` or `volume register` may expand the volume, if the CSI plugin + supports it. + +- `capacity_max` `(string: )` - Option for requesting a maximum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the storage provider. The specific size of the + resulting volume is somewhere between `capacity_min` and `capacity_max`; the + exact behavior is up to the storage provider. If you want to specify an exact + size, you should set `capacity_min` and `capacity_max` to the same + value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not + be supported by all storage providers. + +- `clone_id` `(string: )` - If the storage provider supports cloning, + the external ID of the volume to clone when creating this volume. If omitted, + the volume is created from scratch. The `clone_id` cannot be set if the + `snapshot_id` field is set. Only allowed on volume creation. + +- `context` (map:nil) - An optional key-value map of + strings passed directly to the CSI plugin to validate the volume. The details + of these parameters are specific to each storage provider, so consult the + specific plugin documentation for more information. Only allowed on volume + registration. Note that, like the rest of the volume specification, this block + is declarative, and an update replaces it in its entirety, therefore all + parameters need to be specified. + +- `external_id` `(string: )` - The ID of the physical volume from the + storage provider. For example, the volume ID of an AWS EBS volume or Digital + Ocean volume. Only allowed on volume registration. + +- `id` `(string: )` - The unique ID of the volume. This is how the + [`volume.source`][csi_volume_source] field in a job specification refers to + the volume. + +- `mount_options` ([MountOptions][mount_options]: + <required>) - Options for mounting `file-system` volumes that + don't already have a pre-formatted file system. + +- `name` `(string: )` - The display name of the volume. On volume + creation, this field may be used by the external storage provider to tag the + volume. + +- `namespace` `(string: )` - The namespace of the volume. This field + overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE` + environment variable. Defaults to `"default"` if unset. + +- `parameters` (map:nil) - An optional key-value map + of strings passed directly to the CSI plugin to configure the volume. The + details of these parameters are specific to each storage provider, so consult + the specific plugin documentation for more information. + +- `plugin_id` `(string: )` - The ID of the [CSI plugin][csi_plugin] + that manages this volume. + +- `secrets` (map:nil) - An optional key-value map of + strings used as credentials for publishing and unpublishing volumes. + +- `snapshot_id` `(string: )` - If the storage provider supports + snapshots, the external ID of the snapshot to restore when creating this + volume. If omitted, the volume is created from scratch. The `snapshot_id` + cannot be set if the `clone_id` field is set. Only allowed on volume creation. + +- `topology_request` ([TopologyRequest][topology_request]: nil) - + Specify locations such as region, zone, and rack where the provisioned volume + must be accessible from in the case of volume creation, or the locations where + the existing volume is accessible from in the case of volume registration. + +- `type` `(string: )` - The type of volume. Must be `"csi"` for CSI + volumes. + +## Differences Between Create and Register + +Several fields are set automatically by the plugin when `volume create` or +`volume register` commands are successful and you should not set their values if +they are not supported by the operation. + +On **volume registration** you should not set the [`snapshot_id`](#snapshot_id), +or [`clone_id`](#clone_id) fields. + +On **volume creation** you should not set the [`external_id`](#external_id) or +[`context`](#context) fields. + +## Updating a Volume Definition + +The `volume register` command allows updating a volume definition. However, +after volume registration, you are only allowed to update the following fields: + +* `capacity_min` and `capacity_max`. You may increase the volume size if the CSI + plugin supports it. Expansion may or may not be possible while the volume is + in use, again depending on the plugin. Reducing volume capacity is not + allowed per the CSI spec. +* `mount_options` block if the volume is not in use. +* `secrets` block +* `context` block. The values for this field are typically provided by the CSI + plugin and should not be updated unless recommended by the CSI plugin's + documentation. + +Additionally, you may add or remove `capability` blocks, but only if the +capability is not currently in use by a mounted volume. + +## Volume Expansion + +You may expand CSI Volumes if the CSI controller plugin and node plugin, if +required, has the `EXPAND_VOLUME` capability. + +To trigger a volume expansion, increase [`capacity_min`](#capacity_min) above +the current real capacity of the volume, as received from the [`volume +status`][] command, and re-issue either [`volume create`][] or [`volume +register`][]. + +Nomad reconciles the requested capacity by issuing expand volume requests to the +controller plugin, and if required by the controller, also to the node plugins +for each allocation that has a claim on the volume. + +## Examples + +### Volume registration + +This is an example file used for the [`volume register`][] command. + +```hcl +id = "ebs_prod_db1" +name = "database" +type = "csi" +external_id = "vol-23452345" +plugin_id = "ebs-prod" + +capability { + access_mode = "single-node-reader-only" + attachment_mode = "file-system" +} + +capability { + access_mode = "single-node-writer" + attachment_mode = "file-system" +} + +mount_options { + fs_type = "ext4" + mount_flags = ["noatime"] +} + +topology_request { + required { + topology { segments { "rack" = "R2" } } + topology { segments { "rack" = "R1", "zone" = "us-east-1a"} } + } +} + +secrets { + example_secret = "xyzzy" +} + +parameters { + skuname = "Premium_LRS" +} + +context { + endpoint = "http://192.168.1.101:9425" +} +``` + + +### Example Volume Expansion + +Either [`volume create`][] or [`volume register`][] commands can trigger +an expand to occur, after the volume has already been created or registered. + +This example shows you how to expand a volume using the `volume create` command. + +1. Create a volume configuration file called `volume.hcl` that defines ID, name, + type, plugin ID, and capacity. + + ```hcl + id = "ebs_prod_db1" + name = "database" + type = "csi" + plugin_id = "ebs-prod" + + capacity_min = "50gib" + capacity_max = "50gib" + + # ... etc ... + ``` + +1. Create the volume using the [`volume create`] command. + + ```shell-session + $ nomad volume create volume.hcl + Created external volume vol-asdf1234 with ID ebs_prod_db1 + ``` + +1. Review the volume's current capacity using the [`volume status`] command. + + ```shell-session + $ nomad volume status ebs_prod_db1 | grep Capacity + Capacity = 50 GiB + ``` + +1. Increase volume capacity in the `volume.hcl` file. + + Update the `capacity_min` and `capacity_max` fields. + + ```hcl + id = "ebs_prod_db1" + name = "database" + type = "csi" + plugin_id = "ebs-prod" + # all of the above must remain the same + + capacity_min = "100gib" # double + capacity_max = "100gib" # increased to match + + # ... etc ... + ``` + +1. Expand the volume using the [`volume create`] command. + + ```shell-session + $ nomad volume create volume.hcl + Created external volume vol-asdf1234 with ID ebs_prod_db1 + ``` + +1. Review the new capacity by running the [`volume status`] command. + + ```shell-session + $ nomad volume status ebs_prod_db1 | grep Capacity + Capacity = 100 GiB + ``` + +If you encounter errors that are not clear from the command output, +the CSI plugin allocation logs and Nomad leader server logs may be +helpful. + +[api_volume_create]: /nomad/api-docs/volumes#create-csi-volume +[api_volume_register]: /nomad/api-docs/volumes#register-volume +[capability]: /nomad/docs/other-specifications/volume/capability +[csi_plugin]: /nomad/docs/job-specification/csi_plugin +[csi_volume_source]: /nomad/docs/job-specification/volume#source +[mount_options]: /nomad/docs/other-specifications/volume/mount_options +[topology_request]: /nomad/docs/other-specifications/volume/topology_request +[`volume create`]: /nomad/docs/commands/volume/create +[`volume register`]: /nomad/docs/commands/volume/register +[`volume status`]: /nomad/docs/commands/volume/status diff --git a/website/content/docs/other-specifications/volume/host.mdx b/website/content/docs/other-specifications/volume/host.mdx new file mode 100644 index 000000000..f6943ff5f --- /dev/null +++ b/website/content/docs/other-specifications/volume/host.mdx @@ -0,0 +1,306 @@ +--- +layout: docs +page_title: Dynamic Host Volume Specification +description: |- + Learn about the Nomad dynamic host volume specification. Create and + register dynamic host volumes using the `volume create` and `volume + register` commands and the `PUT /v1/volume/host/create` and + `PUT /v1/volume/host/register` API endpoints. Define capacity, + capability, constraint, node, node pool, and parameters passed directly + to the plugin to configure the volume. Learn how volume creation and + registration are different. Additionally, learn how to place a volume on + specific nodes, update a volume, and expand a volume's capacity. +--- + +# Dynamic Host Volume Specification + +This page provides reference information for the Nomad dynamic host volume +specification. Create and register dynamic host volumes using the [`volume +create`][] and [`volume register`][] commands and the [`PUT +/v1/volume/host/create`][api_volume_create] and [`PUT +/v1/volume/host/register`][api_volume_register] API endpoints. Define capacity, +capability, constraint, node, node pool, and parameters passed directly to the +plugin to configure the volume. Learn how volume creation and registration are +different. Additionally, learn how to place a volume on specific nodes, update a +volume, and expand a volume's capacity. + +Some attributes are only be supported by specific operation, while others may +have a different meaning for each action, so read the documentation for each +attribute carefully. The section [Differences Between Create and +Register](#differences-between-create-and-register) provides a summary of the +differences. + +The file may be provided as either HCL or JSON to the commands and as JSON to +the API. + +## Volume Specification Parameters + +- `capacity` `(string: )` - The size of a volume in bytes. Either the + physical size of a disk or a quota, depending on the plugin. This field must + be between the `capacity_min` and `capacity_max` values unless they are + omitted. Accepts human-friendly suffixes such as `"100GiB"`. Only supported + for volume registration. + +- `capacity_min` `(string: )` - Option for requesting a minimum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the plugin. The specific size of the resulting + volume is somewhere between `capacity_min` and `capacity_max`; the exact + behavior is up to the plugin. If you want to specify an exact size, set + `capacity_min` and `capacity_max` to the same value. Accepts human-friendly + suffixes such as `"100GiB"`. Plugins that cannot restrict the size of volumes + may ignore this field. + +- `capacity_max` `(string: )` - Option for requesting a maximum + capacity, in bytes. The capacity of a volume may be the physical size of a + disk, or a quota, depending on the plugin. The specific size of the resulting + volume is somewhere between `capacity_min` and `capacity_max`; the exact + behavior is up to the plugin. If you want to specify an exact size, set + `capacity_min` and `capacity_max` to the same value. Accepts human-friendly + suffixes such as `"100GiB"`. Plugins that cannot restrict the size of volumes + may ignore this field. + +- `capability` ([Capability][capability]: <required>) - + Option for validating the capability of a volume. + +- `constraint` ([Constraint][constraint]: <optional>) - A + restriction on the eligible nodes where a volume can be created. Refer to the + [volume placement](#volume-placement) section for details. You can provide + multiple `constraint` blocks to add more constraints. Optional for volume + creation and ignored for volume registration. + +- `id` `(string: )` - The ID of a previously created volume to update + via `volume create` or `volume register`. You should never set this field when + initially creating or registering a volume, and you should only use the values + returned from the Nomad API for the ID. + +- `host_path` `(string)` - The path on disk where the volume exists. You should + set this only for volume registration. It is ignored for volume creation. + +- `name` `(string: )` - The name of the volume, which is used as the + [`volume.source`][volume_source] field in job specifications that claim this + volume. Host volume names must be unique per node. Names are visible to any + user with `node:read` ACL, even across namespaces, so they should not be + treated as sensitive values. + +- `namespace` `(string: )` - The namespace of the volume. This field + overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE` + environment variable. Defaults to `"default"` if unset. + +- `node_id` `(string)` - A specific node where you would like the volume to be + created. Refer to the [volume placement](#volume-placement) section for + details. Optional for volume creation but required for volume registration. + +- `node_pool` `(string: )` - A specific node pool where you would like + the volume to be created. Refer to the [volume placement](#volume-placement) + section for details. Optional for volume creation or volume registration. If + you also provide `node_id`, the node must be in the provided `node_pool`. + +- `parameters` (map:nil) - An optional key-value map + of strings passed directly to the plugin to configure the volume. The details + of these parameters are specific to the plugin. + +- `plugin_id` `(string)` - The ID of the [dynamic host volume + plugin][dhv_plugin] that manages this volume. Required for volume creation. + +- `type` `(string: )` - The type of volume. Must be `"host"` for + dynamic host volumes. + +## Differences Between Create and Register + +Several fields are set automatically by Nomad or the plugin when `volume create` +or `volume register` commands, or the equivalent APIs, are successful and you +should not set their values if they are not supported by the operation. + +In **volume creation** you must set the [`plugin_id`](#plugin_id) field. The +[`capacity`](#capacity) and [`host_path`](#host_path) fields are ignored. + +In **volume registration** you must set the [`node_id`](#node_id) and +[`host_path`](#host_path) fields. The [`plugin_id`](#plugin_id) and +[`constraint`](#constraint) fields are ignored. The [`node_pool`](#node_pool), +[`capacity_max`](#capacity_max), and [`capacity_min`](#capacity_min) fields are +ignored but must be consistent if set; the `node_pool` must match the node set +by `node_id`, and the minimum capacity must be less than the `capacity_max` and +`capacity`. + +## Volume Placement + +The `volume create` command creates the volume on a single node. If `node_id` is +set, the volume is created on that node. The `node_pool` must be unset or match +that node. Otherwise, the Nomad server iterates over the available nodes and +place the volume on the first available node based on the following factors: + +* The node cannot already have a host volume with the same name. +* If `node_pool` is set, the selected node must be in that node pool. +* The node must meet any and all constraints defined by the `constraint` fields. + +## Updating a Volume Definition + +The `volume create` and `volume register` commands allow updating a volume +definition. However, after volume registration, you are only allowed to update +the following fields: + +- `plugin_id` +- `capacity_min` and `capacity_max`. You may increase + the volume size if the plugin supports it. Expansion may or may not be + possible while the volume is in use, again depending on the plugin. Reducing + volume capacity is not allowed. Only available for volume creation. +- `capacity`, but only updated for volume registration. +- `constraint` fields. Ignored after the volume is created. + +Additionally, you may add or remove `capability` blocks, but only if the +capability is not currently in use by a mounted volume. + +You cannot update the `name`, `type`, and `node_id` fields. You may only change +the `node_pool` field from empty to the node pool that matches the `node_id` +field. + +## Volume Expansion + +Dynamic host volumes may be expanded if the plugin allows. Reducing the size of +a volume is not permitted. + +To trigger a volume expansion, increase [`capacity_min`](#capacity_min) above +the current real capacity of the volume (as shown with the [`volume status`][] +command), and re-issue [`volume create`][]. + +Nomad reconciles the requested capacity by issuing a create request to the +plugin. + +## Examples + +### Volume creation + +This is an example file used for the [`volume create`][] command. + +```hcl +name = "database" +type = "host" +plugin_id = "lvm-thin-provisioner" +node_pool = "prod" + +capacity_min = "80G" +capacity_max = "100G" + +capability { + access_mode = "single-node-reader-only" + attachment_mode = "file-system" +} + +capability { + access_mode = "single-node-writer" + attachment_mode = "file-system" +} + +parameters { + skuname = "Premium_LRS" +} +``` + +### Volume registration + +This is an example file used for the [`volume register`][] command. + +```hcl +name = "database" +type = "host" +node_id = "a7b4c0ca-cc78-11ef-8b5a-cb6ea67b844c" +host_path = "/var/srv/example" +capacity = "80G" + +capability { + access_mode = "single-node-reader-only" + attachment_mode = "file-system" +} + +capability { + access_mode = "single-node-writer" + attachment_mode = "file-system" +} + +parameters { + skuname = "Premium_LRS" +} +``` + +### Example Volume Expansion + +This example shows how to expand a volume after it has been created or +registered. + +1. Create a volume configuration file called `volume.hcl` that defines name, + type, plugin ID, and capacity. + + ```hcl + name = "database" + type = "host" + plugin_id = "external-plugin" + + capacity_min = "30GiB" + capacity_max = "50GiB" + ``` + +1. Create the volume using the [`volume create`] command. + + ```shell-session + $ nomad volume create ./volume.hcl + ==> Created host volume database with ID 0c903229-311d-ba8a-f77e-45c31b83fab3 + ✓ Host volume "0c903229" ready + + 2025-01-06T16:56:09-05:00 + ID = 0c903229-311d-ba8a-f77e-45c31b83fab3 + Name = database + Namespace = default + Plugin ID = external-plugin + Node ID = cfe033a7-50de-2c46-cd18-12be7429eeb3 + Node Pool = default + Capacity = 50 GiB + State = ready + Host Path = /run/nomad/dev/alloc_mounts/0c903229-311d-ba8a-f77e-45c31b83fab3 + ``` + +1. Review the volume's current capacity using the [`volume status`] command. + + ```shell-session + $ nomad volume status -type=host 0c903229 | grep Capacity + Capacity = 50 GiB + ``` + +1. Increase volume capacity in the `volume.hcl` file. + + Update the `capacity_min` and `capacity_max` fields. + + ```hcl + id = "0c903229-311d-ba8a-f77e-45c31b83fab3" + name = "database" + type = "host" + plugin_id = "external-plugin" + + capacity_min = "100GiB" # double + capacity_max = "100GiB" # increased to match + ``` + +1. Expand the volume using the [`volume create`] command. + + The `volume create` command can trigger an expansion to occur, after the + volume has already been created or registered. + + ```shell-session + $ nomad volume create volume.hcl + Created host volume database with ID 0c903229-311d-ba8a-f77e-45c31b83fab3 + ``` + +1. Review the new capacity by running the [`volume status`] command. + + ```shell-session + $ nomad volume status -type=host 0c903229 | grep Capacity + Capacity = 100 GiB + ``` + +[api_volume_create]: /nomad/api-docs/volumes#create-dynamic-host-volume +[api_volume_register]: /nomad/api-docs/volumes#register-dynamic-host-volume +[volume_source]: /nomad/docs/job-specification/volume#source +[capability]: /nomad/docs/other-specifications/volume/capability +[constraint]: /nomad/docs/job-specification/constraint +[`volume create`]: /nomad/docs/commands/volume/create +[`volume register`]: /nomad/docs/commands/volume/register +[`volume status`]: /nomad/docs/commands/volume/status diff --git a/website/content/docs/other-specifications/volume/index.mdx b/website/content/docs/other-specifications/volume/index.mdx index 49ef3d591..ee0ba73ee 100644 --- a/website/content/docs/other-specifications/volume/index.mdx +++ b/website/content/docs/other-specifications/volume/index.mdx @@ -1,24 +1,32 @@ --- layout: docs page_title: Volume Specification -description: Learn about the Volume specification used to create and register volumes to Nomad. +description: |- + Learn about the Nomad volume specification. Create and register Container + Storage Interface (CSI) and dynamic host volumes using the `volume create` + and `volume register` commands and the equivalent API endpoints. Define + capability, mount options, topology requests, secrets, and parameters. --- # Volume Specification The Nomad volume specification defines the schema for creating and registering -volumes using the [`volume create`] and [`volume register`] commands and the -[`PUT /v1/volume/csi/:volume_id/create`][api_volume_create] and [`PUT -/v1/volume/csi/:volume_id`][api_volume_register] API endpoints. +This page provides reference information for the Nomad volume +specification. Create and register Container Storage Interface (CSI) and dynamic +host volumes using the [`volume create`][] and [`volume register`][] commands +and the equivalent API endpoints. Define capability, mount options, topology +requests, secrets, and parameters. -Some attributes are only be supported by specific operation, while others may -have a different meaning for each action, so read the documentation for each -attribute carefully. The section [Differences Between Create and -Register](#differences-between-create-and-register) provides a summary of the -differences. +Some attributes are only be supported by one volume type or the other, or a +specific operation, while others may have a different meaning for each action, +so read the documentation for each attribute carefully. -The file may be provided as either HCL or JSON to the commands and as JSON to -the API. An example HCL configuration for a `volume create` command: +* [CSI volume specification][csi_spec] +* [Dynamic host volume specification][dhv_spec] + +Provide the file as either HCL or JSON to the commands and as JSON to +the API. An example HCL configuration for a `volume create` command with a CSI +volume: ```hcl id = "ebs_prod_db1" @@ -64,255 +72,7 @@ parameters { } ``` -## Volume Specification Parameters - -- `id` `(string: )` - The unique ID of the volume. This is how the - [`volume.source`][csi_volume_source] field in a job specification will refer - to the volume. - -- `namespace` `(string: )` - The namespace of the volume. This field - overrides the namespace provided by the `-namespace` flag or `NOMAD_NAMESPACE` - environment variable. Defaults to `"default"` if unset. - -- `name` `(string: )` - The display name of the volume. On **volume - creation**, this field may be used by the external storage provider to tag - the volume. - -- `type` `(string: )` - The type of volume. Currently only `"csi"` - is supported. - -- `external_id` `(string: )` - The ID of the physical volume from - the storage provider. For example, the volume ID of an AWS EBS volume or - Digital Ocean volume. Only allowed on **volume registration**. - -- `plugin_id` `(string: )` - The ID of the [CSI plugin][csi_plugin] - that manages this volume. - -- `snapshot_id` `(string: )` - If the storage provider supports - snapshots, the external ID of the snapshot to restore when creating this - volume. If omitted, the volume will be created from scratch. The - `snapshot_id` cannot be set if the `clone_id` field is set. Only allowed on - **volume creation**. - -- `clone_id` `(string: )` - If the storage provider supports cloning, - the external ID of the volume to clone when creating this volume. If omitted, - the volume will be created from scratch. The `clone_id` cannot be set if the - `snapshot_id` field is set. Only allowed on **volume creation**. - -- `capacity_min` `(string: )` - Option for requesting a minimum - capacity, in bytes. The capacity of a volume may be the physical size of a - disk, or a quota, depending on the storage provider. The specific size of the - resulting volume will be somewhere between `capacity_min` and `capacity_max`; - the exact behavior is up to the storage provider. If you want to specify an - exact size, you should set `capacity_min` and `capacity_max` to the same - value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not - be supported by all storage providers. Increasing this value and re-issuing - `volume create` or `volume register` may expand the volume, if the CSI plugin - supports it. - -- `capacity_max` `(string: )` - Option for requesting a maximum - capacity, in bytes. The capacity of a volume may be the physical size of a - disk, or a quota, depending on the storage provider. The specific size of the - resulting volume will be somewhere between `capacity_min` and `capacity_max`; - the exact behavior is up to the storage provider. If you want to specify an - exact size, you should set `capacity_min` and `capacity_max` to the same - value. Accepts human-friendly suffixes such as `"100GiB"`. This field may not - be supported by all storage providers. - -- `capability` ([Capability][capability]: <required>) - - Option for validating the capability of a volume. - -- `mount_options` ([MountOptions][mount_options]: <required>) - - Options for mounting `file-system` volumes that don't already have a - pre-formatted file system. - -- `topology_request` ([TopologyRequest][topology_request]: nil) - - Specify locations (region, zone, rack, etc.) where the provisioned volume - must be accessible from in the case of **volume creation** or the locations - where the existing volume is accessible from in the case of **volume - registration**. - -- `secrets` (map:nil) - An optional key-value map - of strings used as credentials for publishing and unpublishing volumes. - -- `parameters` (map:nil) - An optional key-value - map of strings passed directly to the CSI plugin to configure the volume. The - details of these parameters are specific to each storage provider, so consult - the specific plugin documentation for more information. - -- `context` (map:nil) - An optional key-value map - of strings passed directly to the CSI plugin to validate the volume. The - details of these parameters are specific to each storage provider, so consult - the specific plugin documentation for more information. Only allowed on - **volume registration**. Note that, like the rest of the volume specification, - this block is declarative, and an update replaces it in its entirety, therefore - all parameters need to be specified. - -## Differences Between Create and Register - -Several fields are set automatically by the plugin when `volume create` or -`volume register` commands are successful and you should not set their values -if they are not supported by the operation. - -You should not set the [`snapshot_id`](#snapshot_id), or [`clone_id`](#clone_id) -fields on **volume registration**. - -And you should not set the [`external_id`](#external_id) or -[`context`](#context) fields on **volume creation**. - -## Updating a Volume Definition - -The `volume register` command allows updating a volume definition. But not all -fields can be updated after the volume is registered: - -* The `capacity_min` and `capacity_max` fields can be updated, and may increase - the volume size if the CSI plugin supports it. Expansion may or may not be - possible while the volume is in use, again depending on the plugin. - Reducing volume capacity is not allowed per the CSI spec. -* The `capability` blocks can be added or removed, but only if the capability is - not currently in use by a mounted volume. -* The `mount_options` block can be updated if the volume is not in use. -* The `secrets` block can be updated. -* The `context` block can be updated. The values for this field are typically - provided by the CSI plugin, and should not be updated unless recommended by - the CSI plugin's documentation. - -## Volume Expansion - -CSI Volumes may be expanded (increased in size) if the CSI controller plugin -(and node plugin, if required) has the `EXPAND_VOLUME` capability. - -To trigger a volume expansion, increase [`capacity_min`](#capacity_min) -above the current real capacity of the volume (as seen with the -[`volume status`][] command), and re-issue either [`volume create`][] -or [`volume register`][]. - -Nomad will reconcile the requested capacity by issuing expand volume requests -to the controller plugin, and if required by the controller, also to the -node plugins for each allocation that has a claim on the volume. - -## Examples - -### Volume registration - -This is an example file used for the [`volume register`] command. - -```hcl -id = "ebs_prod_db1" -name = "database" -type = "csi" -external_id = "vol-23452345" -plugin_id = "ebs-prod" - -capability { - access_mode = "single-node-reader-only" - attachment_mode = "file-system" -} - -capability { - access_mode = "single-node-writer" - attachment_mode = "file-system" -} - -mount_options { - fs_type = "ext4" - mount_flags = ["noatime"] -} - -topology_request { - required { - topology { segments { "rack" = "R2" } } - topology { segments { "rack" = "R1", "zone" = "us-east-1a"} } - } -} - -secrets { - example_secret = "xyzzy" -} - -parameters { - skuname = "Premium_LRS" -} - -context { - endpoint = "http://192.168.1.101:9425" -} -``` - - -### Example Volume Expansion - -Either [`volume create`][] or [`volume register`][] commands can trigger -an expand to occur, after the volume has already been created or registered. - -Example using `volume create`, with a `volume.hcl` file: - -```hcl -id = "ebs_prod_db1" -name = "database" -type = "csi" -plugin_id = "ebs-prod" - -capacity_min = "50gib" -capacity_max = "50gib" - -# ... etc ... -``` - -Create the volume: - -```shell-session -$ nomad volume create volume.hcl -Created external volume vol-asdf1234 with ID ebs_prod_db1 -``` - -See its current capacity with [`volume status`][]: - -```shell-session -$ nomad volume status ebs_prod_db1 | grep Capacity -Capacity = 50 GiB -``` - -Update `volume.hcl`: - -```hcl -id = "ebs_prod_db1" -name = "database" -type = "csi" -plugin_id = "ebs-prod" -# all of the above must remain the same - -capacity_min = "100gib" # double -capacity_max = "100gib" # increased to match - -# ... etc ... -``` - -Run `volume create` again: - -```shell-session -$ nomad volume create volume.hcl -Created external volume vol-asdf1234 with ID ebs_prod_db1 -``` - -Check the volume capacity again: - -```shell-session -$ nomad volume status ebs_prod_db1 | grep Capacity -Capacity = 100 GiB -``` - -If you encounter errors that are not clear from the command output, -the CSI plugin allocation logs and/or Nomad leader server logs may be -helpful. - -[api_volume_create]: /nomad/api-docs/volumes#create-csi-volume -[api_volume_register]: /nomad/api-docs/volumes#register-csi-volume -[capability]: /nomad/docs/other-specifications/volume/capability -[csi_plugin]: /nomad/docs/job-specification/csi_plugin -[csi_volume_source]: /nomad/docs/job-specification/volume#source -[mount_options]: /nomad/docs/other-specifications/volume/mount_options -[topology_request]: /nomad/docs/other-specifications/volume/topology_request [`volume create`]: /nomad/docs/commands/volume/create [`volume register`]: /nomad/docs/commands/volume/register -[`volume status`]: /nomad/docs/commands/volume/status +[csi_spec]: /nomad/docs/other-specifications/volume/csi +[dhv_spec]: /nomad/docs/other-specifications/volume/host diff --git a/website/content/docs/other-specifications/volume/mount_options.mdx b/website/content/docs/other-specifications/volume/mount_options.mdx index 6bee03d01..efbb9bee1 100644 --- a/website/content/docs/other-specifications/volume/mount_options.mdx +++ b/website/content/docs/other-specifications/volume/mount_options.mdx @@ -12,8 +12,9 @@ description: The "mount_options" block allows for configuring how a volume is mo ]} /> -Options for mounting `file-system` volumes that don't already have a -pre-formatted file system. +Options for mounting `file-system` CSI volumes that don't already have a +pre-formatted file system. The `mount_options` block is not supported for +dynamic host volumes. ```hcl id = "ebs_prod_db1" diff --git a/website/content/docs/other-specifications/volume/topology_request.mdx b/website/content/docs/other-specifications/volume/topology_request.mdx index 0c1c6a8ae..b25853706 100644 --- a/website/content/docs/other-specifications/volume/topology_request.mdx +++ b/website/content/docs/other-specifications/volume/topology_request.mdx @@ -12,8 +12,10 @@ description: The "topology_request" block allows specifying locations where the ]} /> -Specify locations (region, zone, rack, etc.) where the provisioned volume must -be accessible from or from where an existing volume is accessible from. +Specify locations such as region, zone, and rack, where a provisioned CSI volume must +be accessible, or where an existing volume is accessible. The `topology_request` +block is not supported for dynamic host volumes. + ```hcl id = "ebs_prod_db1" diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 4235f5e11..0ff40ecd2 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -1947,6 +1947,14 @@ "title": "Overview", "path": "other-specifications/volume" }, + { + "title": "CSI", + "path": "other-specifications/volume/csi" + }, + { + "title": "Dynamic Host Volumes", + "path": "other-specifications/volume/host" + }, { "title": "capability", "path": "other-specifications/volume/capability" From 8330d406aa55e9ff60f75bcce4e1a47e74b696ed Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Wed, 15 Jan 2025 16:12:01 -0500 Subject: [PATCH 06/14] docs: dynamic host volume command line (#24814) Dynamic host volumes use some of the same commands as CSI volumes but with different parameters, semantics, and inputs. Ref: https://github.com/hashicorp/nomad/pull/24797 Ref: https://hashicorp.atlassian.net/browse/NET-11482 Co-authored-by: Aimee Ukasick --- .../content/docs/commands/volume/create.mdx | 51 +++++++--- .../content/docs/commands/volume/delete.mdx | 42 +++++---- .../docs/commands/volume/deregister.mdx | 3 + website/content/docs/commands/volume/init.mdx | 3 + .../content/docs/commands/volume/register.mdx | 93 ++++++++++++------- 5 files changed, 132 insertions(+), 60 deletions(-) diff --git a/website/content/docs/commands/volume/create.mdx b/website/content/docs/commands/volume/create.mdx index f383469f0..4e5f46f23 100644 --- a/website/content/docs/commands/volume/create.mdx +++ b/website/content/docs/commands/volume/create.mdx @@ -2,15 +2,19 @@ layout: docs page_title: 'Commands: volume create' description: | - Create volumes with CSI plugins. + The `nomad volume create` command creates storage volumes that are either + Container Storage Interface (CSI) volumes or dynamic host volumes. --- # Command: volume create -The `volume create` command creates external storage volumes with Nomad's -[Container Storage Interface (CSI)][csi] support. Only CSI plugins that -implement the [Controller][csi_plugins_internals] interface support this -command. The volume will also be [registered] when it is successfully created. +The `volume create` command creates storage volumes as either [Container Storage +Interface (CSI)][csi] volumes or dynamic host volumes. + +The [`volume create`][] command can create dynamic host volumes if host volume +plugins are installed on the node, or CSI volumes if the CSI plugins implement +the [Controller][csi_plugins_internals] interface. The `volume create` command +automatically [registers][] the volume as well. ## Usage @@ -18,21 +22,41 @@ command. The volume will also be [registered] when it is successfully created. nomad volume create [options] [file] ``` -The `volume create` command requires a single argument, specifying the path to -a file containing a valid [volume specification][volume_specification]. This -file will be read and the volume will be submitted to Nomad for scheduling. If -the supplied path is "-", the volume file is read from STDIN. Otherwise it is +The `volume create` command requires a single argument, specifying the path to a +file containing a valid [volume specification][volume_specification]. Nomad +reads the file and submits the volume to the server for placement. If the +supplied path is "-", the volume file is read from STDIN. Otherwise the file is read from the file at the supplied path. -When ACLs are enabled, this command requires a token with the -`csi-write-volume` capability for the volume's namespace. +When ACLs are enabled, this command requires a token with the appropriate +capability in the volume's namespace: the `csi-write-volume` capability for CSI +volumes or `host-volume-create` for dynamic host volumes. ## General Options @include 'general_options.mdx' +## Volume Create Options + +- `-detach`: Return immediately instead of entering monitor mode for dynamic + host volumes. After creating a volume, Nomad prints the volume ID to the + screen, which you can use to examine the volume using the `volume status` + command. If `-detach` is omitted or false, the command monitors the state of + the volume until it has been fingerprinted by the client and is ready to be + scheduled. Not valid for CSI volumes. + +- `-verbose`: Display full information when monitoring volume state. Used for + dynamic host volumes only. Not valid for CSI volumes. + +- `-policy-override`: Sets the flag to force override any soft mandatory + Sentinel policies. Used for dynamic host volumes only. Not valid for CSI + volumes. + ## Volume Specification +Refer to the [CSI Volume Specification][csi_vol_spec] and the [Dynamic Host +Volume Specification](host_vol_spec) for further information. +