From 0e3c33110a73016d154d517af46681e883fa2fcc Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Tue, 24 Mar 2020 11:09:46 -0400 Subject: [PATCH] Documentation for CSI command line and jobspec (#7439) Includes: * changes `nomad alloc status` and `nomad node status` * changes to `volume` block in jobspec * new `csi_plugin` block in jobspec * new `nomad volume register/deregister/status` commands * new `nomad plugin status` command --- website/data/docs-navigation.js | 5 +- website/pages/docs/commands/alloc/status.mdx | 18 ++- website/pages/docs/commands/node/status.mdx | 7 ++ website/pages/docs/commands/plugin/index.mdx | 22 ++++ website/pages/docs/commands/plugin/status.mdx | 85 ++++++++++++++ .../pages/docs/commands/volume/deregister.mdx | 31 +++++ website/pages/docs/commands/volume/index.mdx | 26 +++++ .../pages/docs/commands/volume/register.mdx | 90 +++++++++++++++ website/pages/docs/commands/volume/status.mdx | 107 ++++++++++++++++++ .../docs/job-specification/csi_plugin.mdx | 89 +++++++++++++++ .../pages/docs/job-specification/volume.mdx | 49 ++++---- 11 files changed, 500 insertions(+), 29 deletions(-) create mode 100644 website/pages/docs/commands/plugin/index.mdx create mode 100644 website/pages/docs/commands/plugin/status.mdx create mode 100644 website/pages/docs/commands/volume/deregister.mdx create mode 100644 website/pages/docs/commands/volume/index.mdx create mode 100644 website/pages/docs/commands/volume/register.mdx create mode 100644 website/pages/docs/commands/volume/status.mdx create mode 100644 website/pages/docs/job-specification/csi_plugin.mdx diff --git a/website/data/docs-navigation.js b/website/data/docs-navigation.js index 2e72c5c0e..6c8f7098e 100644 --- a/website/data/docs-navigation.js +++ b/website/data/docs-navigation.js @@ -105,6 +105,7 @@ export default [ 'raft-remove-peer' ] }, + { category: 'plugin', content: ['status'] }, { category: 'quota', content: ['apply', 'delete', 'init', 'inspect', 'list', 'status'] @@ -114,7 +115,8 @@ export default [ 'status', { category: 'system', content: ['gc', 'reconcile-summaries'] }, 'ui', - 'version' + 'version', + { category: 'volume', content: ['register', 'deregister', 'status'] } ] }, '----------', @@ -126,6 +128,7 @@ export default [ 'check_restart', 'connect', 'constraint', + 'csi_plugin', 'device', 'dispatch_payload', 'env', diff --git a/website/pages/docs/commands/alloc/status.mdx b/website/pages/docs/commands/alloc/status.mdx index c5486cc99..0a4beea6d 100644 --- a/website/pages/docs/commands/alloc/status.mdx +++ b/website/pages/docs/commands/alloc/status.mdx @@ -8,12 +8,14 @@ description: | # Command: alloc status -The `alloc status` command displays status information and metadata about an -existing allocation and its tasks. It can be useful while debugging to reveal -the underlying reasons for scheduling decisions or failures, as well as the -current state of its tasks. As of Nomad 0.7.1, alloc status also shows -allocation modification time in addition to create time. As of Nomad 0.8, alloc -status shows information about reschedule attempts. +The `alloc status` command displays status information and metadata +about an existing allocation and its tasks. It can be useful while +debugging to reveal the underlying reasons for scheduling decisions or +failures, as well as the current state of its tasks. As of Nomad +0.7.1, alloc status also shows allocation modification time in +addition to create time. As of Nomad 0.8, alloc status shows +information about reschedule attempts. As of Nomad 0.11, alloc status +shows volume claims when a job claims volumes. ## Usage @@ -88,6 +90,10 @@ Task Resources CPU Memory Disk Addresses 1/500 MHz 6.3 MiB/256 MiB 300 MiB db: 127.0.0.1:27908 +CSI Volumes: +ID Plugin Provider Schedulable Mount Options +vol-4150af42 ebs0 aws.ebs true + Task Events: Started At = 07/25/17 16:12:48 UTC Finished At = N/A diff --git a/website/pages/docs/commands/node/status.mdx b/website/pages/docs/commands/node/status.mdx index 085686e9d..b4ba4b6ab 100644 --- a/website/pages/docs/commands/node/status.mdx +++ b/website/pages/docs/commands/node/status.mdx @@ -292,6 +292,13 @@ Drain = false Status = ready Uptime = 17h7m41s +Host Volumes +Name ReadOnly Source + +CSI Volumes +ID Name Plugin ID Schedulable Access Mode Mount Options +402f2c83 vol plug true single-node-writer + Drivers Driver Detected Healthy Message Time docker false false Driver docker is not detected 2018-03-29T17:24:42Z diff --git a/website/pages/docs/commands/plugin/index.mdx b/website/pages/docs/commands/plugin/index.mdx new file mode 100644 index 000000000..fee7edf36 --- /dev/null +++ b/website/pages/docs/commands/plugin/index.mdx @@ -0,0 +1,22 @@ +--- +layout: docs +page_title: 'Commands: plugin' +sidebar_title: plugin +description: | + The plugin command is used to interact with plugins. +--- + +# Command: plugin + +The `plugin` command is used to interact with plugins. + +## Usage + +Usage: `nomad plugin [options]` + +Run `nomad plugin -h` for help on that subcommand. The following +subcommands are available: + +- [`plugin status`][status] - Display status information about a plugin + +[status]: /docs/commands/plugin/status 'Display status information about a plugin' diff --git a/website/pages/docs/commands/plugin/status.mdx b/website/pages/docs/commands/plugin/status.mdx new file mode 100644 index 000000000..9e6f10a95 --- /dev/null +++ b/website/pages/docs/commands/plugin/status.mdx @@ -0,0 +1,85 @@ +--- +layout: docs +page_title: 'Commands: plugin status' +sidebar_title: status +description: | + Display information and status of plugins. +--- + +# Command: plugin status + +The `plugin status` command displays status information for [Container +Storage Interface (CSI)][csi] plugins. + +## Usage + +```plaintext +nomad plugin status [options] [plugin] +``` + +This command accepts an optional plugin ID or prefix as the sole argument. If there +is an exact match based on the provided plugin ID or prefix, then information about +the specific plugin is queried and displayed. Otherwise, a list of matching plugins +and information will be displayed. + +If the ID is omitted, the command lists out all of the existing plugins and a few +of the most useful status fields for each. + +## General Options + +@include 'general_options.mdx' + +## Status Options + +- `-type`: Display only plugins of a particular type. Currently only + the `csi` type is supported, so this option can be omitted when + querying the status of CSI plugins. + +- `-short`: Display short output. Used only when a single plugin is being queried. + Drops verbose plugin allocation data from the output. + +- `-verbose`: Show full information. Allocation create and modify times are + shown in `yyyy/mm/dd hh:mm:ss` format. + +## Examples + +List of all plugins: + +```shell +$ nomad plugin [-type csi] status +ID Provider Controllers Healthy / Expected Nodes Healthy / Expected +ebs-prod aws.ebs 1 / 1 1 / 1 +``` + +Short view of a specific plugin: + +```shell +$ nomad plugin [-type csi] status ebs-prod +ID = ebs-prod +Provider = aws.ebs +Version = 1.0.1 +Controllers Healthy = 1 +Controllers Expected = 1 +Nodes Healthy = 1 +Nodes Expected = 1 +``` + +Full status information of a plugin: + +```shell +$ nomad plugin [-type csi] status ebs-prod +ID = ebs-prod +Provider = aws.ebs +Version = 1.0.1 +Controllers Healthy = 1 +Controllers Expected = 1 +Nodes Healthy = 1 +Nodes Expected = 1 + +Allocations +ID Node ID Task Group Version Desired Status Created Modified +0de05689 95303afc csi 0 run running 1m57s ago 1m19s ago +b206088c 8bf94335 csi 0 run running 1m56s ago 1m19s ago +``` + +[csi]: https://github.com/container-storage-interface/spec diff --git a/website/pages/docs/commands/volume/deregister.mdx b/website/pages/docs/commands/volume/deregister.mdx new file mode 100644 index 000000000..06dc19c3a --- /dev/null +++ b/website/pages/docs/commands/volume/deregister.mdx @@ -0,0 +1,31 @@ +--- +layout: docs +page_title: 'Commands: volume deregister' +sidebar_title: deregister +description: | + Deregister volumes with CSI plugins. +--- + +# Command: volume deregister + +The `volume deregister` command deregisters external storage volumes with +Nomad's [Container Storage Interface (CSI)][csi] support. The volume +must exist on the remote storage provider before it can be deregistered +and used by a task. + +## Usage + +```plaintext +nomad volume deregister [options] [volume] +``` + +The `volume deregister` command requires a single argument, specifying +the ID of volume to be deregistered. Deregistration will fail if the +volume is still in use by an allocation or in the process of being +unpublished. + +## General Options + +@include 'general_options.mdx' + +[csi]: https://github.com/container-storage-interface/spec diff --git a/website/pages/docs/commands/volume/index.mdx b/website/pages/docs/commands/volume/index.mdx new file mode 100644 index 000000000..0f3db8266 --- /dev/null +++ b/website/pages/docs/commands/volume/index.mdx @@ -0,0 +1,26 @@ +--- +layout: docs +page_title: 'Commands: volume' +sidebar_title: volume +description: | + The volume command is used to interact with volumes. +--- + +# Command: volume + +The `volume` command is used to interact with volumes. + +## Usage + +Usage: `nomad volume [options]` + +Run `nomad volume -h` for help on that subcommand. The following +subcommands are available: + +- [`volume register`][register] - Register a volume. +- [`volume deregister`][deregister] - Deregister a volume. +- [`volume status`][status] - Display status information about a volume + +[register]: /docs/commands/volume/register 'Register a volume' +[deregister]: /docs/commands/volume/deregister 'Deregister a volume' +[status]: /docs/commands/volume/status 'Display status information about a volume' diff --git a/website/pages/docs/commands/volume/register.mdx b/website/pages/docs/commands/volume/register.mdx new file mode 100644 index 000000000..90867c26d --- /dev/null +++ b/website/pages/docs/commands/volume/register.mdx @@ -0,0 +1,90 @@ +--- +layout: docs +page_title: 'Commands: volume register' +sidebar_title: register +description: | + Register volumes with CSI plugins. +--- + +# Command: volume register + +The `volume register` command registers external storage volumes with +Nomad's [Container Storage Interface (CSI)][csi] support. The volume +must exist on the remote storage provider before it can be registered +and used by a task. + +## Usage + +```plaintext +nomad volume register [options] [file] +``` + +The `volume register` command requires a single argument, specifying +the path to a file containing a valid [volume +specification][volume_specification]. This file will be read and the +job will be submitted to Nomad for scheduling. If the supplied path is +"-", the job file is read from STDIN. Otherwise it is read from the +file at the supplied path. + +## General Options + +@include 'general_options.mdx' + +## Volume Specification + +The file may be provided as either HCL or JSON. An example HCL configuration: + +```hcl +id = "ebs_prod_db1" +name = "database" +type = "csi" +external_id = "vol-23452345" +plugin_id = "ebs-prod" +access_mode = "single-node-writer" +attachment_mode = "file-system" +mount_options { + fs_type = "ext4" + mount_flags = ["ro"] +} +``` + +## Volume Specification Parameters + +- `id` `(string: )` - The unique ID of the volume. This will + be how [`volume`][csi_volume] stanzas in a jobspec refer to the volume. + +- `name` `(string: )` - The display name of the volume. + +- `type` `(string: )` - The type of volume. Currently only + `"csi"` is supported. + +- `external_id` `(string: )` - The ID of the physical volume + from the storage provider. For example, the volume ID of an AWS EBS + volume or Digital Ocean volume. + +- `plugin_id` `(string: )` - The ID of the [CSI + plugin][csi_plugin] that manages this volume. + +- `access_mode` `(string: )` - Defines whether a volume + should be available concurrently. Can be one of + `"single-node-reader-only"`, `"single-node-writer"`, + `"multi-node-reader-only"`, `"multi-node-single-writer"`, or + `"multi-node-multi-writer"`. Most CSI plugins support only + single-node modes. Consult the documentation of the storage provider + and CSI plugin. + +- `attachment_mode` `(string: )` - The storage API that will + be used by the volume. Most storage providers will support + `"file-system"`, to mount pre-formatted file system volumes. Some + storage providers will support `"block-device"`, which will require + the job be configured with appropriate mount options. + +- `mount_options` ([mount_options][]:nil) - Options for + mounting `block-device`volumes without a pre-formatted file system. + - `fs_type`: file system type (ex. `"ext4"`) + - `mount_flags`: the flags passed to `mount` (ex. `"ro,noatime"`) + +[volume_specification]: #volume-specification +[csi]: https://github.com/container-storage-interface/spec +[csi_plugin]: /docs/job-specification/csi_plugin +[csi_volumes]: /docs/job-specification/volume diff --git a/website/pages/docs/commands/volume/status.mdx b/website/pages/docs/commands/volume/status.mdx new file mode 100644 index 000000000..722d60b6a --- /dev/null +++ b/website/pages/docs/commands/volume/status.mdx @@ -0,0 +1,107 @@ +--- +layout: docs +page_title: 'Commands: volume status' +sidebar_title: status +description: | + Display information and status of volumes. +--- + +# Command: volume status + +The `volume status` command displays status information for [Container +Storage Interface (CSI)][csi] volumes. + +## Usage + +```plaintext +nomad volume status [options] [volume] +``` + +This command accepts an optional volume ID or prefix as the sole argument. If there +is an exact match based on the provided volume ID or prefix, then information about +the specific volume is queried and displayed. Otherwise, a list of matching volumes +and information will be displayed. + +If the ID is omitted, the command lists out all of the existing volumes and a few +of the most useful status fields for each. + +## General Options + +@include 'general_options.mdx' + +## Status Options + +- `-type`: Display only volumes of a particular type. Currently only + the `csi` type is supported, so this option can be omitted when + querying the status of CSI volumes. + +- `-plugin_id`: Display only volumes managed by a particular [CSI + plugin][csi_plugin]. + +- `-short`: Display short output. Used only when a single volume is + being queried. Drops verbose volume allocation data from the + output. + +- `-verbose`: Show full information. Allocation create and modify + times are shown in `yyyy/mm/dd hh:mm:ss` format. + +## Examples + +List of all volumes: + +```shell +$ nomad volume [-type csi] status +ID Name Plugin ID Schedulable Access Mode +ebs_prod_db1 database ebs-prod true single-node-writer +``` + +Short view of a specific volume: + +```shell +$ nomad volume status [-verbose] [-plugin=ebs-prod] ebs_prod_db1 +ID = ebs_prod_db1 +Name = database +Type = csi +External ID = vol-23452345 +Plugin ID = ebs-prod +Provider = aws.ebs +Version = 1.0.1 +Schedulable = true +Controllers Healthy = 1 +Controllers Expected = 1 +Nodes Healthy = 1 +Nodes Expected = 1 +Access Mode = single-node-writer +Attachment Mode = file-system +Mount Options = fs_type: ext4 flags: ro +Namespace = default +``` + +Full status information of a volume: + +```shell +$ nomad volume status [-verbose] [-plugin=ebs-prod] ebs_prod_db1 +ID = ebs_prod_db1 +Name = database +Type = csi +External ID = vol-23452345 +Plugin ID = ebs-prod +Provider = aws.ebs +Version = 1.0.1 +Schedulable = true +Controllers Healthy = 1 +Controllers Expected = 1 +Nodes Healthy = 1 +Nodes Expected = 1 +Access Mode = single-node-writer +Attachment Mode = file-system +Mount Options = fs_type: ext4 flags: ro +Namespace = default + +Allocations +ID Node ID Access Mode Task Group Version Desired [...] +b00fa322 28be17d5 write csi 0 run +``` + +[csi]: https://github.com/container-storage-interface/spec +[csi_plugin]: /docs/job-specification/csi_plugin diff --git a/website/pages/docs/job-specification/csi_plugin.mdx b/website/pages/docs/job-specification/csi_plugin.mdx new file mode 100644 index 000000000..41e6257cd --- /dev/null +++ b/website/pages/docs/job-specification/csi_plugin.mdx @@ -0,0 +1,89 @@ +--- +layout: docs +page_title: csi_plugin Stanza - Job Specification +sidebar_title: csi_plugin +description: >- + The "csi_plugin" stanza allows the task to specify it provides a + Container Storage Interface plugin to the cluster. +--- + +# `csi_plugin` Stanza + + + +The "csi_plugin" stanza allows the task to specify it provides a +Container Storage Interface plugin to the cluster. Nomad will +automatically register the plugin so that it can be used by other jobs +to claim [volumes][csi_volumes]. + +```hcl +csi_plugin { + id = "csi-hostpath" + type = "monolith" + mount_dir = "/csi" +} +``` + +## `csi_plugin` Parameters + +- `id` `(string: )` - This is the ID for the plugin. Some + plugins will require both controller and node plugin types (see + below); you need to use the same ID for both so that Nomad knows the + belong to the same plugin. + +- `type` `(string: )` - One of `node`, `controller`, or + `monolith`. Each plugin supports one or more types. Each Nomad + client node where you want to mount a volume will need a `node` + plugin instance. Some plugins will also require one or more + `controller` plugin instances to communicate with the storage + provider's APIs. Some plugins can serve as both `controller` and + `node` at the same time, and these are called `monolith` + plugins. Refer to your CSI plugin's documentation. + +- `mount_dir` `(string: )` - The directory path inside the + container where the plugin will expect a Unix domain socket for + bidirectional communication with Nomad. + +## `csi_plugin` Examples + +```hcl +job "plugin-efs" { + datacenters = ["dc1"] + + # you can run node plugins as service jobs as well, but running + # as a system job ensures all nodes in the DC have a copy. + type = "system" + + group "nodes" { + task "plugin" { + driver = "docker" + + config { + image = "amazon/aws-efs-csi-driver:latest" + + args = [ + "node", + "--endpoint=unix://csi/csi.sock", + "--logtostderr", + "--v=5", + ] + + # all CSI node plugins will need to run as privileged tasks + # so they can mount volumes to the host. controller plugins + # do not need to be privileged. + privileged = true + } + + csi_plugin { + id = "aws-efs0" + type = "node" + mount_dir = "/csi" # this path /csi matches the --endpoint + # argument for the container + } + } + } +} +``` + +[csi]: https://github.com/container-storage-interface/spec +[csi_volumes]: /docs/job-specification/volume diff --git a/website/pages/docs/job-specification/volume.mdx b/website/pages/docs/job-specification/volume.mdx index 8109ce45f..39772da80 100644 --- a/website/pages/docs/job-specification/volume.mdx +++ b/website/pages/docs/job-specification/volume.mdx @@ -4,22 +4,19 @@ page_title: volume Stanza - Job Specification sidebar_title: volume description: >- The "volume" stanza allows the group to specify that it requires a given - volume - - from the cluster. Nomad will automatically handle ensuring that the volume is - - available and mounted into the task. + volume from the cluster. Nomad will automatically handle ensuring that the + volume is available and mounted into the task. --- # `volume` Stanza -The `volume` stanza allows the group to specify that it requires a given volume -from the cluster. +The `volume` stanza allows the group to specify that it requires a +given volume from the cluster. -The key of the stanza is the name of the volume as it will be exposed to task -configuration. +The key of the stanza is the name of the volume as it will be exposed +to task configuration. ```hcl job "docs" { @@ -33,24 +30,32 @@ job "docs" { } ``` -The Nomad server will ensure that the allocations are only scheduled on hosts -that have a set of volumes that meet the criteria specified in the `volume` -stanzas. +The Nomad server will ensure that the allocations are only scheduled +on hosts that have a set of volumes that meet the criteria specified +in the `volume` stanzas. These may be [host volumes][host_volume] +configured on the client, or [CSI volumes][csi_volume] dynamically +mounted by [CSI plugins][csi_plugin]. -The Nomad client will make the volumes available to tasks according to the -[volume_mount][volume_mount] stanza in the `task` configuration. +The Nomad client will make the volumes available to tasks according to +the [volume_mount][volume_mount] stanza in the `task` configuration. ## `volume` Parameters -- `type` `(string: "")` - Specifies the type of a given volume. Currently the - only possible volume type is `"host"`. +- `type` `(string: "")` - Specifies the type of a given volume. The + valid volume types are `"host"` and `"csi"`. -- `source` `(string: )` - The name of the volume to request. When using - `host_volume`'s this should match the published name of the host volume. +- `source` `(string: )` - The name of the volume to + request. When using `host_volume`'s this should match the published + name of the host volume. When using `csi` volumes, this should match + the ID of the registed volume. -- `read_only` `(bool: false)` - Specifies that the group only requires read only - access to a volume and is used as the default value for the `volume_mount -> read_only` configuration. This value is also used for validating `host_volume` - ACLs and for scheduling when a matching `host_volume` requires `read_only` - usage. +- `read_only` `(bool: false)` - Specifies that the group only requires + read only access to a volume and is used as the default value for + the `volume_mount -> read_only` configuration. This value is also + used for validating `host_volume` ACLs and for scheduling when a + matching `host_volume` requires `read_only` usage. [volume_mount]: /docs/job-specification/volume_mount 'Nomad volume_mount Job Specification' +[host_volume]: /docs/configuration/client/#host_volume-stanza +[csi_volume]: /docs/commands/volume/register +[csi_plugin]: /docs/job-specification/csi_plugin