From eabb47e2d08a9f73c1b3bd4144f8a918e57bc6ca Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 9 Feb 2023 16:03:43 -0800 Subject: [PATCH] Workload Identity, Task API, and Dynamic Node Metadata Docs (#16102) * docs: add dynamic node metadata api docs Also update all paths in the client API docs to explicitly state the `/v1/` prefix. We're inconsistent about that, but I think it's better to display the full path than to only show the fragment. If we ever do a `/v2/` whether or not we explicitly state `/v1/` in our docs won't be our greatest concern. * docs: add task-api docs --- website/content/api-docs/client.mdx | 272 ++++++++++++++---- website/content/api-docs/task-api.mdx | 100 +++++++ .../docs/concepts/workload-identity.mdx | 18 ++ .../docs/job-specification/identity.mdx | 4 - .../content/docs/job-specification/task.mdx | 5 + website/data/api-docs-nav-data.json | 4 + 6 files changed, 345 insertions(+), 58 deletions(-) create mode 100644 website/content/api-docs/task-api.mdx diff --git a/website/content/api-docs/client.mdx b/website/content/api-docs/client.mdx index a03ef02c1..3f5c53222 100644 --- a/website/content/api-docs/client.mdx +++ b/website/content/api-docs/client.mdx @@ -2,13 +2,13 @@ layout: api page_title: Client - HTTP API description: |- - The /client endpoints are used to access client statistics and inspect + The /client endpoints are used to access client information and inspect allocations running on a particular client. --- # Client HTTP API -The `/client` endpoints are used to interact with the Nomad clients. +The `/v1/client` endpoints are used to interact with the Nomad clients. Since Nomad 0.8.0, both a client and server can handle client endpoints. This is particularly useful for when a direct connection to a client is not possible due @@ -20,15 +20,179 @@ When accessing the endpoints via the server, if the desired node is ambiguous based on the URL, an additional `?node_id` query parameter must be provided to disambiguate. +## Read Node Metadata + +This endpoint queries Node metadata on a specific Client agent and responds +with the following fields: + +- `Meta` `(object)` - The Node metadata that will be registered with the Nomad + servers and used by the scheduler (after up to 10 seconds of delay for + batching). This is the merged version of the `Static` and `Dynamic` fields. + +- `Static` `(object)` - The Node metadata set in the Client agent's + configuration file. Only loaded when an agent starts. + +- `Dynamic` `(object)` - The Node metadata set via the API (see below). Unlike + `Meta` and `Static`, this object may contain `null` values to differentiate + "unset" keys from keys with an empty string value (`""`). + +Note that [`/v1/node/:node_id`][api-node-read] only contains the `Meta` object. +It may take up to 10 seconds for dynamic Node metadata to be sent to Servers +and visible through the Node API. Use the Node API to see the version of Node +metadata the scheduler uses. + +| Method | Path | Produces | +| ------ | --------------------- | ------------------ | +| `GET` | `/v1/client/metadata` | `application/json` | + +The table below shows this endpoint's support for +[blocking queries](/nomad/api-docs#blocking-queries) and +[required ACLs](/nomad/api-docs#acls). + +| Blocking Queries | ACL Required | +| ---------------- | ------------- | +| `NO` | `node:read` | + +### Parameters + +- `:node_id` `(string: )` - Specifies the node to query. + This is required when the endpoint is being accessed via a server. Defaults + to the node recieving the request otherwise. This is specified as part of + the URL. Note, this must be the _full_ node ID, not the short 8-character + one. This must be specified as part of the path (`?node_id=...`). + +### Sample Request + +```shell-session +$ nomad operator api /v1/client/metadata +``` + +### Sample Response + +Formatted by appending `?pretty` above. + +```json +{ + "Meta": { + "connect.proxy_concurrency": "1", + "connect.sidecar_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.gateway_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.log_level": "debug", + "foo": "bar" + }, + "Dynamic": { + "key_to_unset": null, + "foo": "bar", + "connect.log_level": "debug" + }, + "Static": { + "connect.sidecar_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.gateway_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.log_level": "info", + "connect.proxy_concurrency": "1" + } +} +``` + + +### Sample Request + +## Update Node Metadata + +This endpoint updates dynamic Node metadata on a specific Client agent. Since +dynamic Node metadata is only periodically synchronized to Nomad Servers, the +`Meta` returned in this API may not be reflected in the +[`/v1/node/:node_id`][api-node-read] API for up to 10 seconds. Scheduling uses +the Node API version of `Meta`. + +For convenience this endpoint returns the same response as a GET. + +| Method | Path | Produces | +| ------ | --------------------- | ------------------ | +| `POST` | `/v1/client/metadata` | `application/json` | + +The table below shows this endpoint's support for +[blocking queries](/nomad/api-docs#blocking-queries) and +[required ACLs](/nomad/api-docs#acls). + +| Blocking Queries | ACL Required | +| ---------------- | ------------- | +| `NO` | `node:write` | + +### Parameters + +- `NodeID` or `:node_id` `(string: )` - Specifies the node to query. + This is required when the endpoint is being accessed via a server. Defaults + to the node recieving the request otherwise. This is specified as part of + the URL. Note, this must be the _full_ node ID, not the short 8-character + one. This may be specified as part of the path (`?node_id=...`) or request + (`NodeID: "..."`). + +- `Meta` `(object: )` - Specifies the Node metadata keys to update. + Only specified keys are updated. + + - `` `(string: )` - Specifies a metadata key to update to a + particular value. Since `""` is a valid value and distinct from unset, + the `null` value is used to mark a key as unset. Keys must be valid + dotted HCL identifiers. For example `connect.log_level` is a valid key + while `some/path` is not. + +### Sample Payload + +```json +{ + "Meta": { + "connect.log_level": "debug", + "key_to_unset": null, + "foo": "bar" + } +} +``` + +### Sample Request + +Assuming the above payload is in a file called `meta.json`. + +```shell-session +$ nomad operator api /v1/client/metadata < meta.json +``` + +### Sample Response + +Formatted by appending `?pretty` above. + +```json +{ + "Meta": { + "connect.proxy_concurrency": "1", + "connect.sidecar_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.gateway_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.log_level": "debug", + "foo": "bar" + }, + "Dynamic": { + "key_to_unset": null, + "foo": "bar", + "connect.log_level": "debug" + }, + "Static": { + "connect.sidecar_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.gateway_image": "envoyproxy/envoy:v${NOMAD_envoy_version}", + "connect.log_level": "info", + "connect.proxy_concurrency": "1" + } +} +``` + ## Read Stats This endpoint queries the actual resources consumed on a node. The API endpoint is hosted by the Nomad client and requests have to be made to the nomad client whose resource usage metrics are of interest. -| Method | Path | Produces | -| ------ | --------------- | ------------------ | -| `GET` | `/client/stats` | `application/json` | +| Method | Path | Produces | +| ------ | ------------------ | ------------------ | +| `GET` | `/v1/client/stats` | `application/json` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -48,8 +212,7 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/stats +$ nomad operator api /v1/client/stats ``` ### Sample Response @@ -203,9 +366,9 @@ $ curl \ The client `allocation` endpoint is used to query the actual resources consumed by an allocation. -| Method | Path | Produces | -| ------ | ------------------------------------ | ------------------ | -| `GET` | `/client/allocation/:alloc_id/stats` | `application/json` | +| Method | Path | Produces | +| ------ | --------------------------------------- | ------------------ | +| `GET` | `/v1/client/allocation/:alloc_id/stats` | `application/json` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -224,8 +387,8 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/stats +$ nomad operator api \ + /v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/stats ``` ### Sample Response @@ -286,9 +449,9 @@ $ curl \ This endpoint reads the contents of a file in an allocation directory. -| Method | Path | Produces | -| ------ | -------------------------- | ------------ | -| `GET` | `/client/fs/cat/:alloc_id` | `text/plain` | +| Method | Path | Produces | +| ------ | ----------------------------- | ------------ | +| `GET` | `/v1/client/fs/cat/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -310,13 +473,13 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99 +$ nomad operator api \ + /v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99 ``` ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=alloc/file.json +$ nomad operator api \ + /v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=alloc/file.json ``` ### Sample Response @@ -330,9 +493,9 @@ $ curl \ This endpoint reads the contents of a file in an allocation directory at a particular offset and limit. -| Method | Path | Produces | -| ------ | ----------------------------- | ------------ | -| `GET` | `/client/fs/readat/:alloc_id` | `text/plain` | +| Method | Path | Produces | +| ------ | -------------------------------- | ------------ | +| `GET` | `/v1/client/fs/readat/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -360,8 +523,8 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/readat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/foo&offset=1323&limit=19303 +$ nomad operator api \ + /v1/client/fs/readat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/foo&offset=1323&limit=19303 ``` ### Sample Response @@ -374,9 +537,9 @@ $ curl \ This endpoint streams the contents of a file in an allocation directory. -| Method | Path | Produces | -| ------ | ----------------------------- | ------------ | -| `GET` | `/client/fs/stream/:alloc_id` | `text/plain` | +| Method | Path | Produces | +| ------ | -------------------------------- | ------------ | +| `GET` | `/v1/client/fs/stream/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -406,8 +569,8 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/stream/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/logs/redis.log +$ nomad operator api \ + /v1/client/fs/stream/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/logs/redis.log ``` ### Sample Response @@ -442,9 +605,9 @@ fields: This endpoint streams a task's stderr/stdout logs. -| Method | Path | Produces | -| ------ | --------------------------- | ------------ | -| `GET` | `/client/fs/logs/:alloc_id` | `text/plain` | +| Method | Path | Produces | +| ------ | ------------------------------ | ------------ | +| `GET` | `/v1/client/fs/logs/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -479,8 +642,8 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/logs/5fc98185-17ff-26bc-a802-0c74fa471c99 +$ nomad operator api \ + /v1/client/fs/logs/5fc98185-17ff-26bc-a802-0c74fa471c99 ``` ### Sample Response @@ -515,9 +678,9 @@ fields: This endpoint lists files in an allocation directory. -| Method | Path | Produces | -| ------ | ------------------------- | ------------ | -| `GET` | `/client/fs/ls/:alloc_id` | `text/plain` | +| Method | Path | Produces | +| ------ | ---------------------------- | ------------ | +| `GET` | `/v1/client/fs/ls/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -539,8 +702,8 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/ls/5fc98185-17ff-26bc-a802-0c74fa471c99 +$ nomad operator api \ + /v1/client/fs/ls/5fc98185-17ff-26bc-a802-0c74fa471c99 ``` ### Sample Response @@ -568,9 +731,9 @@ $ curl \ This endpoint stats a file in an allocation. -| Method | Path | Produces | -| ------ | --------------------------- | ------------ | -| `GET` | `/client/fs/stat/:alloc_id` | `text/plain` | +| Method | Path | Produces | +| ------ | ------------------------------ | ------------ | +| `GET` | `/v1/client/fs/stat/:alloc_id` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -592,8 +755,8 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/fs/stat/5fc98185-17ff-26bc-a802-0c74fa471c99 +$ nomad operator api \ + /v1/client/fs/stat/5fc98185-17ff-26bc-a802-0c74fa471c99 ``` ### Sample Response @@ -614,9 +777,9 @@ This endpoint forces a garbage collection of a particular, stopped allocation on a node. Note that the allocation will still exist on the server and appear in server responses. -| Method | Path | Produces | -| ------ | --------------------------------- | ------------------ | -| `GET` | `/client/allocation/:alloc_id/gc` | `application/json` | +| Method | Path | Produces | +| ------ | ------------------------------------ | ------------------ | +| `GET` | `/v1/client/allocation/:alloc_id/gc` | `application/json` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -635,17 +798,17 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://nomad.rocks/v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/gc +$ nomad operator api \ + /v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/gc ``` ## GC All Allocation This endpoint forces a garbage collection of all stopped allocations on a node. -| Method | Path | Produces | -| ------ | ------------ | ------------ | -| `GET` | `/client/gc` | `text/plain` | +| Method | Path | Produces | +| ------ | --------------- | ------------ | +| `GET` | `/v1/client/gc` | `text/plain` | The table below shows this endpoint's support for [blocking queries](/nomad/api-docs#blocking-queries) and @@ -665,6 +828,7 @@ The table below shows this endpoint's support for ### Sample Request ```shell-session -$ curl \ - https://localhost:4646/v1/client/gc +$ nomad operator api /v1/client/gc ``` + +[api-node-read]: /nomad/api-docs/nodes diff --git a/website/content/api-docs/task-api.mdx b/website/content/api-docs/task-api.mdx new file mode 100644 index 000000000..8384c49ea --- /dev/null +++ b/website/content/api-docs/task-api.mdx @@ -0,0 +1,100 @@ +--- +layout: api +page_title: Task HTTP API +description: |- + Jobs can access Nomad's HTTP API via the Task API. +--- + +# Task API + +Nomad's Task API provides every task managed by Nomad with a Unix Domain Socket +(UDS) to access the local agent's HTTP API. Regardless of agent configuration +the Task API does *not* require [mTLS][], but *always* requires authentication. +See below for details. + +The Unix Domain Socket is located at `${SECRETS_DIR}/api.sock`. + +## Rationale + +Nomad's HTTP API is available on every agent at the configured +[`bind_addr`][bind_addr]. While this is convenient for user access, it is not +always accessible to workloads running on Nomad. These workloads may have a +network configuration that makes it impossible to access the agent HTTP +address, or the agent's HTTP address may be difficult for workloads to discover +in a way that's portable between Nomad nodes and clusters. + +A Unix Domain Socket is a way to expose network services that works with most +runtimes and operating systems and adds minimal complexity or runtime overhead +to Nomad. + +## Security + +Unlike the agent's HTTP API, the Task API *always requires authentication* even +if [ACLs][acl] are disabled. This allows Nomad to always make the Task API +available even if the workload is untrusted. + +Both [ACL Tokens][acl-tokens] and [Workload Identities][workload-id] are +accepted. Once the Task API has authneticated the credentials, the normal +endpoint-specific authorization is applied when ACLs are enabled. + +The Workload Identity should be used by tasks accessing the Task API. + +An ACL Token should be used when an operator is accessing the Task API via +[`nomad alloc exec`][alloc-exec] or when a task is proxying Nomad HTTP requests +on behalf of an authenticated user. The Task API could be used by a proxy +presenting Nomad's UI with a standard TLS certificate for browsers. + +If [`task.user`][task-user] is set in the jobspec, the Task API will only be +usable by that user. Otherwise the Unix Domain Socket is accessible by any +user. + +mTLS is never enabled for the Task API since traffic never leaves the node. + +## Using the Task API + +The following jobspec will use the Task API to set [Dynamic Node Metadata][dnm] +and exit. + +```hcl +job "taskapi-example" { + type = "batch" + + group "taskapi-example" { + + task "taskapi" { + driver = "docker" + + config { + image = "curlimages/curl:7.87.0" + args = [ + "--unix-socket", "${NOMAD_SECRETS_DIR}/api.sock", + "-H", "Authorization: Bearer ${NOMAD_TOKEN}", + "--data-binary", "{\"Meta\": {\"example\": \"Hello World!\"}}", + "--fail-with-body", + "--verbose", + "localhost/v1/client/metadata", + ] + } + + identity { + env = true + } + } + } +} +``` + +If the job was able to run successfully after about 10 seconds you can observe +the outcome by searching for the updated Node's metadata: + +```shell-session +$ nomad node status -filter 'Meta.example == "Hello World!"' +``` + +[acl]: /nomad/docs/concepts/acl +[acl-tokens]: /nomad/docs/concepts/acl#token +[alloc-exec]: /nomad/docs/commands/alloc/exec +[bind_addr]: /nomad/docs/configuration +[mTLS]: /nomad/tutorials/transport-security/security-enable-tls +[task-user]: /nomad/docs/job-specification/task#user +[workload-id]: /nomad/docs/concepts/workload-identity diff --git a/website/content/docs/concepts/workload-identity.mdx b/website/content/docs/concepts/workload-identity.mdx index 7b7725fff..e85f7db87 100644 --- a/website/content/docs/concepts/workload-identity.mdx +++ b/website/content/docs/concepts/workload-identity.mdx @@ -26,6 +26,23 @@ includes the following identity claims: While Nomad always creates and uses workload identities internally, the JWT is not exposed to tasks by default. +To expose Workload Identity to tasks, add an [`identity`][identity-block] block +to your jobspec: + +```hcl +task "example" { + + identity { + # Expose Workload Identity in NOMAD_TOKEN env var + env = true + + # Expose Workload Identity in ${SECRETS_DIR}/nomad_token file + file = true + } + +} +``` + # Workload Associated ACL Policies You can associate additional ACL policies with workload identities by passing @@ -79,6 +96,7 @@ In Nomad 1.4.0 the workload identity is used only for `template` access to [Variables][] and not exposed outside of Nomad. [allocation]: /nomad/docs/concepts/architecture#allocation +[identity-block]: /nomad/docs/job-specification/identity [plan applier]: /nomad/docs/concepts/scheduling/scheduling [Variables]: /nomad/docs/concepts/variables [JSON Web Token (JWT)]: https://datatracker.ietf.org/doc/html/rfc7519 diff --git a/website/content/docs/job-specification/identity.mdx b/website/content/docs/job-specification/identity.mdx index 36cf26fcd..8e0c162d9 100644 --- a/website/content/docs/job-specification/identity.mdx +++ b/website/content/docs/job-specification/identity.mdx @@ -47,9 +47,5 @@ job "docs" { readable by that user. Otherwise the file is readable by everyone but is protected by parent directory permissions. -Note that while both parameters default to `true`, the `identity` block itself -must be present in the job specification or the workload identity will not be -exposed. - [taskuser]: /nomad/docs/job-specification/task#user "Nomad task Block" [Workload Identity]: /nomad/docs/concepts/workload-identity "Nomad Workload Identity" diff --git a/website/content/docs/job-specification/task.mdx b/website/content/docs/job-specification/task.mdx index 4f68089eb..40006df9a 100644 --- a/website/content/docs/job-specification/task.mdx +++ b/website/content/docs/job-specification/task.mdx @@ -51,6 +51,9 @@ job "docs" { - `env` ([Env][]: nil) - Specifies environment variables that will be passed to the running process. +- `identity` ([Identity][]: nil) - Expose [Workload Identity][] to + the task. + - `kill_timeout` `(string: "5s")` - Specifies the duration to wait for an application to gracefully quit before force-killing. Nomad first sends a [`kill_signal`][kill_signal]. If the task does not exit before the configured @@ -207,6 +210,7 @@ task "server" { [affinity]: /nomad/docs/job-specification/affinity 'Nomad affinity Job Specification' [dispatchpayload]: /nomad/docs/job-specification/dispatch_payload 'Nomad dispatch_payload Job Specification' [env]: /nomad/docs/job-specification/env 'Nomad env Job Specification' +[Identity]: /nomad/docs/job-specification/identity 'Nomad identity Job Specification' [meta]: /nomad/docs/job-specification/meta 'Nomad meta Job Specification' [resources]: /nomad/docs/job-specification/resources 'Nomad resources Job Specification' [lifecycle]: /nomad/docs/job-specification/lifecycle 'Nomad lifecycle Job Specification' @@ -224,3 +228,4 @@ task "server" { [user_denylist]: /nomad/docs/configuration/client#user-denylist [max_kill]: /nomad/docs/configuration/client#max_kill_timeout [kill_signal]: /nomad/docs/job-specification/task#kill_signal +[Workload Identity]: /nomad/docs/concepts/workload-identity 'Nomad Workload Identity' diff --git a/website/data/api-docs-nav-data.json b/website/data/api-docs-nav-data.json index 563b4168d..8437b610b 100644 --- a/website/data/api-docs-nav-data.json +++ b/website/data/api-docs-nav-data.json @@ -14,6 +14,10 @@ "title": "JSON Jobs", "path": "json-jobs" }, + { + "title": "Task API", + "path": "task-api" + }, { "divider": true },