docs: add job spec and plugin authoring pages for secrets (#26529)

---------

Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com>

website/content/docs/configuration/client.mdx

@@ -221,6 +221,12 @@ client {
   controls on the behavior of task
   [`template`](/nomad/docs/job-specification/template) blocks.
 
+- `common_plugin_dir` `(string: "")` - Specifies the directory where you should
+  place plugins that conform to the [common plugins interface][common_plugins_interface]. When this
+  parameter is empty, Nomad generates the path using the [top-level
+  `data-dir`][top_level_data_dir] suffixed with `common_plugins`, like
+  `"/opt/nomad/common_plugins"`. This must be an absolute path.
+
 - `host_volume` <code>([host_volume](#host_volume-block): nil)</code> - Exposes
   paths from the host as volumes that can be mounted into jobs.
 
@@ -852,3 +858,4 @@ client {
 [dynamic host volumes]: /nomad/docs/other-specifications/volume/host
 [`volume create`]: /nomad/commands/volume/create
 [`volume register`]: /nomad/commands/volume/register
+[common_plugins_interface]: /nomad/plugins/author#common-plugin-interface

website/content/docs/configuration/index.mdx

@@ -1,10 +1,10 @@
 ---
 layout: docs
-page_title: Nomad Agent Configuration
+page_title: Nomad agent configuration
 description: This section contains reference information for configuring Nomad agents. Learn how Nomad loads and merges multiple configuration files. Configure networking and advertise addresses, Nomad regions and datacenters, data directories, Consul integration, handshake limits, connections per client, logging, plugins, Sentinel policies, security, update check behavior, and Vault integration. Enable debugging, write to syslog, leave on interrupt or terminate, and cross-origin resource sharing (CORS).
 ---
 
-# Nomad Agent Configuration
+# Nomad agent configuration
 
 This page provides an overview of reference information for configuring Nomad
 agents. Learn how Nomad loads and merges multiple configuration files. Configure
@@ -18,7 +18,7 @@ Write configuration files in [HCL][hcl] or specify parameters as command-line
 flags. Nomad can read and combine parameters from multiple configuration files
 or directories to configure the Nomad agent.
 
-## Configuration File Load Order and Merging
+## Configuration file load order and merging
 
 The Nomad agent supports multiple configuration files, which can be provided
 using the `-config` CLI flag. The flag can accept either a file or folder. In
@@ -84,7 +84,11 @@ consul {
 `client` and `server`, although this is supported to simplify development and
 testing.
 
-## General Parameters
+## Configure plugin directories
+
+@include 'plugins/config-dirs.mdx'
+
+## General parameters
 
 - `acl` `(`[`ACL`]`: nil)` - Specifies configuration which is specific to ACLs.
 
@@ -385,7 +389,7 @@ before attempting to reload its configuration.
 
 ## Examples
 
-### Custom Region and Datacenter
+### Custom region and datacenter
 
 This example shows configuring a custom region and data center for the Nomad
 agent:

website/content/docs/job-specification/secret.mdx

@@ -0,0 +1,122 @@
+---
+layout: docs
+page_title: secret block in the job specification
+description: |-
+  The `secret` block allows the task to specify that it requires a secret from a
+  specified location. Nomad automatically retrieves the secret for the task
+  and provides it for variable interpretation.
+---
+
+# `secret` block in the job specification
+
+<Placement
+  groups={[
+    ['job', 'secret'],
+    ['job', 'group', 'secret'],
+    ['job', 'group', 'task', 'secret'],
+  ]}
+/>
+
+The `secret` block allows a task to specify that it requires a secret from
+a specified location. Nomad automatically retrieves the contents of the
+secret, which you may reference as a variable within the task specification.
+If specified at the `group` level, the secret is available to every task
+within the group. If specified at the `job` level, every task in the job
+fetches the secret and is able to interpret it within the task specification.
+
+```hcl
+job "docs" {
+  group "example" {
+    task "server" {
+      secret "my_secret" {
+        provider = "vault"
+        path     = "path/to/secret"
+        config   = {
+          engine = "kv_v2"
+        }
+      }
+    }
+  }
+}
+```
+
+Nomad parses the key value pairs in a secret and makes them available for
+variable interpolation in the following format:
+`secret.<secret_name>.<secret_key>`. To use the preceding task specification
+as an example, you would reference the value in the secret key "foo"  as
+`${secret.my_secret.foo}`.
+
+## Parameters
+
+- `provider` `(string: "")` - Specifies the underlying implementation to use in
+  order for Nomad to interact with a specific secret store.
+
+- `path` `(string: "")` - Specifies the location of the secret within the given
+  secret store.
+
+- `config` `(block: {})` - Specifies any custom attributes used by built-in
+  providers order to fetch the secret. Only available for built-in `vault` and
+  `nomad` providers.
+
+- `env` `(map[string]string)` - Specifies any environment variables to pass to a
+  custom plugin provider. Only available for custom providers.
+
+## Built-in providers
+
+- `nomad` - Query variables from Nomad. Nomad uses the task's [Workload
+  Identity] [Workload Identity] token in order to fetch the variable, so the
+  task must have the necessary access via ACLs to read the variable.
+
+  Config parameters:
+
+   - `namespace` `(string: "default")` - Specifies the namespace where the
+  variable is located.
+
+
+- `vault` - Query secrets from Vault. Similar to the template block, making
+  requests to Vault requires a vault token, which is generated via the
+  [vault][vault] block.
+
+  Config parameters:
+
+    - `engine` `(string: "kv_v2")` - Specifies what Vault storage engine to
+  query. Options are `kv_v1` and `kv_v2`.
+
+
+## Custom providers
+
+Nomad extends the provider list via the use of custom providers plugins. Custom
+providers must implement both the `fingerprint` and `fetch` functionality. You
+must place custom providers in the `common_plugin_dir/secrets/` subdirectory. On
+startup, the Nomad client fingerprints executables in the secrets directory by
+executing it with the first argument of `fingerblock`. When using a custom
+plugin "foo", Nomad looks for an executable "foo" and runs it with the first
+argument being "fetch", and the second argument being the secret's path
+parameter. Refer to the [plugin authoring][secret-provider] guide on secret providers
+for an example custom provider.
+
+A custom provider should print to stdout a fingerprint response:
+
+```json
+{
+  "type": "secrets",
+  "version": "0.0.1" (use appropriate version)
+}
+```
+
+And a valid "fetch" response should have the form:
+
+```json
+{
+  "result": {
+    "key1": "value1",
+    "key2": "value2"
+  },
+  "error": ""
+}
+```
+
+[Workload Identity]: /nomad/docs/concepts/workload-identity
+[vault]: /nomad/docs/job-specification/vault
+[common_plugin_dir]: /nomad/docs/configuration/client#custom_plugin_dir
+[secret-provider]: /nomad/plugins/author/secret-provider

website/content/partials/plugins/config-dirs.mdx

@@ -0,0 +1,13 @@
+You may configure the directories where you place plugin executable binaries.
+The following table lists the Agent configuration field as well as the default
+directory for each plugin type. Note that the default directories use the value
+of the configured [`data_dir`](/nomad/docs/configuration#data_dir) parameter,
+which specifies a local directory.
+
+| Type                                                     | Plugin directory configuration parameter                                                   | Default plugin directory         | Plugin configuration                                                       |
+|----------------------------------------------------------|--------------------------------------------------------------------------------------------|----------------------------------|----------------------------------------------------------------------------|
+| [Device driver](/nomad/plugins/devices)                  | [`plugin_dir`](/nomad/docs/configuration#plugin_dir)                                       | `<data_dir>/plugins`             | [`plugin` block](/nomad/docs/configuration/plugin)                         |
+| [Task driver](/nomad/plugins/drivers)                    | [`plugin_dir`](/nomad/docs/configuration#plugin_dir)                                       | `<data_dir>/plugins`             | [`plugin` block](/nomad/docs/configuration/plugin)                         |
+| [Dynamic host volume](/nomad/plugins/author/host-volume) | [`client.host_volume_plugin_dir`](/nomad/docs/configuration/client#host_volume_plugin_dir) | `<data_dir>/host_volume_plugins` | [ `plugin`  block](/nomad/docs/configuration/plugin)                       |
+| [Secret provider](/nomad/plugins/author/secret-provider) | [`client.common_plugin_dir`/secrets](/nomad/docs/configuration/client#common_plugin_dir)   | `<data_dir>/common_plugins`      | [ `plugin`  block](/nomad/docs/configuration/plugin)                       |
+| [CNI reference](https://www.cni.dev/plugins/current/)    | [`client.cni_path`](/nomad/docs/configuration/client#cni_path)                             | `/opt/cni/bin`                   | [`client.cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) |

website/content/plugins/author/index.mdx

@@ -18,11 +18,15 @@ The following components are currently pluggable within Nomad:
   custom hardware device.
 - [Host volume](/nomad/plugins/author/host-volume): Extend workload storage
   functionality with a plugin specific to your local storage environment.
+- [Secret provider](/nomad/plugins/author/secret-provider): Extend task
+  execution functionality with a custom secret provider.
 - [Task driver](/nomad/plugins/author/task-driver): Extend task execution
   functionality with a custom task driver.
 
 ## Architecture
 
+### Go-plugin
+
 The Nomad task driver and device plugin framework uses the [go-plugin][goplugin]
 project to expose a language-independent plugin interface. Plugins implement a
 set of gRPC services and methods that Nomad manages by running the plugin and
@@ -32,4 +36,19 @@ Go interfaces and structs exist for each plugin type that abstracts away
 go-plugin and the gRPC interface. The guides in this documentation reference
 these abstractions for ease of use.
 
+### Common plugin interface
+
+The Nomad secret provider formalized a pattern first introduced with dynamic
+host volume plugins into a common plugin interface. These plugins differ from
+Go-plugins in that they generally perform a simple task and exit, not needing to
+run for extended periods of time or needing to manage other resources similar to
+task and device drivers. Common plugins must implement a `fingerprint` command
+and implementation-specific commands.
+
+The parent directory of all common plugin implementations is managed via the
+[common_plugin_dir][common_plugin_dir], with each implementation residing in a
+specific subdirectory. Check the specific plugin's documentation for directory details.
+
 [goplugin]: https://github.com/hashicorp/go-plugin
+[common_plugin_dir]: /nomad/docs/configuration/client#common_plugin_dir
+[host_volume_plugin_dir]: /nomad/docs/configuration/client#host_volume_plugin_dir

website/content/plugins/author/secret-provider.mdx

@@ -0,0 +1,246 @@
+---
+layout: docs
+page_title: Create a secret provider plugin
+description: |-
+  Use Nomad's secret provider plugins specification to create a secret provider plugin so that you can use secrets in your tasks from any secrets store.
+---
+
+# Create a secret provider plugin
+
+This page describes the plugin specification for
+[custom secret providers][custom-providers], with examples, so you can write
+your own plugin to utilize secrets from any secrets store.
+
+The full [specification](#specification) follows the examples,
+and is followed by a list of [general considerations](#considerations).
+
+Follow this workflow to create and use a custom secret provider:
+
+1. Develop the plugin code based on the [specification](#specification).
+1. Compile your plugin into an executable binary.
+1. Place the binary in the
+   [`client.common_plugin_dir/secrets/`][common_plugin_dir] directory on each
+   Nomad client node.
+1. Configure the `secret` block in your job specification.
+
+## Example
+
+We wrote the example in Go, but the specification is lean enough to be
+readily fulfilled in any language. The example code is not meant to be inclusive of all error handling.
+
+In this example code, the `aws` secret provider plugin fetches a secret from AWS
+Secret Manager. Refer to the external [AWS Secrets Manager User
+Guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)
+for details on AWS Secrets Manager and [how to fetch secrets in various languages](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html).
+
+### Plugin code
+
+<CodeBlockConfig heading="aws.go" lineNumbers="true">
+
+```golang
+package main
+
+import (
+	"context"
+	"fmt"
+	"os"
+
+	"github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
+	"github.com/aws/aws-sdk-go/aws"
+)
+
+const generalUsage = "You must run with either 'fingerprint' or 'fetch'. \nYou must pass 'fetch' to the path of the secret."
+
+func returnErr(err string) {
+	fmt.Printf(`{"error": "%s"}`, err)
+	os.Exit(1)
+}
+
+func main() {
+	if len(os.Args) < 2 {
+		returnErr("internal error not enough args")
+	}
+
+	switch os.Args[1] {
+	case "fingerprint":
+		fingerprint()
+	case "fetch":
+		// fetch should be called with the correct secret path
+		if len(os.Args) < 3 {
+			returnErr("internal error no path specified")
+		}
+		fetch(os.Args[2])
+	default:
+		returnErr("internal error incorrect function")
+	}
+}
+
+func fingerprint() {
+	fmt.Println(`{"type": "secrets", "version": "0.0.1"}`)
+}
+
+// resource: https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets-go-sdk.html
+func fetch(secretName string) {
+	ctx := context.Background()
+
+	config, err := config.LoadDefaultConfig(ctx, config.WithRegion(os.Getenv("AWS_REGION")))
+	if err != nil {
+		returnErr("error loading default aws config")
+	}
+
+	// Create Secrets Manager client
+	svc := secretsmanager.NewFromConfig(config)
+
+	input := &secretsmanager.GetSecretValueInput{
+		SecretId:     aws.String(secretName),
+		VersionStage: aws.String("AWSCURRENT"),
+	}
+
+	result, err := svc.GetSecretValue(ctx, input)
+	if err != nil {
+		returnErr(err.Error())
+	}
+
+	// Decrypts secret using the associated KMS key.
+	var secretString string = *result.SecretString
+
+	fmt.Printf(`{"result": "%s"}`, secretString)
+}
+```
+
+</CodeBlockConfig>
+
+### Task configuration
+
+Configure your task to fetch the secret with your customer `aws` secret
+provider. Make sure the `provider` value matches the name of your plugin
+executable.
+
+In this example job specification, the task uses your custom `aws`
+secret provider to fetch an AWS secret named `my-aws-secret` from the AWS
+Secrets Manager in region `us-east-2`.
+
+```hcl
+job "docs" {
+  group "example" {
+    task "server" {
+      secret "my_secret" {
+        provider = "aws"
+        path     = "my-aws-secret"
+        env {
+          AWS_REGION = "us-east-2"
+        }
+      }
+    }
+  }
+}
+```
+
+## Specification
+
+Nomad registers a  secret provider plugin if the plugin meets all of the
+following criteria:
+
+- Is an executable file, such as a script or binary
+- Is located in the [`client.common_plugin_dir/secrets/`][common_plugin_dir]
+  directory on Nomad client nodes
+- Responds appropriately to a `fingerprint` call
+
+### Operations
+
+A secrete provider plugin must fulfill all of the following operations:
+
+- [fingerprint](#fingerprint)
+- [fetch](#fetch)
+
+Nomad passes the operation as the first positional argument to the plugin.
+Pass that and other information as environment variables prefixed with `CPI_`.
+
+#### fingerprint
+
+<blockquote style={{borderLeft: "solid 1px #00ca8e"}}>
+
+Nomad calls `fingerprint` to discover valid plugins when the client agent
+starts or is reloaded with a SIGHUP. The version it returns is used to register
+the plugin on the Nomad node.
+
+**CLI arguments:** `$1=fingerprint`
+
+**Environment variables:**
+
+```
+CPI_OPERATION=fingerprint
+```
+
+**Expected stdout:**
+
+```
+{"type": "secrets", "version": "0.0.1"}
+```
+
+**Requirements:**
+
+- Must complete within 10 seconds, or Nomad kills it. It should be much
+  faster, as no actual work should be done.
+- "type" registers this as a secret provider plugin.
+- "version" value must be valid per the [hashicorp/go-version][go-version]
+  golang package.
+
+</blockquote>
+
+#### fetch
+
+<blockquote style={{borderLeft: "solid 1px #00ca8e"}}>
+
+Nomad calls `fetch` when a Nomad job includes a secret block with a provider
+registered as a plugin. When calling fetch, Nomad includes the secret path as
+the second CLI argument. If the `env{}` block was specified, any key/values are
+supplied as environment variables to the plugin process.
+
+**CLI Arguments:** `$1=fetch $2=path`
+
+**Environment variables:**
+
+```
+CPI_OPERATION=fetch
+```
+
+**Expected stdout:**
+
+```
+{"result": {"key1": "value1", "key2": "value2"}}
+```
+
+**Expected stdout on error:**
+
+```
+{"result": {}, "error": "error message"}
+```
+
+Returning an error message is optional. Nomad returns the error message in any
+error returned to the user.
+
+**Requirements:**
+
+- Must complete within 10 seconds, or Nomad kills it.
+- Must return secret contents as a key/value object, even if the secret
+  was not originally in key/value format.
+
+</blockquote>
+
+## Considerations
+
+### Execution
+
+- The plugin is executed as the same user as the Nomad agent (likely root).
+
+[custom-providers]: /nomad/docs/job-specification/secret#custom-providers
+[go-version]: https://pkg.go.dev/github.com/hashicorp/go-version#pkg-constants
+[common_plugin_dir]: /nomad/docs/configuration/client#common_plugin_dir
+
+
+
+
+
+

website/content/plugins/index.mdx

@@ -13,3 +13,6 @@ bundled with Nomad.
 Consult the [Plugin authoring guide](/nomad/plugins/author) for details on how
 to create your own device, task driver, and host volume plugins.
 
+## Configure plugin directories
+
+@include 'plugins/config-dirs.mdx'

website/data/docs-nav-data.json

@@ -1165,6 +1165,10 @@
         },
         "path": "job-specification/schedule"
       },
+      {
+        "title": "secret",
+        "path": "job-specification/secret"
+      },
       {
         "title": "service",
         "path": "job-specification/service"

website/data/plugins-nav-data.json

@@ -128,6 +128,10 @@
         "title": "Host volume",
         "path": "author/host-volume"
       },
+      {
+        "title": "Secret provider",
+        "path": "author/secret-provider"
+      },
       {
         "title": "Task driver",
         "path": "author/task-driver"