From 5dc7e7fe2583030721dace2da6fbcdf64939d534 Mon Sep 17 00:00:00 2001 From: Aimee Ukasick Date: Wed, 30 Jul 2025 09:02:28 -0500 Subject: [PATCH] Docs: Chore: Ent labels (#26323) * replace outdated tutorial links * update more tutorial links * Add CE/ENT or ENT to left nav * remove ce/ent labels * revert enterprise features --- website/content/api-docs/acl/index.mdx | 2 +- website/content/api-docs/acl/policies.mdx | 2 +- website/content/api-docs/acl/tokens.mdx | 2 +- website/content/api-docs/events.mdx | 2 +- website/content/api-docs/index.mdx | 2 +- .../content/api-docs/operator/autopilot.mdx | 8 ++-- website/content/api-docs/search.mdx | 3 +- .../content/api-docs/sentinel-policies.mdx | 2 +- website/content/api-docs/ui.mdx | 4 +- website/content/commands/acl/index.mdx | 4 +- .../content/commands/operator/utilization.mdx | 4 +- .../docs/architecture/security/index.mdx | 6 +-- website/content/docs/concepts/job.mdx | 8 ++-- website/content/docs/concepts/plugins/cni.mdx | 2 +- .../docs/concepts/stateful-deployments.mdx | 2 +- website/content/docs/configuration/acl.mdx | 2 +- website/content/docs/configuration/audit.mdx | 3 +- .../content/docs/configuration/autopilot.mdx | 6 +-- website/content/docs/configuration/client.mdx | 3 -- .../content/docs/configuration/reporting.mdx | 3 +- .../content/docs/configuration/sentinel.mdx | 3 +- .../docs/deploy/clusters/federate-regions.mdx | 4 +- website/content/docs/govern/index.mdx | 2 +- website/content/docs/govern/namespaces.mdx | 2 +- website/content/docs/govern/sentinel.mdx | 4 +- .../content/docs/job-declare/multiregion.mdx | 25 ++++------- .../docs/job-declare/nomad-variables.mdx | 3 +- .../docs/job-specification/lifecycle.mdx | 2 +- .../docs/job-specification/multiregion.mdx | 2 +- .../content/docs/job-specification/update.mdx | 6 +-- website/content/docs/manage/autopilot.mdx | 11 ++--- .../content/docs/monitor/inspect-cluster.mdx | 2 +- .../docs/monitor/inspect-workloads.mdx | 2 +- .../docs/other-specifications/acl-policy.mdx | 2 +- .../docs/other-specifications/variables.mdx | 8 ++-- website/content/docs/quickstart.mdx | 3 -- .../docs/release-notes/nomad/v1-10-x.mdx | 2 +- .../docs/release-notes/nomad/v1_9_x.mdx | 4 +- .../docs/stateful-workloads/csi-volumes.mdx | 2 +- .../content/partials/task-driver-intro.mdx | 2 +- website/content/tools/autoscaling/agent.mdx | 9 ++-- .../agent/dynamic_application_sizing.mdx | 2 +- website/content/tools/autoscaling/cli.mdx | 7 ++- website/content/tools/autoscaling/index.mdx | 2 +- .../plugins/target/app-sizing-nomad.mdx | 10 ++--- website/content/tools/autoscaling/policy.mdx | 4 +- website/data/api-docs-nav-data.json | 25 +++++++++++ website/data/commands-nav-data.json | 5 +++ website/data/docs-nav-data.json | 45 +++++++++++++++++++ website/data/tools-nav-data.json | 25 +++++++++++ 50 files changed, 192 insertions(+), 103 deletions(-) diff --git a/website/content/api-docs/acl/index.mdx b/website/content/api-docs/acl/index.mdx index 318f42b70..86a7bea3f 100644 --- a/website/content/api-docs/acl/index.mdx +++ b/website/content/api-docs/acl/index.mdx @@ -10,6 +10,6 @@ description: |- The `/acl` endpoints provide access to the ACL subsystem which includes ACL bootstrapping, ACL Policies, ACL Roles, ACL Tokens, ACL Auth Methods, and ACL Binding Rules. For more details about ACLs, please see the [ACL -Guide](/nomad/tutorials/access-control). +Guide](/nomad/docs/secure/acl). Please choose a subsection in the navigation for more information. diff --git a/website/content/api-docs/acl/policies.mdx b/website/content/api-docs/acl/policies.mdx index 5788d6bed..97535fc33 100644 --- a/website/content/api-docs/acl/policies.mdx +++ b/website/content/api-docs/acl/policies.mdx @@ -7,7 +7,7 @@ description: The /acl/policy endpoints are used to configure and manage ACL poli # ACL Policies HTTP API The `/acl/policies` and `/acl/policy/` endpoints are used to manage ACL policies. -For more details about ACLs, please see the [ACL Guide](/nomad/tutorials/access-control). +For more details about ACLs, please see the [ACL Guide](/nomad/docs/secure/acl). ## List Policies diff --git a/website/content/api-docs/acl/tokens.mdx b/website/content/api-docs/acl/tokens.mdx index 983ec3003..768a03ec2 100644 --- a/website/content/api-docs/acl/tokens.mdx +++ b/website/content/api-docs/acl/tokens.mdx @@ -7,7 +7,7 @@ description: The /acl/token/ endpoints are used to configure and manage ACL toke # ACL Tokens HTTP API The `/acl/bootstrap`, `/acl/tokens`, and `/acl/token/` endpoints are used to manage ACL tokens. -For more details about ACLs, please see the [ACL Guide](/nomad/tutorials/access-control). +For more details about ACLs, please see the [ACL Guide](/nomad/docs/secure/acl). ## Bootstrap Token diff --git a/website/content/api-docs/events.mdx b/website/content/api-docs/events.mdx index 57e44bdff..43fe25014 100644 --- a/website/content/api-docs/events.mdx +++ b/website/content/api-docs/events.mdx @@ -85,7 +85,7 @@ by default, requiring a management token. | Node | Node | | NodeDrain | Node | | NodePool | NodePool | -| Operator | UtilizationSnapshot (Enterprise only) | +| Operator | UtilizationSnapshot | | Service | Service Registrations | ### Event Types diff --git a/website/content/api-docs/index.mdx b/website/content/api-docs/index.mdx index 7e2f53fa2..88ee01852 100644 --- a/website/content/api-docs/index.mdx +++ b/website/content/api-docs/index.mdx @@ -68,7 +68,7 @@ administration. Several endpoints in Nomad use or require ACL tokens to operate. The token are used to authenticate the request and determine if the request is allowed based on the associated authorizations. Tokens are specified per-request by using the `X-Nomad-Token` request header or with the Bearer scheme in the authorization header set to the `SecretID` of an ACL Token. -For more details about ACLs, please see the [ACL Guide](/nomad/tutorials/access-control). +For more details about ACLs, please see the [ACL Guide](/nomad/docs/secure/acl). ## Authentication diff --git a/website/content/api-docs/operator/autopilot.mdx b/website/content/api-docs/operator/autopilot.mdx index cfe2ea2a9..439355eff 100644 --- a/website/content/api-docs/operator/autopilot.mdx +++ b/website/content/api-docs/operator/autopilot.mdx @@ -106,15 +106,15 @@ The table below shows this endpoint's support for cluster. Only takes effect if all servers are running Raft protocol version 3 or higher. Must be a duration value such as `30s`. -- `EnableRedundancyZones` `(bool: false)` - (Enterprise-only) Specifies whether +- `EnableRedundancyZones` `(bool: false)` - Specifies whether to enable redundancy zones. -- `DisableUpgradeMigration` `(bool: false)` - (Enterprise-only) Disables Autopilot's +- `DisableUpgradeMigration` `(bool: false)` - Disables Autopilot's upgrade migration strategy in Nomad Enterprise of waiting until enough newer-versioned servers have been added to the cluster before promoting any of them to voters. -- `EnableCustomUpgrades` `(bool: false)` - (Enterprise-only) Specifies whether to +- `EnableCustomUpgrades` `(bool: false)` - Specifies whether to enable using custom upgrade versions when performing migrations. ## Read Health @@ -222,7 +222,7 @@ $ curl \ status of 200 will be returned. If `Healthy` is false, then a status of 429 will be returned. - This API endpoint return with more information in Nomad Enterprise. This is + This API endpoint returns with more information in Nomad Enterprise. This is not present in Nomad Community Edition. diff --git a/website/content/api-docs/search.mdx b/website/content/api-docs/search.mdx index 247ed3945..b205ee526 100644 --- a/website/content/api-docs/search.mdx +++ b/website/content/api-docs/search.mdx @@ -585,7 +585,8 @@ $ curl \ If the search Context is `all` when fuzzy searching, the object types that are identified only with UUIDs are also concurrently prefix-searched. Those types include -deployments, evals, volumes, and quotas (Enterprise). +deployments, evaluations, volumes, and resource quotas. Note that resource quotas is +an Enterprise feature. ### Sample Payload (prefix match) diff --git a/website/content/api-docs/sentinel-policies.mdx b/website/content/api-docs/sentinel-policies.mdx index efe2015f9..95a8d3780 100644 --- a/website/content/api-docs/sentinel-policies.mdx +++ b/website/content/api-docs/sentinel-policies.mdx @@ -11,7 +11,7 @@ description: >- The `/sentinel/policies` and `/sentinel/policy/` endpoints are used to manage Sentinel policies. For more details about Sentinel policies, please see the [Sentinel Policy Guide](/nomad/docs/govern/sentinel). -Sentinel endpoints are only available when ACLs are enabled. For more details about ACLs, please see the [ACL Guide](/nomad/tutorials/access-control). +Sentinel endpoints are only available when ACLs are enabled. For more details about ACLs, please see the [ACL Guide](/nomad/docs/secure/acl). diff --git a/website/content/api-docs/ui.mdx b/website/content/api-docs/ui.mdx index f8da33e0c..9a78bef50 100644 --- a/website/content/api-docs/ui.mdx +++ b/website/content/api-docs/ui.mdx @@ -495,9 +495,7 @@ This page lists all allocations for a CSI plugin. Each allocation includes the s - `type` `(string: "")` - Filters the list of allocations to only those with a matching plugin type. Value must be `controller`, `node`, or unset. -## Optimize - -~> **Enterprise Only!** This feature depends on functionality only present in Nomad Autoscaler Enterprise. +## Optimize This page lists all recommendations surfaced by [dynamic application sizing](/nomad/tools/autoscaling#dynamic-application-sizing). This page will automatically redirect to the recommendation summary route for the first recommendation sorted by relevance. diff --git a/website/content/commands/acl/index.mdx b/website/content/commands/acl/index.mdx index b313baace..6500a9a16 100644 --- a/website/content/commands/acl/index.mdx +++ b/website/content/commands/acl/index.mdx @@ -77,6 +77,6 @@ subcommands are available: [roledelete]: /nomad/commands/acl/role/delete [roleinfo]: /nomad/commands/acl/role/info [rolelist]: /nomad/commands/acl/role/list -[secure-guide]: /nomad/tutorials/access-control -[federated]: //nomad/docs/deploy/clusters/federate-regions +[secure-guide]: /nomad/docs/secure/acl +[federated]: /nomad/docs/deploy/clusters/federate-regions [`authoritative_region`]: /nomad/docs/configuration/server#authoritative_region diff --git a/website/content/commands/operator/utilization.mdx b/website/content/commands/operator/utilization.mdx index 0a0c8c1f9..796582e08 100644 --- a/website/content/commands/operator/utilization.mdx +++ b/website/content/commands/operator/utilization.mdx @@ -16,14 +16,12 @@ hours. If ACLs are enabled, this command requires a token with the `operator:write` capability. - + Refer to the [manual license utilization reporting](/nomad/docs/enterprise/license/utilization-reporting) page to learn more about reporting your Nomad Enterprise license utilization. - - ## Options - `-message` `(string; "")` - Provide context about the conditions under which diff --git a/website/content/docs/architecture/security/index.mdx b/website/content/docs/architecture/security/index.mdx index 6097b2812..14c751743 100644 --- a/website/content/docs/architecture/security/index.mdx +++ b/website/content/docs/architecture/security/index.mdx @@ -31,7 +31,7 @@ but the general mechanisms for a secure Nomad deployment revolve around: internal abuse by preventing unauthenticated access to network components within the cluster. -- **[ACLs](/nomad/tutorials/access-control)** Enables authorization for +- **[ACLs](/nomad/docs/secure/acl)** Enables authorization for authenticated connections by granting capabilities to ACL tokens. - **[Namespaces](/nomad/docs/govern/namespaces)** Access to read @@ -50,7 +50,7 @@ granularity may change depending on your team's use case where rigorous roles can be accurately defined and managed using the [Nomad backend secret engine for Vault](/vault/docs/secrets/nomad). This is described further with getting started steps using a development server -[here](/nomad/tutorials/access-control). +[here](/nomad/docs/secure/acl). It's important to note that there's no traditional concept of a user within Nomad itself. @@ -135,7 +135,7 @@ recommendations accordingly. when using `tls.verify_https_client=false`. You can use a reverse proxy or other external means to restrict access to them. -- [ACLs enabled](/nomad/tutorials/access-control) The +- [ACLs enabled](/nomad/docs/secure/acl) The access control list (ACL) system provides a capability-based control mechanism for Nomad administrators allowing for custom roles (typically within Vault) to be tied to an individual human or machine operator diff --git a/website/content/docs/concepts/job.mdx b/website/content/docs/concepts/job.mdx index d73bb8cd0..0ee602638 100644 --- a/website/content/docs/concepts/job.mdx +++ b/website/content/docs/concepts/job.mdx @@ -160,19 +160,21 @@ jobs: configuration and examples. - The [Schedulers] page explains the different types of Nomad schedulers and how those schedulers run jobs. +- The [Declare jobs][deploy-jobs] section contains guides for creating a job, + using task drivers, and deployment strategies. These tutorial resources contain examples of different kinds of jobs and how to configure them: - The [Quick Start][quickstart] tutorials cover deploying an application and its resources, which include parameterized and periodic jobs. -- The [Deploy and Manage Jobs][deploy-jobs] tutorial contains examples of different types of jobs. -- The [Create Nomad Job Specifications][job-spec-tutorial] collection explains several aspects of job creation, including parameterized jobs and deploying a Java app on Nomad. +- The [Create Nomad Job Specifications][job-spec-tutorial] collection contains + guides for migrating a Java application to Nomad. [allocations]: /nomad/docs/glossary#allocation [deployment]: /nomad/docs/glossary/#deployment -[deploy-jobs]: /nomad/tutorials/manage-jobs/ +[deploy-jobs]: /nomad/docs/job-declare [job-spec]: /nomad/docs/job-specification [job-spec-tutorial]: /nomad/tutorials/job-specifications [quickstart]: /nomad/tutorials/get-started/gs-deploy-job diff --git a/website/content/docs/concepts/plugins/cni.mdx b/website/content/docs/concepts/plugins/cni.mdx index 832bdc1ef..dfecdc1bc 100644 --- a/website/content/docs/concepts/plugins/cni.mdx +++ b/website/content/docs/concepts/plugins/cni.mdx @@ -64,4 +64,4 @@ Nomad Networking documentation](/nomad/docs/networking/cni). [cni_spec]: https://www.cni.dev/docs/spec/ [cni_spec_net_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#configuration-format [cni_spec_plugin_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#plugin-configuration-objects -[nomad_install]: /nomad/tutorials/get-started/get-started-install#linux-post-installation-steps +[nomad_install]: /nomad/docs/deploy#linux-post-installation-steps diff --git a/website/content/docs/concepts/stateful-deployments.mdx b/website/content/docs/concepts/stateful-deployments.mdx index 21d6d35bf..bff80bfd5 100644 --- a/website/content/docs/concepts/stateful-deployments.mdx +++ b/website/content/docs/concepts/stateful-deployments.mdx @@ -76,7 +76,7 @@ and volumes: - [Considerations for Stateful Workloads](/nomad/docs/architecture/storage/stateful-workloads) explores the options for persistent storage of workloads running in Nomad. - The [Nomad volume specification][volumes] defines the schema for creating and registering volumes. - The [job specification `volume` block](/nomad/docs/job-specification/volume) lets you configure a group that requires a specific volume from the cluster. -- The [Stateful Workloads](/nomad/tutorials/stateful-workloads) tutorials explore techniques to run jobs that require access to persistent storage. +- The [Stateful Workloads](/nomad/docs/stateful-workloads) guides explore techniques to run jobs that require access to persistent storage. [allocation]: /nomad/docs/glossary#allocation [delete]: /nomad/api-docs/volumes#delete-task-group-host-volume-claims diff --git a/website/content/docs/configuration/acl.mdx b/website/content/docs/configuration/acl.mdx index 6161d1d20..146a44c57 100644 --- a/website/content/docs/configuration/acl.mdx +++ b/website/content/docs/configuration/acl.mdx @@ -66,6 +66,6 @@ acl { TTL value for an ACL token when setting expiration. This is used by the Nomad servers to validate ACL tokens and ACL authentication methods. -[secure-guide]: /nomad/tutorials/access-control +[secure-guide]: /nomad/docs/secure/acl [authoritative-region]: /nomad/docs/configuration/server#authoritative_region [Configure for multiple regions]: /nomad/docs/secure/acl/bootstrap#configure-for-multiple-regions diff --git a/website/content/docs/configuration/audit.mdx b/website/content/docs/configuration/audit.mdx index 4ed021b96..418d5ee8f 100644 --- a/website/content/docs/configuration/audit.mdx +++ b/website/content/docs/configuration/audit.mdx @@ -8,13 +8,14 @@ description: >- # `audit` Block in Agent Configuration - This page provides reference information for configuring audit logging behavior in the `audit` block of a Nomad agent configuration. Enable audit logs, define a sink to stream audit logs to, and change filter rules to exclude events from the audit log. + + ```hcl audit { enabled = true diff --git a/website/content/docs/configuration/autopilot.mdx b/website/content/docs/configuration/autopilot.mdx index 284e67c9e..18a90d3e1 100644 --- a/website/content/docs/configuration/autopilot.mdx +++ b/website/content/docs/configuration/autopilot.mdx @@ -46,17 +46,17 @@ autopilot { cluster. Only takes effect if all servers are running Raft protocol version 3 or higher. Must be a duration value such as `30s`. -- `enable_redundancy_zones` `(bool: false)` - Controls whether +- `enable_redundancy_zones` `(bool: false)` - Controls whether Autopilot separates servers into zones for redundancy, in conjunction with the [redundancy_zone](/nomad/docs/configuration/server#redundancy_zone) parameter. Only one server in each zone can be a voting member at one time. -- `disable_upgrade_migration` `(bool: false)` - Disables Autopilot's +- `disable_upgrade_migration` `(bool: false)` - Disables Autopilot's upgrade migration strategy in Nomad Enterprise of waiting until enough newer-versioned servers have been added to the cluster before promoting any of them to voters. -- `enable_custom_upgrades` `(bool: false)` - Specifies whether to +- `enable_custom_upgrades` `(bool: false)` - Specifies whether to enable using custom upgrade versions when performing migrations, in conjunction with the [upgrade_version](/nomad/docs/configuration/server#upgrade_version) parameter. diff --git a/website/content/docs/configuration/client.mdx b/website/content/docs/configuration/client.mdx index 388af74aa..6d2d4038f 100644 --- a/website/content/docs/configuration/client.mdx +++ b/website/content/docs/configuration/client.mdx @@ -283,9 +283,6 @@ Nomad never attempts to embed the `alloc_dir` in the chroot as doing so would ca ### `options` Parameters -~> Note: In Nomad 0.9 client configuration options for drivers were deprecated. -Refer to the [plugin block][plugin-block] documentation for more information. - The following is not an exhaustive list of options for only the Nomad client. To find the options supported by each individual Nomad driver, refer to the [drivers documentation](/nomad/docs/job-declare/task-driver). diff --git a/website/content/docs/configuration/reporting.mdx b/website/content/docs/configuration/reporting.mdx index 0a2a2bf60..b3d84fb87 100644 --- a/website/content/docs/configuration/reporting.mdx +++ b/website/content/docs/configuration/reporting.mdx @@ -8,11 +8,12 @@ description: >- # `reporting` Block in Agent Configuration - This page provides reference information for enabling automated license utilization reporting in the `reporting` block of a Nomad agent configuration. + + Configuration applies to agents running with [server mode enabled][server_mode_enabled]. diff --git a/website/content/docs/configuration/sentinel.mdx b/website/content/docs/configuration/sentinel.mdx index 8789e0867..a41b6ef14 100644 --- a/website/content/docs/configuration/sentinel.mdx +++ b/website/content/docs/configuration/sentinel.mdx @@ -8,13 +8,14 @@ description: >- # `sentinel` Block in Agent Configuration - This page provides reference information for configuring the Sentinel policy engine in the `sentinel` block of a Nomad agent configuration. Configure the path to the plugin that Nomad uses to import Sentinel policies and specify arguments to pass to that plugin on startup. + + ```hcl sentinel { import "custom-plugin" { diff --git a/website/content/docs/deploy/clusters/federate-regions.mdx b/website/content/docs/deploy/clusters/federate-regions.mdx index 2c23a003d..2ae028306 100644 --- a/website/content/docs/deploy/clusters/federate-regions.mdx +++ b/website/content/docs/deploy/clusters/federate-regions.mdx @@ -137,12 +137,12 @@ Error querying jobs: Unexpected response code: 500 (No path to region) - [Deployment Topology across Multiple Regions][multi-region] -[multi-region]: /nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul#deployment-topology-across-multiple-regions +[multi-region]: /nomad/docs/deploy/production/reference-architecture#deployment-topology-across-multiple-regions [multi-region-pic]: /img/clusters/nomad-multi-region.png [nomad-server-members]: /nomad/commands/server/members [nomad-status]: /nomad/commands/status [nomad-tf]: https://github.com/hashicorp/nomad/tree/master/terraform#provision-a-nomad-cluster-in-the-cloud [ports-used]: /nomad/docs/deploy/production/requirements#ports-used -[reference-arch]: /nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul +[reference-arch]: /nomad/docs/deploy/production/reference-architecture [region-config]: /nomad/docs/configuration#region [server-join]: /nomad/commands/server/join diff --git a/website/content/docs/govern/index.mdx b/website/content/docs/govern/index.mdx index 78f9c1b32..076c490f6 100644 --- a/website/content/docs/govern/index.mdx +++ b/website/content/docs/govern/index.mdx @@ -109,5 +109,5 @@ to be local to each region for low latency. [img_sentinel_overview]: /img/govern/sentinel.jpg -[`sentinel-override` capability]: /nomad/tutorials/access-control#sentinel-override +[`sentinel-override` capability]: /nomad/docs/secure/acl#sentinel-override [`server` stanza]: /nomad/docs/configuration/server diff --git a/website/content/docs/govern/namespaces.mdx b/website/content/docs/govern/namespaces.mdx index a800e2145..b9df280cf 100644 --- a/website/content/docs/govern/namespaces.mdx +++ b/website/content/docs/govern/namespaces.mdx @@ -136,7 +136,7 @@ For specific details about working with namespaces, consult the [namespace commands] and [HTTP API] documentation. -[acls]: /nomad/tutorials/access-control +[acls]: /nomad/docs/secure/acl [http api]: /nomad/api-docs/namespaces [img_ui_ns_dropdown]: /img/govern/nomad-ui-namespace-dropdown.png [namespace commands]: /nomad/commands/namespace diff --git a/website/content/docs/govern/sentinel.mdx b/website/content/docs/govern/sentinel.mdx index 72fdf6620..909ec7216 100644 --- a/website/content/docs/govern/sentinel.mdx +++ b/website/content/docs/govern/sentinel.mdx @@ -197,9 +197,9 @@ For specific details about working with Sentinel, consult the [`nomad sentinel` and [HTTP API] documentation. [`nomad sentinel` sub-commands]: /nomad/commands/sentinel -[`sentinel-override` capability]: /nomad/tutorials/access-control#sentinel-override +[`sentinel-override` capability]: /nomad/docs/secure/acl#sentinel-override [`server` stanza]: /nomad/docs/configuration/server -[acls]: /nomad/tutorials/access-control +[acls]: /nomad/docs/secure/acl [http api]: /nomad/api-docs/sentinel-policies [json job specification]: /nomad/api-docs/json-jobs [nomad enterprise]: https://www.hashicorp.com/products/nomad/ diff --git a/website/content/docs/job-declare/multiregion.mdx b/website/content/docs/job-declare/multiregion.mdx index 84fd52a30..11dda700a 100644 --- a/website/content/docs/job-declare/multiregion.mdx +++ b/website/content/docs/job-declare/multiregion.mdx @@ -8,12 +8,13 @@ description: |- # Configure multi-region deployments -Federated Nomad clusters enable users to submit jobs targeting any region -from any server even if that server resides in a different region. As of Nomad 0.12 -Enterprise, you can also submit jobs that are deployed to multiple -regions. This tutorial demonstrates multi-region deployments, including +Federated Nomad clusters enable you to submit jobs targeting any region +from any server even if that server resides in a different region. You may submit jobs that are deployed to multiple +regions. This guide demonstrates multi-region deployments, including configurable rollout and rollback strategies. + + You can create a multi-region deployment job by adding a [`multiregion`] stanza to the job as shown below. @@ -38,16 +39,6 @@ multiregion { } ``` - - -The functionality described here is available only in [Nomad -Enterprise](https://www.hashicorp.com/products/nomad/pricing/) with the -Multi-Cluster & Efficiency module. To explore Nomad Enterprise features, you can -sign up for a free 30-day trial from -[here](https://www.hashicorp.com/products/nomad/trial). - - - ## Prerequisites To perform the tasks described in this guide, you need to have two Nomad @@ -446,9 +437,9 @@ west f08122e5 successful [nomad-tf]: https://github.com/hashicorp/nomad/tree/master/terraform [server-join]: /nomad/commands/server/join [ports-used]: /nomad/docs/deploy/production/requirements#ports-used -[reference-arch]: /nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul +[reference-arch]: /nomad/docs/deploy/production/reference-architecture [nomad-server-members]: /nomad/commands/server/members -[acls-track]: /nomad/tutorials/access-control +[acls-track]: /nomad/docs/secure/acl [updates-track]: /nomad/docs/job-declare/strategy/ [update-auto-revert]: /nomad/docs/job-specification/update#auto_revert ["task states"]: /nomad/docs/job-specification/update#health_check @@ -456,4 +447,4 @@ west f08122e5 successful [alloc-health-api]: /nomad/api-docs/deployments#set-allocation-health-in-deployment [`update` stanza]: /nomad/docs/job-specification/update [`nomad deployment unblock`]: /nomad/commands/deployment/unblock -[multi-region-topology]: /nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul#multi-region +[multi-region-topology]:/nomad/docs/deploy/production/reference-architecture#multi-region diff --git a/website/content/docs/job-declare/nomad-variables.mdx b/website/content/docs/job-declare/nomad-variables.mdx index a13c74a8c..63a23e3a4 100644 --- a/website/content/docs/job-declare/nomad-variables.mdx +++ b/website/content/docs/job-declare/nomad-variables.mdx @@ -468,8 +468,7 @@ to specify Nomad's behavior when a value changes. ## Next steps Because Nomad Variables use functions in the template block to emit data to -Nomad jobs, consider learning more about templates in Nomad with the [Templates -collection](/nomad/tutorials/templates). +Nomad jobs, consider learning more about templates in Nomad with [Nomad Pack](/nomad/tools/nomad-pack). [Nomad Variables]: /nomad/docs/concepts/variables [Nomad Variables Access Control]: /nomad/tutorials/variables/variables-acls diff --git a/website/content/docs/job-specification/lifecycle.mdx b/website/content/docs/job-specification/lifecycle.mdx index 81e9b30df..9828f9c1a 100644 --- a/website/content/docs/job-specification/lifecycle.mdx +++ b/website/content/docs/job-specification/lifecycle.mdx @@ -49,7 +49,7 @@ Learn more about [Nomad's task dependencies][learn-taskdeps]. lifecycle task is long-lived (`sidecar = true`) and terminates, it will be restarted as long as the allocation is running. -[learn-taskdeps]: /nomad/tutorials/task-deps +[learn-taskdeps]: /nomad/docs/job-declare/task-dependencies [shutdown_delay]: /nomad/docs/job-specification/group#shutdown_delay [leader]: /nomad/docs/job-specification/task#leader diff --git a/website/content/docs/job-specification/multiregion.mdx b/website/content/docs/job-specification/multiregion.mdx index 7c8255b23..9f81b9001 100644 --- a/website/content/docs/job-specification/multiregion.mdx +++ b/website/content/docs/job-specification/multiregion.mdx @@ -14,7 +14,7 @@ The `multiregion` block specifies that a job will be deployed to multiple one specified by the `region` field or the `-region` command line flag to `nomad job run`. - + Federated Nomad clusters are members of the same gossip cluster but not the same raft cluster; they don't share their data stores. Each region in a diff --git a/website/content/docs/job-specification/update.mdx b/website/content/docs/job-specification/update.mdx index 4e2b7fbf4..2f9ecc423 100644 --- a/website/content/docs/job-specification/update.mdx +++ b/website/content/docs/job-specification/update.mdx @@ -269,7 +269,7 @@ group "two" { } ``` -[canary]: /nomad/docs/job-declare/strategy/blue-green-canary 'Nomad Canary Deployments' +[canary]: /nomad/docs/job-declare/strategy/blue-green-canary [checks]: /nomad/docs/job-specification/service#check -[rolling]: /nomad/docs/job-declare/strategy/rolling 'Nomad Rolling Upgrades' -[strategies]: /nomad/tutorials/job-updates 'Nomad Update Strategies' +[rolling]: /nomad/docs/job-declare/strategy/rolling +[strategies]: /nomad/docs/job-declare/strategy diff --git a/website/content/docs/manage/autopilot.mdx b/website/content/docs/manage/autopilot.mdx index a9e781b41..e53461691 100644 --- a/website/content/docs/manage/autopilot.mdx +++ b/website/content/docs/manage/autopilot.mdx @@ -154,12 +154,7 @@ must be healthy and stable for a certain amount of time before being promoted to a full, voting member. This can be configured via the `ServerStabilizationTime` setting. ---- - -~> The following Autopilot features are available only in [Nomad Enterprise] -version 0.8.0 and later. - -## Server read and scheduling scaling +## Server read and scheduling scaling With the [`non_voting_server`] option, a server can be explicitly marked as a non-voter and will never be promoted to a voting member. This can be useful when @@ -168,7 +163,7 @@ have data replicated to it, but it will not be part of the quorum that the leader must wait for before committing log entries. Non voting servers can also act as scheduling workers to increase scheduling throughput in large clusters. -## Redundancy zones +## Redundancy zones Prior to Autopilot, it was difficult to deploy servers in a way that took advantage of isolated failure domains such as AWS Availability Zones; users @@ -196,7 +191,7 @@ Nomad will then use these values to partition the servers by redundancy zone, and will aim to keep one voting server per zone. Extra servers in each zone will stay as non-voters on standby to be promoted if the active voter leaves or dies. -## Upgrade migrations +## Upgrade migrations Autopilot in Nomad Enterprise supports upgrade migrations by default. To disable this functionality, set `DisableUpgradeMigration` to true. diff --git a/website/content/docs/monitor/inspect-cluster.mdx b/website/content/docs/monitor/inspect-cluster.mdx index 61d307c51..3b45c0815 100644 --- a/website/content/docs/monitor/inspect-cluster.mdx +++ b/website/content/docs/monitor/inspect-cluster.mdx @@ -168,5 +168,5 @@ using the Nomad UI. [img-servers-list]: /img/monitor/guide-ui-img-servers-list.png [metadata]: /nomad/docs/configuration/client#meta [performed from the cli]: /nomad/docs/manage/migrate-workloads -[securing the web ui with acls]: /nomad/tutorials/access-control +[securing the web ui with acls]: /nomad/docs/secure/acl [typically three or five]: /nomad/docs/architecture/cluster/consensus#deployment-table diff --git a/website/content/docs/monitor/inspect-workloads.mdx b/website/content/docs/monitor/inspect-workloads.mdx index 81408120a..7f1388e31 100644 --- a/website/content/docs/monitor/inspect-workloads.mdx +++ b/website/content/docs/monitor/inspect-workloads.mdx @@ -218,4 +218,4 @@ the Nomad UI, learn how to inspect the state of your cluster using the Nomad UI. [in the job definition to reschedule]: /nomad/docs/job-specification/reschedule [local restart attempts]: /nomad/docs/job-specification/restart [periodic force]: /nomad/commands/job/periodic-force -[securing the web ui with acls]: /nomad/tutorials/access-control +[securing the web ui with acls]: /nomad/docs/secure/acl diff --git a/website/content/docs/other-specifications/acl-policy.mdx b/website/content/docs/other-specifications/acl-policy.mdx index 0c51850b0..4625cc345 100644 --- a/website/content/docs/other-specifications/acl-policy.mdx +++ b/website/content/docs/other-specifications/acl-policy.mdx @@ -479,7 +479,7 @@ plugin { } ``` -[Secure Nomad with Access Control]: /nomad/tutorials/access-control +[Secure Nomad with Access Control]: /nomad/docs/secure/acl [hcl]: https://github.com/hashicorp/hcl [hcl_syntax_spec]: https://github.com/hashicorp/hcl/blob/main/hclsyntax/spec.md [api_jobs]: /nomad/api-docs/jobs diff --git a/website/content/docs/other-specifications/variables.mdx b/website/content/docs/other-specifications/variables.mdx index aa0b3cab3..7d3e5853d 100644 --- a/website/content/docs/other-specifications/variables.mdx +++ b/website/content/docs/other-specifications/variables.mdx @@ -64,12 +64,14 @@ details on `path` and `items` name restrictions. ## Resources -Visit the [Nomad Variables tutorial][tutorial] to learn how to create variables, -configure access control for variables, and access variables from within job tasks. +- [Create and update Nomad variables](/nomad/tutorials/variables/variables-create) +- [Configure access control for Nomad + variables](/nomad/tutorials/variables/variables-acls) +- [Use Nomad variables in tasks](/nomad/docs/job-declare/nomad-variables) [nv]: /nomad/docs/concepts/variables [var-init]: /nomad/commands/var/init [var-put]: /nomad/commands/var/put [jobspecs]: /nomad/docs/job-specification [var-restrict]: /nomad/commands/var/put#restrictions -[tutorial]: /nomad/tutorials/variables + diff --git a/website/content/docs/quickstart.mdx b/website/content/docs/quickstart.mdx index 8aac8a197..bccbef1d3 100644 --- a/website/content/docs/quickstart.mdx +++ b/website/content/docs/quickstart.mdx @@ -24,8 +24,6 @@ We recommend these tutorials, which provision a Nomad cluster for you. - [Cluster Setup][]: Provision a Nomad cluster on AWS, Azure, or GCP. Enable Consul and access control lists (ACLs). This tutorial series has an associated code repository so you can review the Terraform provisioning scripts. -- [Enable gossip encryption for Nomad][interactive]: Launch an interactive lab - to use Nomad in your browser. ## Cloud installation @@ -73,7 +71,6 @@ Use one of the following methods to run a local Nomad sandbox environment: [installing-binary]: /nomad/docs/deploy [Get Started]: /nomad/tutorials/get-started [Cluster Setup]: /nomad/tutorials/cluster-setup -[interactive]: /nomad/tutorials/interactive/security-gossip-encryption [Provision a Nomad cluster on AWS]: https://github.com/hashicorp/nomad/blob/main/terraform/aws/README.md [Provision a Nomad cluster on Azure]: https://github.com/hashicorp/nomad/blob/main/terraform/azure/README.md diff --git a/website/content/docs/release-notes/nomad/v1-10-x.mdx b/website/content/docs/release-notes/nomad/v1-10-x.mdx index 5fdb70806..587a4aa77 100644 --- a/website/content/docs/release-notes/nomad/v1-10-x.mdx +++ b/website/content/docs/release-notes/nomad/v1-10-x.mdx @@ -217,4 +217,4 @@ These links take you to the changelogs on the GitHub website. [pkce]: https://oauth.net/2/pkce/ [oidc-concepts]: /nomad/docs/secure/authentication/oidc#client-assertions [oidc-trouble]: /nomad/docs/secure/authentication/oidc#oidc-configuration-troubleshooting -[oidc-tutorial]: /nomad/tutorials/single-sign-on/sso-oidc-keycloak +[oidc-tutorial]: /nomad/docs/secure/authentication/sso-pkce-jwt diff --git a/website/content/docs/release-notes/nomad/v1_9_x.mdx b/website/content/docs/release-notes/nomad/v1_9_x.mdx index fa39a1663..5dd37782b 100644 --- a/website/content/docs/release-notes/nomad/v1_9_x.mdx +++ b/website/content/docs/release-notes/nomad/v1_9_x.mdx @@ -17,11 +17,11 @@ make full use of your GPU investment. The device driver automatically detects MIGs. Refer to the [NVIDIA driver docs](/nomad/plugins/devices/nvidia) for details. -- **Quotas for device resources (Enterprise)**: This release extends quotas to +- **Quotas for device resources **: This release extends quotas to allow limiting [device resources](/nomad/docs/v1.9.x/job-specification/device). Refer to [Resource quotas](/nomad/docs/v1.9.x/other-specifications/quota) for configuration details. -- **NUMA awareness for device resources (Enterprise)**: Nomad is able to +- **NUMA awareness for device resources **: Nomad is able to correlate CPU cores with memory nodes and assign tasks to run on specific CPU cores so as to minimize any cross-memory node access patterns. With Nomad 1.9, we are expanding this functionality to also correlate diff --git a/website/content/docs/stateful-workloads/csi-volumes.mdx b/website/content/docs/stateful-workloads/csi-volumes.mdx index 5f07c27b3..20896686d 100644 --- a/website/content/docs/stateful-workloads/csi-volumes.mdx +++ b/website/content/docs/stateful-workloads/csi-volumes.mdx @@ -658,4 +658,4 @@ job’s lifecycle. [k8s-drivers]: https://kubernetes-csi.github.io/docs/drivers.html [nomad-tf]: https://github.com/hashicorp/nomad/tree/master/terraform#provision-a-nomad-cluster-in-the-cloud [password-security]: https://dev.mysql.com/doc/refman/8.0/en/password-security.html -[reference-arch]: /nomad/tutorials/enterprise/production-reference-architecture-vm-with-consul#high-availability +[reference-arch]: /nomad/docs/deploy/production/reference-architecture#high-availability diff --git a/website/content/partials/task-driver-intro.mdx b/website/content/partials/task-driver-intro.mdx index bec6fb920..a79e1ad08 100644 --- a/website/content/partials/task-driver-intro.mdx +++ b/website/content/partials/task-driver-intro.mdx @@ -9,6 +9,6 @@ client CPU, memory, and storage between tasks. Resource isolation effectiveness depends upon individual task driver implementations and underlying client operating systems. Task drivers include various security-related controls but do not use the Nomad client-to-task interface as a security boundary. Refer to the -[access control guide](/nomad/tutorials/access-control) for more information on +[access control guide](/nomad/docs/secure/acl) for more information on how to protect Nomad cluster operations. diff --git a/website/content/tools/autoscaling/agent.mdx b/website/content/tools/autoscaling/agent.mdx index 16bbbdfa0..2ca63b765 100644 --- a/website/content/tools/autoscaling/agent.mdx +++ b/website/content/tools/autoscaling/agent.mdx @@ -51,8 +51,9 @@ namespace "default" { } ``` -If running Nomad Autoscaler Enterprise, the following ACL policy addition is -needed to ensure it can read the Nomad Enterprise license. + + +If you are running Nomad Autoscaler Enterprise, add this ACL policy to ensure Nomad Autoscaler Enterprise is able to read the Nomad Enterprise license. ```hcl operator { @@ -60,6 +61,8 @@ operator { } ``` + + Other APM and target plugins may require additional ACLs; see the plugin documentation for more information. ## Load Order and Merging @@ -120,7 +123,7 @@ following actions. [hcl_v2]: https://github.com/hashicorp/hcl/tree/hcl2 [nomad_namespaces]: /nomad/docs/govern/namespaces -[nomad_acls]: /nomad/tutorials/access-control +[nomad_acls]: /nomad/docs/secure/acl [autoscaler_agent_nomad]: /nomad/tools/autoscaling/agent/nomad [autoscaler_cli_config]: /nomad/tools/autoscaling/cli#config [autoscaler_cli_policy_dir]: /nomad/tools/autoscaling/cli#policy-dir diff --git a/website/content/tools/autoscaling/agent/dynamic_application_sizing.mdx b/website/content/tools/autoscaling/agent/dynamic_application_sizing.mdx index 3acc904e9..04f9f34b0 100644 --- a/website/content/tools/autoscaling/agent/dynamic_application_sizing.mdx +++ b/website/content/tools/autoscaling/agent/dynamic_application_sizing.mdx @@ -12,7 +12,7 @@ description: >- This functionality only exists in Nomad Autoscaler Enterprise. This is not - present in the open source version of Nomad Autoscaler. + present in the Community Edition of Nomad Autoscaler. ~> Note that currently Prometheus is the only APM available for Dynamic Application Sizing diff --git a/website/content/tools/autoscaling/cli.mdx b/website/content/tools/autoscaling/cli.mdx index b5988c78a..7dfc60b40 100644 --- a/website/content/tools/autoscaling/cli.mdx +++ b/website/content/tools/autoscaling/cli.mdx @@ -94,8 +94,11 @@ passed in via CLI arguments. The `agent` command accepts the following arguments - `-policy-eval-workers=`: The number of workers to initialize for each queue, formatted as `:,:`. Nomad Autoscaler supports - `cluster` and `horizontal` queues. Nomad Autoscaler Enterprise supports additional - `vertical_mem` and `vertical_cpu` queues. + the following queues: + - `cluster` + - `horizontal` + - `vertical_mem` + - `vertical_cpu` - `-policy-source-disable-file`: Disable the sourcing of policies from disk. diff --git a/website/content/tools/autoscaling/index.mdx b/website/content/tools/autoscaling/index.mdx index ebf49fbf3..d2abb89c9 100644 --- a/website/content/tools/autoscaling/index.mdx +++ b/website/content/tools/autoscaling/index.mdx @@ -46,7 +46,7 @@ cluster. This functionality only exists in Nomad Autoscaler Enterprise. This is not - present in the open source version of Nomad Autoscaler. + present in the Community Edition of Nomad Autoscaler. Dynamic Application Sizing enables organizations to optimize the resource diff --git a/website/content/tools/autoscaling/plugins/target/app-sizing-nomad.mdx b/website/content/tools/autoscaling/plugins/target/app-sizing-nomad.mdx index d293ba125..4cdbf110b 100644 --- a/website/content/tools/autoscaling/plugins/target/app-sizing-nomad.mdx +++ b/website/content/tools/autoscaling/plugins/target/app-sizing-nomad.mdx @@ -6,16 +6,16 @@ description: The "app-sizing-nomad" target plugin scales a task resource. # Dynamic Application Sizing Nomad Task Target +The `app-sizing-nomad` target plugin reports on the current task resource value +as well as submits recommendations to Nomad via the [recommendations API endpoint][nomad_recommendations_api] +based on the result of the Dynamic Application Sizing strategy calculations. + This functionality only exists in Nomad Autoscaler Enterprise. This is not present in the open source version of Nomad Autoscaler. -~> Note that currently Prometheus is the only APM available for Dynamic Application Sizing - -The `app-sizing-nomad` target plugin reports on the current task resource value -as well as submits recommendations to Nomad via the [recommendations API endpoint][nomad_recommendations_api] -based on the result of the Dynamic Application Sizing strategy calculations. +Note that currently Prometheus is the only APM available for dynamic application sizing. ## Agent Configuration Options diff --git a/website/content/tools/autoscaling/policy.mdx b/website/content/tools/autoscaling/policy.mdx index 77fab0bae..714d0bb87 100644 --- a/website/content/tools/autoscaling/policy.mdx +++ b/website/content/tools/autoscaling/policy.mdx @@ -183,11 +183,11 @@ scaling "azure_cluster_policy" { } ``` -## Task (DAS) `policy` Options +## Task `policy` options for dynamic application sizing This functionality only exists in Nomad Autoscaler Enterprise. This is not - present in the open source version of Nomad Autoscaler. + present in the Community Edition of Nomad Autoscaler. The following options are available when using the Nomad Autoscaler Enterprise diff --git a/website/data/api-docs-nav-data.json b/website/data/api-docs-nav-data.json index 689540408..5b78b65f4 100644 --- a/website/data/api-docs-nav-data.json +++ b/website/data/api-docs-nav-data.json @@ -119,6 +119,11 @@ }, { "title": "License", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "operator/license" }, { @@ -139,6 +144,11 @@ }, { "title": "Utilization Reporting", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "operator/utilization" } ] @@ -149,10 +159,20 @@ }, { "title": "Quotas", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "quotas" }, { "title": "Recommendations", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "recommendations" }, { @@ -169,6 +189,11 @@ }, { "title": "Sentinel Policies", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "sentinel-policies" }, { diff --git a/website/data/commands-nav-data.json b/website/data/commands-nav-data.json index 3d3d210bc..172f0503d 100644 --- a/website/data/commands-nav-data.json +++ b/website/data/commands-nav-data.json @@ -682,6 +682,11 @@ }, { "title": "utilization", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "operator/utilization" } ] diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index affa461e0..70fc0d285 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -720,6 +720,11 @@ }, { "title": "Configure multi-region deployments", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "job-declare/multiregion" }, { @@ -920,6 +925,11 @@ }, { "title": "audit", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "configuration/audit" }, { @@ -965,6 +975,11 @@ }, { "title": "reporting", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "configuration/reporting" }, { @@ -977,6 +992,11 @@ }, { "title": "sentinel", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "configuration/sentinel" }, { @@ -1110,6 +1130,11 @@ }, { "title": "multiregion", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "job-specification/multiregion" }, { @@ -1118,6 +1143,11 @@ }, { "title": "numa", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "job-specification/numa" }, { @@ -1150,6 +1180,11 @@ }, { "title": "schedule", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "job-specification/schedule" }, { @@ -1227,6 +1262,11 @@ }, { "title": "Resource quota", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "other-specifications/quota" }, { @@ -1281,6 +1321,11 @@ }, { "title": "Sentinel policy", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "reference/sentinel-policy" }, { diff --git a/website/data/tools-nav-data.json b/website/data/tools-nav-data.json index 6dc6d6569..fcce32c78 100644 --- a/website/data/tools-nav-data.json +++ b/website/data/tools-nav-data.json @@ -74,6 +74,11 @@ }, { "title": "dynamic_application_sizing", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "autoscaling/agent/dynamic_application_sizing" }, { @@ -163,14 +168,29 @@ }, { "title": "Dynamic Application Sizing Average", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "autoscaling/plugins/strategy/app-sizing-avg" }, { "title": "Dynamic Application Sizing Max", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "autoscaling/plugins/strategy/app-sizing-max" }, { "title": "Dynamic Application Sizing Percentile", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "autoscaling/plugins/strategy/app-sizing-percentile" }, { @@ -208,6 +228,11 @@ }, { "title": "Dynamic Application Sizing", + "badge": { + "text": "ENT", + "type": "outlined", + "color": "neutral" + }, "path": "autoscaling/plugins/target/app-sizing-nomad" }, {