* Move commands from docs to its own root-level directory
* temporarily use modified dev-portal branch with nomad ia changes
* explicitly clone nomad ia exp branch
* retrigger build, fixed dev-portal broken build
* architecture, concepts and get started individual pages
* fix get started section destinations
* reference section
* update repo comment in website-build.sh to show branch
* docs nav file update capitalization
* update capitalization to force deploy
* remove nomad-vs-kubernetes dir; move content to what is nomad pg
* job section
* Nomad operations category, deploy section
* operations category, govern section
* operations - manage
* operations/scale; concepts scheduling fix
* networking
* monitor
* secure section
* remote auth-methods folder and move up pages to sso; linkcheck
* Fix install2deploy redirects
* fix architecture redirects
* Job section: Add missing section index pages
* Add section index pages so breadcrumbs build correctly
* concepts/index fix front matter indentation
* move task driver plugin config to new deploy section
* Finish adding full URL to tutorials links in nav
* change SSO to Authentication in nav and file system
* Docs NomadIA: Move tutorials into NomadIA branch (#26132)
* Move governance and policy from tutorials to docs
* Move tutorials content to job-declare section
* run jobs section
* stateful workloads
* advanced job scheduling
* deploy section
* manage section
* monitor section
* secure/acl and secure/authorization
* fix example that contains an unseal key in real format
* remove images from sso-vault
* secure/traffic
* secure/workload-identities
* vault-acl change unseal key and root token in command output sample
* remove lines from sample output
* fix front matter
* move nomad pack tutorials to tools
* search/replace /nomad/tutorials links
* update acl overview with content from deleted architecture/acl
* fix spelling mistake
* linkcheck - fix broken links
* fix link to Nomad variables tutorial
* fix link to Prometheus tutorial
* move who uses Nomad to use cases page; move spec/config shortcuts
add dividers
* Move Consul out of Integrations; move namespaces to govern
* move integrations/vault to secure/vault; delete integrations
* move ref arch to docs; rename Deploy Nomad back to Install Nomad
* address feedback
* linkcheck fixes
* Fixed raw_exec redirect
* add info from /nomad/tutorials/manage-jobs/jobs
* update page content with newer tutorial
* link updates for architecture sub-folders
* Add redirects for removed section index pages. Fix links.
* fix broken links from linkcheck
* Revert to use dev-portal main branch instead of nomadIA branch
* build workaround: add intro-nav-data.json with single entry
* fix content-check error
* add intro directory to get around Vercel build error
* workound for emtpry directory
* remove mdx from /intro/ to fix content-check and git snafu
* Add intro index.mdx so Vercel build should work
---------
Co-authored-by: Tu Nguyen <im2nguyen@gmail.com>
* fix: initalize the topology of teh processors to avoid nil pointers
* func: initialize topology to avoid nil pointers
* fix: update the new public method for NodeProcessorResources
The RPC handler for deleting dynamic host volumes has a check that any
allocations associated with a volume are client-terminal before deleting the
volume. But the state store delete that happens after we send client RPCs to the
plugin checks that the allocs are non-terminal on both server and client.
This can improperly allow deleting a volume from a client but then not being
able to delete it from the state store because of a time-of-check / time-of-use
bug. If the allocation fails/completes on the client before the server marks its
desired status as terminal, or if the allocation is marked server-terminal
during the client RPC, we can get a volume that passes the first check but not
the second check that happens in the state store and cannot be deleted.
Update the state store delete method to require that any allocation for a volume
is client terminal in order to delete the volume, not just server terminal.
Fixes: https://github.com/hashicorp/nomad/issues/26140
Ref: https://hashicorp.atlassian.net/browse/NMD-883
Nomad agents emit metrics for Consul service and check operations, but these
were not documented. Update the metrics reference table to include these
metrics. Note that the metrics are prefixed `nomad.client` but are present on
all agents, because the server registers itself in Consul as well.
The output of the reconciler stage of scheduling is only visible via debug-level
logs, typically accessible only to the cluster admin. We can give job authors
better ability to understand what's happening to their jobs if we expose this
information to them in the `eval status` command.
Add the reconciler's desired updates to the evaluation struct so it can be
exposed in the API. This increases the size of evals by roughly 15% in the state
store, or a bit more when there are preemptions (but we expect this will be a
small minority of evals).
Ref: https://hashicorp.atlassian.net/browse/NMD-818
Fixes: https://github.com/hashicorp/nomad/issues/15564
The mkdir plugin creates the directory and then chowns it. In the
event the chown command fails, we should attempt to remove the
directory. Without this, we leave directories on the client in
partial failure situations.
The meta client looks for both an environment variable and a CLI
flag when generating a client. The CLI UUID checker needs to do
this also, so we account for users using both env vars and CLI
flag tokens.
When draining nodes allocs are checked for a healthy state and
marked to be drained, with the value in the max parallel setting
determining how many allocs will be migrated. Depending on the
circumstances, however, the max parallel setting may not be
properly respected.
Given a job with max parallel set to one, a group count greater
than one, and allocs on multiple nodes: Draining a single node
will result in one alloc being marked to drain. If another
node is immediately drained the alloc running on the first
node will be seen as "healthy" and another alloc will be
marked to be drained resulting in two allocs being marked
for migration at the same time. This can lead to issues with
service availablility.
To prevent this allocs can only be marked as healthy when the
alloc has not been marked for migration. This prevents migrating
allocs being seen as healthy which results in the max parallel
setting being properly respected.
When performing a graceful shutdown the client drain configuration
is checked for a deadline which is appended to the timeout. When
running as a server the client will not be set. Attempting to get
the drain deadline will result in a panic. This checks for the
client being available prior to fetching the deadline value.
The `killTasks` function will kill all the alloc runners
task runners. If the task of a task runner has already
completed, the killing of the task runner can cause
confusion due to the task event showing that the task
was signaled even though it is already complete.
To prevent this, a check is done when creating the
task event to determine if the task has completed. If
it has no task event is created and when the task
runner is killed, no extra task event is added.
Both the cluster reconciler and node reconciler emit a debug-level log line with
their results, but these are unstructured multi-line logs that are annoying for
operators to parse. Change these to emit structured key-value pairs like we do
everywhere else.
Ref: https://hashicorp.atlassian.net/browse/NMD-818
Ref: https://go.hashi.co/rfc/nmd-212
When debugging an evaluation, you almost always want to know about all the
related evaluations and what allocations were placed by that evaluation (and
where), not just failed placements. We can enrich the command by adding the
`related` query parameter to the API, and having the command query for the
evaluations allocations automatically. Emit this data as a pair of new tables
and expose fields like quota limits, and previous/next/blocked eval without the
`-verbose` flag.
Update the docs to include the full output and remove references to long-removed
behavior of the `-json` flag.
Ref: https://hashicorp.atlassian.net/browse/NMD-818
Ref: https://go.hashi.co/rfc/nmd-212
As part of ongoing work to make the scheduler more legible and more robustly
tested, we're implementing property testing of at least the reconciler. This
changeset provides some infrastructure we'll need for generating the test cases
using `pgregory.net/rapid`, without building out any of the property assertions
yet (that'll be in upcoming PRs over the next couple weeks).
The alloc reconciler generator produces a job, a previous version of the job, a
set of tainted nodes, and a set of existing allocations. The node reconciler
generator produces a job, a set of nodes, and allocations on those
nodes. Reconnecting allocs are not yet well-covered by these generators, and
with ~40 dimensions covered so far we may need to pull those out to their own
tests in order to get good coverage.
Note the scenarios only randomize fields of interest; fields like the job name
that don't impact the reconciler would use up available shrink cycles on failed
tests without actually reducing the scope of the scenario.
Ref: https://hashicorp.atlassian.net/browse/NMD-814
Ref: https://github.com/flyingmutant/rapid
Restoring scaling policies during the start of a stopped job did not account for
jobs that didn't have any scaling policies, and led to a panic when users tried
to restart such jobs.
When a test starts an agent and the client is enabled, we can
wait until this reaches the ready state within the set up method.
This mimics what we already do with leadership and the root
keyring and should reduce flakey tests where it assume the client
is ready as soon as the set up function returns, which is not
guaranteed.
The change exposed a couple of TLS reload tests which were not
using the test agent correctly. They were setting up a client even
though it would never be able to join the cluster due to TLS
configuration issues. These have been fixed.
No matter the passed region identifier, the CLI was always adding
"<role>.global.nomad" to the certificate DNS names. This is not
what we expect and has been removed.
While here, the long deprecated cluster-region flag has been
removed. This removal only impacts CLI functionality, so is safe
to do.
The Nomad server uses an authenticator backend for RPC handling
which includes TLS verification. This verification setting is
configured based on the servers TLS configuration object and is
built when a new server is constructed.
The bug occurs when a servers TLS configuration is reloaded which
can change the desired TLS verification handling. In this case,
the authenticator is not updated, meaning the RPC mTLS verification
is not modified, even if the configuration indicates it should.
This change adds a new function on the authenticator to allow
updating its TLS verification rule. This new function is called
when a servers TLS configuration is reloaded.
In hashicorp/nomad-enterprise#2592 we introduced a
divergence in how Nomad CE and ENT build their binaries. Nomad CE used a more
sophisticated approach, setting uid, gid and home environment variables in the
docker run command. Despite mine (and others) best efforts, we were not able
to do the same in the ENT repo, which relies on special git settings that allow
it to pull dependencies from private repositories, and left a different docker
run command there, that just inherited GHA runner user and copied the resulting
tarball instead of moving it. #26090 then attempted to remedy #25910 resulting
from docker run command ignoring ${{ env.GO_TAGS }} if run with custom
--env, but the resulting backport broke ent builds.
This PR restores ENT behavior of building Nomad builds with GHA runner user,
thus inheriting runner's environment on ent.
For reasons of backwards compatibility, Nomad uses an older branch of
HCL1 (`v1.0.1-nomad`) and HCL2 (`v2.20.2-nomad-1`) and backports a limited set
of changes to those branches.
But the Vault API also has their own HCL1 branch, currently tagged as
`v1.0.1-vault-7`. Normally this isn't a problem because Nomad pins to our own
branch and we don't call any of the Vault API package's HCL code anyways. But in
Vault's branch some functions were changed that break our build unless we
backport them.
We've backported enough of Vault's changes to make our HCL1 branch build, and
now have tags on the HCL repo so that we can pin to specific tags instead of
random commits.
Fixes: https://hashicorp.atlassian.net/browse/NMD-850
Fixes: https://github.com/hashicorp/nomad/pull/26006
Ref: https://github.com/hashicorp/hcl/pull/760
This changeset separates reconciler fields into their own sub-struct to make
testing easier and the code more explicit about what fields relate to which
state.
