In #20007 we fixed a bug where the DNS configuration set by CNI plugins was not
threaded through to the task configuration. This resulted in a regression where
a DNS override set by `dockerd` was not respected for `bridge` mode
networking. Our existing handling of CNI DNS incorrectly assumed that the DNS
field would be empty, when in fact it contains a single empty DNS struct.
Handle this case correctly by checking whether the DNS struct we get back from
CNI has any nameservers, and ignore it if it doesn't. Expand test coverage of
this case.
Fixes: https://github.com/hashicorp/nomad/issues/20174
Replaces #18812
Upgraded with:
```
find . -name '*.go' -exec sed -i s/"github.com\/hashicorp\/go-msgpack\/codec"/"github.com\/hashicorp\/go-msgpack\/v2\/codec/" '{}' ';'
find . -name '*.go' -exec sed -i s/"github.com\/hashicorp\/net-rpc-msgpackrpc"/"github.com\/hashicorp\/net-rpc-msgpackrpc\/v2/" '{}' ';'
go get
go get -v -u github.com/hashicorp/raft-boltdb/v2
go get -v github.com/hashicorp/serf@5d32001edfaa18d1c010af65db707cdb38141e80
```
see https://github.com/hashicorp/go-msgpack/releases/tag/v2.1.0
for details
When loading the client configuration, the user-specified `client.template`
block was not properly merged with the default values. As a result, if the user
set any `client.template` field, all the other field defaulted to their zero
values instead of the documented defaults.
This changeset:
* Adds the missing `Merge` method for the client template config and ensures
it's called.
* Makes a single source of truth for the default template configuration,
instead of two different constructors.
* Extends the tests to cover the merge of a partial block better.
Fixes: https://github.com/hashicorp/nomad/issues/20164
* Move task directory destroy logic from alloc_dir to task_dir
* Update errors to wrap error cause
* Use constants for file permissions
* Make multierror handling consistent.
* Make helpers for directory creation
* Move mount dir unlink to task_dir Unlink method
* Make constant for file mode 710
Co-authored-by: Tim Gross <tgross@hashicorp.com>
Co-authored-by: Michael Schurter <mschurter@hashicorp.com>
Metrics tools that "pull" metrics, such as Prometheus, have a configurable
interval for how frequently they scrape metrics. This should be greater or equal
to the Nomad `telemetry.collection_interval` to avoid re-scraping metrics that
cannot have been updated in that interval.
Fixes: https://github.com/hashicorp/nomad/issues/20055
Multi-node Nomad clusters under test must use the RPC calls to bootstrap
the ACL subsystem. The original implementation of the testcluster tried
naively supersizing the single node behavior in TestRaftRemovePeer,
which mutates the single node's state directly. In the case of a multi-
node cluster, the RPC calls are necessary to ensure that the data is
replicated via Raft to all of the cluster members.
Add information about autopilot health to the `/operator/autopilot/health` API
in Nomad Enterprise.
I've pulled the CE changes required for this feature out of @lindleywhite's PR
in the Enterprise repo. A separate PR will include a new `operator autopilot
health` command that can present this information at the command line.
Ref: https://github.com/hashicorp/nomad-enterprise/pull/1394
Co-authored-by: Lindley <lindley@hashicorp.com>
As of #18754 which shipped in Nomad 1.7, we no longer need to nil-check the
object returned by `ResolveACL` if there's no error return, because in the case
where ACLs are disabled we return a special "ACLs disabled" ACL object. Checking
nil is not a bug but should be discouraged because it opens us up to future bugs
that would bypass ACLs.
While working on an unrelated feature @lindleywhite discovered that we missed
removing the nil check from several endpoints with our semgrep linter. This
changeset fixes that.
Co-Author: Lindley <lindley@hashicorp.com>
This commit introduces the new options for reconciling a reconnecting allocation and its replacement:
Best score (Current implementation)
Keep original
Keep replacement
Keep the one that has run the longest time
It is achieved by adding a new dependency to the allocReconciler that calls the corresponding function depending on the task group's disconnect strategy. For more detailed information, refer to the new stanza for disconnected clientes RFC.
It resolves 15144
While investigating a report around possible consul-template shutdown issues,
which didn't bear fruit, I found that some of the logic around template runner
shutdown is unintuitive.
* Add some doc strings to the places where someone might think we should be
obviously stopping the runner or returning early.
* Mark context argument for `Poststart`, `Stop`, and `Update` hooks as unused.
No functional code changes.
* exec2: add client support for unveil filesystem isolation mode
This PR adds support for a new filesystem isolation mode, "Unveil". The
mode introduces a "alloc_mounts" directory where tasks have user-owned
directory structure which are bind mounts into the real alloc directory
structure. This enables a task driver to use landlock (and maybe the
real unveil on openbsd one day) to isolate a task to the task owned
directory structure, providing sandboxing.
* actually create alloc-mounts-dir directory
* fix doc strings about alloc mount dir paths
* exec: add a client.users configuration block
For now just add min/max dynamic user values; soon we can also absorb
the "user.denylist" and "user.checked_drivers" options from the
deprecated client.options map.
* give the no-op pool implementation a better name
* use explicit error types to make referencing them cleaner in tests
* use import alias to not shadow package name
Although Nomad is not vulnerable to CVE-2024-24786 because it's configured to
discard unknown messages during unmarshaling, we should upgrade so that
third-party vulnerability scanners don't detect the vulnerable version and
complain.
Also update go1.22.1 changelog entry to include CVEs
Nomad has always placed an extremely high priority on backward
compatibility. We have always aimed to support N-2 major releases and
usually gone above and beyond that.
The new https://www.hashicorp.com/long-term-support policy also mentions
that N-2 is what we have always supported, so it's probably time for our
docs to reflect that reality.
* build: upgrade to go1.22
* add cl
* build: use codecgen from go-msgpack v1.1.5+base32 and stringer 0.18.0
for compatability with go1.22
* ci: update golangci-lint to 1.56.2
* build: update hclogvet for go1.22
* build: bump to go1.22.1
* exec2: implement dynamic workload users taskrunner hook
This PR impelements a TR hook for allocating dynamic workload users from
a pool managed by the Nomad client. This adds a new task driver Capability,
DynamicWorkloadUsers - which a task driver must indicate in order to make
use of this feature.
The client config plumbing is coming in a followup PR - in the RFC we
realized having a client.users block would be nice to have, with some
additional unrelated options being moved from the deprecated client.options
config.
* learn to spell
When creating a job ACL, you must supply a job ID if you supply a
namespace. If you try to give a namespace without a job ID, the error
states "JobACL.JobID without Namespace" instead.
* exec2: implement a dynamic users pool
This PR adds an implementation of a Pool from which dynamic users can
be allocated on behalf of tasks making use of an upcoming feature of
Nomad client (dynamic users).
A task hook and client plumbing, etc. will be in follow up PRs.
* no need for randomness assertion
stopAlloc() checks if an allocation represents a system job like this:
```
if *alloc.Job.Type == api.JobTypeSystem {
...
}
```
This caused the cli to crash:
```
==> 2024-02-29T08:45:53+01:00: Restarting 2 allocations
2024-02-29T08:45:54+01:00: Rescheduling allocation "6a9da11a" for group "redacted-group"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x10686affc]
goroutine 36 [running]:
github.com/hashicorp/nomad/command.(*JobRestartCommand).stopAlloc(0x14000b11040, {0x14000996dc0?, 0x0?})
github.com/hashicorp/nomad/command/job_restart.go:968 +0x25c
github.com/hashicorp/nomad/command.(*JobRestartCommand).handleAlloc(0x14000b11040, {0x14000996dc0?, 0x0?})
github.com/hashicorp/nomad/command/job_restart.go:868 +0x34
github.com/hashicorp/nomad/command.(*JobRestartCommand).Run.(*JobRestartCommand).Run.func1.func2()
github.com/hashicorp/nomad/command/job_restart.go:392 +0x28
github.com/hashicorp/go-multierror.(*Group).Go.func1()
github.com/hashicorp/go-multierror@v1.1.1/group.go:23 +0x60
created by github.com/hashicorp/go-multierror.(*Group).Go in goroutine 1
github.com/hashicorp/go-multierror@v1.1.1/group.go:20 +0x84
```
Attaching a debugger revealed that `alloc.Job` was set, but
`alloc.Job.Type` was nil. After guarding the `.Type` check with a
`alloc.Job.Type != nil`, it still crashed. This time, `alloc.Job` was
nil.
I was scrambling to get the job running again, so I didn't have the
opportunity to find out why those values were nil, but this change
ensures the CLI does not crash in these situations.
Fixes#20048
* tests: swap testify for test in plugins/csi/client_test.go
* tests: swap testify for test in testutil/
* tests: swap testify for test in host_test.go
* tests: swap testify for test in plugin_test.go
* tests: swap testify for test in utils_test.go
* tests: swap testify for test in scheduler/
* tests: swap testify for test in parse_test.go
* tests: swap testify for test in attribute_test.go
* tests: swap testify for test in plugins/drivers/
* tests: swap testify for test in command/
* tests: fixup some test usages
* go: run go mod tidy
* windows: cpuset test only on linux
CNI plugins may set DNS configuration, but this isn't threaded through to the
task configuration so that we can write it to the `/etc/resolv.conf` file as
needed. Add the `AllocNetworkStatus` to the alloc hook resources so they're
accessible from the taskrunner. Any DNS entries provided by the user will
override these values.
Fixes: https://github.com/hashicorp/nomad/issues/11102
