* exec: add a client.users configuration block
For now just add min/max dynamic user values; soon we can also absorb
the "user.denylist" and "user.checked_drivers" options from the
deprecated client.options map.
* give the no-op pool implementation a better name
* use explicit error types to make referencing them cleaner in tests
* use import alias to not shadow package name
Although Nomad is not vulnerable to CVE-2024-24786 because it's configured to
discard unknown messages during unmarshaling, we should upgrade so that
third-party vulnerability scanners don't detect the vulnerable version and
complain.
Also update go1.22.1 changelog entry to include CVEs
Nomad has always placed an extremely high priority on backward
compatibility. We have always aimed to support N-2 major releases and
usually gone above and beyond that.
The new https://www.hashicorp.com/long-term-support policy also mentions
that N-2 is what we have always supported, so it's probably time for our
docs to reflect that reality.
* build: upgrade to go1.22
* add cl
* build: use codecgen from go-msgpack v1.1.5+base32 and stringer 0.18.0
for compatability with go1.22
* ci: update golangci-lint to 1.56.2
* build: update hclogvet for go1.22
* build: bump to go1.22.1
* exec2: implement dynamic workload users taskrunner hook
This PR impelements a TR hook for allocating dynamic workload users from
a pool managed by the Nomad client. This adds a new task driver Capability,
DynamicWorkloadUsers - which a task driver must indicate in order to make
use of this feature.
The client config plumbing is coming in a followup PR - in the RFC we
realized having a client.users block would be nice to have, with some
additional unrelated options being moved from the deprecated client.options
config.
* learn to spell
When creating a job ACL, you must supply a job ID if you supply a
namespace. If you try to give a namespace without a job ID, the error
states "JobACL.JobID without Namespace" instead.
* exec2: implement a dynamic users pool
This PR adds an implementation of a Pool from which dynamic users can
be allocated on behalf of tasks making use of an upcoming feature of
Nomad client (dynamic users).
A task hook and client plumbing, etc. will be in follow up PRs.
* no need for randomness assertion
stopAlloc() checks if an allocation represents a system job like this:
```
if *alloc.Job.Type == api.JobTypeSystem {
...
}
```
This caused the cli to crash:
```
==> 2024-02-29T08:45:53+01:00: Restarting 2 allocations
2024-02-29T08:45:54+01:00: Rescheduling allocation "6a9da11a" for group "redacted-group"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x10686affc]
goroutine 36 [running]:
github.com/hashicorp/nomad/command.(*JobRestartCommand).stopAlloc(0x14000b11040, {0x14000996dc0?, 0x0?})
github.com/hashicorp/nomad/command/job_restart.go:968 +0x25c
github.com/hashicorp/nomad/command.(*JobRestartCommand).handleAlloc(0x14000b11040, {0x14000996dc0?, 0x0?})
github.com/hashicorp/nomad/command/job_restart.go:868 +0x34
github.com/hashicorp/nomad/command.(*JobRestartCommand).Run.(*JobRestartCommand).Run.func1.func2()
github.com/hashicorp/nomad/command/job_restart.go:392 +0x28
github.com/hashicorp/go-multierror.(*Group).Go.func1()
github.com/hashicorp/go-multierror@v1.1.1/group.go:23 +0x60
created by github.com/hashicorp/go-multierror.(*Group).Go in goroutine 1
github.com/hashicorp/go-multierror@v1.1.1/group.go:20 +0x84
```
Attaching a debugger revealed that `alloc.Job` was set, but
`alloc.Job.Type` was nil. After guarding the `.Type` check with a
`alloc.Job.Type != nil`, it still crashed. This time, `alloc.Job` was
nil.
I was scrambling to get the job running again, so I didn't have the
opportunity to find out why those values were nil, but this change
ensures the CLI does not crash in these situations.
Fixes#20048
* tests: swap testify for test in plugins/csi/client_test.go
* tests: swap testify for test in testutil/
* tests: swap testify for test in host_test.go
* tests: swap testify for test in plugin_test.go
* tests: swap testify for test in utils_test.go
* tests: swap testify for test in scheduler/
* tests: swap testify for test in parse_test.go
* tests: swap testify for test in attribute_test.go
* tests: swap testify for test in plugins/drivers/
* tests: swap testify for test in command/
* tests: fixup some test usages
* go: run go mod tidy
* windows: cpuset test only on linux
CNI plugins may set DNS configuration, but this isn't threaded through to the
task configuration so that we can write it to the `/etc/resolv.conf` file as
needed. Add the `AllocNetworkStatus` to the alloc hook resources so they're
accessible from the taskrunner. Any DNS entries provided by the user will
override these values.
Fixes: https://github.com/hashicorp/nomad/issues/11102
This PR is the first on two that will implement the new Disconnect block. In this PR the new block is introduced to be backwards compatible with the fields it will replace. For more information refer to this RFC and this ticket.
* Warning and better error handling for invalid-named variables and job templates
* Warning and better error handling for invalid-named variables and job templates
* Tests for variable pathname warnings
* Only show the bad-name warning if the variable is being created and path is editable
While working on #10628 I discovered that the `expose` block was missing an
implementation of Diff, which means it doesn't show up correctly in `job plan`
output.
Also, fix field comparison in `ServiceCheck.Equal`.
This is a bug in the method but it doesn't look like it impacts production code.
In order to provide a DNS address and port to Connect tasks configured for
transparent proxy, we need to fingerprint the Consul DNS address and port. The
client will pass this address/port to the iptables configuration provided to the
`consul-cni` plugin.
Ref: https://github.com/hashicorp/nomad/issues/10628
* Revert "vault: always renew tokens using the renewal loop (#18998)"
This reverts commit 7054fe1a8c.
* test: add case for concurrent Vault token renewal
This PR changes the example of the client config option "fingerprint.denylist"
to include all the cloud environment fingerprinters. Each one contains a
2 second HTTP timeout to a metadata endpoint that does not exist if you are not
in that particular cloud. When run in serial on startup, this results in
an 8 second wait where nothing useful is happening.
Closes#16727
In #19172 we added a check on websocket errors to see if they were one of
several benign "close" messages. This change inadvertently assumed that other
messages used for close would not implement `HTTPCodedError`. When errors like
the following are received:
> msgpack decode error [pos 0]: io: read/write on closed pipe"
they are sent from the inner loop as though they were a "real" error, but the
channel is already being closed with a "close" message.
This allowed many more attempts to pass thru a previously-undiscovered race
condition in the two goroutines that stream RPC responses to the websocket. When
the input stream returns an error for any reason (for example, the command we're
executing has exited), it will unblock the "outer" goroutine and cause a write
to the websocket. If we're concurrently writing the "close error" discussed
above, this results in a panic from the websocket library.
This changeset includes two fixes:
* Catch "closed pipe" error correctly so that we're not sending unnecessary
error messages.
* Move all writes to the websocket into the same response streaming
goroutine. The main handler goroutine will block on a results channel, and the
response streaming goroutine will send on that channel with the final error when
it's done so it can be reported to the user.
The state store's `UpsertPlanResults` method canonicalizes allocations in order
to upgrade them to a new version. But the method does not copy the allocation
before doing so, which can potentially corrupt the state store. This hasn't been
implicated in any known user-facing bugs, but was detected when running Nomad
under a build with Go toolchains data race detection enabled.
When jobs are deregistered with the `purge` flag they are immediately
deleted from the state store instead of just updated to be marked as
stopped.
Without tracking job deletions the event stream would not receive a
`JobDeregistered` event when `purge` was set.
When an allocation can't be placed because of a port collision the
resulting blocked eval is expected to have a metric reporting the port
that caused the conflict, but this metrics was not being emitted when
preemption was enabled.
The value for the executor cgroup CPU weight must be within the limits
imposed by the Linux kernel.
Nomad used the task `resource.cpu`, an unbounded value, directly as the
cgroup CPU weight, causing it to potentially go outside the imposed
values.
This commit clamps the CPU shares values to be within the limits
allowed.
Co-authored-by: Tim Gross <tgross@hashicorp.com>
The Nomad client expects certain cgroups paths to exist in order to
manage tasks. These paths are created when the agent first starts, but
if process fails the agent would just log the error and proceed with its
initialization, despite not being able to run tasks.
This commit surfaces the errors back to the client initialization so the
process can stop early and make clear to operators that something went
wrong.
On Windows, Nomad uses `syscall.NewLazyDLL` and `syscall.LoadDLL` functions to
load a few system DLL files, which does not prevent DLL hijacking
attacks. Hypothetically a local attacker on the client host that can place an
abusive library in a specific location could use this to escalate privileges to
the Nomad process. Although this attack does not fall within the Nomad security
model, it doesn't hurt to follow good practices here.
We can remove two of these DLL loads by using wrapper functions provided by the
stdlib in `x/sys/windows`
Co-authored-by: dduzgun-security <deniz.duzgun@hashicorp.com>
