When a volume is updated, we merge the new definition to the old. But the
volume's context comes from the plugin and is likely not present in the user's
volume specification. Which means that if the user re-submits the volume
specification to make an adjustment to the volume, we wipe out the context field
which might be required for subsequent operations by the CSI plugin. This was
discovered to be a problem with the Terraform provider and fixed there, but it's
also a problem for users of the `volume create` and `volume register` commands.
Update the merge so that we only overwrite the value of the context if it's been
explictly set by the user. We still need to support user-driven updates to
context for the `volume register` workflow.
Ref: https://github.com/hashicorp/terraform-provider-nomad/pull/503
Fixes: https://github.com/democratic-csi/democratic-csi/issues/438
The `volume init` command creates example volume specifications. But one of the
values for `capability.access_mode` is not a valid value. Correct the example to
match the validation logic.
The default node configuration in the client should always set an empty
HostVolumes map. Otherwise callers can panic, e.g.,:
goroutine 179 [running]:
github.com/hashicorp/nomad/client/hostvolumemanager.UpdateVolumeMap({0x36042b0, 0xc000c62a80}, 0x0, {0xc000a802a0, 0xd}, 0xc000691940)
github.com/hashicorp/nomad/client/hostvolumemanager/volume_fingerprint.go:43 +0x1b2
github.com/hashicorp/nomad/client.(*Client).batchFirstFingerprints.func1({0xc000a802a0, 0xd}, 0xc000691940)
github.com/hashicorp/nomad/client/node_updater.go:54 +0xd7
github.com/hashicorp/nomad/client.(*batchNodeUpdates).batchHostVolumeUpdates(0xc000912608?, 0xc0009f2f88)
github.com/hashicorp/nomad/client/node_updater.go:417 +0x152
github.com/hashicorp/nomad/client.(*Client).batchFirstFingerprints(0xc000c2d188)
github.com/hashicorp/nomad/client/node_updater.go:53 +0x1c5
created by github.com/hashicorp/nomad/client.NewClient in goroutine 1
github.com/hashicorp/nomad/client/client.go:557 +0x2069
is a panic of the HVM when restarting a client that doesn't have any static
host volumes, but does have a dynamic host volume.
* func: make windows arch dependant
* func: unify keys and make them cluster grouped
* Update README.md
* Update e2e/terraform/provision-infra/provision-nomad/variables.tf
Co-authored-by: Tim Gross <tgross@hashicorp.com>
* Update .gitignore
* style: add an output with the custer identifier
---------
Co-authored-by: Tim Gross <tgross@hashicorp.com>
Some dynamic host volumes are claimed by allocations with the capability we
borrowed from CSI called `single-node-single-writer`, which says only one
allocation can use the volume, and it can use it in read/write mode. We enforce
this in the scheduler, but if evaluations for different jobs were to be
processed concurrently by the scheduler, it's possible to get plans that would
fail to enforce this requirement. Add a check in the plan applier to ensure that
non-terminal allocations have exclusive access when requested.
let only one of any create/register/delete run at a time per volume ID
* plugins can assume that Nomad will not run concurrent operations for the same volume
* we avoid interleaving client RPCs with raft writes
The volume_mounts test is flaky due to slow starts from the exec-driver and some
incorrect wait code. Refactor the volume_mounts test to use the `e2e/v3` package
helpers, and use these to give it enough time to start the exec tasks.
The Nomad agent used a log filter to ensure logs were written at
the expected level. Since the use of hclog this is not required,
as hclog acts as the gate keeper and filter for logging. All log
writers accept messages from hclog which has already done the
filtering.
The agent syslog write handler was unable to handle JSON log lines
correctly, meaning all syslog entries when using JSON log format
showed as NOTICE level.
This change adds a new handler to the Nomad agent which can parse
JSON log lines and correctly understand the expected log level
entry.
The change also removes the use of a filter from the default log
format handler. This is not needed as the logs are fed into the
syslog handler via hclog, which is responsible for level
filtering.
We can reduce the amount of volume specification configuration many users will
need by setting a default capability on a dynamic host volume if none is
set. The default capability will allow using the volume in read/write mode on
its node, with no further restrictions except those that might be set in the
jobspec.
Enterprise governance checks happen after dynamic host volumes are placed, so if
node pool governance is active and you don't set a node pool or node ID for a
volume, it's possible to get a placement that fails node pool governance even
though there might be other nodes in the cluster that would be valid placements.
Move the node pool governance for host volumes into the placement path, so that
we're checking a specific node pool when node pool or node ID are set, but
otherwise filtering out candidate nodes by node pool.
This changset is the CE version of ENT/2200.
Ref: https://hashicorp.atlassian.net/browse/NET-11549
Ref: https://github.com/hashicorp/nomad-enterprise/pull/2200
Update dynamic host volume validation and update logic to allow for changes to
the node pool and plugin ID. If the client's node pool changes we'll sync up the
correct node pool for the volumes already placed on that client. We'll also
allow the plugin ID to be changed to allow for new versions of plugins
supporting the same volume over time.
The nightly runs for E2E have been failing the recently added dynamic host
volumes tests for a number of reasons:
* Adding timing logs to the tests shows that it can take over 5s (the original
test timeout) for the client fingerprint to show up on the client. This seems
like a lot but seems to be host-dependent because it's much faster locally.
Extend the timeout and leave in the timing logs so that we can keep an eye on
this problem in the future.
* The register test doesn't wait for the dispatched job to complete, and the
dispatched job was actually broken when TLS was in use because we weren't using
the Task API socket. Fix the jobspec for the dispatched job and add waiting
for the dispatched allocation to be marked complete before checking for the
volume on the server.
I've also change both the mounter jobs to batch workloads, so that we don't have
to wait 10s for the deployment to complete.
In #24694 we did a major refactoring of the E2E Terraform configuration. After
deploying a cluster this morning, I noticed a few moved/removed files were not
reflected in the .gitignore files. This changeset updates the .gitignore to have
no unstaged files after applying.
When using the register workflow, `capacity_max` is ignored so is likely
unset. If the volume is then updated later, the check we had for valid updates
assumes that the value was previously. Only perform this check if the value is
set.
We changed the list of access modes available for dynamic host volumes in #24705
but neglected to change them in the API package. Update the API package to
match.
Ref: https://github.com/hashicorp/nomad/pull/24705
* func: move infra provisionining to a module and remove providers
* func: update paths
* func: update more paths
* func: update path inside bootstrap scrip
* style: remove debug prints on bootstrap scripts
* Delete e2e/terraform/csi/input/volume-efs.hcl
* fix: update keys path to use module path instead pf root
* fix: add missing headers
* fix: update keys directory inside provision-nomad
* style; format hcl files
* Update compute.tf
* Update e2e/terraform/main.tf
Co-authored-by: Tim Gross <tgross@hashicorp.com>
* Update e2e/terraform/provision-infra/compute.tf
Co-authored-by: Tim Gross <tgross@hashicorp.com>
* fix: update more paths
* fix: fmt hcl files
* func: final paths revision for running e2e locally
* fix: make path of certs relative to module for the bootstrap
* func: final paths revision for running e2e locally
* Update network.tf
* fix: fix typo and add success message
* fix: remove the test name from token to avoid long names and use name for vol to avoid colisions
* func: unify the uploads folder
* func: make the uploads file one per cluster
* func: Add outputs with all data necessary to connect to the cluster
* fix: make nomad token a sensitive output
* Update bootstrap-nomad.sh
---------
Co-authored-by: Tim Gross <tgross@hashicorp.com>
In anticipation of having quotas for dynamic host volumes, we want the user
experience of the storage limits to feel integrated with the other resource
limits. This is currently prevented by reusing the `Resources` type instead of
having a specific type for `QuotaResources`.
Update the quota limit/usage types to use a `QuotaResources` that includes a new
storage resources quota block. The wire format for the two types are compatible
such that we can migrate the existing variables limit in the FSM.
Also fixes improper parallelism in the quota init test where we change working
directory to avoid file write conflicts but this breaks when multiple tests are
executed in the same process.
Ref: https://github.com/hashicorp/nomad-enterprise/pull/2096
The output of `GetDynamicHostVolumes` is a slice but that slice is constructed
from iterating over a map and isn't sorted. Sort the output in the test to
eliminate a test flake.
Some comment cleanups as we're wrapping up dynamic host volumes work:
* We're not going to implement mount_options for host volumes, as the dynamic
host volumes don't have the equivalent of the stage/publish phase that CSI
volumes do. Users who want that sort of thing will pass them as `parameter`
field during volume create/register.
* The scheduler feasibility check prevents a dynamic host volume being claimed
by a job in the wrong namespace, but the comment incorrectly identifies that
code path as only being about the race between fingerprint and delete. Update
the comment to make the intent clear so that we don't accidentally remove this
behavior in the future.
* Update who-uses-nomad.mdx
Our new contract with Roblox states that we can't mention anywhere on our sites that they use us.
* Update who-uses-nomad.mdx
Edited the sentence above the companies list to more accurately reflect them.
Also added Target to the list with a link to their case study.
Initial end-to-end tests for dynamic host volumes. This includes tests for two
workflows:
* One where a dynamic host volume is created by a plugin and then mounted by a job.
* Another where a dynamic host volume is created out-of-band and registered by a
job, then mounted by another job.
This changeset also moves the existing `volumes` E2E test package to the
better-named `volume_mounts`.
Ref: https://hashicorp.atlassian.net/browse/NET-11551
When we register a volume without a plugin, we need to send a client RPC so that
the node fingerprint can be updated. The registered volume also needs to be
written to client state so that we can restore the fingerprint after a restart.
Co-authored-by: Daniel Bennett <dbennett@hashicorp.com>
