In #24694 we did a major refactoring of the E2E Terraform configuration. After
deploying a cluster this morning, I noticed a few moved/removed files were not
reflected in the .gitignore files. This changeset updates the .gitignore to have
no unstaged files after applying.
When using the register workflow, `capacity_max` is ignored so is likely
unset. If the volume is then updated later, the check we had for valid updates
assumes that the value was previously. Only perform this check if the value is
set.
We changed the list of access modes available for dynamic host volumes in #24705
but neglected to change them in the API package. Update the API package to
match.
Ref: https://github.com/hashicorp/nomad/pull/24705
* func: move infra provisionining to a module and remove providers
* func: update paths
* func: update more paths
* func: update path inside bootstrap scrip
* style: remove debug prints on bootstrap scripts
* Delete e2e/terraform/csi/input/volume-efs.hcl
* fix: update keys path to use module path instead pf root
* fix: add missing headers
* fix: update keys directory inside provision-nomad
* style; format hcl files
* Update compute.tf
* Update e2e/terraform/main.tf
Co-authored-by: Tim Gross <tgross@hashicorp.com>
* Update e2e/terraform/provision-infra/compute.tf
Co-authored-by: Tim Gross <tgross@hashicorp.com>
* fix: update more paths
* fix: fmt hcl files
* func: final paths revision for running e2e locally
* fix: make path of certs relative to module for the bootstrap
* func: final paths revision for running e2e locally
* Update network.tf
* fix: fix typo and add success message
* fix: remove the test name from token to avoid long names and use name for vol to avoid colisions
* func: unify the uploads folder
* func: make the uploads file one per cluster
* func: Add outputs with all data necessary to connect to the cluster
* fix: make nomad token a sensitive output
* Update bootstrap-nomad.sh
---------
Co-authored-by: Tim Gross <tgross@hashicorp.com>
In anticipation of having quotas for dynamic host volumes, we want the user
experience of the storage limits to feel integrated with the other resource
limits. This is currently prevented by reusing the `Resources` type instead of
having a specific type for `QuotaResources`.
Update the quota limit/usage types to use a `QuotaResources` that includes a new
storage resources quota block. The wire format for the two types are compatible
such that we can migrate the existing variables limit in the FSM.
Also fixes improper parallelism in the quota init test where we change working
directory to avoid file write conflicts but this breaks when multiple tests are
executed in the same process.
Ref: https://github.com/hashicorp/nomad-enterprise/pull/2096
The output of `GetDynamicHostVolumes` is a slice but that slice is constructed
from iterating over a map and isn't sorted. Sort the output in the test to
eliminate a test flake.
Some comment cleanups as we're wrapping up dynamic host volumes work:
* We're not going to implement mount_options for host volumes, as the dynamic
host volumes don't have the equivalent of the stage/publish phase that CSI
volumes do. Users who want that sort of thing will pass them as `parameter`
field during volume create/register.
* The scheduler feasibility check prevents a dynamic host volume being claimed
by a job in the wrong namespace, but the comment incorrectly identifies that
code path as only being about the race between fingerprint and delete. Update
the comment to make the intent clear so that we don't accidentally remove this
behavior in the future.
* Update who-uses-nomad.mdx
Our new contract with Roblox states that we can't mention anywhere on our sites that they use us.
* Update who-uses-nomad.mdx
Edited the sentence above the companies list to more accurately reflect them.
Also added Target to the list with a link to their case study.
Initial end-to-end tests for dynamic host volumes. This includes tests for two
workflows:
* One where a dynamic host volume is created by a plugin and then mounted by a job.
* Another where a dynamic host volume is created out-of-band and registered by a
job, then mounted by another job.
This changeset also moves the existing `volumes` E2E test package to the
better-named `volume_mounts`.
Ref: https://hashicorp.atlassian.net/browse/NET-11551
When we register a volume without a plugin, we need to send a client RPC so that
the node fingerprint can be updated. The registered volume also needs to be
written to client state so that we can restore the fingerprint after a restart.
Co-authored-by: Daniel Bennett <dbennett@hashicorp.com>
tl;dr - runtime code is fine but tests should match reality
The Nomad Client Agent is the only consumer of the
`Node.Derive{SI,Vault}Token` RPCs, therefore tests of the RPCs should
match Nomad Client behavior.
- DeriveVaultToken code: a9ee66a6ef/client/client.go (L2904-L2917)
- DeriveSIToken code: a9ee66a6ef/client/client.go (L2988-L2997)
Both of those client code paths include the Node SecretID in both the
request's SecretID field as well as the embedded
`QueryOptions.AuthToken` field.
This patch updates server tests to match that behavior. The tests pass
either way.
When the Nomad client restarts and restores allocations, the network namespace
for an allocation may exist but no longer be correctly configured. For example,
if the host is rebooted and the task was a Docker task using a pause container,
the network namespace may be recreated by the docker daemon.
When we restore an allocation, use the CNI "check" command to verify that any
existing network namespace matches the expected configuration. This requires CNI
plugins of at least version 1.2.0 to avoid a bug in older plugin versions that
would cause the check to fail.
If the check fails, destroy the network namespace and try to recreate it from
scratch once. If that fails in the second pass, fail the restore so that the
allocation can be recreated (rather than silently having networking fail).
This should fix the gap left #24650 for Docker task drivers and any other
drivers with the `MustInitiateNetwork` capability.
Fixes: https://github.com/hashicorp/nomad/issues/24292
Ref: https://github.com/hashicorp/nomad/pull/24650
Adds an additional check in the Keyring.Delete RPC to make sure we're not
trying to delete a key that's been used to encrypt a variable. It also adds a
-force flag for the CLI/API to sidestep that check.
The recent change to collection via a "one-shot" Docker API call
did not update the stream boolean argument. This results in the
PreCPUStats values being zero and therefore breaking the CPU
calculations which rely on this data. The base fix is to update
the passed boolean parameter to match the desired non-streaming
behaviour. The non-streaming API call correctly returns the
PreCPUStats data which can be seen in the added unit test.
The most recent change also modified the behaviour of the
collectStats go routine, so that any error encountered results in
the routine exiting. In the event this was a transient error, the
container will continue to run, however, no stats will be collected
until the task is stopped and replaced. This PR reverts the
behaviour, so that an error encountered during a stats collection
run results in the error being logged but the collection process
continuing with a backoff used.
a node can have only one volume with a given name.
the scheduler prevents duplicates, but can only
do so after the server knows about the volume.
this prevents multiple concurrent creates being
called faster than the fingerprint/heartbeat interval.
users may still modify an existing volume only
if they set the `id` in the volume spec and
re-issue `nomad volume create`
if a *static* vol is added to config with a name
already being used by a dynamic volume, the
dynamic takes precedence, but log a warning.
Adds new topics to the event stream for CSI volumes and CSI plugins. We'll emit
event when either is created or deleted, and when CSI volumes are claimed.
Add a new topic to the event stream for host volumes. We'll emit events when a
dynamic host volume is registered or deregistered, and whenever a node
fingerprints with a changed volume.
Ref: https://hashicorp.atlassian.net/browse/NET-11549
