A Nomad user reported problems with CSI volumes associated with failed
allocations, where the Nomad server did not send a controller unpublish RPC.
The controller unpublish is skipped if other non-terminal allocations on the
same node claim the volume. The check has a bug where the allocation belonging
to the claim being freed was included in the check incorrectly. During a normal
allocation stop for job stop or a new version of the job, the allocation is
terminal. But allocations that fail are not yet marked terminal at the point in
time when the client sends the unpublish RPC to the server.
For CSI plugins that support controller attach/detach, this means that the
controller will not be able to detach the volume from the allocation's host and
the replacement claim will fail until a GC is run. This changeset fixes the
conditional so that the claim's own allocation is not included, and makes the
logic easier to read. Include a test case covering this path.
Also includes two minor extra bugfixes:
* Entities we get from the state store should always be copied before
altering. Ensure that we copy the volume in the top-level unpublish workflow
before handing off to the steps.
* The list stub object for volumes in `nomad/structs` did not match the stub
object in `api`. The `api` package also did not include the current
readers/writers fields that are expected by the UI. True up the two objects and
add the previously undocumented fields to the docs.
When querying the checks for an allocation, the request must be
forwarded to the agent that is running the allocation. If the
initial request is made to a server agent, the request can be made
directly to the client agent running the allocation. If the
request is made to a client agent not running the alloc, the
request needs to be forwarded to a server and then the correct
client.
The map of in-flight RPCs gets cleared by a goroutine in the test without first
locking it to make sure that it's not being accessed concurrently by the stats
fetcher itself. This can cause a panic in tests.
This PR modifies RandomStagger to protect against negative input
values. If the given interval is negative, the value returned will
be somewhere in the stratosphere. Instead, treat negative inputs
like zero, returning zero.
* Bones of a new flyout section
* Basic sidebar behaviour and style edits
* Concept of a refID for service fragments to disambiguate task and group
* A11y audit etc
* Moves health check aggregation to serviceFragment model and retains history
* Has to be a getter
* flyout populated
* Sidebar styling
* Sidebar table and details added
* Mirage fixture
* Active status and table styles
* Unit test mock updated
* Acceptance tests for alloc services table and flyout
* Chart styles closer to mock
* Without a paused test
* Consul and Nomad icons in services table
* Alloc services test updates in light of new column changes
* without using an inherited scenario
* Added to subnav and basic table implemented
* Existing services become service fragments, and services tab aggregated beneath job route
* Index page within jobs/job/services
* Watchable services
* Lintfixes
* Links to clients and individual services set up
* Child service route
* Keyboard shortcuts on service page
* Model that shows consul services as well, plus level and provider cols
* lintfix
* Level as query param
* Watch job for service name changes too
* Group level service fixtures established
* Progress at task level and job-linked services
* Task and group services on update
* Fixture side-effect cleanup
* Basic acceptance tests for job services
* Testmodel cleanup
* Disabled mirage logging
* New cluster type specifically for services
* Without explicit job-model binding
* Trying to isolate a tostring error
* Account for new tab in keyboardnav
* More test isolation attempts
* Remove skipped tests and link task to parent group by id
ui: add service health viz to table (#14369)
* ui: add service-status-bar
* test: service-status-bar
* refact: update component api for new data struct
* ui: format service health struct
* ui: add service health viz to table
* temp: add placeholder to remind conditional watcher
* test: write tests for transformation algorithm
* refact: update transformation algo
* ui: conditionally long poll checks endpoint
* refact: add conditional logic for nomad provider
refact: update service-fragment model to include owner info
ui: differentiate between task and group-level in derived state comp
test: add test to document behavior
refact: update tests for api change
refact: update integration test for API change
chore: remove unsused vars
chore: elvis operator to protect mirage
refact: create refId instead of internalModel
refact: update algo
refact: update conditional template logic
refact: update test for api change:
chore: cant use if and not in hbs conditional
* Added to subnav and basic table implemented
* Existing services become service fragments, and services tab aggregated beneath job route
* Index page within jobs/job/services
* Watchable services
* Lintfixes
* Links to clients and individual services set up
* Child service route
* Keyboard shortcuts on service page
* Model that shows consul services as well, plus level and provider cols
* lintfix
* Level as query param
* Watch job for service name changes too
* Lintfix
* Testfixes
* Placeholder mirage route
Includes:
* Remove leader upgrade raft version test, as older versions of raft are now
incompatible with our autopilot library.
* Remove attempt to assert initial non-voter status on the `PromoteNonVoter`
test, as this happens too quickly to reliably detect.
* Unskip some previously-skipped tests which we should make stable.
* Remove the `consul/sdk` retry helper for these tests; this uses panic recovery
in a kind of a clever/gross way to reduce LoC but it seems to introduce some
timing issues in the process.
* Add more test step logging and reduce logging noise from the scheduler
goroutines to make it easier to debug failing tests.
* Be more consistent about using the `waitForStableLeadership` helper so that we
can assert the cluster is fully stable and not just that we've added peers.
When configuring Consul Service Mesh, it's sometimes necessary to
provide dynamic value that are only known to Nomad at runtime. By
interpolating configuration values (in addition to configuration keys),
user are able to pass these dynamic values to Consul from their Nomad
jobs.
Nomad's original autopilot was importing from a private package in Consul. It
has been moved out to a shared library. Switch Nomad to use this library so that
we can eliminate the import of Consul, which is necessary to build Nomad ENT
with the current version of the Consul SDK. This also will let us pick up
autopilot improvements shared with Consul more easily.
These options are mutually exclusive but, since `-hcl2-strict` defaults
to `true` users had to explicitily set it to `false` when using `-hcl1`.
Also return `255` when job plan fails validation as this is the expected
code in this situation.
Nomad is generally compliant with the CSI specification for Container
Orchestrators (CO), except for unimplemented features. However, some storage
vendors have built CSI plugins that are not compliant with the specification or
which expect that they're only deployed on Kubernetes. Nomad cannot vouch for
the compatibility of any particular plugin, so clarify this in the docs.
Co-authored-by: Derek Strickland <1111455+DerekStrickland@users.noreply.github.com>
Log lines which include an error should use the full term "error"
as the context key. This provides consistency across the codebase
and avoids a Go style which operators might not be aware of.
In order to add an E2E test to cover token expiration, the server
config has been updated to include a low minimum allowed TTL
value. For ease of reading, the max value is also set.
The ACL command docs are now found within a sub-dir like the
operator command docs. Updates to the ACL token commands to
accommodate token expiry have also been added.
The ACL API docs are now found within a sub-dir like the operator
API docs. The ACL docs now include the ACL roles endpoint as well
as updated ACL token endpoints for token expiration.
The configuration section is also updated to accommodate the new
ACL and server parameters for the new ACL features.
This adds a new ACL test suite to the e2e framework which includes
an initial test for ACL roles. The ACL test includes a helper to
track and clean created Nomad resources which keeps the test
cluster clean no matter if the test fails early or not.
PR #12130 refactored the test to use the `wantPeers` helper, but this
function only returns the number of voting peers, which in this test
should be equal to 2.
I think the tests were passing back them because of a bug in Raft
(https://github.com/hashicorp/raft/pull/483) where a non-voting server
was able to transition to candidate state.
One possible evidence of this is that a successful test run would have
the following log line:
```
raft@v1.3.5/raft.go:1058: nomad.raft: updating configuration: command=AddVoter server-id=127.0.0.1:9101 server-addr=127.0.0.1:9101 servers="[{Suffrage:Voter ID:127.0.0.1:9107 Address:127.0.0.1:9107} {Suffrage:Voter ID:127.0.0.1:9105 Address:127.0.0.1:9105} {Suffrage:Voter ID:127.0.0.1:9103 Address:127.0.0.1:9103} {Suffrage:Voter ID:127.0.0.1:9101 Address:127.0.0.1:9101}]"
```
This commit reverts the test logic to check for peer count, regardless
of voting status.
