This changeset fixes three potential data integrity issues between allocations
and their Nomad native service registrations.
* When a node is marked down because it missed heartbeats, we remove Vault and
Consul tokens (for the pre-Workload Identity workflows) after we've written
the node update to Raft. This is unavoidably non-transactional because the
Consul and Vault servers aren't in the same Raft cluster as Nomad itself. But
we've unnecessarily mirrored this same behavior to deregister Nomad
services. This makes it possible for the leader to successfully write the node
update to Raft without removing services.
To address this, move the delete into the same Raft transaction. One minor
caveat with this approach is the upgrade path: if the leader is upgraded first
and a node is marked down during this window, older followers will have stale
information until they are also upgraded. This is unavoidable without
requiring the leader to unconditionally make an extra Raft write for every
down node until 2 LTS versions after Nomad 1.8.0. This temporary reduction in
data integrity for stale reads seems like a reasonable tradeoff.
* When an allocation is marked client-terminal from the client in
`UpdateAllocsFromClient`, we have an opportunity to ensure data integrity by
deregistering services for that allocation.
* When an allocation is deleted during eval garbage collection, we have an
opportunity to ensure data integrity by deregistering services for that
allocation. This is a cheap no-op if the allocation has been previously marked
client-terminal.
This changeset does not address client-side retries for the originally reported
issue, which will be done in a separate PR.
Ref: https://github.com/hashicorp/nomad/issues/16616
