vault: avoid continual renewal of invalid token (#18985)

A series of errors may happen when a token is invalidated while the
Vault client is waiting to renew it. The token may have been invalidated
for several reasons, such as the alloc finished running and it's now
terminal or the token may have been change directly on Vault
out-of-band.

Most of the errors are caused by retries that will never succeed until
Vault fully removes the token from its state.

This commit prevents the retries by making the error `invalid lease ID`
a fatal error.

In earlier versions of Vault, this case was covered by the error `lease
not found or lease is not renewable`, which is already considered to be
a fatal error by Nomad:

2d0cde4ccc/vault/expiration.go (L636-L639)

But https://github.com/hashicorp/vault/pull/5346 introduced an earlier
`nil` check that generates a different error message:

750ab337ea/vault/expiration.go (L1362-L1364)

Both errors happen for the same reason (`le == nil`) and so should be
considered fatal on renewal.
This commit is contained in:
Luiz Aoqui
2023-11-07 19:50:19 -05:00
committed by GitHub
parent 7054fe1a8c
commit ab36cf031c
2 changed files with 4 additions and 0 deletions

3
.changelog/18985.txt Normal file
View File

@@ -0,0 +1,3 @@
```release-note:bug
vault: Fixed an issue that could cause Nomad to attempt to renew a Vault token that is already expired
```

View File

@@ -512,6 +512,7 @@ func (c *vaultClient) renew(req *vaultClientRenewalRequest) error {
fatal := false
if renewalErr != nil &&
(strings.Contains(renewalErr.Error(), "lease not found or lease is not renewable") ||
strings.Contains(renewalErr.Error(), "invalid lease ID") ||
strings.Contains(renewalErr.Error(), "lease is not renewable") ||
strings.Contains(renewalErr.Error(), "token not found") ||
strings.Contains(renewalErr.Error(), "permission denied")) {