mirror of
https://github.com/kemko/nomad.git
synced 2026-01-01 16:05:42 +03:00
f52912454d
The CSI specification says that we "SHOULD" send no more than one in-flight request per *volume* at a time, with an allowance for losing state (ex. leadership transitions) which the plugins "SHOULD" handle gracefully. We mostly successfully serialize node and controller RPCs for the same volume, except when Nomad clients are lost. (See also https://github.com/container-storage-interface/spec/issues/512) These concurrency requirements in the spec fall short because Storage Provider APIs aren't necessarily safe to call concurrently on the same host even for _different_ volumes. For example, concurrently attaching AWS EBS volumes to an EC2 instance results in a race for device names, which results in failure to attach (because the device name is taken already and the API call fails) and confused results when releasing claims. So in practice many CSI plugins rely on k8s-specific sidecars for serializing storage provider API calls globally. As a result, we have to be much more conservative about concurrency in Nomad than the spec allows. This changeset includes four major changes to fix this: * Add a serializer method to the CSI volume RPC handler. When the RPC handler makes a destructive CSI Controller RPC, we send the RPC thru this serializer and only one RPC is sent at a time. Any other RPCs in flight will block. * Ensure that requests go to the same controller plugin instance whenever possible by sorting by lowest client ID out of the plugin instances. * Ensure that requests go to _healthy_ plugin instances only. * Ensure that requests for controllers can go to a controller on any _live_ node, not just ones eligible for scheduling (which CSI controllers don't care about) Fixes: #15415
12 lines
376 B
Plaintext
12 lines
376 B
Plaintext
```release-note:bug
|
|
csi: Fixed a bug in sending concurrent requests to CSI controller plugins by serializing them per plugin
|
|
```
|
|
|
|
```release-note:bug
|
|
csi: Fixed a bug where CSI controller requests could be sent to unhealthy plugins
|
|
```
|
|
|
|
```release-note:bug
|
|
csi: Fixed a bug where CSI controller requests could not be sent to controllers on nodes ineligible for scheduling
|
|
```
|