Commit Graph

20955 Commits

Author SHA1 Message Date
Tim Gross
6455d9bfd7 CSI: fix misleading HTTP test
The HTTP test to create CSI volumes depends on having a controller plugin to
talk to, but the test was using a node-only plugin, which allows it to
silently ignore the missing controller.
2021-03-31 16:37:09 -04:00
Tim Gross
783ed980ea CSI: fingerprint detailed controller capabilities
In order to support new controller RPCs, we need to fingerprint volume
capabilities in more detail and perform controller RPCs only when the specific
capability is present. This fixes a bug in Ceph support where the plugin can
only suport create/delete but we assume that it also supports attach/detach.
2021-03-31 16:37:09 -04:00
Tim Gross
38767549a9 CSI: volume creation/registration should not validate attachment
The CSI specification requires that we validate a list of `Capability` (access
mode + accessibility) when we create volume, but the existing volume
registration workflow incorrectly validates a single capability. The
specific capability required by a volume claim is checked at the time we make
the claim, so remove the check for `AttachmentMode`/`AcccessMode`.
2021-03-31 16:37:09 -04:00
Tim Gross
d7e80380de CSI: HTTP handlers for create/delete/list 2021-03-31 16:37:09 -04:00
Tim Gross
41258e54de CSI: create/delete/list volume RPCs
This commit implements the RPC handlers on the client that talk to the CSI
plugins on that client for the Create/Delete/List RPC.
2021-03-31 16:37:09 -04:00
Tim Gross
bb194cb91d test infrastructure for mock client RPCs (#10193)
This commit includes a new test client that allows overriding the RPC
protocols. Only the RPCs that are passed in are registered, which lets you
implement a mock RPC in the server tests. This commit includes an example of
this for the ClientCSI RPC server.
2021-03-31 16:37:09 -04:00
Tim Gross
f6ed7950e1 CSI: protobuffer mappings for Create/Delete/List volume RPCs
Note that unset proto fields for volume create should be nil. The CSI spec
handles empty fields and nil fields in the protobuf differently, which may
result in validation failures for creating volumes with no prior source (and
does in testing with the AWS EBS plugin). Refactor the `CreateVolumeRequest`
mapping to the protobuf in the plugin client to avoid this bug.
2021-03-31 16:37:09 -04:00
Tim Gross
2a4d075522 docs: CLI commands for volume create/delete 2021-03-31 16:37:09 -04:00
Tim Gross
f7263f6e3c drivers/docker: fix flaky image coordinator test
The test assertion that we don't have a delete future remaining races with the
code its testing, because the removal of the image and the removal of the
future are not atomic. Move this assertion into a `WaitForResult` to avoid
test flakes which we're seeing on CI on Windows in particular.
2021-03-31 15:59:01 -04:00
zhsj
46b335d652 deps: update runc to v1.0.0-rc93
includes updates for breaking changes in runc v1.0.0-rc93
2021-03-31 10:57:02 -04:00
Drew Bailey
9f0a4d4964 Docs/licensing reference docs (#10260)
* add a license docs page

* license reference docs

* update wording

* use new docs-nav

* rm file held over from rebase, fix path
2021-03-31 10:24:27 -04:00
Mahmood Ali
e3ea51622f Merge pull request #10247 from hashicorp/f-memory-oversubscription-2
Memory oversubscription
2021-03-31 09:57:19 -04:00
Bryce Kalow
ee79587a67 feat(website): migrates to new nav data format (#10264) 2021-03-31 08:43:17 -05:00
Mahmood Ali
6d891317ad fixup! oversubscription: Add MemoryMaxMB to internal structs 2021-03-31 09:26:26 -04:00
Mahmood Ali
699cd7a120 fixup! oversubscription: Add MemoryMaxMB to internal structs 2021-03-31 08:52:20 -04:00
Mahmood Ali
43549b46fc driver/exec: set soft memory limit
Linux offers soft memory limit:
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#soft-limits
, and
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html?highlight=memory.low
.

We can set soft memory limits through libcontainer
`Resources.MemoryReservation`: https://pkg.go.dev/github.com/opencontainers/runc@v0.1.1/libcontainer/configs#Resources
2021-03-30 16:55:58 -04:00
Mahmood Ali
750e665a1d oversubscription: e2e tests! 2021-03-30 16:55:58 -04:00
Mahmood Ali
5e3fbd5774 oversubscription: driver/exec to honor MemoryMaxMB 2021-03-30 16:55:58 -04:00
Mahmood Ali
b1ff06fd19 oversubscription: docker to honor MemoryMaxMB values 2021-03-30 16:55:58 -04:00
Mahmood Ali
b383f92188 oversubscription: set the linux memory limit
Use the MemoryMaxMB as the LinuxResources limit. This is intended to ease
drivers implementation and adoption of the features: drivers that use
`resources.LinuxResources.MemoryLimitBytes` don't need to be updated.

Drivers that use NomadResources will need to updated to track the new
field value. Given that tasks aren't guaranteed to use up the excess
memory limit, this is a reasonable compromise.
2021-03-30 16:55:58 -04:00
Mahmood Ali
18e1a599ed oversubscription: add memory_max to hclv1
Allow specifying the `memory_max` field in HCL under the resources block.

Though HCLv1 is deprecated, I've updated them to ease our testing.
2021-03-30 16:55:58 -04:00
Mahmood Ali
7df90da9c8 oversubscription: adds CLI and API support
This commit updates the API to pass the MemoryMaxMB field, and the CLI to show
the max set for the task.

Also, start parsing the MemoryMaxMB in hcl2, as it's set by tags.

A sample CLI output; note the additional `Max: ` for "task":

```
$ nomad alloc status 96fbeb0b
ID                  = 96fbeb0b-a0b3-aa95-62bf-b8a39492fd5c
[...]

Task "cgroup-fetcher" is "running"
Task Resources
CPU        Memory         Disk     Addresses
0/500 MHz  32 MiB/20 MiB  300 MiB

Task Events:
[...]

Task "task" is "running"
Task Resources
CPU        Memory          Disk     Addresses
0/500 MHz  176 KiB/20 MiB  300 MiB
           Max: 30 MiB

Task Events:
[...]
```
2021-03-30 16:55:58 -04:00
Mahmood Ali
80faab0f79 oversubscription: Add MemoryMaxMB to internal structs
Start tracking a new MemoryMaxMB field that represents the maximum memory a task
may use in the client. This allows tasks to specify a memory reservation (to be
used by scheduler when placing the task) but use excess memory used on the
client if the client has any.

This commit adds the server tracking for the value, and ensures that allocations
AllocatedResource fields include the value.
2021-03-30 16:55:58 -04:00
Tim Gross
14568b3e00 deps: bump gopsutil to v3.21.2 2021-03-30 16:02:51 -04:00
Tim Gross
f1645dd53a changelog: add section for backports from 1.1.0 to 1.0.5
* Fixed order
* Added missing changlog entry for GH-10145
* Fixed misclassified entry for GH-10158
2021-03-30 15:53:17 -04:00
Bryce Kalow
dee1664c83 fix(circle): change website branch filter to main (#10261) 2021-03-30 10:29:25 -05:00
Seth Hoenig
7da9ece876 Merge pull request #10243 from apollo13/issue10239
Automatically populate `CONSUL_HTTP_ADDR` for connect native tasks in host networking mode.
2021-03-30 09:00:17 -05:00
Nick Ethier
0a25e153c7 Merge pull request #10203 from hashicorp/f-cpu-cores
Reserved Cores [1/4]: Structs and scheduler implementation
2021-03-29 14:05:54 -04:00
Zachary Shilton
5ca4acff06 website: use global featured-slider component (#10254)
* website: use global featured-slider component

* website: delete unused local featured-slider
2021-03-29 11:37:48 -04:00
Bryce Kalow
9dc47fd8b7 website: add should-build script (#10194) 2021-03-29 10:06:20 -05:00
Dave May
5e6cb151c5 debug: Remove extra linefeed in monitor.log (#10252) 2021-03-29 09:22:27 -04:00
Florian Apolloner
f21ab14690 Automatically populate CONSUL_HTTP_ADDR for connect native tasks in host networking mode. Fixes #10239 2021-03-28 14:34:31 +02:00
Chris Baker
b72bafaaed Merge pull request #10125 from hashicorp/docs-autoscaler-409-plugins
autoscaler plugin docs
2021-03-28 07:07:24 -05:00
Luiz Aoqui
233dba8bc2 docs: minor updates on autoscaling plugin guides 2021-03-26 16:55:24 -04:00
Tim Gross
3d3980eb64 redirect from HTTP root to UI should include query params
The OTT feature relies on having a query parameter for a one-time token which
gets handled by the UI. We need to make sure that query param is preserved
when redirecting from the root URL to the `/ui/` URI.
2021-03-26 14:54:41 -04:00
Tim Gross
8dc695987c docs: note that Connect requires a hard-coded port
Co-authored-by: Kris Hicks <khicks@hashicorp.com>
2021-03-26 14:43:29 -04:00
Shishir
12fc27cc08 Update containerd task driver docs. (#10244) 2021-03-26 14:42:27 -04:00
Drew Bailey
fe0716ed43 e2e license smoke test (#10242) 2021-03-26 13:21:47 -04:00
Buck Doyle
3d9464023d Update to v4 of Ember Power Select (#10226)
This closes #10146.

Because of cibernox/ember-power-select#1203, which documents
the current impossibility of attaching test selectors to a
PowerSelect invocation, this uses test selectors on parent
containers instead, occasionally adding wrappers when needed.
I chose to leave the existing test selectors in the hopes that
we can return to using them eventually, but I could easily
remove them if it seems like extra noise now.

Presumably for the same reason, @class no longer works, so
this adjusts the scoping of global search CSS to preserve the style
of the search control.

I also included an update to the latest version of
ember-test-selectors, since we were far behind and I tried
that before finding the aforelinked issue.

Finally, this replaces ember-cli-uglify with ember-cli-terser to address
production build failures as described at ember-cli/ember-cli#9290.
2021-03-26 08:55:12 -05:00
Mahmood Ali
274e795be5 Merge pull request #10145 from hashicorp/b-periodic-init-status
periodic: always reset periodic children status
2021-03-26 09:19:08 -04:00
James Rasell
db3761782b Merge pull request #10230 from hashicorp/f-add-allocation-terminal-status-funcs-api
api: add Allocation client and server terminal status funcs.
2021-03-26 08:03:37 +01:00
Mahmood Ali
d81df3dd1a dispatched parameterized job should clear status too 2021-03-25 15:14:21 -04:00
Drew Bailey
15cf77ad66 e2e allow setting an enterprise license environment variable (#10233)
* allow setting an enterprise license environment variable

* update comment

* address pr comments
2021-03-25 14:35:55 -04:00
Taylor Dolezal
554f69394c Change enable to enabled 2021-03-25 14:10:54 -04:00
Michael Lange
c4a7728e03 Merge pull request #10208 from hashicorp/f-ui/use-new-chart-stuff
UI: Showed host reservations and allocation utilization by task on existing stats charts
2021-03-25 09:18:42 -07:00
Mahmood Ali
032945b718 Add a test for parameterized summary counts 2021-03-25 11:27:09 -04:00
Mahmood Ali
15fd4f3a8e periodic: always reset periodic children status
Fixes a bug where Nomad reports negative or incorrect running children
counts for periodic jobs.

The periodic dispatcher derives a child job without reseting the status.
If the periodic job has a `running` status, the derived job will start
as `running` status and transition to `pending`.  Since this is
unexpected transition, the counting in StateStore.setJobSummary gets out of sync and
result in negative/incorrect values.

Note that this only affects periodic jobs after a leader transition.
During the first job registration, the job is added with `pending` or
`""` status. However, after a leader transition, the new leader
repopulates the dispatcher heap with `"running"` status and triggers the
bug.
2021-03-25 11:27:09 -04:00
James Rasell
788f9f23a0 api: add Allocation client and server terminal status funcs. 2021-03-25 08:52:59 +01:00
Michael Lange
c15bf65ef0 Remove errant log line 2021-03-24 15:34:46 -07:00
Michael Lange
d9426df7af Correctly sort tasks in alloc stat tracker so the main task takes precedence 2021-03-24 15:10:46 -07:00