Commit Graph

627 Commits

Author SHA1 Message Date
Tim Gross
f7263f6e3c drivers/docker: fix flaky image coordinator test
The test assertion that we don't have a delete future remaining races with the
code its testing, because the removal of the image and the removal of the
future are not atomic. Move this assertion into a `WaitForResult` to avoid
test flakes which we're seeing on CI on Windows in particular.
2021-03-31 15:59:01 -04:00
zhsj
46b335d652 deps: update runc to v1.0.0-rc93
includes updates for breaking changes in runc v1.0.0-rc93
2021-03-31 10:57:02 -04:00
Mahmood Ali
43549b46fc driver/exec: set soft memory limit
Linux offers soft memory limit:
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#soft-limits
, and
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html?highlight=memory.low
.

We can set soft memory limits through libcontainer
`Resources.MemoryReservation`: https://pkg.go.dev/github.com/opencontainers/runc@v0.1.1/libcontainer/configs#Resources
2021-03-30 16:55:58 -04:00
Mahmood Ali
5e3fbd5774 oversubscription: driver/exec to honor MemoryMaxMB 2021-03-30 16:55:58 -04:00
Mahmood Ali
b1ff06fd19 oversubscription: docker to honor MemoryMaxMB values 2021-03-30 16:55:58 -04:00
Tim Gross
14568b3e00 deps: bump gopsutil to v3.21.2 2021-03-30 16:02:51 -04:00
Charlie Voiselle
d914990e5f Fixup uses of sanity (#10187)
* Fixup uses of `sanity`
* Remove unnecessary comments.

These checks are better explained by earlier comments about
the context of the test. Per @tgross, moved the tests together
to better reinforce the overall shared context.

* Update nomad/fsm_test.go
2021-03-16 18:05:08 -04:00
Florian Apolloner
8b3ea4ea9a docker: support configuring default log driver in plugin options 2021-03-12 16:04:33 -05:00
Adrian Todorov
2748d2a895 driver/docker: add extra labels ( job name, task and task group name) 2021-03-08 08:59:52 -05:00
Seth Hoenig
836ee9e4a2 drivers/exec+java: Add task configuration to restore previous PID/IPC isolation behavior
This PR adds pid_mode and ipc_mode options to the exec and java task
driver config options. By default these will defer to the default_pid_mode
and default_ipc_mode agent plugin options created in #9969. Setting
these values to "host" mode disables isolation for the task. Doing so
is not recommended, but may be necessary to support legacy job configurations.

Closes #9970
2021-02-08 14:26:35 -06:00
Seth Hoenig
6dd5de4b69 docs: fixup comments, var names 2021-02-08 10:58:44 -06:00
Seth Hoenig
b682371a22 drivers/exec+java: Add configuration to restore previous PID/IPC namespace behavior.
This PR adds default_pid_mode and default_ipc_mode options to the exec and java
task drivers. By default these will default to "private" mode, enabling PID and
IPC isolation for tasks. Setting them to "host" mode disables isolation. Doing
so is not recommended, but may be necessary to support legacy job configurations.

Closes #9969
2021-02-05 15:52:11 -06:00
Nick Ethier
0dd2feb1db drivers/docker: support mapping multiple host ports to the same container port 2021-02-02 22:54:23 -05:00
Chris Baker
7f06adf1af Merge tag 'v1.0.3' into post-release-1.0.3
Version 1.0.3
2021-01-29 19:30:08 +00:00
vagrant
c1910544be attempting to fix flaky tests caused by pid isolation 2021-01-28 12:03:20 +00:00
Kris Hicks
f4fe79ad89 Backfill unit test for NEWIPC 2021-01-28 12:03:19 +00:00
Chris Baker
6a067a33ff modify exec driver test TestExecDriver_StartWaitStop in light of the fact that signaling sleep with SIGINT doesn't work if it's PID1 2021-01-28 12:03:19 +00:00
Chris Baker
611abc33a5 modify exec driver test TestExecDriver_DestroyKills all in light of the fact that PID namespacing means that the kernel does this now 2021-01-28 12:03:19 +00:00
Chris Baker
109fb53e50 put exec process in a new IPC namespace 2021-01-28 12:03:19 +00:00
Kris Hicks
41681fa1d2 Add unit test for container namespacing 2021-01-28 12:03:19 +00:00
Kris Hicks
75cea8ad48 Always check that resource constraints were applied 2021-01-28 12:03:19 +00:00
Kris Hicks
677353a205 Add PID namespacing and e2e test 2021-01-28 12:03:19 +00:00
Tim Gross
153522f47f prefer TrimPrefix to checking HasPrefix first 2021-01-22 13:41:28 -05:00
Huan Wang
c4d8bc49c9 fix the inconsistency handling between infra image and normal task image 2021-01-22 13:41:28 -05:00
Kris Hicks
2a40c1f912 Don't prepend https to docker cred helper call (#9852)
Some credential helpers, like the ECR helper, will strip the protocol if
given. Others, like the linux "pass" helper, do not.
2021-01-21 11:46:59 -08:00
Charlie Voiselle
ba233328dd Enable network namespaces for QEMU driver (#9861)
* Enable network namespaces for QEMU driver
* Add CHANGELOG entry
2021-01-21 14:05:46 -05:00
Kris Hicks
bcd4752fc9 executor_linux: Remove unreachable PATH= code (#9778)
This has to have been unused because the HasPrefix operation is
backwards, meaning a Command.Env that includes PATH= never would have
worked; the default path was always used.
2021-01-15 11:19:09 -08:00
Mahmood Ali
8879645ab9 docker: introduce a new hcl2-friendly mount syntax (#9635)
Introduce a new more-block friendly syntax for specifying mounts with a new `mount` block type with the target as label:

```hcl
config {
  image = "..."

  mount {
    type = "..."
    target = "target-path"
    volume_options { ... }
  }
}
```

The main benefit here is that by `mount` being a block, it can nest blocks and avoids the compatibility problems noted in https://github.com/hashicorp/nomad/pull/9634/files#diff-2161d829655a3a36ba2d916023e4eec125b9bd22873493c1c2e5e3f7ba92c691R128-R155 .

The intention is for us to promote this `mount` blocks and quietly deprecate the `mounts` type, while still honoring to preserve compatibility as much as we could.

This addresses the issue in https://github.com/hashicorp/nomad/issues/9604 .
2020-12-15 14:13:50 -05:00
Kris Hicks
7747124ef0 Apply some suggested fixes from staticcheck (#9598) 2020-12-10 07:29:18 -08:00
Kris Hicks
85ed8ddd4f Add gosimple linter (#9590) 2020-12-09 11:05:18 -08:00
Kris Hicks
071f4c7596 Add gocritic to golangci-lint config (#9556) 2020-12-08 12:47:04 -08:00
Tim Gross
a1bfc0320e docker: kill signal API should include timeout context
When the Docker driver kills as task, we send a request via the Docker API for
dockerd to fire the signal. We send that signal and then block for the
`kill_timeout` waiting for the container to exit. But if the Docker API
blocks, we will block indefinitely because we haven't configured the API call
with the same timeout.

This changeset is a minimal intervention to add the timeout to the Docker API
call _only_ when we have the `kill_timeout` set. Future work should examine
whether we should be threading contexts through other `go-dockerclient` API
calls.
2020-12-02 16:51:57 -05:00
Mahmood Ali
6d57a1c69b use comment ignores (#9448)
Use targetted ignore comments for the cases where we are bound by
backward compatibility.

I've left some file based linters, especially when the file is riddled
with linter voilations (e.g. enum names), or if it's a property of the
file (e.g. package and file names).

I encountered an odd behavior related to RPC_REQUEST_RESPONSE_UNIQUE and
RPC_REQUEST_STANDARD_NAME.  Apparently, if they target a `stream` type,
we must separate them into separate lines so that the ignore comment
targets the type specifically.
2020-11-25 16:03:01 -05:00
Mahmood Ali
edaa16589b honor task user when execing into raw_exec task (#9439)
Fix #9210 .

This update the executor so it honors the User when using nomad alloc exec. The bug was that the exec task didn't honor the init command when execing.
2020-11-25 09:34:10 -05:00
Nick Ethier
f8aabc8bd2 command: use correct port mapping syntax in examples 2020-11-23 10:25:30 -06:00
Mahmood Ali
de5a21f269 Merge pull request #8291 from shishir-a412ed/cpusets
Add cpuset_cpus to docker driver.
2020-11-11 17:13:27 -05:00
Mahmood Ali
bd745fa3e5 raw_exec: don't use cgroups when no_cgroup is set (#9328)
When raw_exec is configured with [`no_cgroups`](https://www.nomadproject.io/docs/drivers/raw_exec#no_cgroups), raw_exec shouldn't attempt to create a cgroup.

Prior to this change, we accidentally always required freezer cgroup to do stats PID tracking. We already have the proper fallback in place for metrics, so only need to ensure that we don't create a cgroup for the task.

Fixes https://github.com/hashicorp/nomad/issues/8565
2020-11-11 16:20:34 -05:00
Shishir Mahajan
8f27c08efd Fix review comments. 2020-11-11 12:30:00 -08:00
Shishir Mahajan
850694634f Fix circleci. 2020-11-11 12:30:00 -08:00
Shishir Mahajan
161bec4ada Add cpuset_cpus to docker driver. 2020-11-11 12:30:00 -08:00
Tim Gross
306cfabb62 docker: disallow volume mounts from host by default (#9321)
The default behavior for `docker.volumes.enabled` is intended to be `false`,
but the HCL schema defaults to `true` if the value is unset. Set the default
literal value to `true`.

Additionally, Docker driver mounts of type "volume" (but not "bind") are not
being properly sandboxed with that setting. Disable Docker mounts with type
"volume" entirely whenever the `docker.volumes.enabled` flag is set to
false. Note this is unrelated to the `volume_mount` feature, which is
constrained to preconfigured host volumes or whatever is mounted by a CSI
plugin.

This changeset includes updates to unit tests that should have been failing
under the documented behavior but were not.
2020-11-11 10:03:46 -05:00
Mahmood Ali
669f9cbcbe Merge pull request #9304 from hashicorp/b-legacy-executors-are-executors
Legacy executors are executors after all
2020-11-10 12:54:03 -05:00
Kris Hicks
b3825e74b5 protos: Update .proto files not to use Go package name (#9301)
Previously, it was required that you `go get github.com/hashicorp/nomad` to be
able to build protos, as the protoc invocation added an include directive that
pointed to `$GOPATH/src`, which is how dependent protos were discovered. As
Nomad now uses Go modules, it won't necessarily be cloned to `$GOPATH`.
(Additionally, if you _had_ go-gotten Nomad at some point, protoc compilation
would have possibly used the _wrong_ protos, as those wouldn't necessarily be
the most up-to-date ones.)

This change modifies the proto files and the `protoc` invocation to handle
discovering dependent protos via protoc plugin modifier statements that are
specific to the protoc plugin being used.

In this change, `make proto` was run to recompile the protos, which results in
changes only to the gzipped `FileDescriptorProto`.
2020-11-10 08:42:35 -08:00
Mahmood Ali
5847b93c94 Legacy executors are executors after all
This fixes a bug where pre-0.9 executors fail to recover after an
upgrade.

The bug is that legacyExecutorWrappers didn't get updated with
ExecStreaming function, and thus failed to implement the Executor
function. Sadly, this meant that all recovery attempts fail, as the
runtime check in
b312aacbc9/drivers/shared/executor/utils.go (L103-L110)
.
2020-11-10 10:20:07 -05:00
Russell Rollins
93c7909951 Use Dockerhub Mirror. (#9220)
Dockerhub is going to rate limit unauthenticated pulls.

Use our HashiCorp internal mirror for builds run through CircleCI.

Co-authored-by: Mahmood Ali <mahmood@hashicorp.com>
2020-11-02 09:28:02 -05:00
Charlie Voiselle
b5e6326819 Fix for Java fingerprinter on macOS (#9225)
Use alternative test for macOS JVM with /usr/libexec/java_home
2020-11-01 13:20:31 -05:00
Tim Gross
a9979db1d4 docker: image_delay default missing without gc stanza (#9101)
In the Docker driver plugin config for garbage collection, the `image_delay`
field was missing from the default we set if the entire `gc` stanza is
missing. This results in a default of 0s and immediate GC of Docker images.

Expanded docker gc config test fields.
2020-10-15 12:36:01 -04:00
Michael Schurter
f44c04ecd1 s/0.13/1.0/g
1.0 here we come!
2020-10-14 15:17:47 -07:00
Yoan Blanc
c14c616194 use allow/deny instead of the colored alternatives (#9019)
Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2020-10-12 08:47:05 -04:00
Seth Hoenig
a299b76c5a docs: document docker signal fix, add tests
This PR adds a version specific upgrade note about the docker stop
signal behavior. Also adds test for the signal logic in docker driver.

Closes #8932 which was fixed in #8933
2020-10-02 10:06:43 -05:00