Commit Graph

75 Commits

Author SHA1 Message Date
Nick Ethier
5b14d24bf4 executor v2 (#4656)
* client/executor: refactor client to remove interpolation

* executor: POC libcontainer based executor

* vendor: use hashicorp libcontainer fork

* vendor: add libcontainer/nsenter dep

* executor: updated executor interface to simplify operations

* executor: implement logging pipe

* logmon: new logmon plugin to manage task logs

* driver/executor: use logmon for log management

* executor: fix tests and windows build

* executor: fix logging key names

* executor: fix test failures

* executor: add config field to toggle between using libcontainer and standard executors

* logmon: use discover utility to discover nomad executable

* executor: only call libcontainer-shim on main in linux

* logmon: use seperate path configs for stdout/stderr fifos

* executor: windows fixes

* executor: created reusable pid stats collection utility that can be used in an executor

* executor: update fifo.Open calls

* executor: fix build

* remove executor from docker driver

* executor: Shutdown func to kill and cleanup executor and its children

* executor: move linux specific universal executor funcs to seperate file

* move logmon initialization to a task runner hook

* client: doc fixes and renaming from code review


* taskrunner: use shared config struct for logmon fifo fields

* taskrunner: logmon only needs to be started once per task
2018-10-16 16:53:31 -07:00
Nick Ethier
e9f3f2cfee Update runc/libcontainer and friends (#4655)
* vendor: bump libcontainer and docker to remove Sirupsen imports

* vendor: fix bad vendoring of archive package

* vendor: fix api changes to cgroups in executor

* vendor: fix docker api changes

* vendor: update github.com/Azure/go-ansiterm to use non capitalized logrus import
2018-10-16 16:53:30 -07:00
Alex Dadgar
207fece24e Use more appropriate name than cgroup 2018-05-31 18:05:03 -07:00
Alex Dadgar
c5d2c7def5 Only use freezer/devices when in the basic cgroup only 2018-05-31 18:05:03 -07:00
Alex Dadgar
1adf3f9047 Use cgroup when possible 2018-05-31 18:05:03 -07:00
Fabian Holler
fbc461b904 fix: disable swap for executor_linux allocations
A comment in the nomad source code states that swapping for
executor_linux allocations is disabled but it wasn't.

Nomad wrote -1 to the memsw.limit_in_bytes cgroup file to disable
swapping.
This has the following problems:

1.) Writing -1 to the file does not disable swapping. It sets
    the limit for memory and swap to unlimited.
2.) On common Linux distributions like Ubuntu 16.04 LTS the
    memsw.limit_in_bytes cgroup file does not exist by default.
    The memsw.limit_in_bytes file only exist if the Linux kernel is
    build with CONFIG_MEMCG_SWAP=yes and either
    CONFIG_MEMCG_SWAP_ENABLED=yes or when the kernel parameter
    swapaccount=1 is passed during boot.
    Most Linux distributions disable swap accounting by default because
    of higher memory usage.
    Nomad silently ignores if writing to the memsw.limit_in_bytes file
    fails. The allocation succeeds, no message is logged to notify the
    user.

To ensure that disabling swap works on common Linux kernels, disable
swapping by writing 0 to the memory.swappiness file.
Using the memory.swappiness file only requires that the kernel is
compiled with CONFIG_MEMCG=yes. This is the default in common Linux
kernels.
2018-03-13 10:52:50 +01:00
Michael Schurter
04b8f8e7fc Remove structs import from api
Goes a step further and removes structs import from api's tests as well
by moving GenerateUUID to its own package.
2017-09-29 10:36:08 -07:00
Alex Dadgar
67c07c9932 Various fixes for setting user/group of task
This PR fixes two issues:
* Folder permissions in -dev mode were incorrect and not suitable for
running as a particular user.
* Was not setting the group membership properly for the launched
process.

Fixes https://github.com/hashicorp/nomad/issues/2160
2017-03-20 14:21:13 -07:00
Michael Schurter
de7351b959 Move chroot building into TaskRunner
* Refactor AllocDir to have a TaskDir struct per task.
* Drivers expose filesystem isolation preference
* Fix lxc mounting of `secrets/`
2017-01-05 16:31:49 -08:00
Alex Dadgar
17cab6e932 Fix env vars relating to secretdir 2016-11-08 12:28:43 -08:00
Alex Dadgar
335cababe1 Revert "Introduce a Secret/ directory" 2016-09-01 17:23:15 -07:00
Alex Dadgar
dfab22cd9f environment variables 2016-08-31 13:56:11 -07:00
Jay Oster
593669ada2 Address review comments
- Simplify map length check in Linux Executor
- Added a `chroot_env` test for config parser
- Moved `ChrootEnv` field from ExecutorCommand to ExecutorContext
- Added a test for `chroot_env` functionality
2016-08-04 15:33:06 -07:00
Jay Oster
1876fc2b10 Add config field to specify chroot mapping for exec driver
- Same format as used by the internal chroot mapping
- Map: source_path -> dest_path
- Example HCL:

client {
  chroot_env {
    "/etc" = "/etc"
    "/lib" = "/lib"
    "/opt/projects/foo/bin" = "/usr/bin"
  }
}
2016-08-03 17:17:17 -07:00
Sean Chittenden
875c090ed9 Rename resourceContainer{,Context} and resCon{,Ctx}. 2016-07-11 00:02:55 -07:00
Sean Chittenden
5ede387a70 Update comments and pushdown a lock into the resource container 2016-07-10 00:12:59 -07:00
Sean Chittenden
1baa5de6af Push down the Linux-specific bits into resourceContainer 2016-07-10 00:06:53 -07:00
Alex Dadgar
54f6f57448 Merge pull request #1262 from hashicorp/remove-artifact-check
Removing artifact check for java and qemu drivers
2016-06-12 11:21:18 -07:00
Diptanu Choudhury
f795e215b4 Setting a flag to indicate whether fs isolation is indeed happening 2016-06-12 15:43:24 +02:00
Diptanu Choudhury
12f3da9ad4 Not converting the abs path relative to task dir for drivers which enforce FS isolation only in linux 2016-06-12 13:54:30 +02:00
Alex Dadgar
1a7df4e7d2 Allocation resources returned in a struct 2016-06-11 21:04:10 -07:00
Diptanu Choudhury
446d3f22f2 Pruning out pids which are no longer present 2016-06-11 01:40:52 +02:00
Diptanu Choudhury
d31034941a Not reset-ing the list of pids if they don't change 2016-06-11 01:19:50 +02:00
Diptanu Choudhury
4ac1c673aa Implementing the total ticks per task for the docker driver 2016-06-10 23:33:25 +02:00
Diptanu Choudhury
c13e750a02 Calculating the cpu ticks in nomad client 2016-06-10 22:22:32 +02:00
Alex Dadgar
693c8f9e42 Alloc-status only shows measured statistics and fixes to CPU calculations 2016-06-10 10:38:29 -07:00
Diptanu Choudhury
d594d95012 Fixed merge conflicts 2016-06-10 04:47:34 +01:00
Diptanu Choudhury
06e90cb6c0 Using named variables 2016-06-10 04:45:16 +01:00
Diptanu Choudhury
8d52390aca Fixed calculation of cpu usage percent in user and sys mode for pids 2016-06-10 04:06:32 +01:00
Alex Dadgar
17af581d93 Annotate what is measured 2016-06-09 19:45:41 -07:00
Diptanu Choudhury
993675dd4e Added a test for docker 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
f16191c297 comments 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
c99733e8c7 Fixed the compilation on linux 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
584c1e34fb Incorporated review comments for executor 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
6132ccc230 Added pidstats in task resource usage struct 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
3192e318e9 Renamed CpuUsage to CpuStats 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
16f298fb95 Fixed the percentage calculation for cgroups 2016-05-28 19:59:20 -07:00
Diptanu Choudhury
98068678f1 Implemented nomad cpu percentage calculator 2016-05-28 19:59:03 -07:00
Diptanu Choudhury
30cbfe1f71 Implemented cpu stats 2016-05-28 19:51:22 -07:00
Diptanu Choudhury
0fdff61d2d Implemented stats for raw_exec 2016-05-28 19:42:34 -07:00
Diptanu Choudhury
7d8196d7ea Added some documentation 2016-05-28 19:42:34 -07:00
Diptanu Choudhury
f390261cd0 Reporting time series of stats 2016-05-28 19:42:34 -07:00
Diptanu Choudhury
50250b1455 Added the nomad stats command 2016-05-28 19:42:34 -07:00
Diptanu Choudhury
a485a38029 Added cpu stats 2016-05-28 19:42:34 -07:00
Diptanu Choudhury
01e0ae7753 Added a client API to display resource usage of an allocation 2016-05-28 19:42:34 -07:00
Diptanu Choudhury
1cae57aee9 Add the Stats api to driverhandle 2016-05-28 19:42:34 -07:00
Alex Dadgar
44627677f7 Freeze the cgroup and cleanup around Shutdown 2016-04-19 13:48:02 -07:00
Alex Dadgar
7616be8094 Put the executor into the cgroup to avoid a fork race 2016-04-18 17:59:08 -07:00
Alex Dadgar
9f1976ef3e remove systemd cgroup manager 2016-04-15 14:55:30 -07:00
Diptanu Choudhury
36d8c7e831 Setting the cg path after application of pid 2016-04-02 12:37:19 -07:00