Commit Graph

236 Commits

Author SHA1 Message Date
Nick Ethier
e26192ad49 Driver networking support
Adds support for passing network isolation config into drivers and
implements support in the rawexec driver as a proof of concept
2019-07-31 01:03:20 -04:00
Nick Ethier
9fa47daf5c ar: fix lint errors 2019-07-31 01:03:19 -04:00
Nick Ethier
da3978b377 plugins/driver: make DriverNetworkManager interface optional 2019-07-31 01:03:19 -04:00
Nick Ethier
bc71cafcee plugin/driver: fix tests and add new dep to vendor 2019-07-31 01:03:17 -04:00
Nick Ethier
4a8a96fa1a ar: initial driver based network management 2019-07-31 01:03:17 -04:00
Nick Ethier
e20fa7ccc1 Add network lifecycle management
Adds a new Prerun and Postrun hooks to manage set up of network namespaces
on linux. Work still needs to be done to make the code platform agnostic and
support Docker style network initalization.
2019-07-31 01:03:17 -04:00
Jasmine Dahilig
e31db578e0 add formatting for hcl parsing error messages (#5972) 2019-07-19 10:04:39 -07:00
Mahmood Ali
d8a43b2066 Signal plugin shutdown for driver.TaskStats
The driver plugin stub client must call `grpcutils.HandleGrpcErr` to handle plugin
shutdown similar to other functions.  This ensures that TaskStats returns
`ErrPluginShutdown` when plugin shutdown.
2019-07-11 13:57:35 +08:00
Chris Baker
3a96683131 docker: DestroyTask was not cleaning up Docker images because it was erroring early due to an attempt to inspect an image that had already been removed 2019-06-03 19:04:27 +00:00
Mahmood Ali
3a77502cae Add basic drivers conformance tests
Add consolidated testing package to serve as conformance tests for all drivers.
2019-05-09 16:49:08 -04:00
Mahmood Ali
94ed649489 implemment streaming exec handling in driver grpc handlers
Also add a helper that converts the adapts the high level interface to the
low-level interface of nomad exec interfaces.
2019-05-09 16:49:08 -04:00
Mahmood Ali
6d711d054b add nomad streaming exec core data structures and interfaces
In this commit, we add two driver interfaces for supporting `nomad exec`
invocation:

* A high level `ExecTaskStreamingDriver`, that operates on io reader/writers.
  Drivers should prefer using this interface
* A low level `ExecTaskStreamingRawDriver` that operates on the raw stream of
  input structs; useful when a driver delegates handling to driver backend (e.g.
  across RPC/grpc).

The interfaces are optional for a driver, as `nomad exec` support is opt-in.
Existing drivers continue to compile without exec support, until their
maintainer add such support.

Furthermore, we create protobuf structures to represent exec stream entities:
`ExecTaskStreamingRequest` and `ExecTaskStreamingResponse`.  We aim to reuse the
protobuf generated code as much as possible, without translation to avoid
conversion overhead.

`ExecTaskStream` abstract fetching and sending stream entities.  It's influenced
by the grpc bi-directional stream interface, to avoid needing any adapter.  I
considered using channels, but the asynchronisity and concurrency makes buffer
reuse too complicated, which would put more pressure on GC and slows exec operation.
2019-04-30 14:02:29 -04:00
Mahmood Ali
b8f80e5124 Simplify proto conversion and handle swap
Convert all cpu and memory usage fields regardless of stated measured
fields, and handle swap fields
2019-03-30 15:18:28 -04:00
Mahmood Ali
484ccfafab deserialize total ticks 2019-03-30 07:14:57 -04:00
Mahmood Ali
a880f93a61 Always report TotalTicks when percent is measured
Fix a case where TotalTicks doesn't get serialized across executor grpc
calls.

Here, I opted to implicit add field, rather than explicitly mark it as a
measured field, because it's a derived field and to preserve 0.8
behavior where total ticks aren't explicitly marked as a measured field.
2019-03-29 22:34:28 -04:00
Mahmood Ali
7299e0015a Merge pull request #5428 from hashicorp/b-dropped-logs-on-task-restart
client/logmon: restart log collection correctly when a task is restarted
2019-03-21 14:02:08 -04:00
Nick Ethier
cc12c8443f logmon: fix logmon handling in driver test harness 2019-03-20 21:14:08 -04:00
Mahmood Ali
eb5ab38ae5 Regenerate Proto files (#5421)
Noticed that the protobuf files are out of sync with ones generated by 1.2.0 protoc go plugin.

The cause for these files seem to be related to release processes, e.g. [0.9.0-beta1 preperation](ecec3d38de (diff-da4da188ee496377d456025c2eab4e87)), and [0.9.0-beta3 preperation](b849d84f2f).

This restores the changes to that of the pinned protoc version and fails build if protobuf files are out of sync.  Sample failing Travis job is that of the first commit change: https://travis-ci.org/hashicorp/nomad/jobs/506285085
2019-03-14 10:56:27 -04:00
Michael Schurter
b849d84f2f Generate files for 0.9.0-beta3 release 2019-02-26 09:44:49 -08:00
Michael Schurter
c9fe5d26b3 plugins: squelch context Canceled error logs
As far as I can tell this is the most straightforward and resilient way
to skip error logging on context cancellation with grpc streams. You
cannot compare the error against context.Canceled directly as it is of
type `*status.statusError`. The next best solution I found was:

```go
resp, err := stream.Recv()
if code, ok := err.(interface{ Code() code.Code }); ok {
	if code.Code == code.Canceled {
		return
	}
}
```

However I think checking ctx.Err() directly makes the code much easier
to read and is resilient against grpc API changes.
2019-02-21 15:32:18 -08:00
Michael Schurter
1d17fbc681 simplify hcl2 parsing helper
No need to pass in the entire eval context
2019-02-04 11:07:57 -08:00
Alex Dadgar
ecec3d38de Nomad 0.9.0-beta1 generated code 2019-01-30 10:49:44 -08:00
Nick Ethier
8aa451806b executor: fix bug and add tests for incorrect stats timestamp reporting 2019-01-28 21:57:45 -05:00
Mahmood Ali
b70442a325 drivers: pass logger through driver plugin client
This fixes a panic whenever driver plugin attempts to log a message.
2019-01-25 09:38:41 -05:00
Nick Ethier
f38612c3b3 plugins/drivers: change stats interval to duration type in proto 2019-01-24 22:19:18 -05:00
Mahmood Ali
4781f1dd7b Merge pull request #5229 from hashicorp/r-grabbag-201901019
Grab bag of small changes
2019-01-23 13:06:51 -05:00
Mahmood Ali
cabe5e90f2 spell check 2019-01-23 10:54:52 -05:00
Michael Schurter
158c74887e goimports until make check is happy 2019-01-23 06:27:14 -08:00
Michael Schurter
0d61ff0fb9 move pluginutils -> helper/pluginutils
I wanted a different color bikeshed, so I get to paint it
2019-01-22 15:50:08 -08:00
Alex Dadgar
95297c608c goimports 2019-01-22 15:44:31 -08:00
Alex Dadgar
fe2fa21a7d gofmt 2019-01-22 15:43:34 -08:00
Alex Dadgar
72f8851254 Split hclspec 2019-01-22 15:43:34 -08:00
Alex Dadgar
8be7057177 move hclutils 2019-01-22 15:43:34 -08:00
Alex Dadgar
e46d67a889 Driver tests do not use hcl2/hcl, hclspec, or hclutils 2019-01-22 15:43:34 -08:00
Alex Dadgar
2d23f4a038 move reattach config 2019-01-22 15:11:58 -08:00
Alex Dadgar
c19cd2e5cf loader and singleton 2019-01-22 15:11:57 -08:00
Alex Dadgar
b9f36134dc move catalog + grpcutils 2019-01-22 15:11:57 -08:00
Alex Dadgar
7e81a9e3c4 increase log level 2019-01-22 14:20:54 -08:00
Nick Ethier
f3ab65947e drivers: regen proto 2019-01-18 18:53:45 -05:00
Nick Ethier
994c66f7d7 drivers: use consts for task handle version 2019-01-18 18:31:01 -05:00
Nick Ethier
7c440aa437 drivers: fix test 2019-01-18 18:31:01 -05:00
Nick Ethier
2f91ac88f7 cleanup code comments and small fixes from refactor 2019-01-18 18:31:01 -05:00
Nick Ethier
07cdedec2f driver: add pre09 migration logic 2019-01-18 18:31:01 -05:00
Nick Ethier
2118d94b2d executor: add pre 0.9 client and wrapper 2019-01-18 18:30:58 -05:00
Mahmood Ali
9f7619344e Merge pull request #5190 from hashicorp/f-memory-usage
Track Basic Memory Usage as reported by cgroups
2019-01-18 16:46:02 -05:00
Chris Baker
0502f255c7 renamed mispelled function, fixed incorrect comment 2019-01-18 20:00:49 +00:00
Danielle Tomlinson
9b3456011a Merge pull request #5204 from hashicorp/dani/loader-typo-2
Fix typo in PluginLoader
2019-01-18 11:22:22 +01:00
Danielle Tomlinson
1d5c04ee1d pluginloader: typo: s/validePluginConfig/validatePluginConfig 2019-01-17 19:10:32 +01:00
Danielle Tomlinson
8ffd94ca55 plugins: Require an exe extension on windows 2019-01-17 18:43:14 +01:00
Danielle Tomlinson
3287c3f019 chore: General Cleanup 2019-01-17 18:43:14 +01:00