Commit Graph

1698 Commits

Author SHA1 Message Date
Mahmood Ali
ec3650fe17 Merge pull request #9142 from hashicorp/f-hclv2-2.3
Support HCLv2 for Nomad jobs
2020-10-22 12:26:28 -05:00
Drew Bailey
3347b40d11 remove event durability (#9147)
* remove event durability

temporarily removing go-memdb event durability until a new strategy is developed on how to best handled increased durability needs

* drop events table schema and state store methods

* fix neweventbuffer invocations
2020-10-22 12:21:03 -04:00
Mahmood Ali
9fe7423403 api: update /render api to parse hclv2 2020-10-21 15:46:57 -04:00
Drew Bailey
7ce0b5017c Events/msgtype cleanup (#9117)
* use msgtype in upsert node

adds message type to signature for upsert node, update tests, remove placeholder method

* UpsertAllocs msg type test setup

* use upsertallocs with msg type in signature

update test usage of delete node

delete placeholder msgtype method

* add msgtype to upsert evals signature, update test call sites with test setup msg type

handle snapshot upsert eval outside of FSM and ignore eval event

remove placeholder upsertevalsmsgtype

handle job plan rpc and prevent event creation for plan

msgtype cleanup upsertnodeevents

updatenodedrain msgtype

msg type 0 is a node registration event, so set the default  to the ignore type

* fix named import

* fix signature ordering on upsertnode to match
2020-10-19 09:30:15 -04:00
Drew Bailey
baf7260eaf event buffer size and durable count must be non negative 2020-10-15 16:34:33 -04:00
Nick Ethier
7b50685cf7 Consul with CNI and host_network addresses (#9095)
* consul: advertise cni and multi host interface addresses

* structs: add service/check address_mode validation

* ar/groupservices: fetch networkstatus at hook runtime

* ar/groupservice: nil check network status getter before calling

* consul: comment network status can be nil
2020-10-15 15:32:21 -04:00
Michael Schurter
b3c8deeac4 Merge pull request #9094 from hashicorp/f-1.0
s/0.13/1.0/g
2020-10-15 08:53:33 -07:00
James Rasell
1cfeeba9f2 Merge pull request #9083 from hashicorp/b-fix-enterprise-config-merge
agent: fix enterprise config overlay merging.
2020-10-15 08:40:49 +02:00
Michael Schurter
f44c04ecd1 s/0.13/1.0/g
1.0 here we come!
2020-10-14 15:17:47 -07:00
Michael Schurter
116b2b8b35 Merge pull request #9055 from hashicorp/f-9017-resources
api: add field filters to /v1/{allocations,nodes}
2020-10-14 14:49:39 -07:00
Michael Schurter
0695801256 unify boolean parameter parsing 2020-10-14 12:23:25 -07:00
Drew Bailey
3c15f41411 filter on additional filter keys, remove switch statement duplication
properly wire up durable event count

move newline responsibility

moves newline creation from NDJson to the http handler, json stream only encodes and sends now

ignore snapshot restore if broker is disabled

enable dev mode to access event steam without acl

use mapping instead of switch

use pointers for config sizes, remove unused ttl, simplify closed conn logic
2020-10-14 14:14:33 -04:00
Michael Schurter
a55f46e9ba api: add field filters to /v1/{allocations,nodes}
Fixes #9017

The ?resources=true query parameter includes resources in the object
stub listings. Specifically:

- For `/v1/nodes?resources=true` both the `NodeResources` and
  `ReservedResources` field are included.
- For `/v1/allocations?resources=true` the `AllocatedResources` field is
  included.

The ?task_states=false query parameter removes TaskStates from
/v1/allocations responses. (By default TaskStates are included.)
2020-10-14 10:35:22 -07:00
Drew Bailey
71028207db namespace filtering 2020-10-14 12:44:43 -04:00
Drew Bailey
39ef3263ca Add EvictCallbackFn to handle removing entries from go-memdb when they
are removed from the event buffer.

Wire up event buffer size config, use pointers for structs.Events
instead of copying.
2020-10-14 12:44:42 -04:00
Drew Bailey
1288b18b27 rehydrate event publisher on snapshot restore
address pr feedback
2020-10-14 12:44:41 -04:00
Drew Bailey
8a3130a356 event durability count and cfg 2020-10-14 12:44:40 -04:00
Drew Bailey
004f634868 use Events to wrap index and events, store in events table 2020-10-14 12:44:39 -04:00
Drew Bailey
e7e6df394f wire up enable_event_publisher 2020-10-14 12:44:38 -04:00
Drew Bailey
8d16846695 Events/deployment events (#9004)
* Node Drain events and Node Events (#8980)

Deployment status updates

handle deployment status updates (paused, failed, resume)

deployment alloc health

generate events from apply plan result

txn err check, slim down deployment event

one ndjson line per index

* consolidate down to node event + type

* fix UpdateDeploymentAllocHealth test invocations

* fix test
2020-10-14 12:44:37 -04:00
Drew Bailey
b825ba3bf1 Event Stream API/RPC (#8947)
This Commit adds an /v1/events/stream endpoint to stream events from.

The stream framer has been updated to include a SendFull method which
does not fragment the data between multiple frames. This essentially
treats the stream framer as a envelope to adhere to the stream framer
interface in the UI.

If the `encode` query parameter is omitted events will be streamed as
newline delimted JSON.
2020-10-14 12:44:36 -04:00
James Rasell
d89b09c7e2 agent: fix enterprise config overlay merging. 2020-10-14 09:35:16 +02:00
Chris Baker
797543ad4b removed backwards-compatible/untagged metrics deprecated in 0.7 2020-10-13 20:18:39 +00:00
Seth Hoenig
bdeb73cd2c consul/connect: dynamically select envoy sidecar at runtime
As newer versions of Consul are released, the minimum version of Envoy
it supports as a sidecar proxy also gets bumped. Starting with the upcoming
Consul v1.9.X series, Envoy v1.11.X will no longer be supported. Current
versions of Nomad hardcode a version of Envoy v1.11.2 to be used as the
default implementation of Connect sidecar proxy.

This PR introduces a change such that each Nomad Client will query its
local Consul for a list of Envoy proxies that it supports (https://github.com/hashicorp/consul/pull/8545)
and then launch the Connect sidecar proxy task using the latest supported version
of Envoy. If the `SupportedProxies` API component is not available from
Consul, Nomad will fallback to the old version of Envoy supported by old
versions of Consul.

Setting the meta configuration option `meta.connect.sidecar_image` or
setting the `connect.sidecar_task` stanza will take precedence as is
the current behavior for sidecar proxies.

Setting the meta configuration option `meta.connect.gateway_image`
will take precedence as is the current behavior for connect gateways.

`meta.connect.sidecar_image` and `meta.connect.gateway_image` may make
use of the special `${NOMAD_envoy_version}` variable interpolation, which
resolves to the newest version of Envoy supported by the Consul agent.

Addresses #8585 #7665
2020-10-13 09:14:12 -05:00
Yoan Blanc
c14c616194 use allow/deny instead of the colored alternatives (#9019)
Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2020-10-12 08:47:05 -04:00
Seth Hoenig
d538559c60 Merge pull request #9033 from pierreca/verify-remove-checks
Do not double-remove checks removed by Consul
2020-10-06 10:16:13 -05:00
James Rasell
87bfe0e558 Merge pull request #9027 from hashicorp/f-gh-9026
cli: move tests to use NewMockUi func.
2020-10-06 08:28:18 +02:00
Pierre Cauchois
6c5d2c3d85 Do not double-remove checks removed by Consul
When deregistering a service, consul also deregisters the associated
checks. The current state keeps track of all services and all checks
separately and deregisters them in sequence, which leads, whether during
syncs or shutdowns, to check deregistrations happening twice and failing
the second time (generating errors in logs)

This fix includes:
- a fix to the sync logic that just pulls the checks *after* the
services have been synced
- a fix to the shutdown mechanism that gets an updated list of checks
after deregistering the services, so that we get a cleaner check
deregistration process.
2020-10-06 00:30:29 +00:00
Chris Baker
b0c2e5176a updated docs and validation to further prohibit null chars in region, datacenter, and job name 2020-10-05 18:01:50 +00:00
James Rasell
f5567b3261 cli: move tests to use NewMockUi func. 2020-10-05 16:07:41 +02:00
Kent 'picat' Gruber
ea397cb042 Merge pull request #8998 from hashicorp/keygen-32-bytes
Use 32-byte key for gossip encryption to enable AES-256
2020-10-02 17:17:55 -04:00
Kent 'picat' Gruber
a38e33a9c8 Fix panic in test due to the agent's logger not being initialized yet
So a null logger is used to avoid the problem.
2020-10-02 11:10:27 -04:00
Fredrik Hoem Grelland
8238b9f864 update consul-template to v0.25.1 (#8988) 2020-10-01 14:08:49 -04:00
Kent 'picat' Gruber
614729bd4c Fix other usages of initKeyring func to use logger as third argument 2020-10-01 11:13:06 -04:00
Kent 'picat' Gruber
2ae72441b5 Log AES-128 and AES-192 key sizes during keyring initialization 2020-10-01 11:12:14 -04:00
Michael Schurter
902b0b5673 jobspec: lower min cpu resources from 10->1
Since CPU resources are usually a soft limit it is desirable to allow
setting it as low as possible to allow tasks to run only in "idle" time.

Setting it to 0 is still not allowed to avoid potential unintentional
side effects with allowing a zero value. While there may not be any side
effects this commit attempts to minimize risk by avoiding the issue.

This does *not* change the defaults.
2020-09-30 12:15:13 -07:00
Michael Schurter
97d516701c Merge pull request #8862 from hashicorp/release-0.12.4
Prepare for 0.13 development cycle
2020-09-10 09:14:44 -07:00
Mahmood Ali
d7f14d12c1 Upgrade to golang 1.15 (#8858)
Upgrade to golang 1.15

Starting with golang 1.5, setting Ctty value result in `Setctty set but Ctty not valid in child` error, as part of https://github.com/golang/go/issues/29458 .
This commit lifts the fix in https://github.com/creack/pty/pull/97 .
2020-09-09 15:59:29 -04:00
Nomad Release bot
77f3b0cd74 Generate files for 0.12.4-rc1 release 2020-09-03 02:59:23 +00:00
Tim Gross
5a01ab312d MRD: move 'job stop -global' handling into RPC (#8776)
The initial implementation of global job stop for MRD looped over all the
regions in the CLI for expedience. This changeset includes the OSS parts of
moving this into the RPC layer so that API consumers don't have to implement
this logic themselves.
2020-08-28 14:28:13 -04:00
Lang Martin
dd7016b847 csi: plugins track jobs in addition to allocations, and use job information to set expected counts (#8699)
* nomad/structs/csi: add explicit job support
* nomad/state/state_store: capture job updates directly
* api/nodes: CSIInfo needs the AllocID
* command/agent/csi_endpoint: AllocID was missing
Co-authored-by: Tim Gross <tgross@hashicorp.com>
2020-08-27 17:20:00 -04:00
Seth Hoenig
a46ba7f4ac Merge branch 'master' into f-cc-ingress 2020-08-26 15:31:05 -05:00
Seth Hoenig
abd38b3a86 consul/connect: fixup some comments and context timeout 2020-08-26 13:17:16 -05:00
Tim Gross
d6ad3be8bd csi: fix panic in serializing nil allocs in volume API (#8735)
- fix panic in serializing nil allocs in volume API
- prevent potential panic in serializing plugin allocs
2020-08-25 10:13:05 -04:00
Seth Hoenig
db8020f4eb consul/connect: fixup tests to use new consul sdk 2020-08-24 12:02:41 -05:00
Seth Hoenig
36a743f19d consul/connect: remove envoy dns option from gateway proxy config 2020-08-24 09:11:55 -05:00
Yoan Blanc
e1ee6a45b1 fixup! vendor: consul/api, consul/sdk v1.6.0
Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2020-08-24 08:59:03 +02:00
Seth Hoenig
9ffdeed904 consul/connect: add initial support for ingress gateways
This PR adds initial support for running Consul Connect Ingress Gateways (CIGs) in Nomad. These gateways are declared as part of a task group level service definition within the connect stanza.

```hcl
service {
  connect {
    gateway {
      proxy {
        // envoy proxy configuration
      }
      ingress {
        // ingress-gateway configuration entry
      }
    }
  }
}
```

A gateway can be run in `bridge` or `host` networking mode, with the caveat that host networking necessitates manually specifying the Envoy admin listener (which cannot be disabled) via the service port value.

Currently Envoy is the only supported gateway implementation in Consul, and Nomad only supports running Envoy as a gateway using the docker driver.

Aims to address #8294 and tangentially #8647
2020-08-21 16:21:54 -05:00
Nick Ethier
2be03bc0d9 Update UI to use new allocated ports fields (#8631)
* nomad: canonicalize alloc shared resources to populate ports

* ui: network ports

* ui: remove unused task network references and update tests with new shared ports model

* ui: lint

* ui: revert auto formatting

* ui: remove unused page objects

* structs: remove unrelated test from bad conflict resolution

* ui: formatting
2020-08-20 11:07:13 -04:00
Tim Gross
a8b144cb91 mrd: remove redundant validation in HTTP endpoint (#8685)
The `regionForJob` function in the HTTP job endpoint overrides the region for
multiregion jobs to `global`, which is used as a sentinel value in the
server's job endpoint to avoid re-registration loops. This changeset removes
an extraneous check that results in errors in the web UI and makes
round-tripping through the HTTP API cumbersome for all consumers.
2020-08-18 16:48:09 -04:00