From 170b322d03acaa8efc60c364f7c514e4dfe71fe2 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 12 Apr 2018 11:42:23 -0700 Subject: [PATCH 1/5] docs: add drain upgrade guide --- .../source/docs/upgrade/upgrade-specific.html.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/website/source/docs/upgrade/upgrade-specific.html.md b/website/source/docs/upgrade/upgrade-specific.html.md index dad0032eb..c31aaacc4 100644 --- a/website/source/docs/upgrade/upgrade-specific.html.md +++ b/website/source/docs/upgrade/upgrade-specific.html.md @@ -51,6 +51,16 @@ Raft Protocol versions supported by each Consul version: In order to enable all [Autopilot](/guides/cluster/autopilot.html) features, all servers in a Nomad cluster must be running with Raft protocol version 3 or later. +### Node Draining Improvements + +Node draining via the [`node drain`][drain-cli] command or the [drain +API][drain-api] has been substantially changed in Nomad 0.8. In Nomad 0.7.1 and +earlier draining a node would immediately stop all allocations on the node +being drained. Nomad 0.8 now supports a [`migrate`][migrate] stanza in job +specifications to control how many allocations may be migrated at once. + +See the [Decommissioning Nodes guide](/guides/node-draining.html) for details. + ### Periods in Environment Variable Names No Longer Escaped *Applications which expect periods in environment variable names to be replaced @@ -210,3 +220,7 @@ draining the node so no tasks are running on it. This can be verified by running `nomad node status ` and verify there are no tasks in the `running` state. Once that is done the client can be killed, the `data_dir` should be deleted and then Nomad 0.3.0 can be launched. + +[drain-api]: /api/nodes.html#drain-node +[drain-cli]: /docs/commands/node/drain.html +[migrate]: /docs/job-specification/migrate.html From 06a5dd9087cf3f39d22aaeea6c6d75c3a06ba81d Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 12 Apr 2018 15:39:46 -0700 Subject: [PATCH 2/5] docs: add default migrate stanza to upgrade docs --- .../source/docs/upgrade/upgrade-specific.html.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/website/source/docs/upgrade/upgrade-specific.html.md b/website/source/docs/upgrade/upgrade-specific.html.md index c31aaacc4..9a06024b4 100644 --- a/website/source/docs/upgrade/upgrade-specific.html.md +++ b/website/source/docs/upgrade/upgrade-specific.html.md @@ -57,9 +57,20 @@ Node draining via the [`node drain`][drain-cli] command or the [drain API][drain-api] has been substantially changed in Nomad 0.8. In Nomad 0.7.1 and earlier draining a node would immediately stop all allocations on the node being drained. Nomad 0.8 now supports a [`migrate`][migrate] stanza in job -specifications to control how many allocations may be migrated at once. +specifications to control how many allocations may be migrated at once. For +service jobs with a `count > 1` the default migrate stanza will be used: -See the [Decommissioning Nodes guide](/guides/node-draining.html) for details. +```hcl + migrate { + max_parallel = 1 + health_check = "checks" + min_healthy_time = "10s" + healthy_deadline = "5m" + } +``` + +See the [`migrate` stanza documentation][migrate] and [Decommissioning Nodes +guide](/guides/node-draining.html) for details. ### Periods in Environment Variable Names No Longer Escaped From 9c446185de178ff4e98cba67347fa48cf01549a6 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 12 Apr 2018 15:54:21 -0700 Subject: [PATCH 3/5] docs: compare old and new drain methods --- website/source/docs/upgrade/upgrade-specific.html.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/website/source/docs/upgrade/upgrade-specific.html.md b/website/source/docs/upgrade/upgrade-specific.html.md index 9a06024b4..726911c7d 100644 --- a/website/source/docs/upgrade/upgrade-specific.html.md +++ b/website/source/docs/upgrade/upgrade-specific.html.md @@ -69,6 +69,12 @@ service jobs with a `count > 1` the default migrate stanza will be used: } ``` +The old drain behavior made it impossible to drain multiple nodes at once +without risking service outages and lots of churn as draining allocations could +be placed on nodes that are about to be drained. Nomad 0.8's drain behavior is +intended to fix these issues and offer both job authors and cluster operators +more control over drains. + See the [`migrate` stanza documentation][migrate] and [Decommissioning Nodes guide](/guides/node-draining.html) for details. From c9aaab2ada78b1dd0ecc4be397b1572f6ba40ab3 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 12 Apr 2018 16:24:57 -0700 Subject: [PATCH 4/5] docs: only mention what changed with drain --- .../docs/upgrade/upgrade-specific.html.md | 21 +++++-------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/website/source/docs/upgrade/upgrade-specific.html.md b/website/source/docs/upgrade/upgrade-specific.html.md index 726911c7d..836c8166b 100644 --- a/website/source/docs/upgrade/upgrade-specific.html.md +++ b/website/source/docs/upgrade/upgrade-specific.html.md @@ -57,23 +57,12 @@ Node draining via the [`node drain`][drain-cli] command or the [drain API][drain-api] has been substantially changed in Nomad 0.8. In Nomad 0.7.1 and earlier draining a node would immediately stop all allocations on the node being drained. Nomad 0.8 now supports a [`migrate`][migrate] stanza in job -specifications to control how many allocations may be migrated at once. For -service jobs with a `count > 1` the default migrate stanza will be used: +specifications to control how many allocations may be migrated at once and the +default will be used for existing jobs. -```hcl - migrate { - max_parallel = 1 - health_check = "checks" - min_healthy_time = "10s" - healthy_deadline = "5m" - } -``` - -The old drain behavior made it impossible to drain multiple nodes at once -without risking service outages and lots of churn as draining allocations could -be placed on nodes that are about to be drained. Nomad 0.8's drain behavior is -intended to fix these issues and offer both job authors and cluster operators -more control over drains. +The `drain` command now blocks until the drain completes. To get the Nomad +0.7.1 and earlier drain behavior use the command: `nomad node drain -enable +-force -detach ` See the [`migrate` stanza documentation][migrate] and [Decommissioning Nodes guide](/guides/node-draining.html) for details. From 58ad3e84614fd545bf7c7a3de1a916d8133aad20 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 12 Apr 2018 16:34:55 -0700 Subject: [PATCH 5/5] docs: link upgrade docs to node drain guide --- website/source/docs/upgrade/index.html.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/website/source/docs/upgrade/index.html.md b/website/source/docs/upgrade/index.html.md index d45c8aee1..758f20a25 100644 --- a/website/source/docs/upgrade/index.html.md +++ b/website/source/docs/upgrade/index.html.md @@ -101,11 +101,11 @@ Use the same actions in step #2 above to confirm cluster health. ### 5. Upgrade clients Following the successful upgrade of the servers you can now update your -clients using a similar process as the servers. If you wish to gracefully -move tasks on a client use the `nomad node drain ` command to -gracefully migrate jobs to another client in the cluster. The `node drain` -command prevents new tasks from being allocated to the client and begins -migrating existing allocations to another client. +clients using a similar process as the servers. You may either upgrade clients +in-place or start new nodes on the new version. See the [Decommissioning Nodes +guide](/guides/node-draining.html) for instructions on how to migrate running +allocations from the old nodes to the new nodes with the [`nomad node +drain`](/docs/commands/node/drain.html) command. ## Done!