Michael Schurter
4a70644df0
drain: fix node drain monitoring
...
The whole approach to monitoring drains has ordering issues and lacks
state to output useful error messages.
AFAICT to get the tests passing reliably I needed to change the behavior
of monitoring.
Parts of these tests are skipped in CI, and they should be rewritten as
e2e tests.
2019-01-08 09:35:16 -08:00
Mahmood Ali
40164b3dc6
Use max 3 precision in displaying floats
...
When formating floats in `nomad node status`, use a maximum precision of
3.
2018-12-10 12:18:24 -05:00
Mahmood Ali
a802984c9a
device attributes in nomad node status -verbose
...
This reports device attributes like the following:
```
$ nomad node status -self -verbose
ID = f7adb958-29e1-2a5a-2303-9d61ffaab33a
Name = mars.local
Class = <none>
DC = dc1
Drain = false
Eligibility = eligible
Status = ready
Uptime = 12h40m13s
Drivers
Driver Detected Healthy Message Time
docker true true healthy 2018-12-10T11:47:19-05:00
...
Attributes
cpu.arch = amd64
cpu.frequency = 2200
cpu.modelname = Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
cpu.numcores = 12
...
Device Group Attributes
Device Group = nomad/file/mock
block_device = sda1
filesystem = ext4
size = 63.2 GB
Meta
```
2018-12-10 12:18:24 -05:00
Mahmood Ali
276149eaa7
device stats summary in node status
...
Sample output with a mock device:
```
Host Resource Utilization
CPU Memory Disk
2651/26400 MHz 9.6 GiB/16 GiB 98 GiB/234 GiB
Device Resource Utilization
nomad/file/mock[README.md] 511 bytes
nomad/file/mock[e2e.go] 239 bytes
nomad/file/mock[e2e_test.go] 128 bytes
Allocations
No allocations placed
```
2018-11-14 22:13:23 -05:00
Mahmood Ali
c212716dda
Add NodeResource Device types in api package
2018-11-14 14:42:36 -05:00
Mahmood Ali
ba3fe15f7e
Add Client Device Stats structs in api package
2018-11-14 14:41:19 -05:00
Alex Dadgar
262ccbcc10
rename api TotalShares -> CpuShares
2018-10-16 17:25:55 -07:00
Alex Dadgar
5e67b37aad
use int64
2018-10-16 15:34:32 -07:00
Preetha Appan
3ca71ae935
Change CPU/Disk/MemoryMB to int everywhere in new resource structs
2018-10-16 16:21:42 -05:00
Alex Dadgar
f969298854
Node reserved resources
2018-09-29 18:44:55 -07:00
Alex Dadgar
b310a54aa6
Node resources on client
2018-09-29 17:23:41 -07:00
Alex Dadgar
3402f2539f
code review
2018-06-06 14:52:26 -07:00
Alex Dadgar
3c9ad243a4
Monitoring non-draining node exits
2018-06-05 17:58:44 -07:00
Preetha Appan
bcf34c3449
Set node eligibility to true when old client calls disable
2018-05-30 16:54:07 -05:00
Nick Ethier
faeb612bd9
api: emit different monitor message if node's drain strategy is never set
2018-05-24 06:39:09 -04:00
Chelsea Holland Komlo
eab7a1f7d3
Add driver health information to node list stub
2018-05-09 11:21:54 -04:00
James Rasell
805b5278e7
Update node-drain logging message to clearer for operators.
...
This change updates the console log message when performing a node
drain and particulary when a node has marked all allocs for
migration. Previously it logged 'drain complete' which was a little
confusing to operators as the node is not drained at this point.
Closes #4183
2018-04-24 07:50:01 +01:00
Michael Schurter
2c1f24c1ce
cli: remove unreachable drain message
2018-03-30 14:15:12 -07:00
Michael Schurter
6dd601791c
cli: log if a node goes down during draining
2018-03-30 14:02:42 -07:00
Michael Schurter
482f0b9fc7
drain: fix monitor node index handling
2018-03-30 12:43:53 -07:00
Michael Schurter
7ff790bc7d
cli: differentiate normal output vs info
2018-03-30 11:42:11 -07:00
Michael Schurter
55b98ee299
cli: add color to drain output
2018-03-30 11:15:12 -07:00
Michael Schurter
efbaf2cc2d
drain: fix cli blocking when allocs already stopped
2018-03-30 10:18:14 -07:00
Michael Schurter
4cefb6f06a
drain: block cli until all allocs stop
...
Before the drain CLI would block until the node was marked as completing
drain operations. While technically correct, it could lead operators (or
more likely: scripts) to shutdown drained nodes before all of its
allocations had *actually* terminated.
This change makes the CLI block until all allocations have terminated
(unless ignoring system jobs).
2018-03-29 10:56:09 -07:00
Alex Dadgar
400cbcf128
Create evals for system jobs when drain is unset
...
This PR creates evals for system jobs when:
* Drain is unset and mark eligible is true
* Eligibility is restored to the node
2018-03-27 15:53:24 -07:00
Chelsea Holland Komlo
041786360e
use time.Time for node events for compatibility
2018-03-27 15:43:57 -04:00
Michael Schurter
3496bcf766
docs: improve DrainRequest.MarkEligible comment
2018-03-21 16:55:22 -07:00
Alex Dadgar
45e7e88558
Fix deadline handling
2018-03-21 16:51:44 -07:00
Alex Dadgar
d15371405f
Toggle Drain allows resetting eligibility
...
This PR allows marking a node as eligible for scheduling while toggling
drain. By default the `nomad node drain -disable` commmand will mark it
as eligible but the drainer will maintain in-eligibility.
2018-03-21 16:51:44 -07:00
Alex Dadgar
3fe3c6eff7
Improve DeadlineTime helper
2018-03-21 16:51:44 -07:00
Alex Dadgar
d6399cb733
Add eligibility to node view
2018-03-21 16:51:44 -07:00
Alex Dadgar
0fb9ba7732
HTTP and API
2018-03-21 16:51:44 -07:00
Alex Dadgar
2bdeacebff
Drain cli, api, http
2018-03-21 16:51:43 -07:00
Chelsea Holland Komlo
1570972cb3
add concept of health checks to fingerprinters and nodes
...
fix up feedback from code review
add driver info for all drivers to node
2018-03-21 15:15:25 -04:00
Alex Dadgar
8ba5ba33de
small cleanup
2018-03-13 18:08:22 -07:00
Alex Dadgar
eceabb66f8
nodeevents -> events
2018-03-13 18:08:22 -07:00
Chelsea Holland Komlo
705ff980ad
code review feedback
2018-03-13 18:05:40 -07:00
Chelsea Holland Komlo
fc819754a4
Ensure node updates don't strip node events
...
Add node events to CLI
2018-03-13 18:05:40 -07:00
Alex Dadgar
4f2725bb98
Enhance API pkg to utilize Server's Client Tunnel
...
This PR enhances the API package by having client only RPCs route
through the server when they are low cost and for filesystem access to
first attempt a direct connection to the node and then falling back to
a server routed request.
2018-02-15 13:59:03 -08:00
James Rasell
3868e88518
Update node-status verbose command to include node address.
...
This change updates the `nomad node-status -verbose` command to
also include the addreess of the node. This is helpful for cluster
administrators to quickly discover information and access nodes
when required.
2017-12-21 08:58:35 +00:00
Alex Dadgar
a9e3a41407
Enable more linters
2017-09-26 15:26:33 -07:00
Michael Schurter
2f30827205
Refactor GetNodeClient weirdness
...
- No need to for a pointer to a pointer
- Properly set and use QueryOptions.Region
2017-08-28 14:41:21 -07:00
Michael Schurter
1bb8a54ffb
Fix TLS support in api pkg / cli
...
Fixes #3013
It's a little weird that Client now has a method for returning a
NewClient, but it's a convenient way to dedupe the logic to
connect-directly-to-a-node which is nontrivial and had sutble
differences between locations.
2017-08-28 11:46:28 -07:00
James Rasell
fe481ae0dd
Updates based on feedback provided by dadgar.
2017-08-16 22:19:31 +01:00
“James
113e9b0c33
Add the Nomad agent version to the node-status CLI putput.
2017-08-10 08:27:26 +01:00
Diptanu Choudhury
fca8c2eb9a
Added the API for GC of allocations and nodes
2017-01-12 16:18:29 -08:00
Diptanu Choudhury
b2bbd63d87
Fixed a bunch of TLS related failures
2016-10-26 14:08:46 -07:00
Diptanu Choudhury
722433b4f9
Making the cli use TLS if the client has enabled TLS
2016-10-26 11:13:53 -07:00
Diptanu Choudhury
3ef112b30e
Added the status updated at timestamp
2016-07-27 11:50:06 -07:00
Alex Dadgar
020f8b05d3
only support latest and remove ring buffer
2016-06-12 09:32:38 -07:00
