kemko/nomad
1
0
Fork 0
mirror of https://github.com/kemko/nomad.git synced 2026-01-07 10:55:42 +03:00
Code Issues Packages Projects Releases Wiki Activity

add a section about memory oversubscription (#10573)

Browse Source 
add a section about memory oversubscription

Co-authored-by: Tim Gross <tgross@hashicorp.com>
This commit is contained in:
Mahmood Ali
2021-05-13 13:35:51 -04:00
committed by GitHub
parent a9ec864a1a
commit 02e585a611
1 changed files with 47 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
website/content/docs/job-specification/resources.mdx
View File

@@ -39,7 +39,7 @@ job "docs" {
- `memory` `(int: 300)` - Specifies the memory required in MB.
- `memory_max` <code>(`int`: &lt;optional&gt;)</code> <sup>1.1 Beta</sup> - Optionally, specifies the maximum memory the task may use, if the client has excess memory capacity, in MB.
- `memory_max` <code>(`int`: &lt;optional&gt;)</code> <sup>1.1 Beta</sup> - Optionally, specifies the maximum memory the task may use, if the client has excess memory capacity, in MB. See [Memory Oversubscription](#memory-oversubscription) for more details.
- `device` <code>([Device][]: &lt;optional&gt;)</code> - Specifies the device
requirements. This may be repeated to request multiple device types.
@@ -86,5 +86,51 @@ resources {
}
}
```
## Memory Oversubscription
Setting task memory limits requires balancing the risk of interrupting tasks
against the risk of wasting resources. If a task memory limit is set too low,
the task may exceed the limit and be interrupted; if the task memory is too
high, the cluster is left underutilized.
To help maximize cluster memory utilization while allowing a safety margin for
unexpected load spikes, Nomad 1.1. lets job authors set two separate memory
limits:
* `memory`: the reserve limit to represent the task’s typical memory usage —
this number is used by the Nomad scheduler to reserve and place the task
* `memory_max`: the maximum memory the task may use, if the client has excess
available memory, and may be terminated if it exceeds
If a client's memory becomes contended or low, the operating system will
pressure the running tasks to free up memory. If the contention persists, Nomad
may kill oversubscribed tasks and reschedule them to other clients. The exact
mechanism for memory pressure is specific to the task driver, operating system,
and application runtime.
The new max limit attribute is currently supported by the official `docker`,
`exec`, and `java` task drivers. Consult the documentation of
community-supported task drivers for their memory oversubscription support.
Memory oversubscription is opt-in. Nomad operators can enable [Memory Oversubscription in the scheduler
configuration](/api-docs/operator/scheduler#update-scheduler-configuration). Enterprise customers can use [Resource
Quotas](https://learn.hashicorp.com/tutorials/nomad/quotas) to limit the memory
oversubscription.
To avoid degrading the cluster experience, we recommend examining and monitoring
resource utilization and considering the following suggestions:
* Set `oom_score_adj` for Linux host services that aren't managed by Nomad, e.g.
Docker, logging services, and the Nomad agent itself. For Systemd services, you can use the [`OOMScoreAdj` field](https://github.com/hashicorp/nomad/blob/v1.0.0/dist/systemd/nomad.service#L25).
* Monitor hosts for memory utilization and set alerts on Out-Of-Memory errors
* Set the [client `reserved`](/docs/configuration/client#reserved) with enough
memory for host services that aren't managed by Nomad as well as a buffer
for the memory excess. For example, if the client reserved memory is 1GB,
the allocations on the host may exceed their soft memory limit by almost
1GB in aggregate before the memory becomes contended and allocations get
killed.
[device]: /docs/job-specification/device 'Nomad device Job Specification'