kemko/nomad
1
0
Fork 0
mirror of https://github.com/kemko/nomad.git synced 2026-01-06 18:35:44 +03:00
Code Issues Packages Projects Releases Wiki Activity

Device job stanza

Browse Source
This commit is contained in:
Alex Dadgar
2019-01-22 16:30:57 -08:00
parent 4781f1dd7b
commit a0a52f6198
5 changed files with 429 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
website/source/docs/devices/community.html.md Normal file
View File

@@ -0,0 +1,16 @@
---
layout: "docs"
page_title: "Drivers: Custom"
sidebar_current: "docs-devices-community"
description: |-
Create custom task drivers for Nomad.
---
# Custom Drivers
Nomad does not currently support pluggable task drivers, however the
interface that a task driver must implement is minimal. In the short term,
custom drivers can be implemented in Go and compiled into the binary,
however in the long term we plan to expose a plugin interface such that
task drivers can be dynamically registered without recompiling the Nomad binary.

24
website/source/docs/devices/index.html.md Normal file
View File

@@ -0,0 +1,24 @@
---
layout: "docs"
page_title: "Device Plugins"
sidebar_current: "docs-devices"
description: |-
Device Plugins are used to expose devices to tasks in Nomad.
---
# Device Plugins
Task drivers are used by Nomad clients to execute a task and provide resource
isolation. By having extensible task drivers, Nomad has the flexibility to
support a broad set of workloads across all major operating systems.
The list of supported task drivers is provided on the left of this page.
Each task driver documents the configuration available in a
[job specification](/docs/job-specification/index.html), the environments it
can be used in, and the resource isolation mechanisms available.
Nomad strives to mask the details of running a task from users and instead
provides a clean abstraction. It is possible for the same task to be executed
with different isolation levels depending on the client running the task.
The goal is to use the strictest isolation available and gracefully degrade
protections where necessary.

118
website/source/docs/devices/nvidia.html.md Normal file
View File

@@ -0,0 +1,118 @@
---
layout: "docs"
page_title: "Drivers: Raw Exec"
sidebar_current: "docs-devices-nvidia"
description: |-
The Raw Exec task driver simply fork/execs and provides no isolation.
---
# Raw Fork/Exec Driver
Name: `raw_exec`
The `raw_exec` driver is used to execute a command for a task without any
isolation. Further, the task is started as the same user as the Nomad process.
As such, it should be used with extreme care and is disabled by default.
## Task Configuration
```hcl
task "webservice" {
driver = "raw_exec"
config {
command = "my-binary"
args = ["-flag", "1"]
}
}
```
The `raw_exec` driver supports the following configuration in the job spec:
* `command` - The command to execute. Must be provided. If executing a binary
that exists on the host, the path must be absolute. If executing a binary that
is downloaded from an [`artifact`](/docs/job-specification/artifact.html), the
path can be relative from the allocations's root directory.
* `args` - (Optional) A list of arguments to the `command`. References
to environment variables or any [interpretable Nomad
variables](/docs/runtime/interpolation.html) will be interpreted before
launching the task.
## Examples
To run a binary present on the Node:
```
task "example" {
driver = "raw_exec"
config {
# When running a binary that exists on the host, the path must be absolute/
command = "/bin/sleep"
args = ["1"]
}
}
```
To execute a binary downloaded from an [`artifact`](/docs/job-specification/artifact.html):
```
task "example" {
driver = "raw_exec"
config {
command = "name-of-my-binary"
}
artifact {
source = "https://internal.file.server/name-of-my-binary"
options {
checksum = "sha256:abd123445ds4555555555"
}
}
}
```
## Client Requirements
The `raw_exec` driver can run on all supported operating systems. For security
reasons, it is disabled by default. To enable raw exec, the Nomad client
configuration must explicitly enable the `raw_exec` driver in the client's
[options](/docs/configuration/client.html#options):
```
client {
options = {
"driver.raw_exec.enable" = "1"
}
}
```
## Client Options
* `driver.raw_exec.enable` - Specifies whether the driver should be enabled or
disabled.
* `driver.raw_exec.no_cgroups` - Specifies whether the driver should not use
cgroups to manage the process group launched by the driver. By default,
cgroups are used to manage the process tree to ensure full cleanup of all
processes started by the task. The driver only uses cgroups when Nomad is
launched as root, on Linux and when cgroups are detected.
## Client Attributes
The `raw_exec` driver will set the following client attributes:
* `driver.raw_exec` - This will be set to "1", indicating the driver is available.
## Resource Isolation
The `raw_exec` driver provides no isolation.
If the launched process creates a new process group, it is possible that Nomad
will leak processes on shutdown unless the application forwards signals
properly. Nomad will not leak any processes if cgroups are being used to manage
the process tree. Cgroups are used on Linux when Nomad is being run with
appropriate priviledges, the cgroup system is mounted and the operator hasn't
disabled cgroups for the driver.

255
website/source/docs/job-specification/device.html.md Normal file
View File

@@ -0,0 +1,255 @@
---
layout: "docs"
page_title: "device Stanza - Job Specification"
sidebar_current: "docs-job-specification-device"
description: |-
The "device" stanza is used to require a certain device be made available
to the task.
---
# `device` Stanza
<table class="table table-bordered table-striped">
<tr>
<th width="120">Placement</th>
<td>
<code>job -> group -> task -> resources -> **device**</code>
</td>
</tr>
</table>
The `device` stanza is used to create both a scheduling and runtime requirement
that the given task has access to the specified devices. A device is a hardware
device that is attached to the node and may be made available to the task.
Examples are GPUs, FPGAs, and TPUs.
When a `device` stanza is added, Nomad will schedule the task onto a node that
contains the set of device(s) that meet the specified requirements. The `device` stanza
allows the operator to specify as little as just the type of device required,
such as `gpu`, all the way to specifying arbitrary constraints and affinities.
Once the scheduler has placed the allocation on a suitable node, the Nomad
Client will invoke the device plugin to retrieve information on how to mount the
device and what environment variables to expose. For more information on the
runtime environment, please consult the individual device plugin's documentation.
See the [device plugin's documentation][devices] for a list of supported devices.
```hcl
job "docs" {
group "example" {
task "server" {
resources {
device "nvidia/gpu" {
count = 2
constraint {
attribute = "${driver.attr.memory}"
operator = ">="
value = "2 GiB"
}
affinity {
attribute = "${driver.attr.memory}"
operator = ">="
value = "4 GiB"
weight = 75
}
}
}
}
}
}
```
In the above example, the task is requesting two GPUs, from the Nvidia vendor,
but is not specifying the specific model required. Instead it is placing a hard
constraint that the device has at least 2 GiB of memory and that it would prefer
to use GPUs that have at least 4 GiB. This examples shows how expressive the
`device` stanza can be.
~> Device supported is currently limited to Linux, and container based drivers
due to the ability to isolate devices to specific tasks.
## `device` Parameters
- `name` `(string: "")` - Specifies the device required. The following inputs
are valid:
* `<device_type>`: If a single value is given, it is assumed to be the device
type, such as "gpu", or "fpga".
* `<vendor>/<device_type>`: If two values are given separated by a `/`, the
given device type will be selected, constraining on the provided vendor.
Examples include "nvidia/gpu" or "amd/gpu".
* `<vendor>/<device_type>/<model>`: If three values are given separated by a `/`, the
given device type will be selected, constraining on the provided vendor, and
model name. Examples include "nvidia/gpu/1080ti" or "nvidia/gpu/2080ti".
- `count` `(int: 1)` - Specifies the number of instances of the given device
that are required.
- `constraint` <code>([Constraint][]: nil)</code> - Constraints to restrict
which devices are eligible. This can be provided multiple times to define
additional constraints. See below for available attributes.
- `affinity` <code>([Affinity][]: nil)</code> - Affinity to specify a preference
for which devices get selected. This can be provided multiple times to define
additional affinities. See below for available attributes.
## `device` Constraint and Affinity Attributes
The set of attributes available for use in a `constraint` or `affinity` are as
follows:
<table class="table table-bordered table-striped">
<tr>
<th>Variable</th>
<th>Description</th>
<th>Example Value</th>
</tr>
<tr>
<td><tt>${device.type}</tt></td>
<td>The type of device</td>
<td><tt>"gpu", "tpu", "fpga"</tt></td>
</tr>
<tr>
<td><tt>${device.vendor}</tt></td>
<td>The device's vendor</td>
<td><tt>"amd", "nvidia", "intel"</tt></td>
</tr>
<tr>
<td><tt>${device.model}</tt></td>
<td>The device's model</td>
<td><tt>"1080ti"</tt></td>
</tr>
<tr>
<td><tt>${device.attr.&lt;property&gt;}</tt></td>
<td>Property of the device</td>
<td><tt>${device.attr.memory} => 8 GiB</tt></td>
</tr>
</table>
For the set of attributes available, please see the individual [device plugin's
documentation][devices].
### Attribute Units and Conversions
Devices report their attributes with strict types and can also provide unit
information. For example, when a GPU is reporting its memory, it can report that
it is "4096 MiB". Since Nomad has the associated unit information, a constraint
that requires greater than "3.5 GiB" can match since Nomad can convert between
these units.
The units Nomad supports is as follows:
<table class="table table-bordered table-striped">
<tr>
<th>Base Unit</th>
<th>Values</th>
</tr>
<tr>
<td><tt>Byte</tt></td>
<td><tt>**Base 2**: KiB, MiB, GiB, TiB, PiB, EiB<br>**Base 10**: kB, KB (equivalent to kB), MB, GB, TB, PB, EB</tt>
</tr>
<tr>
<td><tt>Byte Rates</tt></td>
<td><tt>**Base 2**: KiB/s, MiB/s, GiB/s, TiB/s, PiB/s, EiB/s<br>**Base 10**: kB/s, KB/s (equivalent to kB/s), MB/s, GB/s, TB/s, PB/s, EB/s</tt>
</tr>
<tr>
<td><tt>Hertz</tt></td>
<td><tt>MHz, GHz</tt></td>
</tr>
<tr>
<td><tt>Watts</tt></td>
<td><tt>mW, W, kW, MW, GW</tt></td>
</tr>
</table>
Conversion is only possible within the same base unit.
## `device` Examples
The following examples only show the `device` stanzas. Remember that the
`device` stanza is only valid in the placements listed above.
### Single Nvidia GPU
This example schedules a task with a single Nvidia GPU made available.
```hcl
device "nvidia/gpu" {}
```
### Multiple Nvidia GPU
This example schedules a task with a two Nvidia GPU made available.
```hcl
device "nvidia/gpu" {
count = 2
}
```
### Single Nvidia GPU with Specific Model
This example schedules a task with a single Nvidia GPU made available and uses
the name to specify the exact model to be used.
```hcl
device "nvidia/gpu/1080ti" {}
```
This is a simplification of the following:
```hcl
device "gpu" {
count = 1
constraint {
attribute = "${device.vendor}"
value = "nvidia"
}
constraint {
attribute = "${device.model}"
value = "1080ti"
}
}
```
### Affinity with Unit Conversion
This example uses an affinity to tell the scheduler it would prefer if the GPU
had at least 1.5 GiB of memory. The following are both equivalent as Nomad can
do unit conversions.
Specified in `GiB`:
```hcl
device "nvidia/gpu" {
affinity {
attribute = "${device.attr.memory}"
operator = ">="
value = "1.5 GiB"
weight = 75
}
}
```
Specified in `MiB`:
```hcl
device "nvidia/gpu" {
affinity {
attribute = "${device.attr.memory}"
operator = ">="
value = "1500 MiB"
weight = 75
}
}
```
[affinity]: /docs/job-specification/affinity.html "Nomad affinity Job Specification"
[constraint]: /docs/job-specification/constraint.html "Nomad constraint Job Specification"
[devices]: /docs/devices/index.html "Nomad Device Plugins"