Files
nomad/website/content/docs/job-specification/artifact.mdx
Seth Hoenig f1ce127524 jobspec: add a chown option to artifact block (#24157)
* jobspec: add a chown option to artifact block

This PR adds a boolean 'chown' field to the artifact block.

It indicates whether the Nomad client should chown the downloaded files
and directories to be owned by the task.user. This is useful for drivers
like raw_exec and exec2 which are subject to the host filesystem user
permissions structure. Before, these drivers might not be able to use or
manage the downloaded artifacts since they would be owned by the root
user on a typical Nomad client configuration.

* api: no need for pointer of chown field
2024-10-11 11:30:27 -05:00

288 lines
9.3 KiB
Plaintext

---
layout: docs
page_title: artifact Block - Job Specification
description: |-
The "artifact" block instructs Nomad to fetch and unpack a remote resource,
such as a file, tarball, or binary, and permits downloading artifacts from a
variety of locations using a URL as the input source.
---
# `artifact` Block
<Placement groups={['job', 'group', 'task', 'artifact']} />
The `artifact` block instructs Nomad to fetch and unpack a remote resource,
such as a file, tarball, or binary. Nomad downloads artifacts using the popular
[`go-getter`][go-getter] library, which permits downloading artifacts from a
variety of locations using a URL as the input source.
```hcl
job "docs" {
group "example" {
task "server" {
artifact {
source = "https://example.com/file.tar.gz"
destination = "local/some-directory"
options {
checksum = "md5:df6a4178aec9fbdc1d6d7e3634d1bc33"
}
}
}
}
}
```
Nomad supports downloading `http`, `https`, `git`, `hg` and `S3` artifacts. If
these artifacts are archived (`zip`, `tgz`, `bz2`, `xz`), they are
automatically unarchived before the starting the task.
## `artifact` Parameters
- `destination` `(string: "local/")` - Specifies the directory path to
download the artifact, relative to the root of the [task's working
directory]. If omitted, the default value is to place the artifact in
`local/`. The destination is treated as a directory unless `mode` is set to
`file`. Source files will be downloaded into that directory path. For more
details on how the `destination` interacts with task drivers, see the
[Filesystem internals] documentation.
- `mode` `(string: "any")` - One of `any`, `file`, or `dir`. If set to `file`
the `destination` must be a file, not a directory. By default the
`destination` will be `local/<filename>`.
- `options` `(map<string|string>: nil)` - Specifies configuration parameters to
fetch the artifact. The key-value pairs map directly to parameters appended to
the supplied `source` URL. Please see the [`go-getter`
documentation][go-getter] for a complete list of options and examples.
- `headers` `(map<string|string>: nil)` - Specifies HTTP headers to set when
fetching the artifact using `http` or `https` protocol. Please see the
[`go-getter` headers documentation][go-getter-headers] for more information.
- `source` `(string: <required>)` - Specifies the URL of the artifact to download.
See [`go-getter`][go-getter] for details.
- `chown` `(bool: false)` - Specifies whether Nomad should recursively `chown`
the downloaded artifact to be owned by the [`task.user`][task_user] uid and gid.
## Operation Limits
The client [`artifact`][client_artifact] configuration can set limits to
specific artifact operations to prevent excessive data download or operation
time.
If a task's `artifact` retrieval exceeds one of those limits, the task will be
interrupted and fail to start. Refer to the task events for more information.
## `artifact` Examples
The following examples only show the `artifact` blocks. Remember that the
`artifact` block is only valid in the placements listed above.
### Download File
This example downloads the artifact from the provided URL and places it in
`local/file.txt`. The `local/` path is relative to the [task's working
directory].
```hcl
artifact {
source = "https://example.com/file.txt"
}
```
To set HTTP headers in the request for the source the optional `headers` field
can be configured.
```hcl
artifact {
source = "https://example.com/file.txt"
headers {
User-Agent = "nomad-[${NOMAD_JOB_ID}]-[${NOMAD_GROUP_NAME}]-[${NOMAD_TASK_NAME}]"
X-Nomad-Alloc = "${NOMAD_ALLOC_ID}"
}
}
```
To use HTTP basic authentication, preprend the username and password to the
hostname in the URL. All special characters, including the username and
password, must be URL encoded. For example, for a username `exampleUser` and
the password `pass/word!`:
```hcl
artifact {
source = "https://exampleUser:pass%2Fword%21@example.com/file.txt"
}
```
### Download using git
This example downloads the artifact from the provided GitHub URL and places it at
`local/repo`, as specified by the optional `destination` parameter.
```hcl
artifact {
source = "git::https://github.com/hashicorp/nomad-guides"
destination = "local/repo"
}
```
To download from a private repo, sshkey needs to be set. The key must be
base64-encoded string. On Linux, you can run `base64 -w0 <file>` to encode the
file. Or use [HCL2](/nomad/docs/job-specification/hcl2)
expressions to read and encode the key from a file on your machine:
```hcl
artifact {
# The git:: prefix forces go-getter's protocol detection to use the git ssh
# protocol. It can also automatically detect the protocol from the domain of
# some git hosting providers (such as GitHub) without the prefix.
source = "git::git@bitbucket.org:example/nomad-examples"
destination = "local/repo"
options {
# Make sure that the system known hosts file is populated:
# ssh-keyscan github.com | sudo tee -a /etc/ssh/ssh_known_hosts
# https://github.com/hashicorp/go-getter/issues/55
sshkey = "${base64encode(file("/path/to/private-key"))}"
}
}
```
To clone specific refs or at a specific depth, use the `ref` and `depth`
options:
```hcl
artifact {
source = "git::https://github.com/hashicorp/nomad-guides"
destination = "local/repo"
options {
ref = "main"
depth = 1
}
}
```
### Download and Unarchive
This example downloads and unarchives the result in `local/file`. Because the
source URL is an archive extension, Nomad will automatically decompress it:
```hcl
artifact {
source = "https://example.com/file.tar.gz"
}
```
To disable automatic unarchiving, set the `archive` option to false:
```hcl
artifact {
source = "https://example.com/file.tar.gz"
options {
archive = false
}
}
```
### Download and Verify Checksums
This example downloads an artifact and verifies the resulting artifact's
checksum before proceeding. If the checksum is invalid, an error will be
returned.
```hcl
artifact {
source = "https://example.com/file.zip"
options {
checksum = "md5:df6a4178aec9fbdc1d6d7e3634d1bc33"
}
}
```
### Download from an S3-compatible Bucket
These examples download artifacts from Amazon S3. There are several different
types of [S3 bucket addressing][s3-bucket-addr] and [S3 region-specific
endpoints][s3-region-endpoints]. Non-Amazon S3-compatible endpoints like [Minio]
are supported, but you must explicitly set the "s3::" prefix.
This example uses path-based notation on a publicly-accessible bucket:
```hcl
artifact {
source = "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
}
```
If a bucket requires authentication, you can avoid the use of credentials by
using [EC2 IAM instance profiles][iam-instance-profiles]. If this is not possible,
credentials may be supplied via the `options` parameter:
```hcl
artifact {
options {
aws_access_key_id = "<id>"
aws_access_key_secret = "<secret>"
aws_access_token = "<token>"
}
}
```
To force the S3-specific syntax, use the `s3::` prefix:
```hcl
artifact {
source = "s3::https://my-bucket-example.s3-eu-west-1.amazonaws.com/my_app.tar.gz"
}
```
Alternatively you can use virtual hosted style:
```hcl
artifact {
source = "s3://my-bucket-example.s3-eu-west-1.amazonaws.com/my_app.tar.gz"
}
```
[DigitalOcean Spaces][do_spaces] provide S3-compatible object storage and can be
used in conjunction with the `artifact` block. Given a bucket named
`my-bucket-example` located in the `sfo3` region and a file named `my_app.tar.gz`,
the following artifact block will work, provided the bucket and file are public.
This uses the origin endpoint as detailed directly from Digital Ocean.
```hcl
artifact {
source = "https://my-bucket-example.sfo3.digitaloceanspaces.com/my_app.tar.gz"
}
```
If the bucket or file are private, the artifact source string needs to be
modified to add the `s3::` prefix. The bucket name is also moved to form part
of the URL path component.
```hcl
artifact {
source = "s3::https://sfo3.digitaloceanspaces.com/my-bucket-example/my_app.tar.gz"
}
```
## Environment
The `artifact` downloader by default does not have access to the environment variables
set for the Nomad client. Inheritance of environment variables can be managed through the [`artifact.set_environment_variables`][client_artifact]
client configuration.
[client_artifact]: /nomad/docs/configuration/client#artifact-parameters
[go-getter]: https://github.com/hashicorp/go-getter 'HashiCorp go-getter Library'
[go-getter-headers]: https://github.com/hashicorp/go-getter#headers 'HashiCorp go-getter Headers'
[minio]: https://www.minio.io/
[s3-bucket-addr]: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro 'Amazon S3 Bucket Addressing'
[s3-region-endpoints]: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region 'Amazon S3 Region Endpoints'
[iam-instance-profiles]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html 'EC2 IAM instance profiles'
[task's working directory]: /nomad/docs/runtime/environment#task-directories 'Task Directories'
[task_user]: /nomad/docs/job-specification/task#user
[filesystem internals]: /nomad/docs/concepts/filesystem#templates-artifacts-and-dispatch-payloads
[do_spaces]: https://www.digitalocean.com/products/spaces