kemko/nomad
1
0
Fork 0
mirror of https://github.com/kemko/nomad.git synced 2026-01-03 17:05:43 +03:00
Code Issues Packages Projects Releases Wiki Activity

update vault integration docs (#8543)

Browse Source 
* update vault integration docs

docs/integrations/vault-integration was a copy of the learn guide. Remove that and move /docs/vault-integration to this location instead

fix link

fix link

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Michael Schurter <mschurter@hashicorp.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Michael Schurter <mschurter@hashicorp.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>

Update website/pages/docs/integrations/vault-integration.mdx

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>

* revert accidental deletion

Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
This commit is contained in:
Drew Bailey
2020-08-10 14:23:43 -04:00
committed by GitHub
parent bf67737f56
commit c08555247f
4 changed files with 321 additions and 907 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
website/_redirects
View File

@@ -474,3 +474,6 @@
# Guide Catch-all Redirects
/guides https://learn.hashicorp.com/nomad 301!
/guides/* https://learn.hashicorp.com/nomad 301!
# Vault Integration
/docs/vault-integration /docs/integrations/vault-integration

1
website/data/docs-navigation.js
View File

@@ -249,7 +249,6 @@ export default [
]
},
{ category: 'telemetry', content: ['metrics'] },
{ category: 'vault-integration' },
'------------',
{ category: 'enterprise' },
'faq'

874
website/pages/docs/integrations/vault-integration.mdx
View File

@@ -1,131 +1,74 @@
---
layout: docs
page_title: Vault Integration and Retrieving Dynamic Secrets
sidebar_title: Vault
description: |-
Learn how to deploy an application in Nomad and retrieve dynamic credentials
by integrating with Vault.
page_title: Vault Integration
sidebar_title: Vault Integration
description: >-
Learn how to integrate Nomad with HashiCorp Vault and retrieve Vault tokens
for
tasks.
---
# Vault Integration
Nomad integrates seamlessly with [Vault][vault] and allows your application to
retrieve dynamic credentials for various tasks. In this guide, you will deploy a
web application that needs to authenticate against [PostgreSQL][postgresql] to
display data from a table to the user.
Many workloads require access to tokens, passwords, certificates, API keys, and
other secrets. To enable secure, auditable and easy access to your secrets,
Nomad integrates with HashiCorp's [Vault][]. Nomad servers and clients
coordinate with Vault to derive a Vault token that has access to only the Vault
policies the tasks needs. Nomad clients make the token available to the task and
handle the tokens renewal. Further, Nomad's [`template` block][template] can
retrieve secrets from Vault making it easier than ever to secure your
infrastructure.
## Reference Material
Note that in order to use Vault with Nomad, you will need to configure and
install Vault separately from Nomad. Nomad does not run Vault for you.
- [Vault Integration Documentation][vault-integration]
- [Nomad Template Stanza Integration with Vault][nomad-template-vault]
- [Secrets Task Directory][secrets-task-directory]
-> **Note:** Vault integration requires Vault version 0.6.2 or higher.
## Estimated Time to Complete
## Vault Configuration
20 minutes
To use the Vault integration, Nomad servers must be provided a Vault token. This
token can either be a root token or a periodic token with permissions to create
from a token role. The root token is the easiest way to get started, but we
recommend a token role based token for production installations. Nomad servers
will renew the token automatically. **Note that the Nomad clients do not need to
be provided with a Vault token.**
## Challenge
-> **Note:** See the [Enterprise specific section][ent] for configuring Vault Enterprise
Think of a scenario where a Nomad operator needs to deploy an application that
can quickly and safely retrieve dynamic credentials to authenticate against a
database and return information.
### Root Token Integration
## Solution
If Nomad is given a [root
token](https://www.vaultproject.io/docs/concepts/tokens#root-tokens), no
further configuration is needed as Nomad can derive a token for jobs using any
Vault policies. Best practices recommend using a periodic token with the minimal
permissions necessary instead of providing Nomad the root vault token.
Deploy Vault and configure the nodes in your Nomad cluster to integrate with it.
Use the appropriate [templating syntax][nomad-template-vault] to retrieve
credentials from Vault and then store those credentials in the
[secrets][secrets-task-directory] task directory to be consumed by the Nomad
task.
### Token Role based Integration
## Prerequisites
Vault's [Token Authentication Backend][auth] supports a concept called "roles".
Token roles allow policies to be grouped together and token creation to be
delegated to a trusted service such as Nomad. By creating a token role, the set
of policies that tasks managed by Nomad can access may be limited compared to
giving Nomad a root token. Token roles allow both white-list and blacklist
management of policies accessible to the role.
To perform the tasks described in this guide, you need to have a Nomad
environment with Consul and Vault installed. You can use this [repo][repo] to
easily provision a sandbox environment. This guide will assume a cluster with
one server node and three client nodes.
To configure Nomad and Vault to create tokens against a role, the following must
occur:
-> **Please Note:** This guide is for demo purposes and is only using a single
Nomad server with Vault installed alongside. In a production cluster, 3 or 5
Nomad server nodes are recommended along with a separate Vault cluster.
1. Create a "nomad-server" policy used by Nomad to create and manage tokens.
## Steps
2. Create a Vault token role with the configuration described below.
### Step 1: Initialize Vault Server
3. Configure Nomad to use the created token role.
Run the following command to initialize Vault server and receive an
[unseal][seal] key and initial root [token][token]. Be sure to note the unseal
key and initial root token as you will need these two pieces of information.
4. Give Nomad servers a periodic token with the "nomad-server" policy created
above.
```shell-session
$ vault operator init -key-shares=1 -key-threshold=1
```
#### Required Vault Policies
The `vault operator init` command above creates a single Vault unseal key for
convenience. For a production environment, it is recommended that you create at
least five unseal key shares and securely distribute them to independent
operators. The `vault operator init` command defaults to five key shares and a
key threshold of three. If you provisioned more than one server, the others will
become standby nodes but should still be unsealed.
### Step 2: Unseal Vault
Run the following command and then provide your unseal key to Vault.
```shell-session
$ vault operator unseal
```
The output of unsealing Vault will look similar to the following:
```text
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 0.11.4
Cluster Name vault-cluster-d12535e5
Cluster ID 49383931-c782-fdc6-443e-7681e7b15aca
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
```
### Step 3: Log in to Vault
Use the [login][login] command to authenticate yourself against Vault using the
initial root token you received earlier. You will need to authenticate to run
the necessary commands to write policies, create roles, and configure a
connection to your database.
```shell-session
$ vault login <your initial root token>
```
If your login is successful, you will see output similar to what is shown below:
```text
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
...
```
### Step 4: Write the Policy for the Nomad Server Token
To use the Vault integration, you must provide a Vault token to your Nomad
servers. Although you can provide your root token to easily get started, the
recommended approach is to use a token [role][role] based token. This first
requires writing a policy that you will attach to the token you provide to your
Nomad servers. By using this approach, you can limit the set of
[policies][policy] that tasks managed by Nomad can access.
For this exercise, use the following policy for the token you will create for
your Nomad server. Place this policy in a file named `nomad-server-policy.hcl`.
The token Nomad receives must have the capabilities listed below. An explanation
for the use of each capability is given.
```hcl
# Allow creating tokens under "nomad-cluster" token role. The token role name
@@ -171,48 +114,31 @@ path "auth/token/renew-self" {
}
```
You can now write a policy called `nomad-server` by running the following
command:
The above [`nomad-server` policy](/data/vault/nomad-server-policy.hcl) is
available for download. Below is an example of writing this policy to Vault:
```shell-session
# Download the policy
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
```
You should see the following output:
#### Vault Token Role Configuration
```text
Success! Uploaded policy: nomad-server
```
A Vault token role must be created for use by Nomad. The token role can be used
to manage what Vault policies are accessible by jobs submitted to Nomad. The
policies can be managed as a whitelist by using `allowed_policies` in the token
role definition or as a blacklist by using `disallowed_policies`.
You will generate the actual token in the next few steps.
If using `allowed_policies`, tasks may only request Vault policies that are in
the list. If `disallowed_policies` is used, task may request any policy that is
not in the `disallowed_policies` list. There are trade-offs to both approaches
but generally it is easier to use the blacklist approach and add policies that
you would not like tasks to have access to into the `disallowed_policies` list.
### Step 5: Create a Token Role
At this point, you must create a Vault token role that Nomad can use. The token
role allows you to limit what Vault policies are accessible by jobs
submitted to Nomad. We will use the following token role:
```json
{
"allowed_policies": "access-tables",
"token_explicit_max_ttl": 0,
"name": "nomad-cluster",
"orphan": true,
"token_period": 259200,
"renewable": true
}
```
Please notice that the `access-tables` policy is listed under the
`allowed_policies` key. We have not created this policy yet, but it will be used
by our job to retrieve credentials to access the database. A job running in our
Nomad cluster will only be allowed to use the `access-tables` policy.
If you would like to allow all policies to be used by any job in the Nomad
cluster except for the ones you specifically prohibit, then use the
`disallowed_policies` key instead and simply list the policies that should not
be granted. If you take this approach, be sure to include `nomad-server` in the
disallowed policies group. An example of this is shown below:
An example token role definition is given below:
```json
{
@@ -225,485 +151,321 @@ disallowed policies group. An example of this is shown below:
}
```
Save the policy in a file named `nomad-cluster-role.json` and create the token
role named `nomad-cluster`.
##### Token Role Requirements
Nomad checks that token role has an appropriate configuration for use by the
cluster. Fields that are checked are documented below as well as descriptions of
the important fields. See Vault's [Token Authentication Backend][auth]
documentation for all possible fields and more complete documentation.
- `allowed_policies` - Specifies the list of allowed policies as a
comma-separated string. This list should contain all policies that jobs running
under Nomad should have access to.
- `disallowed_policies` - Specifies the list of disallowed policies as a
comma-separated string. This list should contain all policies that jobs running
under Nomad should **not** have access to. The policy created above that
grants Nomad the ability to generate tokens from the token role should be
included in list of disallowed policies. This prevents tokens created by
Nomad from generating new tokens with different policies than those granted
by Nomad.
A regression occurred in Vault 0.6.4 when validating token creation using a
token role with `disallowed_policies` such that it is not usable with
Nomad. This was remedied in 0.6.5 and does not effect earlier versions
of Vault.
- `token_explicit_max_ttl` - Specifies the max TTL of a token. **Must be set to `0`** to
allow periodic tokens.
- `name` - Specifies the name of the policy. We recommend using the name
`nomad-cluster`. If a different name is chosen, replace the token role in the
above policy.
- `orphan` - Specifies whether tokens created against this token role will be
orphaned and have no parents. Nomad does not enforce the value of this field
but understanding the implications of each value is important.
If set to false, all tokens will be revoked when the Vault token given to
Nomad expires. This makes it easy to revoke all tokens generated by Nomad but
forces all Nomad servers to use the same Vault token, even through upgrades of
Nomad servers. If the Vault token that was given to Nomad and used to generate
a tasks token expires, the token used by the task will also be revoked which
is not ideal.
When set to true, the tokens generated for tasks will not be revoked when
Nomad's token is revoked. However Nomad will still revoke tokens when the
allocation is no longer running, minimizing the lifetime of any task's token.
With orphaned enabled, each Nomad server may also use a unique Vault token,
making bootstrapping and upgrading simpler. As such, **setting `orphan = true`
is the recommended setting**.
- `token_period` - Specifies the length the TTL is extended by each renewal in
seconds. It is suggested to set this value on the order of magnitude of 3 days
(259200 seconds) to avoid a large renewal request rate to Vault. **Must be set
to a positive value**.
- `renewable` - Specifies whether created tokens are renewable. **Must be set to
`true`**. This allows Nomad to renew tokens for tasks.
The above [`nomad-cluster` token role](/data/vault/nomad-cluster-role.json) is
available for download. Below is an example of writing this role to Vault:
```shell-session
# Download the token role
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
You should see the following output:
#### Example Configuration
```text
Success! Data written to: auth/token/roles/nomad-cluster
To make getting started easy, the basic [`nomad-server`
policy](/data/vault/nomad-server-policy.hcl) and
[`nomad-cluster` role](/data/vault/nomad-cluster-role.json) described above are
available for download.
The below example assumes Vault is accessible, unsealed and the operator has
appropriate permissions.
```shell-session
# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
### Step 6: Generate the Token for the Nomad Server
#### Retrieving the Token Role based Token
Run the following command to create a token for your Nomad server:
After the token role is created, a token suitable for the Nomad servers may be
retrieved by issuing the following Vault command:
```shell-session
$ vault token create -policy nomad-server -period 72h -orphan
Key Value
--- -----
token f02f01c2-c0d1-7cb7-6b88-8a14fada58c0
token_accessor 8cb7fcb3-9a4f-6fbf-0efc-83092bb0cb1c
token_duration 259200s
token_renewable true
token_policies [default nomad-server]
```
The `-orphan` flag is included when generating the Nomad server token above to
prevent revocation of the token when its parent expires. Vault typically creates
tokens with a parent-child relationship. When an ancestor token is revoked, all
of its descendant tokens and their associated leases are revoked as well.
prevent revocation of the token when its parent expires. Vault typically
creates tokens with a parent-child relationship. When an ancestor token is
revoked, all of its descendant tokens and their associated leases are revoked
as well.
If everything works, you should see output similar to the following:
When generating Nomad's Vault token, we need to ensure that revocation of the
parent token does not revoke Nomad's token. To prevent this behavior we
specify the `-orphan` flag when we create the Nomad's Vault token. All
other tokens generated by Nomad for jobs will be generated using the policy
default of `orphan = false`.
```text
Key Value
--- -----
token 1gr0YoLyTBVZl5UqqvCfK9RJ
token_accessor 5fz20DuDbxKgweJZt3cMynya
token_duration 72h
token_renewable true
token_policies ["default" "nomad-server"]
identity_policies []
policies ["default" "nomad-server"]
More information about creating orphan tokens can be found in
[Vault's Token Hierarchies and Orphan Tokens documentation][tokenhierarchy].
The token can then be set in the server configuration's
[`vault` stanza][config], as a command-line flag, or via an environment
variable.
```shell-session
$ VAULT_TOKEN=f02f01c2-c0d1-7cb7-6b88-8a14fada58c0 nomad agent -config /path/to/config
```
### Step 7: Edit the Nomad Server Configuration to Enable Vault Integration
At this point, you are ready to edit the [vault stanza][vault-stanza] in the
Nomad Server's configuration file located at `/etc/nomad.d/nomad.hcl`. Provide
the token you generated in the previous step in the `vault` stanza of your Nomad
server configuration. The token can also be provided as an environment variable
called `VAULT_TOKEN`. Be sure to specify the `nomad-cluster-role` in the
[create_from_role][create-from-role] option. If using
[Vault Namespaces](https://www.vaultproject.io/docs/enterprise/namespaces),
modify both the client and server configuration to include the namespace;
alternatively, it can be provided in the environment variable `VAULT_NAMESPACE`.
After following these steps and enabling Vault, the `vault` stanza in your Nomad
server configuration will be similar to what is shown below:
An example of what may be contained in the configuration is shown below. For
complete documentation please see the [Nomad agent Vault integration][config]
configuration.
```hcl
vault {
enabled = true
address = "http://active.vault.service.consul:8200"
task_token_ttl = "1h"
enabled = true
ca_path = "/etc/certs/ca"
cert_file = "/var/certs/vault.crt"
key_file = "/var/certs/vault.key"
address = "https://vault.service.consul:8200"
create_from_role = "nomad-cluster"
token = "<your nomad server token>"
namespace = "<vault namespace for the cluster>"
}
```
Restart the Nomad server
## Agent Configuration
```shell-session
$ sudo systemctl restart nomad
To enable Vault integration, please see the [Nomad agent Vault
integration][config] configuration.
## Vault Definition Syntax
To configure a job to retrieve Vault tokens, please see the [`vault` job
specification documentation][vault-spec].
## Troubleshooting
### Invalid Vault token
Upon startup, Nomad will attempt to connect to the specified Vault server. Nomad
will lookup the passed token and if the token is from a token role, the token
role will be validated. Nomad will not shutdown if given an invalid Vault token,
but will log the reasons the token is invalid and disable Vault integration.
### Permission Denied errors
If you are using a Vault version less than 0.7.1 with a Nomad version greater than or equal to 0.6.1, you will need to update your task's policy (listed in [the `vault` stanza of the job specification][vault-spec]) to add the following:
```hcl
path "sys/leases/renew" {
capabilities = ["update"]
}
```
NOTE: Nomad servers will renew the token automatically.
This is included in Vault's "default" policy beginning with Vault 0.7.1 and is relied upon by Nomad's Vault integration beginning with Nomad 0.6.1. If you're using a newer Nomad version with an older Vault version, your default policy may not automatically include this and you will see "permission denied" errors in your Nomad logs similar to the following:
Vault integration needs to be enabled on the client nodes as well, but this has
been configured for you already in this environment. You will see the `vault`
stanza in your Nomad clients' configuration (located at
`/etc/nomad.d/nomad.hcl`) looks similar to the following:
```plaintext
Code: 403. Errors:
URL: PUT https://vault:8200/v1/sys/leases/renew
* permission denied
```
### No Secret Exists
Vault has two APIs for secrets, [`v1` and `v2`][vault-secrets-version]. Each version
has different paths, and Nomad does not abstract this for you. As such you will
need to specify the path as reflected by Vault's HTTP API, rather than the path
used in the `vault kv` command.
You can see examples of `v1` and `v2` syntax in the
[template documentation][vault-kv-templates].
## Enterprise Configuration
Nomad Enterprise 0.12.2 introduced the ability for jobs to use multiple Vault Namespaces.
There are a few configuration settings to consider when using this functionality.
### Example Configuration
Below is an example for creating two Namespaces within Vault.
```shell-session
# Create a namespace "engineering" within Vault
$ vault namespace create engineering
# Create a child namespace "frontend" under "engineering"
$ vault namespace create -namespace=engineering frontend
```
### Required Vault Policies
Policies are configured per Vault namespace. We will apply the policy in the example above to each namespace—engineering and engineering/frontend.
```shell-session
# Create the "nomad-server" policy in the "engineering" namespace
$ vault policy write -namespace=engineering nomad-server nomad-server-policy.hcl
# Create the "nomad-server" policy in the "engineering/frontend" namespace
$ vault policy write -namespace=engineering/frontend nomad-server nomad-server-policy.hcl
```
We will also configure the previously configured `nomad-cluster` role with each Namespace
```shell-session
# Create the "nomad-cluster" token role in the "engineering" namespace
$ vault write -namespace=engineering /auth/token/roles/nomad-cluster @nomad-cluster-role.json
# Create the "nomad-cluster" token role in the "engineering/frontend" namespace
$ vault write -namespace=engineering/frontend /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
The [Nomad agent Vault integration][config] configuration supports specifying a Vault Namespace, but since
we will be using multiple it can be left blank. By default Nomad will interact with Vault's root Namespace, but individual jobs may specify other Vault Namespaces to use.
```hcl
vault {
enabled = true
address = "http://active.vault.service.consul:8200"
enabled = true
ca_path = "/etc/certs/ca"
cert_file = "/var/certs/vault.crt"
key_file = "/var/certs/vault.key"
address = "https://vault.service.consul:8200"
create_from_role = "nomad-cluster"
allow_unauthenticated = false # Disabling allow_unauthenticated is a best practice for securing your cluster
}
```
Please note that the Nomad clients do not need to be provided with a Vault
token.
The same steps can be taken to inject a Vault token from the [Retrieving the Token Role based Token](#retrieving-the-token-role-based-token) steps.
### Step 8: Deploy Database
### Submitting a job with a Vault Namespace
The next few steps will involve configuring a connection between Vault and our
database, so let's deploy one that we can connect to. Create a Nomad job called
`db.nomad` with the following content:
Since [`allow_unauthenticated`][allow_unauth] is set to `false` job submitters will need to provide a sufficiently privileged token when submitting a job.
[allow_unauth]: /docs/configuration/vault#allow_unauthenticated
The example job file below specifies to use the `engineering` Namespace in Vault. It will then read the value at secret/foo and fetch the value for key `bar`
```hcl
job "postgres-nomad-demo" {
datacenters = ["dc1"]
group "db" {
task "server" {
driver = "docker"
config {
image = "hashicorp/postgres-nomad-demo:latest"
port_map {
db = 5432
}
}
resources {
network {
port "db"{
static = 5432
}
}
}
service {
name = "database"
port = "db"
check {
type = "tcp"
interval = "2s"
timeout = "2s"
}
}
}
}
}
```
Run the job as shown below:
```shell-session
$ nomad run db.nomad
```
Verify the job is running with the following command:
```shell-session
$ nomad status postgres-nomad-demo
```
The result of the status command will look similar to the output below:
```text
ID = postgres-nomad-demo
Name = postgres-nomad-demo
Submit Date = 2018-11-15T21:01:00Z
Type = service
Priority = 50
Datacenters = dc1
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
db 0 0 1 0 0 0
Allocations
ID Node ID Task Group Version Desired Status Created Modified
701e2699 5de1330c db 0 run running 1m56s ago 1m33s ago
```
Now we can move on to configuring the connection between Vault and our database.
### Step 9: Enable the Database Secrets Engine
We are using the database secrets engine for Vault in this exercise so that we
can generate dynamic credentials for our PostgreSQL database. Run the following command to enable it:
```shell-session
$ vault secrets enable database
```
If the previous command was successful, you will see the following output:
```text
Success! Enabled the database secrets engine at: database/
```
### Step 10: Configure the Database Secrets Engine
Create a file named `connection.json` and placed the following information into
it:
```json
{
"plugin_name": "postgresql-database-plugin",
"allowed_roles": "accessdb",
"connection_url": "postgresql://{{username}}:{{password}}@database.service.consul:5432/postgres?sslmode=disable",
"username": "postgres",
"password": "postgres123"
}
```
The information above allows Vault to connect to our database and create users
with specific privileges. We will specify the `accessdb` role soon. In a
production setting, it is recommended to give Vault credentials with enough
privileges to generate database credentials dynamically and and manage their
lifecycle.
Run the following command to configure the connection between the database
secrets engine and our database:
```shell-session
$ vault write database/config/postgresql @connection.json
```
If the operation is successful, there will be no output.
### Step 11: Create a Vault Role to Manage Database Privileges
Recall from the previous step that we specified `accessdb` in the
`allowed_roles` key of our connection information. Let's set up that role now. Create a file called `accessdb.sql` with the following content:
```shell
CREATE USER "{{name}}" WITH ENCRYPTED PASSWORD '{{password}}' VALID UNTIL
'{{expiration}}';
GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO "{{name}}";
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "{{name}}";
GRANT ALL ON SCHEMA public TO "{{name}}";
```
The SQL above will be used in the [creation_statements][creation-statements]
parameter of our next command to specify the privileges that the dynamic
credentials being generated will possess. In our case, the dynamic database user
will have broad privileges that include the ability to read from the tables that
our application will need to access.
Run the following command to create the role:
```shell-session
$ vault write database/roles/accessdb db_name=postgresql \
creation_statements=@accessdb.sql default_ttl=1h max_ttl=24h
```
You should see the following output after running the previous command:
```text
Success! Data written to: database/roles/accessdb
```
### Step 12: Generate PostgreSQL Credentials
You should now be able to generate dynamic credentials to access your database.
Run the following command to generate a set of credentials:
```shell-session
$ vault read database/creds/accessdb
```
The previous command should return output similar to what is shown below:
```text
Key Value
--- -----
lease_id database/creds/accessdb/3JozEMSMqw0vHHhvla15sKTW
lease_duration 1h
lease_renewable true
password A1a-3pMGjpDXHZ2Qzuf7
username v-root-accessdb-5LA65urB4daA8KYy2xku-1542318363
```
Congratulations! You have configured Vault's connection to your database and can
now generate credentials with the previously specified privileges. Now we need
to deploy our application and make sure that it will be able to communicate with
Vault and obtain the credentials as well.
### Step 13: Create the `access-tables` Policy for Your Nomad Job to Use
Recall from [Step 5][step-5] that we specified a policy named `access-tables` in
our `allowed_policies` section of the token role. We will create this policy now
and give it the capability to read from the `database/creds/accessdb` endpoint
(the same endpoint we read from in the previous step to generate credentials for
our database). We will then specify this policy in our Nomad job which will
allow it to retrieve credentials for itself to access the database.
On the Nomad server (which is also running Vault), create a file named
`access-tables-policy.hcl` with the following content:
```hcl
path "database/creds/accessdb" {
capabilities = ["read"]
}
```
Create the `access-tables` policy with the following command:
```shell-session
$ vault policy write access-tables access-tables-policy.hcl
```
You should see the following output:
```text
Success! Uploaded policy: access-tables
```
### Step 14: Deploy Your Job with the Appropriate Policy and Templating
Now we are ready to deploy our web application and give it the necessary policy
and configuration to communicate with our database. Create a file called
`web-app.nomad` and save the following content in it.
```hcl
job "nomad-vault-demo" {
job "vault" {
datacenters = ["dc1"]
group "demo" {
task "server" {
task "task" {
vault {
policies = ["access-tables"]
namespace = "engineering"
policies = ["access-kv"]
}
driver = "docker"
driver = "raw_exec"
config {
image = "hashicorp/nomad-vault-demo:latest"
port_map {
http = 8080
}
volumes = [
"secrets/config.json:/etc/demo/config.json"
]
command = "/usr/bin/cat"
args = ["secrets/config.txt"]
}
template {
data = <<EOF
{{ with secret "database/creds/accessdb" }}
{
"host": "database.service.consul",
"port": 5432,
"username": "{{ .Data.username }}",
{{ /* Ensure password is a properly escaped JSON string. */ }}
"password": {{ .Data.password | toJSON }},
"db": "postgres"
}
data = <<EOF
{{ with secret "secret/foo" }}
SOME_VAL={{.Data.bar}}
{{ end }}
EOF
destination = "secrets/config.json"
}
resources {
network {
port "http" {}
}
}
service {
name = "nomad-vault-demo"
port = "http"
tags = [
"urlprefix-/",
]
check {
type = "tcp"
interval = "2s"
timeout = "2s"
}
destination = "secrets/config.txt"
}
}
}
}
```
There are a few key points to note here:
- We have specified the `access-tables` policy in the [vault][vault-jobspec]
stanza of this job. The Nomad client will receive a token with this policy
attached. Recall from the previous step that this policy will allow our
application to read from the `database/creds/accessdb` endpoint in Vault and
retrieve credentials.
- We are using the [template][template] stanza's [vault
integration][nomad-template-vault] to populate the JSON configuration file
that our application needs. The underlying tool being used is [Consul
Template][consul-template]. You can use Consul Template's documentation to
learn more about the [syntax][consul-temp-syntax] needed to interact with
Vault. Please note that although we have defined the template
[inline][inline], we can use the template stanza [in conjunction with the
artifact stanza][remote-template] to download an input template from a remote
source such as an S3 bucket.
- We are using the `toJSON` function to ensure the password is encoded as a JSON
string. Any templated value which may contain special characters (like quotes
or newlines) should be passed through the `toJSON` function.
- Finally, note that that [destination][destination] of our template is the
[secrets/][secrets-task-directory] task directory. This ensures the data is
not accessible with a command like [nomad alloc fs][nomad-alloc-fs] or
filesystem APIs.
Use the following command to run the job:
To Submit this job a token that has the `access-kv` policy in the Namespace `engineering`
```shell-session
$ nomad run web-app.nomad
$ vault token create -policy access-kv -namespace=engineering -period 72h -oprhan
Key Value
--- -----
token s.H39hfS7eHSbb1GpkdzOQLTmz.fvuLy
token_accessor VsKtJwaShwtTo1r9nWV9Rlad.fvuLy
token_duration 72h
token_renewable true
token_policies ["access-kv" "default"]
identity_policies []
policies ["access-kv" "default"]
```
### Step 15: Confirm the Application is Accessing the Database
At this point, you can visit your application at the path `/names` to confirm
the appropriate data is being accessed from the database and displayed to you.
There are several ways to do this.
- Use the `dig` command to query the SRV record of your service and obtain the
port it is using. Then `curl` your service at the appropriate port and `names` path.
The token can then be submitted with our job
```shell-session
$ dig +short SRV nomad-vault-demo.service.consul
1 1 30478 ip-172-31-58-230.node.dc1.consul.
$ VAULT_TOKEN=s.H39hfS7eHSbb1GpkdzOQLTmz.fvuLy nomad job run vault.nomad
```
```shell-session
$ curl nomad-vault-demo.service.consul:30478/names
<!DOCTYPE html>
<html>
<body>
<h1> Welcome! </h1>
<h2> If everything worked correctly, you should be able to see a list of names
below </h2>
<hr>
<h4> John Doe </h4>
<h4> Peter Parker </h4>
<h4> Clifford Roosevelt </h4>
<h4> Bruce Wayne </h4>
<h4> Steven Clark </h4>
<h4> Mary Jane </h4>
</body>
<html>
```
- You can also deploy [fabio][fabio] and visit any Nomad client at its public IP
address using a fixed port. The details of this method are beyond the scope of
this guide, but you can refer to the [Load Balancing with Fabio][fabio-lb]
guide for more information on this topic. Alternatively, you could use the
`nomad` [alloc status][alloc-status] command along with the AWS console to
determine the public IP and port your service is running (remember to open the
port in your AWS security group if you choose this method).
[![Web Service][web-service]][web-service]
[alloc-status]: /docs/commands/alloc/status
[consul-template]: https://github.com/hashicorp/consul-template
[consul-temp-syntax]: https://github.com/hashicorp/consul-template#secret
[create-from-role]: /docs/configuration/vault#create_from_role
[creation-statements]: https://www.vaultproject.io/api/secret/databases#creation_statements
[destination]: /docs/job-specification/template#destination
[fabio]: https://github.com/fabiolb/fabio
[fabio-lb]: https://learn.hashicorp.com/nomad/load-balancing/fabio
[inline]: /docs/job-specification/template#inline-template
[login]: https://www.vaultproject.io/docs/commands/login
[nomad-alloc-fs]: /docs/commands/alloc/fs
[nomad-template-vault]: /docs/job-specification/template#vault-integration
[policy]: https://www.vaultproject.io/docs/concepts/policies
[postgresql]: https://www.postgresql.org/about/
[remote-template]: /docs/job-specification/template#remote-template
[repo]: https://github.com/hashicorp/nomad/tree/master/terraform
[role]: https://www.vaultproject.io/docs/auth/token
[seal]: https://www.vaultproject.io/docs/concepts/seal
[secrets-task-directory]: /docs/runtime/environment#secrets
[step-5]: /docs/integrations/vault-integration#step-5-create-a-token-role
[template]: /docs/job-specification/template
[token]: https://www.vaultproject.io/docs/concepts/tokens
[vault]: https://www.vaultproject.io/
[vault-integration]: /docs/vault-integration
[vault-jobspec]: /docs/job-specification/vault
[vault-stanza]: /docs/configuration/vault
[web-service]: /img/nomad-demo-app.png
[auth]: https://www.vaultproject.io/docs/auth/token 'Vault Authentication Backend'
[config]: /docs/configuration/vault 'Nomad Vault Configuration Block'
[createfromrole]: /docs/configuration/vault#create_from_role 'Nomad vault create_from_role Configuration Flag'
[template]: /docs/job-specification/template 'Nomad template Job Specification'
[vault]: https://www.vaultproject.io/ 'Vault by HashiCorp'
[vault-spec]: /docs/job-specification/vault 'Nomad Vault Job Specification'
[tokenhierarchy]: https://www.vaultproject.io/docs/concepts/tokens#token-hierarchies-and-orphan-tokens 'Vault Tokens - Token Hierarchies and Orphan Tokens'
[vault-secrets-version]: https://www.vaultproject.io/docs/secrets/kv 'KV Secrets Engine'
[vault-kv-templates]: /docs/job-specification/template#vault-kv-api-v1 'Vault KV API v1'
[ent]: /docs/integrations/vault-integration#enterprise-configuration

350
website/pages/docs/vault-integration/index.mdx
View File

@@ -1,350 +0,0 @@
---
layout: docs
page_title: Vault Integration
sidebar_title: Vault Integration
description: >-
Learn how to integrate Nomad with HashiCorp Vault and retrieve Vault tokens
for
tasks.
---
# Vault Integration
Many workloads require access to tokens, passwords, certificates, API keys, and
other secrets. To enable secure, auditable and easy access to your secrets,
Nomad integrates with HashiCorp's [Vault][]. Nomad servers and clients
coordinate with Vault to derive a Vault token that has access to only the Vault
policies the tasks needs. Nomad clients make the token available to the task and
handle the tokens renewal. Further, Nomad's [`template` block][template] can
retrieve secrets from Vault making it easier than ever to secure your
infrastructure.
Note that in order to use Vault with Nomad, you will need to configure and
install Vault separately from Nomad. Nomad does not run Vault for you.
-> **Note:** Vault integration requires Vault version 0.6.2 or higher.
## Vault Configuration
To use the Vault integration, Nomad servers must be provided a Vault token. This
token can either be a root token or a periodic token with permissions to create
from a token role. The root token is the easiest way to get started, but we
recommend a token role based token for production installations. Nomad servers
will renew the token automatically. **Note that the Nomad clients do not need to
be provided with a Vault token.**
### Root Token Integration
If Nomad is given a [root
token](https://www.vaultproject.io/docs/concepts/tokens#root-tokens), no
further configuration is needed as Nomad can derive a token for jobs using any
Vault policies.
### Token Role based Integration
Vault's [Token Authentication Backend][auth] supports a concept called "roles".
Token roles allow policies to be grouped together and token creation to be
delegated to a trusted service such as Nomad. By creating a token role, the set
of policies that tasks managed by Nomad can access may be limited compared to
giving Nomad a root token. Token roles allow both white-list and blacklist
management of policies accessible to the role.
To configure Nomad and Vault to create tokens against a role, the following must
occur:
1. Create a "nomad-server" policy used by Nomad to create and manage tokens.
2. Create a Vault token role with the configuration described below.
3. Configure Nomad to use the created token role.
4. Give Nomad servers a periodic token with the "nomad-server" policy created
above.
#### Required Vault Policies
The token Nomad receives must have the capabilities listed below. An explanation
for the use of each capability is given.
```hcl
# Allow creating tokens under "nomad-cluster" token role. The token role name
# should be updated if "nomad-cluster" is not used.
path "auth/token/create/nomad-cluster" {
capabilities = ["update"]
}
# Allow looking up "nomad-cluster" token role. The token role name should be
# updated if "nomad-cluster" is not used.
path "auth/token/roles/nomad-cluster" {
capabilities = ["read"]
}
# Allow looking up the token passed to Nomad to validate # the token has the
# proper capabilities. This is provided by the "default" policy.
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow looking up incoming tokens to validate they have permissions to access
# the tokens they are requesting. This is only required if
# `allow_unauthenticated` is set to false.
path "auth/token/lookup" {
capabilities = ["update"]
}
# Allow revoking tokens that should no longer exist. This allows revoking
# tokens for dead tasks.
path "auth/token/revoke-accessor" {
capabilities = ["update"]
}
# Allow checking the capabilities of our own token. This is used to validate the
# token upon startup.
path "sys/capabilities-self" {
capabilities = ["update"]
}
# Allow our own token to be renewed.
path "auth/token/renew-self" {
capabilities = ["update"]
}
```
The above [`nomad-server` policy](/data/vault/nomad-server-policy.hcl) is
available for download. Below is an example of writing this policy to Vault:
```shell-session
# Download the policy
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
```
#### Vault Token Role Configuration
A Vault token role must be created for use by Nomad. The token role can be used
to manage what Vault policies are accessible by jobs submitted to Nomad. The
policies can be managed as a whitelist by using `allowed_policies` in the token
role definition or as a blacklist by using `disallowed_policies`.
If using `allowed_policies`, tasks may only request Vault policies that are in
the list. If `disallowed_policies` is used, task may request any policy that is
not in the `disallowed_policies` list. There are trade-offs to both approaches
but generally it is easier to use the blacklist approach and add policies that
you would not like tasks to have access to into the `disallowed_policies` list.
An example token role definition is given below:
```json
{
"disallowed_policies": "nomad-server",
"token_explicit_max_ttl": 0,
"name": "nomad-cluster",
"orphan": true,
"token_period": 259200,
"renewable": true
}
```
##### Token Role Requirements
Nomad checks that token role has an appropriate configuration for use by the
cluster. Fields that are checked are documented below as well as descriptions of
the important fields. See Vault's [Token Authentication Backend][auth]
documentation for all possible fields and more complete documentation.
- `allowed_policies` - Specifies the list of allowed policies as a
comma-separated string. This list should contain all policies that jobs running
under Nomad should have access to.
- `disallowed_policies` - Specifies the list of disallowed policies as a
comma-separated string. This list should contain all policies that jobs running
under Nomad should **not** have access to. The policy created above that
grants Nomad the ability to generate tokens from the token role should be
included in list of disallowed policies. This prevents tokens created by
Nomad from generating new tokens with different policies than those granted
by Nomad.
A regression occurred in Vault 0.6.4 when validating token creation using a
token role with `disallowed_policies` such that it is not usable with
Nomad. This was remedied in 0.6.5 and does not effect earlier versions
of Vault.
- `token_explicit_max_ttl` - Specifies the max TTL of a token. **Must be set to `0`** to
allow periodic tokens.
- `name` - Specifies the name of the policy. We recommend using the name
`nomad-cluster`. If a different name is chosen, replace the token role in the
above policy.
- `orphan` - Specifies whether tokens created against this token role will be
orphaned and have no parents. Nomad does not enforce the value of this field
but understanding the implications of each value is important.
If set to false, all tokens will be revoked when the Vault token given to
Nomad expires. This makes it easy to revoke all tokens generated by Nomad but
forces all Nomad servers to use the same Vault token, even through upgrades of
Nomad servers. If the Vault token that was given to Nomad and used to generate
a tasks token expires, the token used by the task will also be revoked which
is not ideal.
When set to true, the tokens generated for tasks will not be revoked when
Nomad's token is revoked. However Nomad will still revoke tokens when the
allocation is no longer running, minimizing the lifetime of any task's token.
With orphaned enabled, each Nomad server may also use a unique Vault token,
making bootstrapping and upgrading simpler. As such, **setting `orphan = true`
is the recommended setting**.
- `token_period` - Specifies the length the TTL is extended by each renewal in
seconds. It is suggested to set this value on the order of magnitude of 3 days
(259200 seconds) to avoid a large renewal request rate to Vault. **Must be set
to a positive value**.
- `renewable` - Specifies whether created tokens are renewable. **Must be set to
`true`**. This allows Nomad to renew tokens for tasks.
The above [`nomad-cluster` token role](/data/vault/nomad-cluster-role.json) is
available for download. Below is an example of writing this role to Vault:
```shell-session
# Download the token role
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
#### Example Configuration
To make getting started easy, the basic [`nomad-server`
policy](/data/vault/nomad-server-policy.hcl) and
[`nomad-cluster` role](/data/vault/nomad-cluster-role.json) described above are
available for download.
The below example assumes Vault is accessible, unsealed and the operator has
appropriate permissions.
```shell-session
# Download the policy and token role
$ curl https://nomadproject.io/data/vault/nomad-server-policy.hcl -O -s -L
$ curl https://nomadproject.io/data/vault/nomad-cluster-role.json -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
```
#### Retrieving the Token Role based Token
After the token role is created, a token suitable for the Nomad servers may be
retrieved by issuing the following Vault command:
```shell-session
$ vault token create -policy nomad-server -period 72h -orphan
Key Value
--- -----
token f02f01c2-c0d1-7cb7-6b88-8a14fada58c0
token_accessor 8cb7fcb3-9a4f-6fbf-0efc-83092bb0cb1c
token_duration 259200s
token_renewable true
token_policies [default nomad-server]
```
The `-orphan` flag is included when generating the Nomad server token above to
prevent revocation of the token when its parent expires. Vault typically
creates tokens with a parent-child relationship. When an ancestor token is
revoked, all of its descendant tokens and their associated leases are revoked
as well.
When generating Nomad's Vault token, we need to ensure that revocation of the
parent token does not revoke Nomad's token. To prevent this behavior we
specify the `-orphan` flag when we create the Nomad's Vault token. All
other tokens generated by Nomad for jobs will be generated using the policy
default of `orphan = false`.
More information about creating orphan tokens can be found in
[Vault's Token Hierarchies and Orphan Tokens documentation][tokenhierarchy].
The token can then be set in the server configuration's
[`vault` stanza][config], as a command-line flag, or via an environment
variable.
```shell-session
$ VAULT_TOKEN=f02f01c2-c0d1-7cb7-6b88-8a14fada58c0 nomad agent -config /path/to/config
```
An example of what may be contained in the configuration is shown below. For
complete documentation please see the [Nomad agent Vault integration][config]
configuration.
```hcl
vault {
enabled = true
ca_path = "/etc/certs/ca"
cert_file = "/var/certs/vault.crt"
key_file = "/var/certs/vault.key"
address = "https://vault.service.consul:8200"
create_from_role = "nomad-cluster"
}
```
## Agent Configuration
To enable Vault integration, please see the [Nomad agent Vault
integration][config] configuration.
## Vault Definition Syntax
To configure a job to retrieve Vault tokens, please see the [`vault` job
specification documentation][vault-spec].
## Troubleshooting
### Invalid Vault token
Upon startup, Nomad will attempt to connect to the specified Vault server. Nomad
will lookup the passed token and if the token is from a token role, the token
role will be validated. Nomad will not shutdown if given an invalid Vault token,
but will log the reasons the token is invalid and disable Vault integration.
### Permission Denied errors
If you are using a Vault version less than 0.7.1 with a Nomad version greater than or equal to 0.6.1, you will need to update your task's policy (listed in [the `vault` stanza of the job specification][vault-spec]) to add the following:
```hcl
path "sys/leases/renew" {
capabilities = ["update"]
}
```
This is included in Vault's "default" policy beginning with Vault 0.7.1 and is relied upon by Nomad's Vault integration beginning with Nomad 0.6.1. If you're using a newer Nomad version with an older Vault version, your default policy may not automatically include this and you will see "permission denied" errors in your Nomad logs similar to the following:
```plaintext
Code: 403. Errors:
URL: PUT https://vault:8200/v1/sys/leases/renew
* permission denied
```
### No Secret Exists
Vault has two APIs for secrets, [`v1` and `v2`][vault-secrets-version]. Each version
has different paths, and Nomad does not abstract this for you. As such you will
need to specify the path as reflected by Vault's HTTP API, rather than the path
used in the `vault kv` command.
You can see examples of `v1` and `v2` syntax in the
[template documentation][vault-kv-templates].
[auth]: https://www.vaultproject.io/docs/auth/token 'Vault Authentication Backend'
[config]: /docs/configuration/vault 'Nomad Vault Configuration Block'
[createfromrole]: /docs/configuration/vault#create_from_role 'Nomad vault create_from_role Configuration Flag'
[template]: /docs/job-specification/template 'Nomad template Job Specification'
[vault]: https://www.vaultproject.io/ 'Vault by HashiCorp'
[vault-spec]: /docs/job-specification/vault 'Nomad Vault Job Specification'
[tokenhierarchy]: https://www.vaultproject.io/docs/concepts/tokens#token-hierarchies-and-orphan-tokens 'Vault Tokens - Token Hierarchies and Orphan Tokens'
[vault-secrets-version]: https://www.vaultproject.io/docs/secrets/kv 'KV Secrets Engine'
[vault-kv-templates]: /docs/job-specification/template#vault-kv-api-v1 'Vault KV API v1'