diff --git a/e2e/README.md b/e2e/README.md index 2774c356b..b8ede4f44 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -4,69 +4,31 @@ This package contains integration tests. Unlike tests alongside Nomad code, these tests expect there to already be a functional Nomad cluster accessible (either on localhost or via the `NOMAD_ADDR` env var). -The `terraform` folder has provisioning code to spin up a Nomad cluster on AWS. -The tests work with the `NOMAD_ADDR` environment variable which can be set -either to a local dev Nomad agent or a Nomad client on AWS. +See [`framework/doc.go`](framework/doc.go) for how to write tests. The `NOMAD_E2E=1` environment variable must be set for these tests to run. -## Local Nomad Development - -When developing tests locally, provisioning is not required when only the tests -change. See [`framework/doc.go`](framework/doc.go) for how to write tests. - -When making changes to the Nomad agent itself, use `./bin/update $(which nomad) -/usr/local/bin/nomad` and `./bin/run sudo systemctl restart nomad` to -destructively modify the provisioned cluster. - ## Provisioning Test Infrastructure on AWS -You'll need Terraform and AWS credentials (`AWS_ACCESS_KEY_ID` and -`AWS_SECRET_ACCESS_KEY`) to setup AWS instances on which e2e tests -will run. See the [README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md) -for details. The number of servers and clients is configurable, as is -the configuration file for each client and server. +The `terraform/` folder has provisioning code to spin up a Nomad cluster on +AWS. You'll need both Terraform and AWS credentials to setup AWS instances on +which e2e tests will run. See the +[README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md) +for details. The number of servers and clients is configurable, as is the +specific build of Nomad to deploy and the configuration file for each client +and server. -## Provisioning e2e Framework Nomad Cluster +## Provisioning Local Clusters -You can use the Terraform output from the [previous step](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md) -to generate a provisioning configuration file for the e2e framework. +To run tests against a local cluster, you'll need to make sure the following +environment variables are set: -```sh -# from the ./e2e/terraform directory -terraform output provisioning | jq . > ../provisioning.json -``` +* `NOMAD_ADDR` should point to one of the Nomad servers +* `CONSUL_HTTP_ADDR` should point to one of the Consul servers +* `NOMAD_E2E=1` -By default the `provisioning.json` will not include the Nomad version -that will be deployed to each node. You can pass the following flags -to `go test` to set the version for all nodes: - -- `-nomad.local_file=string`: provision this specific local binary of - Nomad. This is a path to a Nomad binary on your own - host. Ex. `-nomad.local_file=/home/me/nomad` -- `-nomad.sha=string`: provision this specific sha from S3. This is a - Nomad binary identified by its full commit SHA that's stored in a - shared s3 bucket that Nomad team developers can access. That commit - SHA can be from any branch that's pushed to - remote. Ex. `-nomad.sha=0b6b475e7da77fed25727ea9f01f155a58481b6c` -- `-nomad.version=string`: provision this version from - [releases.hashicorp.com](https://releases.hashicorp.com/nomad). Ex. `-nomad.version=0.10.2` - -Then deploy Nomad to the cluster by passing `-provision.terraform` -without a Nomad version flag: - -```sh -NOMAD_E2E=1 go test -v . \ - -timeout 20m \ - -nomad.local_file=$(which nomad) \ - -provision.terraform=./provisioning.json \ - -skipTests -``` - -- `-skipTests`: provisioning can take time, so it's best to skip tests - -- `-timeout 20m`: depending on your cluster size and upload bandwidth the - default 10m timeout may not be long enough for provisioning to finish +_TODO: the scripts in `./bin` currently work only with Terraform, it would be +nice for us to have a way to deploy Nomad to Vagrant or local clusters._ ## Running @@ -98,7 +60,7 @@ names in the full name of the tests: go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup' ^ ^ ^ ^ | | | | - Component | | Test func + Component | | Test func | | Go Package Struct ``` @@ -118,49 +80,38 @@ Run `terraform output` for IP addresses and details. ### ...Deploy a Cluster of Mixed Nomad Versions -The `provisioning.json` file output by Terraform has a blank field for -`nomad_sha` for each node of the cluster (server and client). You can -manually edit the file to replace this value with a `nomad_sha`, -`nomad_local_binary`, or `nomad_version` for each node to create a -cluster of mixed versions. The provisioning framework accepts any of -the following options for those fields: +The `variables.tf` file describes the `nomad_sha`, `nomad_version`, and +`nomad_local_binary` variable that can be used for most circumstances. But if +you want to deploy mixed Nomad versions, you can provide a list of versions in +your `terraform.tfvars` file. -- `nomad_sha`: This is a Nomad binary identified by its full commit - SHA that's stored in a shared s3 bucket that Nomad team developers - can access. That commit SHA can be from any branch that's pushed to - remote. (Ex. `"nomad_sha": - "0b6b475e7da77fed25727ea9f01f155a58481b6c"`) -- `nomad_local_binary`: This is a path to a Nomad binary on your own - host. (Ex. `"nomad_local_binary": "/home/me/nomad"`) -- `nomad_version`: This is a version number of Nomad that's been - released to HashiCorp. (Ex. `"nomad_version": "0.10.2"`) +For example, if you want to provision 3 servers all using Nomad 0.12.1, and 2 +Linux clients using 0.12.1 and 0.12.2, you can use the following variables: -Then deploy Nomad to the cluster by passing `-provision.terraform` -without a Nomad version flag: +```hcl +# will be used for servers +nomad_version = "0.12.1" -```sh -go test -v . -provision.terraform ./provisioning.json -skipTests +# will override the nomad_version for Linux clients +nomad_version_client_linux = [ + "0.12.1", + "0.12.2" +] ``` ### ...Deploy Custom Configuration Files -The `provisioning.json` file includes a `bundles` section for each -node of the cluster (server and client). You can manually edit this -file to add, remove, or replace - -```json -"bundles": [ - { - "destination": "/ops/shared/nomad/base.hcl", - "source": "/home/me/custom.hcl" - } -] -``` +Set the `profile` field to `"custom"` and put the configuration files in +`./terraform/config/custom/` as described in the +[README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md#Profiles). ### ...Deploy More Than 4 Linux Clients -Right now the framework doesn't support this out-of-the-box because of -the way the provisioning script adds specific client configurations to -each client node (for constraint testing). You'll need to add -additional configuration files to -`./e2e/terraform/shared/nomad/indexed`. +Use the `"custom"` profile as described above. + +### ...Change the Nomad Version After Provisioning + +You can update the `nomad_sha` or `nomad_version` variables, or simply rebuild +the binary you have at the `nomad_local_binary` path so that Terraform picks +up the changes. Then run `terraform plan`/`terraform apply` again. This will +update Nomad in place, making the minimum amount of changes necessary. diff --git a/e2e/terraform/Makefile b/e2e/terraform/Makefile index cd1b4c8ea..08094de6a 100644 --- a/e2e/terraform/Makefile +++ b/e2e/terraform/Makefile @@ -2,16 +2,31 @@ NOMAD_SHA ?= $(shell git rev-parse HEAD) PKG_PATH = $(shell pwd)/../../pkg/linux_amd64/nomad dev-cluster: - terraform apply -auto-approve -var-file=terraform.tfvars.dev - terraform output provisioning | jq . > ../provisioning.json - cd .. && NOMAD_E2E=1 go test -v . -nomad.sha=$(NOMAD_SHA) -provision.terraform ./provisioning.json -skipTests - terraform output message + terraform apply -auto-approve \ + -var="nomad_sha=$(NOMAD_SHA)" \ + -var-file=terraform.tfvars.dev + terraform output environment dev-cluster-from-local: - terraform apply -auto-approve -var-file=terraform.tfvars.dev - terraform output provisioning | jq . > ../provisioning.json - cd .. && NOMAD_E2E=1 go test -v . -nomad.local_file=$(PKG_PATH) -provision.terraform ./provisioning.json -skipTests - terraform output message + terraform apply -auto-approve \ + -var="nomad_local_binary=$(PKG_PATH)" \ + -var-file=terraform.tfvars.dev + terraform output environment clean: terraform destroy -auto-approve + +full-cluster: + terraform apply -auto-approve \ + -var="nomad_sha=$(NOMAD_SHA)" \ + -var-file=terraform.tfvars + +plan-dev-cluster: + terraform plan \ + -var="nomad_sha=$(NOMAD_SHA)" \ + -var-file=terraform.tfvars.dev + +plan-full-cluster: + terraform plan \ + -var="nomad_sha=$(NOMAD_SHA)" \ + -var-file=terraform.tfvars diff --git a/e2e/terraform/README.md b/e2e/terraform/README.md index ca82782c3..d47840f54 100644 --- a/e2e/terraform/README.md +++ b/e2e/terraform/README.md @@ -1,16 +1,11 @@ # Terraform infrastructure -This folder contains terraform resources for provisioning EC2 instances on AWS -to use as the target of end-to-end tests. +This folder contains Terraform resources for provisioning a Nomad cluster on +EC2 instances on AWS to use as the target of end-to-end tests. -Terraform provisions the AWS infrastructure only, whereas the Nomad -cluster is deployed to that infrastructure by the e2e -framework. Terraform's output will include a `provisioning` stanza -that can be written to a JSON file used by the e2e framework's -provisioning step. - -You can use Terraform to output the provisioning parameter JSON file the e2e -framework uses. +Terraform provisions the AWS infrastructure assuming that EC2 AMIs have +already been built via Packer. It deploys a specific build of Nomad to the +cluster along with configuration files for Nomad, Consul, and Vault. ## Setup @@ -28,6 +23,7 @@ instance_type = "t2.medium" server_count = "3" client_count = "4" windows_client_count = "1" +profile = "dev-cluster" ``` Run Terraform apply to deploy the infrastructure: @@ -37,6 +33,67 @@ cd e2e/terraform/ terraform apply ``` +> Note: You will likely see "Connection refused" or "Permission denied" errors +> in the logs as the provisioning script run by Terraform hits an instance +> where the ssh service isn't yet ready. That's ok and expected; they'll get +> retried. In particular, Windows instances can take a few minutes before ssh +> is ready. + +## Nomad Version + +You'll need to pass one of the following variables in either your +`terraform.tfvars` file or as a command line argument (ex. `terraform apply +-var 'nomad_version=0.10.2+ent'`) + +* `nomad_local_binary`: provision this specific local binary of Nomad. This is + a path to a Nomad binary on your own host. Ex. `nomad_local_binary = + "/home/me/nomad"`. +* `nomad_sha`: provision this specific sha from S3. This is a Nomad binary + identified by its full commit SHA that's stored in a shared s3 bucket that + Nomad team developers can access. That commit SHA can be from any branch + that's pushed to remote. Ex. `nomad_sha = + "0b6b475e7da77fed25727ea9f01f155a58481b6c"` +* `nomad_version`: provision this version from + [releases.hashicorp.com](https://releases.hashicorp.com/nomad). Ex. `nomad_version + = "0.10.2+ent"` + +## Profiles + +The `profile` field selects from a set of configuration files for Nomad, +Consul, and Vault by uploading the files found in `./config/`. The +profiles are as follows: + +* `full-cluster`: This profile is used for nightly E2E testing. It assumes at + least 3 servers and includes a unique config for each Nomad client. +* `dev-cluster`: This profile is used for developer testing of a more limited + set of clients. It assumes at least 3 servers but uses the one config for + all the Linux Nomad clients and one config for all the Windows Nomad + clients. +* `custom`: This profile is used for one-off developer testing of more complex + interactions between features. You can build your own custom profile by + writing config files to the `./config/custom` directory, which are protected + by `.gitignore` + +For each profile, application (Nomad, Consul, Vault), and agent type +(`server`, `client_linux`, or `client_windows`), the agent gets the following +configuration files, ignoring any that are missing. + +* `./config///*`: base configurations shared between all + servers and clients. +* `./config////*`: base configurations shared + between all agents of this type. +* `./config////indexed/*.`: a + configuration for that particular agent, where the index value is the index + of that agent within the total count. + +For example, with the `full-cluster` profile, 2nd Nomad server would get the +following configuration files: +* `./config/full-cluster/nomad/base.hcl` +* `./config/full-cluster/nomad/server/indexed/server-1.hcl` + +The directory `./config/full-cluster/nomad/server` has no configuration files, +so that's safely skipped. + ## Outputs After deploying the infrastructure, you can get connection information @@ -50,8 +107,6 @@ about the cluster: client node IPs. - `terraform output windows_clients` will output the list of Windows client node IPs. -- `terraform output provisioning | jq .` will output the JSON used by - the e2e framework for provisioning. ## SSH diff --git a/e2e/terraform/nomad.tf b/e2e/terraform/nomad.tf new file mode 100644 index 000000000..ffa21a088 --- /dev/null +++ b/e2e/terraform/nomad.tf @@ -0,0 +1,91 @@ +module "nomad_server" { + + source = "./provision-nomad" + depends_on = [aws_instance.server] + count = var.server_count + + platform = "linux_amd64" + profile = var.profile + role = "server" + index = count.index + + # The specific version of Nomad deployed will default to whichever one of + # nomad_sha, nomad_version, or nomad_local_binary is set, but if you want to + # deploy multiple versions you can use the nomad_*_server variables to + # provide a list of builds + nomad_version = count.index < length(var.nomad_version_server) ? var.nomad_version_server[count.index] : var.nomad_version + + nomad_sha = count.index < length(var.nomad_sha_server) ? var.nomad_sha_server[count.index] : var.nomad_sha + + nomad_local_binary = count.index < length(var.nomad_local_binary_server) ? var.nomad_local_binary_server[count.index] : var.nomad_local_binary + + connection = { + type = "ssh" + user = "ubuntu" + host = "${aws_instance.server[count.index].public_ip}" + port = 22 + private_key = "${path.root}/keys/${local.random_name}.pem" + } +} + +# TODO: split out the different Linux targets (ubuntu, centos, arm, etc.) when +# they're available +module "nomad_client_linux" { + + source = "./provision-nomad" + depends_on = [aws_instance.client_linux] + count = var.client_count + + platform = "linux_amd64" + profile = var.profile + role = "client-linux" + index = count.index + + # The specific version of Nomad deployed will default to whichever one of + # nomad_sha, nomad_version, or nomad_local_binary is set, but if you want to + # deploy multiple versions you can use the nomad_*_client_linux + # variables to provide a list of builds + nomad_version = count.index < length(var.nomad_version_client_linux) ? var.nomad_version_client_linux[count.index] : var.nomad_version + + nomad_sha = count.index < length(var.nomad_sha_client_linux) ? var.nomad_sha_client_linux[count.index] : var.nomad_sha + + nomad_local_binary = count.index < length(var.nomad_local_binary_client_linux) ? var.nomad_local_binary_client_linux[count.index] : var.nomad_local_binary + + connection = { + type = "ssh" + user = "ubuntu" + host = "${aws_instance.client_linux[count.index].public_ip}" + port = 22 + private_key = "${path.root}/keys/${local.random_name}.pem" + } +} + +# TODO: split out the different Windows targets (2016, 2019) when they're +# available +module "nomad_client_windows" { + + source = "./provision-nomad" + depends_on = [aws_instance.client_windows] + count = var.windows_client_count + + platform = "windows_amd64" + profile = var.profile + role = "client-windows" + index = count.index + + # The specific version of Nomad deployed will default to whichever one of + # nomad_sha, nomad_version, or nomad_local_binary is set, but if you want to + # deploy multiple versions you can use the nomad_*_client_windows + # variables to provide a list of builds + nomad_version = count.index < length(var.nomad_version_client_windows) ? var.nomad_version_client_windows[count.index] : var.nomad_version + + nomad_sha = count.index < length(var.nomad_sha_client_windows) ? var.nomad_sha_client_windows[count.index] : var.nomad_sha + + connection = { + type = "ssh" + user = "Administrator" + host = "${aws_instance.client_windows[count.index].public_ip}" + port = 22 + private_key = "${path.root}/keys/${local.random_name}.pem" + } +} diff --git a/e2e/terraform/outputs.tf b/e2e/terraform/outputs.tf index fbd1e0227..794e7d6ab 100644 --- a/e2e/terraform/outputs.tf +++ b/e2e/terraform/outputs.tf @@ -35,7 +35,7 @@ EOM output "environment" { description = "get connection config by running: $(terraform output environment)" - value = </dev/null \ + | sudo xargs -I % ln -fs % "$3" +} + +install_config_profile() { + + if [ -d /tmp/custom ]; then + rm -rf /opt/config/custom + sudo mv /tmp/custom /opt/config/ + fi + + # we're removing the whole directory and recreating to avoid + # any quirks around dotfiles that might show up here. + sudo rm -rf /etc/nomad.d + sudo rm -rf /etc/consul.d + sudo rm -rf /etc/vault.d + + sudo mkdir -p /etc/nomad.d + sudo mkdir -p /etc/consul.d + sudo mkdir -p /etc/vault.d + + sym "${NOMAD_PROFILE}/nomad/" '*' /etc/nomad.d + sym "${NOMAD_PROFILE}/consul/" '*' /etc/consul.d + sym "${NOMAD_PROFILE}/vault/" '*' /etc/vault.d + + if [ -n "$NOMAD_ROLE" ]; then + sym "${NOMAD_PROFILE}/nomad/${NOMAD_ROLE}/" '*' /etc/nomad.d + sym "${NOMAD_PROFILE}/consul/${NOMAD_ROLE}/" '*' /etc/consul.d + sym "${NOMAD_PROFILE}/vault/${NOMAD_ROLE}/" '*' /etc/vault.d + fi + if [ -n "$NOMAD_INDEX" ]; then + sym "${NOMAD_PROFILE}/nomad/${NOMAD_ROLE}/indexed/" "*${NOMAD_INDEX}*" /etc/nomad.d + sym "${NOMAD_PROFILE}/consul/${NOMAD_ROLE}/indexed/" "*${NOMAD_INDEX}*" /etc/consul.d + sym "${NOMAD_PROFILE}/vault/${NOMAD_ROLE}/indexed/" "*${NOMAD_INDEX}*" /etc/vault.d + fi +} + + while [[ $# -gt 0 ]] do opt="$1" @@ -89,6 +133,21 @@ opt="$1" install_fn=install_from_uploaded_binary shift 2 ;; + --config_profile) + if [ -z "$2" ]; then echo "Missing profile parameter"; usage; fi + NOMAD_PROFILE="/opt/config/${2}" + shift 2 + ;; + --role) + if [ -z "$2" ]; then echo "Missing role parameter"; usage; fi + NOMAD_ROLE="$2" + shift 2 + ;; + --index) + if [ -z "$2" ]; then echo "Missing index parameter"; usage; fi + NOMAD_INDEX="$2" + shift 2 + ;; --nostart) # for initial packer builds, we don't want to start Nomad START=0 @@ -98,6 +157,16 @@ opt="$1" esac done -# call the appropriate instalation function -if [ -z "$install_fn" ]; then echo "Missing install option"; usage; fi -$install_fn +# call the appropriate installation function +if [ -n "$install_fn" ]; then + $install_fn +fi +if [ -n "$NOMAD_PROFILE" ]; then + install_config_profile +fi + +if [ $START == "1" ]; then + # sudo systemctl restart vault + sudo systemctl restart consul + sudo systemctl restart nomad +fi diff --git a/e2e/terraform/packer/linux/setup.sh b/e2e/terraform/packer/linux/setup.sh index a55923302..a2696e185 100755 --- a/e2e/terraform/packer/linux/setup.sh +++ b/e2e/terraform/packer/linux/setup.sh @@ -72,6 +72,7 @@ mkdir_for_root $NOMAD_PLUGIN_DIR sudo mv /tmp/linux/nomad.service /etc/systemd/system/nomad.service echo "Install Nomad" +sudo mv /tmp/config /opt/ sudo mv /tmp/linux/provision.sh /opt/provision.sh sudo chmod +x /opt/provision.sh /opt/provision.sh --nomad_version $NOMADVERSION --nostart diff --git a/e2e/terraform/packer/packer-windows.json b/e2e/terraform/packer/packer-windows.json index 047c97f1f..53368b216 100644 --- a/e2e/terraform/packer/packer-windows.json +++ b/e2e/terraform/packer/packer-windows.json @@ -46,6 +46,11 @@ { "type": "windows-restart" }, + { + "type": "file", + "source": "../config", + "destination": "/opt" + }, { "type": "file", "source": "./windows/provision.ps1", diff --git a/e2e/terraform/packer/packer.json b/e2e/terraform/packer/packer.json index accf322c0..2fae1dd2c 100644 --- a/e2e/terraform/packer/packer.json +++ b/e2e/terraform/packer/packer.json @@ -19,6 +19,11 @@ "source": "./linux", "destination": "/tmp/linux" }, + { + "type": "file", + "source": "../config", + "destination": "/tmp/config" + }, { "type": "shell", "script": "./linux/setup.sh" diff --git a/e2e/terraform/packer/windows/provision.ps1 b/e2e/terraform/packer/windows/provision.ps1 index c96dfd57e..e1745b8ec 100755 --- a/e2e/terraform/packer/windows/provision.ps1 +++ b/e2e/terraform/packer/windows/provision.ps1 @@ -2,21 +2,27 @@ param( [string]$nomad_sha, [string]$nomad_version, [string]$nomad_binary, + [string]$config_profile, + [string]$role, + [string]$index, [switch]$nostart = $false ) Set-StrictMode -Version latest $ErrorActionPreference = "Stop" - - $usage = @" Usage: provision.ps1 [options...] Options (use one of the following): - --nomad_sha SHA full git sha to install from S3 - --nomad_version VERSION release version number (ex. 0.12.4+ent) - --nomad_binary FILEPATH path to file on host - --nostart do not start or restart Nomad + -nomad_sha SHA full git sha to install from S3 + -nomad_version VERSION release version number (ex. 0.12.4+ent) + -nomad_binary FILEPATH path to file on host + +Options for configuration: + -config_profile FILEPATH path to config profile directory + -role ROLE role within config profile directory + -index INDEX count of instance, for profiles with per-instance config + -nostart do not start or restart Nomad "@ $RunningAsAdmin = ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator") @@ -61,10 +67,6 @@ function InstallFromS3 { New-Item -ItemType Directory -Force -Path C:\opt\nomad.d New-Item -ItemType Directory -Force -Path C:\opt\nomad Write-Output "Installed Nomad." - - if (!($nostart)) { - StartNomad - } } Catch { Write-Error "Failed to install Nomad." $host.SetShouldExit(-1) @@ -80,10 +82,6 @@ function InstallFromUploadedBinary { New-Item -ItemType Directory -Force -Path C:\opt\nomad.d New-Item -ItemType Directory -Force -Path C:\opt\nomad Write-Output "Installed Nomad." - - if (!($nostart)) { - StartNomad - } } Catch { Write-Error "Failed to install Nomad." $host.SetShouldExit(-1) @@ -119,10 +117,6 @@ function InstallFromRelease { New-Item -ItemType Directory -Force -Path C:\opt\nomad.d New-Item -ItemType Directory -Force -Path C:\opt\nomad Write-Output "Installed Nomad." - - if (!($nostart)) { - StartNomad - } } Catch { Write-Error "Failed to install Nomad." $host.SetShouldExit(-1) @@ -130,7 +124,48 @@ function InstallFromRelease { } } -function StartNomad { + +function ConfigFiles($src, $dest) { + Get-ChildItem -Path "$src" -Name -Attributes !Directory -ErrorAction Ignore` + | ForEach-Object { ` + New-Item -ItemType SymbolicLink -Path "${dest}\$_" -Target "${src}\$_" } +} + +function InstallConfigProfile { + + if ( Test-Path -Path 'C:\tmp\custom' -PathType Container ) { + Remote-Item 'C:\opt\config\custom' -Force -ErrorAction Ignore + Move-Item -Path 'C:\tmp\custom' -Destination 'C:\opt\config\custom' -Force + } + + $cfg = "C:\opt\config\${config_profile}" + + Remove-Item "C:\opt\nomad.d\*" -Force -ErrorAction Ignore + Remove-Item "C:\opt\consul.d\*" -Force -ErrorAction Ignore + + ConfigFiles "${cfg}\nomad" "C:\opt\nomad.d" + ConfigFiles "${cfg}\consul" "C:\opt\consul.d" + + if ( "" -ne $role ) { + ConfigFiles "${cfg}\nomad\${role}" "C:\opt\nomad.d" + ConfigFiles "${cfg}\consul\${role}" "C:\opt\consul.d" + } + + if ( "" -ne $index ) { + ConfigFiles "${cfg}\nomad\${role}\indexed\*${index}*" "C:\opt\nomad.d" + ConfigFiles "${cfg}\consul\${role}\indexed\*${index}*" "C:\opt\consul.d" + } +} + +function CreateConsulService { + New-Service ` + -Name "Consul" ` + -BinaryPathName "C:\opt\consul.exe agent -config-dir C:\opt\consul.d" ` + -StartupType "Automatic" ` + -ErrorAction Ignore +} + +function CreateNomadService { New-NetFirewallRule ` -DisplayName 'Nomad HTTP Inbound' ` -Profile @('Public', 'Domain', 'Private') ` @@ -145,21 +180,27 @@ function StartNomad { -BinaryPathName "C:\opt\nomad.exe agent -config C:\opt\nomad.d" ` -StartupType "Automatic" ` -ErrorAction Ignore - - Start-Service "Nomad" } if ( "" -ne $nomad_sha ) { InstallFromS3 - return + CreateNomadService } if ( "" -ne $nomad_version ) { InstallFromRelease - return + CreateNomadService } if ( "" -ne $nomad_binary ) { InstallFromUploadedBinary - return + CreateNomadService +} +if ( "" -ne $config_profile) { + InstallConfigProfile } -Usage +if (!($nostart)) { + CreateConsulService + CreateNomadService + Restart-Service "Consul" + Restart-Service "Nomad" +} diff --git a/e2e/terraform/provision-nomad/main.tf b/e2e/terraform/provision-nomad/main.tf new file mode 100644 index 000000000..2f630dee3 --- /dev/null +++ b/e2e/terraform/provision-nomad/main.tf @@ -0,0 +1,119 @@ +locals { + provision_script = var.platform == "windows_amd64" ? "C:/opt/provision.ps1" : "/opt/provision.sh" + + custom_path = abspath("${var.config_path}/custom/") + + custom_config_files = compact(setunion( + fileset(local.custom_path, "nomad/*.hcl"), + fileset(local.custom_path, "nomad/${var.role}/*.hcl"), + fileset(local.custom_path, "nomad/${var.role}/indexed/*${var.index}.hcl"), + fileset(local.custom_path, "consul/*.json"), + fileset(local.custom_path, "consul/${var.role}/*.json"), + fileset(local.custom_path, "consul${var.role}indexed/*${var.index}*.json"), + fileset(local.custom_path, "vault/*.hcl"), + fileset(local.custom_path, "vault${var.role}*.hcl"), + fileset(local.custom_path, "vault${var.role}indexed/*${var.index}.hcl"), + )) + + # abstract-away platform-specific parameter expectations + _arg = var.platform == "windows_amd64" ? "-" : "--" +} + +resource "null_resource" "provision_nomad" { + + depends_on = [ + null_resource.upload_custom_configs, + null_resource.upload_nomad_binary + ] + + # no need to re-run if nothing changes + triggers = { + script = data.template_file.provision_script.rendered + } + + # Run the provisioner as a local-exec'd ssh command as a workaround for + # Windows remote-exec zero-byte scripts bug: + # https://github.com/hashicorp/terraform/issues/25634 + # + # The retry behavior and explicit PasswordAuthenticaiton flag here are to + # workaround a race with the Windows userdata script that installs the + # authorized_key. Unfortunately this still results in a bunch of "permission + # denied" errors while waiting for those keys to be configured. + provisioner "local-exec" { + command = "until ssh -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${var.connection.private_key} -p ${var.connection.port} ${var.connection.user}@${var.connection.host} ${data.template_file.provision_script.rendered}; do sleep 5; done" + } + +} + +data "template_file" "provision_script" { + template = "${local.provision_script}${data.template_file.arg_nomad_sha.rendered}${data.template_file.arg_nomad_version.rendered}${data.template_file.arg_nomad_binary.rendered}${data.template_file.arg_profile.rendered}${data.template_file.arg_role.rendered}${data.template_file.arg_index.rendered}" +} + +data "template_file" "arg_nomad_sha" { + template = var.nomad_sha != "" ? " ${local._arg}nomad_sha ${var.nomad_sha}" : "" +} + +data "template_file" "arg_nomad_version" { + template = var.nomad_version != "" ? " ${local._arg}nomad_version ${var.nomad_version}" : "" +} + +data "template_file" "arg_nomad_binary" { + template = var.nomad_local_binary != "" ? " ${local._arg}nomad_binary ${var.nomad_local_binary}" : "" +} + +data "template_file" "arg_profile" { + template = var.profile != "" ? " ${local._arg}config_profile ${var.profile}" : "" +} + +data "template_file" "arg_role" { + template = var.role != "" ? " ${local._arg}role ${var.role}" : "" +} + +data "template_file" "arg_index" { + template = var.index != "" ? " ${local._arg}index ${var.index}" : "" +} + +resource "null_resource" "upload_nomad_binary" { + + count = var.nomad_local_binary != "" ? 1 : 0 + depends_on = [null_resource.upload_custom_configs] + triggers = { + nomad_binary_sha = filemd5(var.nomad_local_binary) + } + + connection { + type = "ssh" + user = var.connection.user + host = var.connection.host + port = var.connection.port + private_key = var.connection.private_key + timeout = "15m" + } + + provisioner "file" { + source = var.nomad_local_binary + destination = "/tmp/nomad" + } +} + +resource "null_resource" "upload_custom_configs" { + + count = var.profile == "custom" ? 1 : 0 + triggers = { + hashes = "${join(",", [for file in local.custom_config_files : filemd5(file)])}" + } + + connection { + type = "ssh" + user = var.connection.user + host = var.connection.host + port = var.connection.port + private_key = var.connection.private_key + timeout = "15m" + } + + provisioner "file" { + source = local.custom_path + destination = "/tmp/" + } +} diff --git a/e2e/terraform/provision-nomad/variables.tf b/e2e/terraform/provision-nomad/variables.tf new file mode 100644 index 000000000..c6e2d5802 --- /dev/null +++ b/e2e/terraform/provision-nomad/variables.tf @@ -0,0 +1,58 @@ +variable "platform" { + type = string + description = "Platform ID (ex. \"linux_amd64\" or \"windows_amd64\")" + default = "linux_amd64" +} + +variable "nomad_version" { + type = string + description = "Nomad release version (ex. \"0.10.3\")" + default = "" +} + +variable "nomad_sha" { + type = string + description = "Nomad build full SHA (ex. \"fef22bdbfa094b5d076710354275e360867261aa\")" + default = "" +} + +variable "nomad_local_binary" { + type = string + description = "Path to local Nomad build (ex. \"/home/me/bin/nomad\")" + default = "" +} + +variable "profile" { + type = string + description = "The name of the configuration profile (ex. 'full-cluster')" + default = "" +} + +variable "role" { + type = string + description = "The role in the configuration profile for this instance (ex. 'client-linux')" + default = "" +} + +variable "index" { + type = string # note that we have string here so we can default to "" + description = "The count of this instance for indexed configurations" + default = "" +} + +variable "config_path" { + type = string + description = "The path to the config directory" + default = "../config" +} + +variable "connection" { + type = object({ + type = string + user = string + host = string + port = number + private_key = string + }) + description = "ssh connection information for remote target" +} diff --git a/e2e/terraform/terraform.tfvars b/e2e/terraform/terraform.tfvars index 69e4aea71..fc54622ee 100644 --- a/e2e/terraform/terraform.tfvars +++ b/e2e/terraform/terraform.tfvars @@ -3,3 +3,4 @@ instance_type = "t2.medium" server_count = "3" client_count = "4" windows_client_count = "1" +profile = "full-cluster" diff --git a/e2e/terraform/terraform.tfvars.dev b/e2e/terraform/terraform.tfvars.dev index c89a8aa83..89dd0302e 100644 --- a/e2e/terraform/terraform.tfvars.dev +++ b/e2e/terraform/terraform.tfvars.dev @@ -3,3 +3,8 @@ instance_type = "t2.medium" server_count = "3" client_count = "2" windows_client_count = "0" +profile = "dev-cluster" + +# Example overrides: +# nomad_local_binary = "../../pkg/linux_amd/nomad" +# nomad_local_binary_client_windows = ["../../pkg/windows_amd64/nomad.exe"] diff --git a/e2e/terraform/variables.tf b/e2e/terraform/variables.tf index 3e44cf470..a383823c7 100644 --- a/e2e/terraform/variables.tf +++ b/e2e/terraform/variables.tf @@ -13,11 +13,6 @@ variable "availability_zone" { default = "us-east-1a" } -variable "indexed" { - description = "Different configurations per client/server" - default = true -} - variable "instance_type" { description = "The AWS instance type to use for both clients and servers." default = "t2.medium" @@ -38,11 +33,6 @@ variable "windows_client_count" { default = "1" } -variable "nomad_sha" { - description = "The sha of Nomad to write to provisioning output" - default = "" -} - variable "aws_assume_role_arn" { description = "The AWS IAM role to assume (not used by human users)" default = "" @@ -57,3 +47,87 @@ variable "aws_assume_role_external_id" { description = "The AWS IAM external ID to assume (not used by human users)" default = "" } + +variable "profile" { + description = "A default Nomad/Consul/Vault configuration profile" + type = string + default = "" +} + +# ---------------------------------------- +# The specific version of Nomad deployed will default to whichever one of +# nomad_sha, nomad_version, or nomad_local_binary is set + +variable "nomad_sha" { + description = "The sha of Nomad to provision" + default = "" +} + +variable "nomad_version" { + description = "The release version of Nomad to provision" + default = "" +} + +variable "nomad_local_binary" { + description = "The path to a local binary to provision" + default = "" +} + +# ---------------------------------------- +# If you want to deploy multiple versions you can use these variables to +# provide a list of builds to override the values of nomad_sha, nomad_version, +# or nomad_local_binary. Most of the time you can ignore these variables! + +variable "nomad_version_server" { + description = "A list of Nomad versions to deploy to servers, to override nomad_version" + type = list(string) + default = [] +} + +variable "nomad_sha_server" { + description = "A list of Nomad SHAs to deploy to servers, to override nomad_sha" + type = list(string) + default = [] +} + +variable "nomad_local_binary_server" { + description = "A list of Nomad SHAs to deploy to servers, to override nomad_sha" + type = list(string) + default = [] +} + +variable "nomad_version_client_linux" { + description = "A list of Nomad versions to deploy to Linux clients, to override nomad_version" + type = list(string) + default = [] +} + +variable "nomad_sha_client_linux" { + description = "A list of Nomad SHAs to deploy to Linux clients, to override nomad_sha" + type = list(string) + default = [] +} + +variable "nomad_local_binary_client_linux" { + description = "A list of Nomad SHAs to deploy to Linux clients, to override nomad_sha" + type = list(string) + default = [] +} + +variable "nomad_version_client_windows" { + description = "A list of Nomad versions to deploy to Windows clients, to override nomad_version" + type = list(string) + default = [] +} + +variable "nomad_sha_client_windows" { + description = "A list of Nomad SHAs to deploy to Windows clients, to override nomad_sha" + type = list(string) + default = [] +} + +variable "nomad_local_binary_client_windows" { + description = "A list of Nomad SHAs to deploy to Windows clients, to override nomad_sha" + type = list(string) + default = [] +}