0b6b475e7d
The e2e framework instantiates clients for Nomad/Consul but the provisioning of the actual Nomad cluster is left to Terraform. The Terraform provisioning process uses `remote-exec` to deploy specific versions of Nomad so that we don't have to bake an AMI every time we want to test a new version. But Terraform treats the resulting instances as immutable, so we can't use the same tooling to update the version of Nomad in-place. This is a prerequisite for upgrade testing. This changeset extends the e2e framework to provide the option of deploying Nomad (and, in the future, Consul/Vault) with specific versions to running infrastructure. This initial implementation is focused on deploying to a single cluster via `ssh` (because that's our current need), but provides interfaces to hook the test run at the start of the run, the start of each suite, or the start of a given test case. Terraform work includes: * provides Terraform output that written to JSON used by the framework to configure provisioning via `terraform output provisioning`. * provides Terraform output that can be used by test operators to configure their shell via `$(terraform output environment)` * drops `remote-exec` provisioning steps from Terraform * makes changes to the deployment scripts to ensure they can be run multiple times w/ different versions against the same host.
End to End Tests
This package contains integration tests.
The terraform folder has provisioning code to spin up a Nomad cluster on AWS. The tests work with the NOMAD_ADDR environment variable which can be set either to a local dev Nomad agent or a Nomad client on AWS.
Local Development
The workflow when developing end to end tests locally is to run the provisioning step described below once, and then run the tests as described below.
When making local changes, use ./bin/update $(which nomad) /usr/local/bin/nomad and ./bin/run sudo systemctl restart nomad to destructively modify the provisioned cluster.
Provisioning
You'll need AWS credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) to create the Nomad cluster. See the README for details. The number of servers and clients is configurable, as is the configuration file for each client and server.
Running
After completing the provisioning step above, you should see CLI output showing the IP addresses of Nomad client machines. To run the tests, set the NOMAD_ADDR variable to http://[client IP]:4646/
$ NOMAD_ADDR=<> NOMAD_E2E=1 go test -v