39555db45f
This is a follow up work to https://github.com/hashicorp/nomad/pull/4813 to fix #4809 and fix a regression introduced in 0.9 in marking files in libcontainer executable. #4809 bug is that `lookupBin` uses `exec.LookPath` when not inspecting task dir files. `exec.LookPath` only returns a file if it's already marked as an executable path in https://github.com/golang/go/blob/go1.12.1/src/os/exec/lp_unix.go#L24-L27 . This affects raw exec as if passed an absolute path to file, `lookupBin` returns an error if file isn't already an executable. This explains why the error manifests when an absolute interpolated path is used (e.g. `${NOMAD_TASK_DIR}/hellov1`) but not when using a task rel dir (e.g. `local/hellov1`) in the above examples used in ticket. PR #4813 remedied this problem for raw exec but inadvertably broke libcontainer executor, as it made `lookupBin` returns the paths to host files rather than ones found inside chroot. This PR reorders the evaluation, so we go back to 0.8 behavior of looking up task directories first, but then check for host paths before using `exec.LookPath`. This PR is broken into three commits to illustrate evolution and confirming hypothesis: 1.9adab75ac8: Adding a test illustrating how libcontainer executor fails at marking processes as executable in https://travis-ci.org/hashicorp/nomad/jobs/514942694 - note that the test doesn't depend on artifacts or interpolated paths 2.d441cdd52f: reverting PR #4809 and showing the test fail now with raw_exec case (as expected) in https://travis-ci.org/hashicorp/nomad/jobs/514944065 2.244544b735: in where we add the check in appropriate place next to `exec.LookPath(...)` for absolute paths and have a green job in https://travis-ci.org/hashicorp/nomad/jobs/514945024 ## Future work Inspecting `lookupBin` in 0.8 and 0.9 case, we have a bug in using `exec.LookPath` for the libcontainer executor case. We should be looking up paths based on the container chroot and container PATH rather than the host's. However, this is not a 0.9.0 regression and was present in 0.8; so punting to fix it post 0.9.
Nomad 
- Website: www.nomadproject.io
- Mailing list: Google Groups
Nomad is a cluster manager, designed for both long lived services and short lived batch processing workloads. Developers use a declarative job specification to submit work, and Nomad ensures constraints are satisfied and resource utilization is optimized by efficient task packing. Nomad supports all major operating systems and virtualized, containerized, or standalone applications.
The key features of Nomad are:
-
Docker Support: Jobs can specify tasks which are Docker containers. Nomad will automatically run the containers on clients which have Docker installed, scale up and down based on the number of instances requested, and automatically recover from failures.
-
Operationally Simple: Nomad runs as a single binary that can be either a client or server, and is completely self contained. Nomad does not require any external services for storage or coordination. This means Nomad combines the features of a resource manager and scheduler in a single system.
-
Multi-Datacenter and Multi-Region Aware: Nomad is designed to be a global-scale scheduler. Multiple datacenters can be managed as part of a larger region, and jobs can be scheduled across datacenters if requested. Multiple regions join together and federate jobs making it easy to run jobs anywhere.
-
Flexible Workloads: Nomad has extensible support for task drivers, allowing it to run containerized, virtualized, and standalone applications. Users can easily start Docker containers, VMs, or application runtimes like Java. Nomad supports Linux, Windows, BSD, and OSX, providing the flexibility to run any workload.
-
Built for Scale: Nomad was designed from the ground up to support global scale infrastructure. Nomad is distributed and highly available, using both leader election and state replication to provide availability in the face of failures. Nomad is optimistically concurrent, enabling all servers to participate in scheduling decisions which increases the total throughput and reduces latency to support demanding workloads. Nomad has been proven to scale to cluster sizes that exceed 10k nodes in real-world production environments.
-
HashiCorp Ecosystem: HashiCorp Ecosystem: Nomad integrates with the entire HashiCorp ecosystem of tools. Like all HashiCorp tools, Nomad follows the UNIX design philosophy of doing something specific and doing it well. Nomad integrates with Terraform, Consul, and Vault for provisioning, service discovery, and secrets management.
For more information, see the introduction section of the Nomad website.
Getting Started & Documentation
All documentation is available on the Nomad website.
Developing Nomad
If you wish to work on Nomad itself or any of its built-in systems, you will first need Go installed on your machine (version 1.10.2+ is required).
Developing with Vagrant There is an included Vagrantfile that can help bootstrap the process. The created virtual machine is based off of Ubuntu 16, and installs several of the base libraries that can be used by Nomad.
To use this virtual machine, checkout Nomad and run vagrant up from the root
of the repository:
$ git clone https://github.com/hashicorp/nomad.git
$ cd nomad
$ vagrant up
The virtual machine will launch, and a provisioning script will install the needed dependencies.
Developing locally
For local dev first make sure Go is properly installed, including setting up a
GOPATH. After setting up Go, clone this
repository into $GOPATH/src/github.com/hashicorp/nomad. Then you can
download the required build tools such as vet, cover, godep etc by bootstrapping
your environment.
$ make bootstrap
...
Afterwards type make test. This will run the tests. If this exits with exit status 0,
then everything is working!
$ make test
...
To compile a development version of Nomad, run make dev. This will put the
Nomad binary in the bin and $GOPATH/bin folders:
$ make dev
Optionally run Consul to enable service discovery and health checks:
$ sudo consul agent -dev
And finally start the nomad agent:
$ sudo bin/nomad agent -dev
If the Nomad UI is desired in the development version, run make dev-ui. This will build the UI from source and compile it into the dev binary.
$ make dev-ui
...
$ bin/nomad
...
To compile protobuf files, installing protoc is required: See
https://github.com/google/protobuf for more information.
Note: Building the Nomad UI from source requires Node, Yarn, and Ember CLI. These tools are already in the Vagrant VM. Read the UI README for more info.
To cross-compile Nomad, run make prerelease and make release.
This will generate all the static assets, compile Nomad for multiple
platforms and place the resulting binaries into the ./pkg directory:
$ make prerelease
$ make release
...
$ ls ./pkg
...