nomad/CHANGELOG.md

1.10.1 (May 13, 2025)

BREAKING CHANGES:

• api: The non-functional option -peer-address has been removed from the operator raft remove-peer command and equivalent API [GH-25599]
• core: Errors encountered when reloading agent configuration will now cause agents to exit. Before configuration errors during reloads were only logged. This could lead to agents running but unable to communicate [GH-25721]

SECURITY:

• build: Update Go to 1.24.3 to address CVE-2025-22873 [GH-25818]
• sentinel (Enterprise): Fixed a bug where in some cases hard-mandatory policies could be overridden with -policy-override. CVE-2025-3744 [GH-2618]

IMPROVEMENTS:

• command: added priority flag to job dispatch command [GH-25622]

BUG FIXES:

• agent: Fixed a bug where reloading the agent with systemd notification enabled would cause the agent to be killed by system [GH-25636]
• cli: Respect NOMAD_REGION environment variable in operator debug command [GH-25716]
• client: fix failure cleaning up namespace on batch jobs [GH-25714]
• docker: Fix missing stats for rss, cache and swap memory for cgroups v1 [GH-25741]
• encrypter: Refactor startup decryption task handling to avoid timing problems with task addition on FSM restore [GH-25795]
• java: Fixed a bug where the default task user was set to 'nobody' on Windows [GH-25648]
• metrics: Fixed a bug where RSS and cache stats would not be reported for docker, exec, and java drivers under Linux cgroups v2 [GH-25751]
• scheduler: Fixed a bug in accounting for resources.cores that could prevent placements on nodes with available cores [GH-25705]
• scheduler: Fixed a bug where draining a node with canaries could result in a stuck deployment [GH-25726]
• scheduler: Fixed a bug where updating the rescheduler tracker could corrupt the state store [GH-25698]
• scheduler: Use core ID when selecting cores. This fixes a panic in the scheduler when the reservable_cores is not a contiguous list of core IDs. [GH-25340]
• server: Added a new server configuration option named start_timeout with a default value of 30s. This duration is used to monitor the server setup and startup processes which must complete before it is considered healthy, such as keyring decryption. If these processes do not complete before the timeout is reached, the server process will exit. [GH-25803]
• ui: Fixed a bug where the job list page incorrectly calculated if a job had paused tasks. [GH-25742]

1.10.0 (April 09, 2025)

FEATURES:

• Dynamic Host Volumes: Nomad now supports creating host volumes via the API [GH-24479]
• OIDC Login: Nomad now enables PKCE for OIDC logins, and supports the private key JWT / client assertion option in the OIDC authentication flow. [GH-25231]
• Stateful Deployments: Nomad now supports stateful deployments when using dynamic host volumes. [GH-24993]

BREAKING CHANGES:

• agent: Plugins stored within the plugin_dir will now only be executed when they have a corresponding plugin configuration block. Any plugin found without a corresponding configuration block will be skipped. [GH-18530]
• api: QuotaSpec.RegionLimit is now of type QuotaResources instead of Resources [GH-24785]
• consul: Identities are no longer added to tasks by default when they include a template block. Please see Nomad's upgrade guide for more detail. [GH-25298]
• consul: The deprecated token-based authentication workflow for allocations has been removed. Please see Nomad's upgrade guide for more detail. [GH-25217]
• disconnected nodes: ignore the previously deprecated disconnect group fields in favor of the disconnect block introduced in Nomad 1.8 [GH-25284]
• drivers: remove remote task support for task drivers [GH-24909]
• sentinel: The sentinel apply command now requires the -scope option [GH-24601]
• vault: The deprecated token-based authentication workflow for allocations has been removed. Please see Nomad's upgrade guide for more detail. [GH-25155]

IMPROVEMENTS:

• build: Updated Go to 1.24.2 [GH-25623]
• cli: Add -group option to alloc exec, alloc logs, alloc fs commands [GH-25568]
• cli: Added UI URL hints to the end of common CLI commands and a -ui flag to auto-open them [GH-24454]
• client: Fixed a bug where JSON formatted logs would not show the requested and overlapping cores when failing to reserve cores [GH-25523]
• client: Improve memory usage by dropping references to task environment [GH-25373]
• cni: Add a warning log when CNI check commands fail [GH-25581]
• csi: Accept ID prefixes and wildcard namespace for the volume delete command [GH-24997]
• csi: Added CSI volume and plugin events to the event stream [GH-24724]
• csi: Show volume capabilities in the volume status command [GH-25173]
• drivers/docker: adds image_pull_timeout to plugin config options [GH-25489]
• drivers/rawexec: adds denied_envvars to driver and task config options [GH-25511]
• rawexec: add support for setting the task user on windows platform [GH-25496]
• rpc: Added ability to configure yamux session parameters [GH-25466]
• ui: Added Dynamic Host Volumes to the web UI [GH-25224]
• ui: Added a scope selector for sentinel policy page [GH-25390]
• ui: Makes jobs list filtering case-insensitive [GH-25378]
• ui: Updated icons to the newest design system [GH-25353]

DEPRECATIONS:

• api: QuotaSpec.VariablesLimit field is deprecated and will be removed in Nomad 1.12.0. Use QuotaSpec.RegionLimit.Storage.Variables instead. [GH-24785]
• quotas: the variables_limit field in the quota specification is deprecated and replaced by a new storage block under the region_limit block, with a variables field. The variables_limit field will be removed in Nomad 1.12.0 [GH-24785]

BUG FIXES:

• client: fixed a bug where AMD CPUs were not correctly fingerprinting base speed [GH-24415]
• client: remove blocking call during client gc [GH-25123]
• client: skip a task groups shutdown_delay when all tasks have already been deregistered [GH-25157]
• csi: Fixed a CSI ExpandVolume bug where the namespace was left out of the staging path [GH-25253]
• csi: Fixed a bug where GC would attempt and fail to delete plugins that had volumes [GH-25432]
• csi: Fixed a bug where cleaning up volume claims on GC'd nodes would cause errors on the leader [GH-25428]
• csi: Fixed a bug where in-flight CSI RPCs would not be cancelled on client GC or dev agent shutdown [GH-25472]
• drivers: set -1 exit code in case of executor failure for the exec, raw_exec, java, and qemu task drivers [GH-25453]
• job: Ensure migrate block difference is added to planning diff object [GH-25528]
• scheduler: Fixed a bug that made affinity and spread updates destructive [GH-25109]
• server: Validate num_schedulers configuration parameter is between 0 and the number of CPUs available on the machine [GH-25441]
• services: Fixed a bug where Nomad native services would not be correctly interpolated during in-place updates [GH-25373]
• services: Fixed a bug where task-level services, checks, and identities could interpolate jobspec values from other tasks in the same group [GH-25373]

1.9.9 Enterprise (May 13, 2025)

BREAKING CHANGES:

• core: Errors encountered when reloading agent configuration will now cause agents to exit. Before configuration errors during reloads were only logged. This could lead to agents running but unable to communicate [GH-25721]

SECURITY:

• build: Update Go to 1.24.3 to address CVE-2025-22873 [GH-25818]
• sentinel (Enterprise): Fixed a bug where in some cases hard-mandatory policies could be overridden with -policy-override. CVE-2025-3744.

BUG FIXES:

• agent: Fixed a bug where reloading the agent with systemd notification enabled would cause the agent to be killed by system [GH-25636]
• api: Fixed pagination bug which could result in duplicate results [GH-25792]
• cli: Respect NOMAD_REGION environment variable in operator debug command [GH-25716]
• client: fix failure cleaning up namespace on batch jobs [GH-25714]
• docker: Fix missing stats for rss, cache and swap memory for cgroups v1 [GH-25741]
• encrypter: Refactor startup decryption task handling to avoid timing problems with task addition on FSM restore [GH-25795]
• metrics: Fixed a bug where RSS and cache stats would not be reported for docker, exec, and java drivers under Linux cgroups v2 [GH-25751]
• scheduler: Fixed a bug in accounting for resources.cores that could prevent placements on nodes with available cores [GH-25705]
• scheduler: Fixed a bug where draining a node with canaries could result in a stuck deployment [GH-25726]
• scheduler: Fixed a bug where updating the rescheduler tracker could corrupt the state store [GH-25698]
• scheduler: Use core ID when selecting cores. This fixes a panic in the scheduler when the reservable_cores is not a contiguous list of core IDs. [GH-25340]
• server: Added a new server configuration option named start_timeout with a default value of 30s. This duration is used to monitor the server setup and startup processes which must complete before it is considered healthy, such as keyring decryption. If these processes do not complete before the timeout is reached, the server process will exit. [GH-25803]
• ui: Fixed a bug where the job list page incorrectly calculated if a job had paused tasks. [GH-25742]

1.9.8 Enterprise (April 9, 2025)

IMPROVEMENTS:

• build: Updated Go to 1.24.2 [GH-25623]
• client: Improve memory usage by dropping references to task environment [GH-25373]
• cni: Add a warning log when CNI check commands fail [GH-25581]
• ui: Makes jobs list filtering case-insensitive [GH-25378]

BUG FIXES:

• client: remove blocking call during client gc [GH-25123]
• client: skip a task groups shutdown_delay when all tasks have already been deregistered [GH-25157]
• csi: Fixed a CSI ExpandVolume bug where the namespace was left out of the staging path [GH-25253]
• csi: Fixed a bug where GC would attempt and fail to delete plugins that had volumes [GH-25432]
• csi: Fixed a bug where cleaning up volume claims on GC'd nodes would cause errors on the leader [GH-25428]
• csi: Fixed a bug where in-flight CSI RPCs would not be cancelled on client GC or dev agent shutdown [GH-25472]
• drivers: set -1 exit code in case of executor failure for the exec, raw_exec, java, and qemu task drivers [GH-25453]
• job: Ensure migrate block difference is added to planning diff object [GH-25528]
• server: Validate num_schedulers configuration parameter is between 0 and the number of CPUs available on the machine [GH-25441]
• services: Fixed a bug where Nomad native services would not be correctly interpolated during in-place updates [GH-25373]
• services: Fixed a bug where task-level services, checks, and identities could interpolate jobspec values from other tasks in the same group [GH-25373]

1.9.7 (March 11, 2025)

BREAKING CHANGES:

• node: The node attribute consul.addr.dns has been changed to unique.consul.addr.dns. The node attribute nomad.advertise.address has been changed to unique.advertise.address. [GH-24942]

SECURITY:

• auth: Redact OIDC client secret from API responses and event stream (CVE-2025-1296) [GH-25328]

IMPROVEMENTS:

• build: Updated Go to 1.24.1 [GH-25249]
• config: Allow disabling wait in client config [GH-25255]
• cpustats: Add config "cpu_disable_dmidecode" to disable cpu detection using dmidecode [GH-25108]
• metrics: Fix the process lookup for raw_exec when running rootless [GH-25198]
• ui: System, Batch and Sysbatch jobs get a "Revert to prev version" button on their main pages [GH-25104]

BUG FIXES:

• cli: Add node_prefix read when setting up the task workload identity Consul policy [GH-25310]
• cni: Fixed a bug where CNI state was not migrated after upgrade, resulting in IP collisions [GH-25093]
• csi: Fixed a bug where plugins that failed initial fingerprints would not be restarted [GH-25307]
• fingerprint: Fixed a bug where Consul/Vault would never be fingerprinted if not available on agent start [GH-25102]
• hcl: Avoid panics by checking null values on durations [GH-25294]
• rpc: Fixed a bug that would cause the reader side of RPC connections to hang indefinitely [GH-25201]
• scheduler: Fixed a bug where node class hashes included unique attributes, making scheduling more costly [GH-24942]
• template: Fixed a bug where unset client.template retry blocks ignored defaults [GH-25113]
• template: Updated the consul-template dependency to v0.40.0 which included a bug fix in the quiescence timers. This bug could cause increased Nomad client CPU usage for tasks which use two or more template blocks. [GH-25140]

1.9.6 (February 11, 2025)

BREAKING CHANGES:

• fingerprint: Consul and Vault fingerprints no longer reload periodically [GH-24526]

SECURITY:

• api: sanitize the SignedIdentities in allocations of events to clean the identity token. [GH-24966]
• build: Updated Go to 1.23.6 [GH-25041]
• event stream: fixes vulnerability CVE-2025-0937, where using a wildcard namespace to subscribe to the events API grants a user with "read" capabilites on any namespace, the ability to read events from all namespaces. [GH-25089]

IMPROVEMENTS:

• auth: adds VerboseLogging option to auth-method config for debugging SSO [GH-24892]
• cli: Added actions available to a job when running nomad job status command [GH-24959]
• event stream: adds ability to authenticate using workload identities [GH-24849]
• services: Nomad service checks now support the tls_skip_verify parameter [GH-24781]
• task schedule: The task being paused no longer impacts restart attempts [GH-25085]
• ui: Contextualizes the Start Job button on whether it is startable, revertable, or not [GH-24985]

BUG FIXES:

• agent: Fixed a bug where Nomad error log messages within syslog showed via the notice priority [GH-24820]
• agent: Fixed a bug where all syslog entries were marked as notice when using JSON logging format [GH-24865]
• client: Fixed a bug where temporary RPC errors cause the client to poll for changes more frequently thereafter [GH-25039]
• csi: Fixed a bug where volume context from the plugin would be erased on volume updates [GH-24922]
• docker: Fixed a bug that prevented image_pull_timeout from being applied [GH-24991]
• docker: Fixed a bug where "error reading image pull progress" caused the allocation to get stuck pending [GH-24981]
• reporting (Enterprise): Updated the reporting metric to utilize node active heartbeat count. [GH-24919]
• state store: fix for setting correct status for a job version when reverting, and also fixes an issue where jobs were briefly marked dead during restarts [GH-24974]
• taskrunner: fix panic when a task with dynamic user is recovered [GH-24739]
• ui: Ensure pending service check blocks are filled [GH-24818]
• ui: Remove unrequired node read API call when attempting to stream task logs [GH-24973]
• vault: Fixed a bug where successful renewal was logged as an error [GH-25040]

1.9.5 (January 14, 2025)

IMPROVEMENTS:

• client: Add noswap mount option to secrets directory where supported on Linux [GH-24645]
• deps: Upgraded aws-sdk-go from v1 to v2 [GH-24720]
• keyring: Warn if deleting a key previously used to encrypt an existing variable [GH-24766]
• ui: Added possibility to supply HCL variable values on job submission [GH-24622]
• ui: add leadership status for servers in other regions [GH-24723]

BUG FIXES:

• docker: Fix a bug where images with port number and no tags weren't parsed correctly [GH-24547]
• driver/docker: Fix container CPU stats collection where previous CPU stats were missing and causing incorrect calculations [GH-24768]
• drivers: validate logmon plugin during reattach [GH-24798]
• networking: check network namespaces on Linux during client restarts and fail the allocation if an existing namespace is invalid [GH-24658]

1.9.4 (December 18, 2024)

SECURITY:

• api: sanitize the SignedIdentities in allocations to prevent privilege escalation through unredacted workload identity token impersonation associated with ACL policies. [GH-24683]
• security: Added more host environment variables to the default deny list for tasks [GH-24540]
• security: Explicitly set 'Content-Type' header to mitigate XSS vulnerability [GH-24489]
• security: add executeTemplate to default template function_denylist [GH-24541]

IMPROVEMENTS:

• actions: Nomad Actions names now accept a wider range of names [GH-24642]
• api: Sanitise hcl variables before storage on JobSubmission [GH-24423]
• client: Emit telemetry from prerun and prestart hooks for monitoring and alerting [GH-24556]
• cni: Add Nomad specific workload information to CNI_ARGS [GH-24319]
• core: add the possibility to scale system jobs between 0 and 1 [GH-24363]
• ui: Add an Edit From Version button as an option when reverting from an older job version [GH-24168]
• ui: Adds metadata tables to Task Group and Task pages [GH-24594]

BUG FIXES:

• agent: Fixed a bug where retry_join gave up after a single failure, rather than retrying until max attempts had been reached [GH-24561]
• api: Fixed a bug where alloc exec/logs/fs APIs would return errors for non-global regions [GH-24644]
• cli: Ensure the operator autopilot health command only outputs JSON when the json flag is supplied [GH-24655]
• consul: Fixed a bug where failures when syncing Consul checks could panic the Nomad agent [GH-24513]
• consul: Fixed a bug where non-root Nomad agents could not recreate a task's Consul token on task restart [GH-24410]
• csi: Fixed a bug where drivers that emit multiple topology segments would cause placements to fail [GH-24522]
• csi: Removed redundant namespace output from volume status command [GH-24432]
• discovery: Fixed a bug where IPv6 addresses would not be accepted from cloud autojoin [GH-24649]
• drivers: fix executor leak when drivers error starting tasks [GH-24495]
• executor: validate executor on reattach to avoid possibility of killing non-Nomad processes [GH-24538]
• keyring: Fixed a bug when decrypting aead with an empty RSA block on state upserts [GH-24442]
• networking: use a tmpfs location for the state of CNI IPAM plugin used by bridge mode, to fix a bug where allocations would fail to restore after host reboot [GH-24650]
• scheduler: Fix bug where forced garbage collection does not ignore GC thresholds [GH-24456]
• scheduler: take all assigned cpu cores into account instead of only those part of the largest lifecycle [GH-24304]
• ui: Fix a bug where namespaced jobs wouldn't show diffs on the versions page [GH-24466]
• ui: Fix an issue where 2 parent jobs would see the others dispatches if it were otherwise empty [GH-24668]
• ui: Fix an issue where cmd+click or ctrl+click would double-open a var [GH-24316]
• ui: Fix an issue where system jobs with garbage-collected allocations were showing as Scaled Down [GH-24620]
• ui: Fix an issue where volumes weren't navigable [GH-24542]
• vault: Fixed a bug where expired secret leases were treated as non-fatal and retried [GH-24409]
• windows: Restore process accounting logic from Nomad 1.6.x [GH-24494]

1.9.3 (November 11, 2024)

BUG FIXES:

• fix: handles consul template re-renders on client restart [GH-24399]
• fsm: Fix bug in snapshot restore that could partially wipe state [GH-24412]

1.9.2 (November 08, 2024)

SECURITY:

• csi: Fixed a bug where a user with csi-write-volume permissions to one namespace can create volumes in another namespace (CVE-2024-10975) [GH-24396]

IMPROVEMENTS:

• api: new parameterized dispatch endpoint sends raw HTTP request body as Payload [GH-24312]
• connect: Able to accept go-sockaddr address for consul grpc address [GH-24280]
• consul: add support for service weight [GH-24186]
• drivers: Add work_dir config to exec/rawexec/java drivers for setting the working directory of processes in a task [GH-24249]
• drivers: Move executor process out of task cgroup after task starts on cgroups v1 [GH-24340]
• drivers: add posibility to restrict user and group for exec and rawexec [GH-20073]
• fingerprint gce: fingerprint preemptibility [GH-24169]
• getter: Added option to chown artifact(s) to task user [GH-24157]
• ui: Indicates prestart/poststart tasks by running/failed/pending status [GH-24133]
• ui: Show region in header when only one region exists, and set it immediately upon logging in with a token [GH-24320]
• ui: When your token expires, upon signing back in, redirect to your original route [GH-24374]

BUG FIXES:

• cli: Corrected an ordering mistake in job tag unset help text [GH-24272]
• connect: add validation to ensure that connect native services specify a port [GH-24329]
• deps: Fixed a bug where restarting Nomad could cause an unrelated process with the same PID as a failed executor to be killed [GH-24265]
• keyring: Fixed a panic on server startup when decrypting AEAD key data with empty RSA block [GH-24383]
• scheduler: fixed a bug where resource calculation did not account correctly for poststart tasks [GH-24297]
• state: Fixed setting GC threshold to more than 72hrs being ignored [GH-24112]

1.9.1 (October 21, 2024)

IMPROVEMENTS:

• cli: Added synopsis for operator root and operator gossip command [GH-23671]
• cli: Updated example job specifications in nomad job init [GH-24232]

BUG FIXES:

• consul: Fixed a bug where broken Consul ACL tokens could block registration and deregistration of services and checks [GH-24166]
• consul: Fixed a bug where service deregistration could fail because Consul ACL tokens were revoked during allocation GC [GH-24166]
• docker: Always negotiate API version when initializing clients [GH-24237]
• docker: Fix incorrect auth parsing for private registries [GH-24215]
• docker: Fixed a bug where alloc exec could leak a goroutine [GH-24244]
• docker: Fixed a bug where alloc exec with stdin would hang [GH-24202]
• docker: Fixed a bug where task CPU stats were reported incorrectly [GH-24229]
• heartbeat: Fixed a bug where failed nodes would not be marked down [GH-24241]
• scheduler: fixes reconnecting allocations not getting picked correctly when replacements failed [GH-24165]
• ui: Fix an issue where a dropdown on the variables page would appear underneath table headers [GH-24162]
• ui: Put a max-width on token name so it doesn't collide with the search box in the top nav [GH-24240]
• windows: Fixed a bug where a crashed executor would orphan task processes [GH-24214]

1.9.0 (October 10, 2024)

BREAKING CHANGES:

• heartbeats: clients older than 1.6.0 will fail heartbeats to 1.9.0+ servers [GH-23838]
• jobspec: Removed support for HCLv1 [GH-23912]
• services: Clients older than 1.5.0 will fail to read Nomad native services via template blocks [GH-23910]
• tls: Removed deprecated tls.prefer_server_cipher_suites field from agent configuration [GH-23712]

SECURITY:

• security: Fixed a bug in client FS API where the check to prevent reads from the secrets dir could be bypassed on case-insensitive file systems [GH-24125]

IMPROVEMENTS:

• cli: Added redaction options to operator snapshot commands [GH-24023]
• cli: Increase default log level and duration when capturing logs with operator debug [GH-23850]
• deps: Upgraded yamux to v0.1.2 to fix a bug where RPC connections could deadlock [GH-24058]
• docker: Use official docker SDK instead of a 3rd party client [GH-23966]
• identity: Added filepath parameter to identity block for persisting workload identities [GH-24038]
• jobs: Added Version Tags to job versions, to prevent them from being garbage collected and allow for diffs [GH-24055]
• keyring: Stored wrapped data encryption keys in Raft [GH-23977]
• metrics: introduce client config to include alloc metadata as part of the base labels [GH-23964]
• networking: Added an option to ignore static port collisions when scheduling, for programs that use the SO_REUSEPORT unix socket option [GH-23956]
• networking: IPv6 can now be enabled on the Nomad bridge network mode [GH-23882]
• quotas (Enterprise): Added the possibility to set device count limits [GH-23894]
• raft: Bump raft to v1.7.1 which includes pre-vote. This should make servers more stable after network partitions [GH-24029]

BUG FIXES:

• bug: Allow client template config block to be parsed when using json config [GH-24007]
• cli: Fixed a bug in job status command where -t would act as though -json was also set [GH-24054]
• scaling: Fixed a bug where scaling policies would not get created during job submission unless namespace field was set in jobspec [GH-24065]
• state: Fixed a bug where compatibility updates for node topology for nodes older than 1.7.0 were not being correctly applied [GH-24127]
• task: adds node.pool attribute to interpretable values in task env [GH-24052]
• template: Fixed a panic on client restart when using change_mode=script [GH-24057]
• ui: Fixes an issue where variables paths would not let namespaced users write variables unless they also had wildcard namespace variable write permissions [GH-24073]

1.8.13 Enterprise (May 13, 2025)

BREAKING CHANGES:

• core: Errors encountered when reloading agent configuration will now cause agents to exit. Before configuration errors during reloads were only logged. This could lead to agents running but unable to communicate [GH-25721]

SECURITY:

• build: Update Go to 1.24.3 to address CVE-2025-22873 [GH-25818]
• sentinel (Enterprise): Fixed a bug where in some cases hard-mandatory policies could be overridden with -policy-override. CVE-2025-3744.

BUG FIXES:

• agent: Fixed a bug where reloading the agent with systemd notification enabled would cause the agent to be killed by system [GH-25636]
• api: Fixed pagination bug which could result in duplicate results [GH-25792]
• cli: Respect NOMAD_REGION environment variable in operator debug command [GH-25716]
• client: fix failure cleaning up namespace on batch jobs [GH-25714]
• metrics: Fixed a bug where RSS and cache stats would not be reported for docker, exec, and java drivers under Linux cgroups v2 [GH-25751]
• scheduler: Fixed a bug in accounting for resources.cores that could prevent placements on nodes with available cores [GH-25705]
• scheduler: Fixed a bug where draining a node with canaries could result in a stuck deployment [GH-25726]
• scheduler: Fixed a bug where updating the rescheduler tracker could corrupt the state store [GH-25698]
• scheduler: Use core ID when selecting cores. This fixes a panic in the scheduler when the reservable_cores is not a contiguous list of core IDs. [GH-25340]
• ui: Fixed a bug where the job list page incorrectly calculated if a job had paused tasks. [GH-25742]

1.8.12 Enterprise (April 9, 2025)

IMPROVEMENTS:

• build: Updated Go to 1.24.2 [GH-25623]
• client: Improve memory usage by dropping references to task environment [GH-25373]
• cni: Add a warning log when CNI check commands fail [GH-25581]

BUG FIXES:

• client: remove blocking call during client gc [GH-25123]
• client: skip a task groups shutdown_delay when all tasks have already been deregistered [GH-25157]
• csi: Fixed a CSI ExpandVolume bug where the namespace was left out of the staging path [GH-25253]
• csi: Fixed a bug where GC would attempt and fail to delete plugins that had volumes [GH-25432]
• csi: Fixed a bug where cleaning up volume claims on GC'd nodes would cause errors on the leader [GH-25428]
• csi: Fixed a bug where in-flight CSI RPCs would not be cancelled on client GC or dev agent shutdown [GH-25472]
• drivers: set -1 exit code in case of executor failure for the exec, raw_exec, java, and qemu task drivers [GH-25453]
• job: Ensure migrate block difference is added to planning diff object [GH-25528]
• server: Validate num_schedulers configuration parameter is between 0 and the number of CPUs available on the machine [GH-25441]
• services: Fixed a bug where Nomad native services would not be correctly interpolated during in-place updates [GH-25373]
• services: Fixed a bug where task-level services, checks, and identities could interpolate jobspec values from other tasks in the same group [GH-25373]

1.8.11 Enterprise (March 11, 2025)

BREAKING CHANGES:

• node: The node attribute consul.addr.dns has been changed to unique.consul.addr.dns. The node attribute nomad.advertise.address has been changed to unique.advertise.address. [GH-24942]

SECURITY:

• auth: Redact OIDC client secret from API responses and event stream (CVE-2025-1296) [GH-25328]

IMPROVEMENTS:

• build: Updated Go to 1.24.1 [GH-25249]
• metrics: Fix the process lookup for raw_exec when running rootless [GH-25198]

BUG FIXES:

• cli: Add node_prefix read when setting up the task workload identity Consul policy [GH-25310]
• cni: Fixed a bug where CNI state was not migrated after upgrade, resulting in IP collisions [GH-25093]
• csi: Fixed a bug where plugins that failed initial fingerprints would not be restarted [GH-25307]
• rpc: Fixed a bug that would cause the reader side of RPC connections to hang indefinitely [GH-25201]
• scheduler: Fixed a bug where node class hashes included unique attributes, making scheduling more costly [GH-24942]
• template: Fixed a bug where unset client.template retry blocks ignored defaults [GH-25113]
• template: Updated the consul-template dependency to v0.40.0 which included a bug fix in the quiescence timers. This bug could cause increased Nomad client CPU usage for tasks which use two or more template blocks. [GH-25140]

1.8.10 (February 11, 2025)

SECURITY:

• api: sanitize the SignedIdentities in allocations of events to clean the identity token. [GH-24966]
• build: Updated Go to 1.23.6 [GH-25041]
• event stream: fixes vulnerability CVE-2025-0937, where using a wildcard namespace to subscribe to the events API grants a user with "read" capabilites on any namespace, the ability to read events from all namespaces. [GH-25089]

IMPROVEMENTS:

• auth: adds VerboseLogging option to auth-method config for debugging SSO [GH-24892]
• event stream: adds ability to authenticate using workload identities [GH-24849]

BUG FIXES:

• agent: Fixed a bug where Nomad error log messages within syslog showed via the notice priority [GH-24820]
• agent: Fixed a bug where all syslog entries were marked as notice when using JSON logging format [GH-24865]
• client: Fixed a bug where temporary RPC errors cause the client to poll for changes more frequently thereafter [GH-25039]
• csi: Fixed a bug where volume context from the plugin would be erased on volume updates [GH-24922]
• networking: check network namespaces on Linux during client restarts and fail the allocation if an existing namespace is invalid [GH-24658]
• reporting (Enterprise): Updated the reporting metric to utilize node active heartbeat count. [GH-24919]
• state store: fix for setting correct status for a job version when reverting, and also fixes an issue where jobs were briefly marked dead during restarts [GH-24974]
• taskrunner: fix panic when a task with dynamic user is recovered [GH-24739]
• ui: Ensure pending service check blocks are filled [GH-24818]
• ui: Remove unrequired node read API call when attempting to stream task logs [GH-24973]
• vault: Fixed a bug where successful renewal was logged as an error [GH-25040]

1.8.9 (January 14, 2025)

IMPROVEMENTS:

• api: Sanitise hcl variables before storage on JobSubmission [GH-24423]
• deps: Upgraded aws-sdk-go from v1 to v2 [GH-24720]

BUG FIXES:

• drivers: validate logmon plugin during reattach [GH-24798]

1.8.8 Enterprise (December 18, 2024)

SECURITY:

• api: sanitize the SignedIdentities in allocations to prevent privilege escalation through unredacted workload identity token impersonation associated with ACL policies. [GH-24683]
• security: Added more host environment variables to the default deny list for tasks [GH-24540]
• security: Explicitly set 'Content-Type' header to mitigate XSS vulnerability [GH-24489]
• security: add executeTemplate to default template function_denylist [GH-24541]

BUG FIXES:

• agent: Fixed a bug where retry_join gave up after a single failure, rather than retrying until max attempts had been reached [GH-24561]
• api: Fixed a bug where alloc exec/logs/fs APIs would return errors for non-global regions [GH-24644]
• cli: Ensure the operator autopilot health command only outputs JSON when the json flag is supplied [GH-24655]
• consul: Fixed a bug where failures when syncing Consul checks could panic the Nomad agent [GH-24513]
• consul: Fixed a bug where non-root Nomad agents could not recreate a task's Consul token on task restart [GH-24410]
• csi: Fixed a bug where drivers that emit multiple topology segments would cause placements to fail [GH-24522]
• csi: Removed redundant namespace output from volume status command [GH-24432]
• discovery: Fixed a bug where IPv6 addresses would not be accepted from cloud autojoin [GH-24649]
• drivers: fix executor leak when drivers error starting tasks [GH-24495]
• executor: validate executor on reattach to avoid possibility of killing non-Nomad processes [GH-24538]
• fix: handles consul template re-renders on client restart [GH-24399]
• networking: use a tmpfs location for the state of CNI IPAM plugin used by bridge mode, to fix a bug where allocations would fail to restore after host reboot [GH-24650]
• scheduler: take all assigned cpu cores into account instead of only those part of the largest lifecycle [GH-24304]
• vault: Fixed a bug where expired secret leases were treated as non-fatal and retried [GH-24409]

1.8.7 Enterprise (November 8, 2024)

SECURITY:

• csi: Fixed a bug where a user with csi-write-volume permissions to one namespace can create volumes in another namespace (CVE-2024-10975) [GH-24396]

BUG FIXES:

• connect: add validation to ensure that connect native services specify a port [GH-24329]
• keyring: Fixed a panic on server startup when decrypting AEAD key data with empty RSA block [GH-24383]
• scheduler: fixed a bug where resource calculation did not account correctly for poststart tasks [GH-24297]

1.8.6 Enterprise(October 21, 2024)

IMPROVEMENTS:

• cli: Added synopsis for operator root and operator gossip command [GH-23671]

BUG FIXES:

• consul: Fixed a bug where broken Consul ACL tokens could block registration and deregistration of services and checks [GH-24166]
• consul: Fixed a bug where service deregistration could fail because Consul ACL tokens were revoked during allocation GC [GH-24166]
• deps: Fixed a bug where restarting Nomad could cause an unrelated process with the same PID as a failed executor to be killed [GH-24265]
• scheduler: fixes reconnecting allocations not getting picked correctly when replacements failed [GH-24165]
• windows: Fixed a bug where a crashed executor would orphan task processes [GH-24214]

1.8.5 Enterprise (October 10, 2024)

SECURITY:

• security: Fixed a bug in client FS API where the check to prevent reads from the secrets dir could be bypassed on case-insensitive file systems [GH-24125]

IMPROVEMENTS:

• cli: Increase default log level and duration when capturing logs with operator debug [GH-23850]

BUG FIXES:

• bug: Allow client template config block to be parsed when using json config [GH-24007]
• cli: Fixed a bug in job status command where -t would act as though -json was also set [GH-24054]
• licensing: Fixed a bug where environment variable to opt-out of reporting was not respected
• scaling: Fixed a bug where scaling policies would not get created during job submission unless namespace field was set in jobspec [GH-24065]
• state: Fixed a bug where compatibility updates for node topology for nodes older than 1.7.0 were not being correctly applied [GH-24127]
• task: adds node.pool attribute to interpretable values in task env [GH-24052]
• template: Fixed a panic on client restart when using change_mode=script [GH-24057]

1.8.4 (September 17, 2024)

BREAKING CHANGES:

• docker: The default infra_image for pause containers is now registry.k8s.io/pause [GH-23927]

IMPROVEMENTS:

• build: update to go1.22.6 [GH-23805]
• cgroups: Allow clients with delegated cgroups check that required cgroup v2 controllers exist [GH-23803]
• docker: Disable cpuset management for non-root clients [GH-23804]
• identity: Added support for server-configured additional claims on the Vault default_identity block [GH-23675]
• namespaces: Allow enabling/disabling allowed network modes per namespace [GH-23813]
• ui: Badge added for Scaled Down jobs [GH-23829]

DEPRECATIONS:

• api: the JobParseRequest.HCLv1 field will be removed in Nomad 1.9.0 [GH-23913]
• jobspec: using the -hcl1 flag for HCLv1 job specifications will now emit a warning at the command line. This feature will be removed in Nomad 1.9.0 [GH-23913]

BUG FIXES:

• identity: Fixed a bug where dispatch and periodic jobs would have their job ID and not parent job ID used when creating the subject claim [GH-23902]
• identity: Fixed a bug where dispatch and periodic jobs would have their job ID and not parent job ID used when interpolating vault.default_identity.extra_claims [GH-23817]
• node: Fixed bug where sysbatch allocations were started prematurely [GH-23858]
• ui: Fix an issue where cmd+click or ctrl+click would double-open a job [GH-23832]

1.8.3 (August 13, 2024)

SECURITY:

• security: Fix symlink escape during unarchiving by removing existing paths within the same allocdir. Compromising the Nomad client agent at the source allocation first is a prerequisite for leveraging this issue. [GH-23738]

IMPROVEMENTS:

• acl: Submitting a policy with a leading / in a variable path will now return an error to prevent improperly working policies. [GH-23757]
• cli: Added option to return original HCL in job inspect command [GH-23699]
• cli: Added support for updating the roles for an ACL token [GH-18532]
• cli: acl token create will now emit a warning if the token has a policy that does not yet exist [GH-16437]
• keyring: Added support for encrypting the keyring via Vault transit or external KMS [GH-23580]
• keyring: Added support for prepublishing keys [GH-23577]
• identity: Added support for server-configured additional claims on the Vault default_identity block [GH-23675]
• metrics: Added client.tasks metrics to track task states [GH-23773]
• resources: Added resources.secrets field to configure size of secrets directory on Linux [GH-23696]
• tls: Allow setting the tls_min_version field to "tls13" [GH-23713]
• ui: added a Pack badge to the jobs index page for jobs run via Nomad Pack [GH-23404]

BUG FIXES:

• api: Fixed a bug where an api.Config targeting a unix domain socket could not be reused between clients [GH-23785]
• cni: .conf and .json config files are now parsed properly [GH-23629]
• cni: network.cni jobspec updates now replace allocs to apply the new network config [GH-23764]
• docker: Fixed a bug where plugin SELinux labels would conflict with read-only volume options [GH-23750]
• identity: Fixed a bug where a missing default task identity could panic the leader [GH-23763]
• keyring: Fixed a bug where keys could be garbage collected before workload identities expire [GH-23577]
• keyring: Fixed a bug where keys would never exit the "rekeying" state after a rotation with the -full flag [GH-23577]
• keyring: Fixed a bug where periodic key rotation would not occur [GH-23577]
• networking: The same static port can now be used more than once on host networks with multiple IPs [GH-23693]
• scaling: Fixed a bug where state store corruption could occur when writing scaling events [GH-23673]
• template: Fixed a bug where change_mode = "script" would not execute after a client restart [GH-23663]
• ui: Fixed storage/plugin 404s by unescaping a slash character in the request URL [GH-23625]
• windows: Fix bug with containers capabilities on Docker CE [GH-23599]

1.8.2 (July 16, 2024)

BREAKING CHANGES:

• docker: default to hyper-v isolation mode on Windows [GH-23452]

SECURITY:

• build: Updated Go to 1.22.5 to address CVE-2024-24791 [GH-23498]
• migration: Added a check for relative paths escaping the allocation directory when unpacking archive during migration, to harden clients against compromised peer clients sending malicious archives [GH-23319]
• security: Removed insecure TLS cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA25 and TLS_RSA_WITH_AES_128_CBC_SHA256. [GH-23551]

IMPROVEMENTS:

• client: add a preferred_address_family config to prefer ipv4 or ipv6 when deducing IP from network interface [GH-23389]
• cni: allow users to input CNI args in job specification [GH-23538]
• deps: Updated Consul API to 1.29.1. [GH-23436]
• deps: Updated consul-template to 0.39 to allow admin partition and sameness groups queries. [GH-23436]
• docker: Validate that unprivileged containers aren't running as ContainerAdmin on Windows [GH-23443]
• namespaces: Added warnings if deleting namespaces that have existing objects associated with them [GH-23499]
• quota (Enterprise): Allow CPU cores to be configured within a quota [GH-23543]
• scaling: Added -check-index support to job scale command [GH-23457]
• ui: Allow users to create Global ACL tokens from the Administration UI [GH-23506]
• ui: Update headers in the Admin section to use the HashiCorp Design System [GH-23366]
• ui: allow for multiple namespaces in jobs index filters [GH-23468]

BUG FIXES:

• api: Fixed bug where newlines in JobSubmission vars weren't encoded correctly [GH-23560]
• cli: Fixed bug where the plugin status command would fail if the plugin ID was a prefix of another plugin ID [GH-23502]
• cli: Fixed bug where the quota status and quota inspect commands would fail if the quota name was a prefix of another quota name [GH-23502]
• cli: Fixed bug where the scaling policy info command would fail if the policy ID was a prefix of another policy ID [GH-23502]
• cli: Fixed bug where the service info command would fail if the service name was a prefix of another service name in the same namespace [GH-23502]
• cli: Fixed bug where the volume deregister, volume detach, and volume status commands would fail if the volume ID was a prefix of another volume ID in the same namespace [GH-23502]
• consul: Fixed a bug where service registration and Envoy bootstrap would not wait for Consul ACL tokens and services to be replicated to the local agent [GH-23381]
• plugins: Fix panic on systems that don't support NUMA [GH-23399]
• qemu: Fixed a bug that prevented qemu tasks from running on Linux [GH-23466]
• quota (Enterprise): Fixed a bug where a task's resource core count was not translated to CPU MHz and checked against its quota when performing a job plan [GH-18876]
• scheduler: Fix a bug where reserved resources are not calculated correctly [GH-23386]
• server: Fixed a bug where expiring heartbeats for garbage collected nodes could panic the server [GH-23383]
• template: Fix template rendering on Windows [GH-23432]
• ui: Actions run from jobs with explicit name properties now work from the web UI [GH-23553]
• ui: Don't show keyboard nav hints when taking a screenshot [GH-23365]
• ui: Fix an issue where a remotely purged job would prevent redirect from taking place in the web UI [GH-23492]
• ui: Fix an issue where access to Job Templates in the UI was restricted to variable.write access [GH-23458]
• ui: Fix the Upload Jobspec button on the Run Job page [GH-23548]
• ui: Fixed support for namespace parameter on job statuses API [GH-23456]
• ui: fix an issue where gateway timeouts would cause the jobs list to revert to null, gives users a Pause Fetch option [GH-23427]
• vault: Fixed a bug where requests to derive or renew tokens could be sent to the wrong namespace [GH-23491]

1.8.1 (June 19, 2024)

SECURITY:

• build: Updated Go to 1.22.4 to address Go stdlib vulnerabilities CVE-2024-24789 and CVE-2024-24790 [GH-23172]

IMPROVEMENTS:

• api: Add support for setting Notes field for Consul health checks [GH-22397]
• cli: operator snapshot inspect now includes details of data in snapshot [GH-18372]
• docker: Added container_exists_attempts plugin configuration variable [GH-22419]
• docker: Added support for oom_score_adj [GH-23297]
• exec: Fixed a bug where exec driver tasks would fail on older versions of glibc [GH-23331]
• metrics (Enterprise): Publish quota utilization as metrics [GH-22912]
• raw_exec: Added support for oom_score_adj [GH-23308]
• ui: adds a Stopped label for jobs that a user has manually stopped [GH-23328]
• ui: namespace dropdown gets a search field and supports many namespaces [GH-20626]
• ui: shorten client/node metadata/attributes display and make parent-terminal attributes show up [GH-23290]

BUG FIXES:

• acl: Fix plugin policy validation when checking write permissions [GH-23274]
• api: (Enterprise) fixed Allocations.GetPauseState method discarding the task argument [GH-23377]
• client: Fixed a bug where empty task directories would be left behind [GH-23237]
• connect: fix validation with multiple socket paths [GH-22312]
• consul: (Enterprise) Fixed a bug where gateway config entries were written before Sentinel policies were enforced [GH-22228]
• consul: Fixed a bug where Consul admin partition was not used to login via Consul JWT auth method [GH-22226]
• consul: Fixed a bug where gateway config entries were written to the Nomad server agent's Consul partition and not the client's partition [GH-22228]
• driver: Fixed a bug where the exec, java, and raw_exec drivers would not configure cgroups to allow access to devices provided by device plugins [GH-22518]
• scheduler: Fixed a bug where rescheduled allocations that could not be placed would later ignore their reschedule policy limits [GH-12319]
• task schedule: Fixed a bug where schedules wrongly errored as invalid on the last day of the month [GH-23329]
• ui: unbind job detail running allocations count from job-summary endpoint [GH-23306]

1.8.0 (May 28, 2024)

IMPROVEMENTS:

• agent: Added support for systemd readiness notifications [GH-20528]
• api: new /v1/jobs/statuses endpoint collates details about jobs' allocs and latest deployment, intended for use in the updated UI jobs index page [GH-20130]
• artifact: Added support for downloading artifacts without validating the TLS certificate [GH-20126]
• autopilot: Added operator autopilot health command to review Autopilot health data [GH-20156]
• cli: Add -jwks-ca-file argument to setup consul/vault commands [GH-20518]
• client/volumes: Add a mount volume level option for selinux tags on volumes [GH-19839]
• client: expose network namespace bridge/cni configuration values as task env vars [GH-11810]
• connect: Added support for volume_mount blocks on sidecar task overrides [GH-20575]
• consul/connect: Attempt autodetection of podman task driver for Connect gateways [GH-20611]
• consul: provide tasks that have Consul tokens the CONSUL_HTTP_TOKEN environment variable [GH-20519]
• core: Do not create evaluations within batch deregister endpoint during job garbage collection [GH-20510]
• csi: Added support for wildcard namespace to plugin status command [GH-20551]
• deps: Update msgpack to v2 [GH-20173]
• deps: Updated docker dependency to 26.0.1 [GH-20389]
• driver/rawexec: Allow specifying custom cgroups [GH-20481]
• func: Allow custom paths to be added the the getter landlock [GH-20315]
• jobspec: Add a schedule{} block for time based task execution (Enterprise) [GH-22201]
• metrics: Added tracking of enqueue and dequeue times of evaluations to the broker [GH-20329]
• networking: Inject constraints on CNI plugins when using bridge networking [GH-15473]
• scheduler: Added a new configuration to avoid rescheduling allocations if a nodes misses one or more heartbits [GH-19101]
• server: Add new options for reconcilation in case of disconnected nodes [GH-20029]
• ui: Added a UI for creating, editing and deleting Sentinel Policies [GH-20483]
• ui: Added a copy button on Action output [GH-19496]
• ui: Added a new UI block to job spec in order to provide description and links in the Web UI [GH-18292]
• ui: Added token.name information to the top nav for ease of operator debugging [GH-20539]
• ui: Improve error and warning messages for invalid variable and job template paths/names [GH-19989]
• ui: Overhaul of the Jobs Index list page, with live updates, more informative statuses, filter expressions, and pagination [GH-20452]
• ui: Prompt a user before they close an exec window to prevent accidental close-browser-tab shortcuts that overlap with terminal ones [GH-19985]
• ui: Replaced single-line variable value fields with multi-line textarea blocks [GH-19544]
• ui: Updated the style of components in the Variables web ui [GH-19544]
• ui: change the State filter on clients page to split out eligibility and drain status [GH-18607]

BUG FIXES:

• cli: Fix handling of scaling jobs which don't generate evals [GH-20479]
• client: Fix unallocated CPU metric calculation when client reserved CPU is set [GH-20543]
• client: terminate old exec task processes before starting new ones, to avoid accidentally leaving running processes in case of an error [GH-20500]
• config: Fixed a panic triggered by registering a job specifying a Vault cluster that has not been configured within the server [GH-22227]
• core: Fix multiple incorrect type conversion for potential overflows [GH-20553]
• csi: Fixed a bug where concurrent mount and unmount operations could unstage volumes needed by another allocation [GH-20550]
• csi: Fixed a bug where plugins would not be deleted on GC if their job updated the plugin ID [GH-20555]
• csi: Fixed a bug where volumes in different namespaces but the same ID would fail to stage on the same client [GH-20532]
• job endpoint: fix implicit constraint mutation for task-level services [GH-22229]
• quota (Enterprise): Fixed a bug where quota usage would not be freed if a job was purged
• services: Added retry to Nomad service deregistration RPCs during alloc stop [GH-20596]
• services: Fixed bug where Nomad services might not be deregistered when nodes are marked down or allocations are terminal [GH-20590]
• structs: Fix job canonicalization for array type fields [GH-20522]
• ui: Fix a bug where the UI would prompt a user to promote a deployment with unplaced canaries [GH-20408]
• ui: Fixed an issue where keynav would not trigger evaluation sidebar expand [GH-20047]
• ui: Show the namespace in the web UI exec command hint [GH-20218]
• windows: Fixed a regression where scanning task processes was inefficient [GH-20619]

Unsupported Versions

Versions of Nomad before 1.8.0 are no longer supported. See CHANGELOG-unsupported.md for their changelogs.