--- layout: docs page_title: Garbage collection description: |- Nomad garbage collects Access Control List (ACL) tokens, allocations, deployments, encryption root keys, evaluations, jobs, nodes, plugins, and Container Storage Interface (CSI) volumes. Learn about server-side and client-side garbage collection processes, including configuration and triggers. --- # Garbage collection Nomad garbage collection is not the same as garbage collection in a programming language, but the motivation behind its design is similar: garbage collection frees up memory allocated for objects that the schedular no longer references or needs. Nomad only garbage collects objects that are in a terminal state and only after a delay to allow inspection or debugging. Nomad runs garbage collection processes on servers and on client nodes. You may also manually trigger garbage collection on the server. Nomad garbage collects the following objects: - [ACL token](#configuration) - [Allocation](#client-side-garbage-collection) - [CSI Plugin](#configuration) - [Deployment](#configuration) - [Encryption root key](#configuration) - [Evaluation](#configuration) - [Job](#configuration) - [Node](#configuration) - [Volume](#configuration) ## Cascading garbage collection Nomad's scheduled garbage collection processes generally handle each resource type independently. However, there is an implicit cascading relationship because of how objects reference each other. In practice, when Nomad garbage collects a higher-level object, Nomad also removes the object's associated sub-objects to prevent orphaned objects. For example, garbage collecting a job also causes Nomad to drop all of that job's remaining evaluations, deployments, and allocation records from the state. Nomad garbage collects those objects, either as part of the job garbage collection process or by each object's own garbage collection processes running immediately after. Nomad's scheduled garbage collection processes only garbage collect objects after they are terminal for at least the specified time threshold and no longer needed for future scheduling decisions. Note that when you force garbage collection by running the `nomad system gc` command, Nomad ignores the specified time threshold. ## Server-side garbage collection The Nomad server leader starts periodic garbage collection processes that clean objects marked for garbage collection from memory. Nomad automatically marks some objects, like evaluations, for garbage collection. Alternatively, you may manually mark jobs for garbage collection by running `nomad system gc`, which runs the garbage collection process. ### Configuration These settings govern garbage collection behavior on the server nodes. You may review the intervals in the [`config.go` class](https://github.com/hashicorp/nomad/blob/b11619010e1c83488e14e2785569e515b2769062/nomad/config.go#L564) for objects without a configurable interval setting. | Object | Interval | Threshold | |---|---|---| | **ACL token** | 5 minutes | [`acl_token_gc_threshold`](/nomad/docs/configuration/server#acl_token_gc_threshold)
Default: 1 hour | | **CSI Plugin** | 5 minutes | [`csi_plugin_gc_threshold`](/nomad/docs/configuration/server#csi_plugin_gc_threshold)
Default: 1 hour | | **Deployment** | 5 minutes | [`deployment_gc_threshold`](/nomad/docs/configuration/server#deployment_gc_threshold)
Default: 1 hour | | **Encryption root key** | [`root_key_gc_interval`](/nomad/docs/configuration/server#root_key_gc_interval)
Default: 10 minutes | [`root_key_gc_threshold`](/nomad/docs/configuration/server#root_key_gc_threshold)
Default: 1 hour | | **Evaluation** | 5 minutes | [`eval_gc_threshold`](/nomad/docs/configuration/server#eval_gc_threshold)
Default: 1 hour | | **Evaluation, batch** | 5 minutes | [`batch_eval_gc_threshold`](/nomad/docs/configuration/server#batch_eval_gc_threshold)
Default: 24 hours | | **Job** | [`job_gc_interval`](/nomad/docs/configuration/server#job_gc_interval)
Default: 5 minutes | [`job_gc_threshold`](/nomad/docs/configuration/server#job_gc_threshold)
Default: 4 hours | | **Node** | 5 minutes | [`node_gc_threshold`](/nomad/docs/configuration/server#node_gc_threshold)
Default: 24 hours | | **Volume** | [`csi_volume_claim_gc_interval`](/nomad/docs/configuration/server#csi_volume_claim_gc_interval)
Default: 5 minutes| [`csi_volume_claim_gc_threshold`](/nomad/docs/configuration/server#csi_volume_claim_gc_threshold)
Default: 1 hour | ### Triggers The server garbage collection processes wake up at configured intervals to scan for any expired or terminal objects to permanently delete, provided the object's time in a terminal state exceeds its garbage collection threshold. For example, a job's default garbage collection threshold is four hours, so the job must be in a terminal state for at least four hours before the garbage collection process permanently deletes the job and its dependent objects. When you force garbage collection by manually running the `nomad system gc` command, you are telling the garbage collection process to ignore thresholds and immediately purge all terminal objects on all servers and clients. ## Client-side garbage collection On each client node, Nomad must clean up resources from terminated allocations to free disk and memory on the machine. ### Configuration These settings govern allocation garbage collection behavior on each client node. | Parameter | Default | Description | | -------- | ------- | ------------- | | [`gc_interval`](/nomad/docs/configuration/client#gc_interval) | 1 minute | Interval at which Nomad attempts to garbage collect terminal allocation directories | | [ `gc_disk_usage_threshold` ](/nomad/docs/configuration/client#gc_disk_usage_threshold) | 80 | Disk usage percent which Nomad tries to maintain by garbage collecting terminal allocations | | [ `gc_inode_usage_threshold` ](/nomad/docs/configuration/client#gc_inode_usage_threshold) | 70 | Inode usage percent which Nomad tries to maintain by garbage collecting terminal allocations | | [ `gc_max_allocs` ](/nomad/docs/configuration/client#gc_max_allocs) | 50 | Maximum number of allocations which a client will track before triggering a garbage collection of terminal allocations | | [ `gc_parallel_destroys ` ](/nomad/docs/configuration/client#gc_parallel_destroys) | 2 | Maximum number of parallel destroys allowed by the garbage collector | Refer to the [client block in agent configuration reference](/nomad/docs/configuration/client) for complete parameter descriptions and examples. Note that there is no time-based retention setting for allocations. Unlike jobs or evaluations, you cannot specify a time to keep allocations alive before garbage collection. As soon as an allocation is terminal, it becomes eligible for cleanup if the configured thresholds demand it. ### Triggers Nomad's client runs allocation garbage collection based on these triggers: - Scheduled interval The garbage collection process launches a ticker based on the configured `gc_interval`. On each tick, the garbage collection process checks to see if it needs to remove terminal allocations. - Terminal state When an allocation transitions to a terminal state, Nomad marks the allocation for garbage collection and then signals the garbage collection process to run immediately. - Allocation placement Nomad may preemptively run garbage collection to make room for new allocations. The client garbage collects older, terminal allocations if adding new allocations would exceed the `gc_max_allocs` limit. - Forced garbage collection When you force garbage collection by running the `nomad system gc` command, the garbage collection process removes all terminal objects on all servers and clients, ignoring thresholds. Nomad does not continuously monitor disk or inode usage to trigger garbage collection. Instead, Nomad only checks disk and inode thresholds when one of the aforementioned triggers invokes the garbage collection process. The `gc_inode_usage_threshold` and `gc_disk_usage_threshold` values do not trigger garbage collection; rather, those values influence how the garbage collector behaves during a collection run. ### Allocation selection When the garbage collection process runs, Nomad destroys as many finished allocations as needed to meet the resource thresholds. The client maintains a priority queue of terminal allocations ordered by the time they were marked finished, oldest first. The process repeatedly evicts allocations from the queue until the conditions are back within configured limits. Specifically, the garbage collection loop checks, in order: 1. If disk usage exceeds `gc_disk_usage_threshold` value 1. If inode usage exceeds `gc_inode_usage_threshold` value 1. If the count of allocations exceeds `gc_max_allocs` value If any one of these conditions is true, the garbage collector selects the oldest finished allocation for removal. After deleting one allocation, the loop re-checks the metrics and continues removing the next-oldest allocation until all thresholds are satisfied or until there are no more terminal allocs. This means in a single run, the garbage collection removes multiple allocations back-to-back if the node was far over the limits. The evictions happen in termination-time order, which is oldest completed allocations first. If node's usage and allocation count are under the limits, a normal garbage collection cycle does not remove any allocations. In other words, periodic and event-driven garbage collection does not delete allocations just because they are finished. There has to be pressure or a limit reached. The exception is when an administrative command or server-side removal triggers client-side garbage collection. Aside from that forced scenario, the default behavior is threshold-driven: Nomad leaves allocations on disk until it needs to reclaim those allocations due to space, inode, or count limits being hit. ### Task driver resources garbage collection Most task drivers do not have their own garbage collection process. When an allocation is terminal, the client garbage collection process communicates with the task driver to ensure the task's resources have been cleaned up. Note that the Docker task driver periodically cleans up its own resources. Refer to the [Docker task driver plugin options](https://developer.hashicorp.com/nomad/docs/drivers/docker#gc) for details. When a task has configured restart attempts and the task fails, the Nomad client attempts an in-place task restart within the same allocation. The task driver starts a new process or container for the task. If the task continues to fail and exceeds the configured restart attempts, Nomad terminates the task and marks the allocation as terminal. The task driver then cleans up its resources, such as a Docker container or cgroups. When the garbage collection process runs, it makes sure that the task driver cleanup is done before deleting the allocation. If a task driver fails to clean up properly, Nomad logs errors but continues the garbage collection process. Task driver cleanup failure issues can influence when the allocation truly frees up. For instance, if volumes are not detached, disk space might not be fully reclaimed until fixed. ## Resources - [Nomad's internal garbage collection and optimization discovery during the Nomad Bench project blog post](https://www.hashicorp.com/en/blog/nomad-garbage-collection-optimization-discovery-during-nomad-bench) - Configuration - [client Block in Agent Configuration](/nomad/docs/configuration/client) - [server Block in Agent Configuration](/nomad/docs/configuration/server) - [the `nomad system gc` command reference](/nomad/docs/commands/system/gc) - [System HTTP API Force GC](/nomad/api-docs/system#force-gc)