mirror of
https://github.com/kemko/nomad.git
synced 2026-01-03 00:45:43 +03:00
02d98b9357
The `nomad operator debug` command saves a CPU profile for each interval, and names these files based on the interval. The same functions takes a goroutine profile, heap profile, etc. but is missing the logic to interpolate the file name with the interval. This results in the operator debug command making potentially many expensive profile requests, and then overwriting the data. Update the command to save every profile it scrapes, and number them similarly to the existing CPU profile. Additionally, the command flags for `-pprof-interval` and `-pprof-duration` were validated backwards, which meant that we always coerced the `-pprof-interval` to be the same as the `-pprof-duration`, which always resulted in a single profile being taken at the start of the bundle. Correct the check as well as change the defaults to be more sensible. Fixes: https://github.com/hashicorp/nomad/issues/20151
178 lines
6.5 KiB
Plaintext
178 lines
6.5 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: 'Commands: operator debug'
|
|
description: |
|
|
Build an archive of debug data.
|
|
---
|
|
|
|
# Command: operator debug
|
|
|
|
The `operator debug` command builds an archive containing Nomad cluster
|
|
configuration and state information, Nomad server and client node
|
|
logs, and pprof profiles from the selected servers and client nodes.
|
|
|
|
If no selection option is specified, the debug archive contains only
|
|
cluster meta information.
|
|
|
|
## Usage
|
|
|
|
```plaintext
|
|
nomad operator debug [options]
|
|
```
|
|
|
|
This command accepts comma separated `server-id` and `node-id` IDs for
|
|
monitoring and pprof profiling. If IDs are provided, the command will
|
|
monitor logs for the `duration`, saving a snapshot of Nomad state
|
|
every `interval`. Captured logs and configurations are subjected to
|
|
redaction, but may still contain sensitive information and the archive
|
|
contents should be reviewed before sharing.
|
|
|
|
If an `output` path is provided, `debug` will create a timestamped
|
|
directory in that path instead of an archive. By default, the command
|
|
creates a compressed tar archive in the current directory.
|
|
|
|
Consul and Vault status and version information are included if
|
|
configured.
|
|
|
|
If ACLs are enabled, this command will require a token with the 'node:read'
|
|
capability to run. In order to collect information, the token will also
|
|
require the 'agent:read' and 'operator:read' capabilities, as well as the
|
|
'list-jobs' capability for all namespaces. To collect pprof profiles the
|
|
token will also require 'agent:write', or enable_debug configuration set to
|
|
true.
|
|
|
|
## General Options
|
|
|
|
@include 'general_options.mdx'
|
|
|
|
## Debug Options
|
|
|
|
- `-duration=2m`: Set the duration of the debug capture. Logs will be captured from
|
|
specified servers and nodes at `log-level`. Defaults to `2m`.
|
|
|
|
- `-interval=30s`: The interval between snapshots of the Nomad state.
|
|
If unspecified, only one snapshot is captured. Defaults to `30s`.
|
|
|
|
- `-log-level=DEBUG`: The log level to monitor. Defaults to `DEBUG`.
|
|
|
|
- `-log-include-location`: Include file and line information in each log line
|
|
monitored. The default is `true`.
|
|
|
|
- `-max-nodes=<count>`: Cap the maximum number of client nodes included
|
|
in the capture. Defaults to 10, set to 0 for unlimited.
|
|
|
|
- `-node-class=<node-class>`: Filter client nodes based on node class.
|
|
|
|
- `-node-id=<node1>,<node2>`: Comma separated list of Nomad client node ids
|
|
to monitor for logs, API outputs, and pprof profiles. Accepts id prefixes, and
|
|
"all" to select all nodes (up to count = max-nodes). Defaults to `all`.
|
|
|
|
- `-pprof-duration=<duration>`: Duration for pprof
|
|
collection. Defaults to 1s or `-duration`, whichever is less.
|
|
|
|
- `-pprof-interval=30s`: The interval between pprof collections. Set interval
|
|
equal to pprof duration to capture a single snapshot. Defaults to 30s or
|
|
`-pprof-duration`, whichever is more.
|
|
|
|
- `-server-id=<server1>,<server2>`: Comma separated list of Nomad server names to
|
|
monitor for logs, API outputs, and pprof profiles. Accepts server names, "leader", or
|
|
"all". Defaults to `all`.
|
|
|
|
- `-stale=<true|false>`: If "false", the default, get membership data from the
|
|
cluster leader. If the cluster is in an outage unable to establish
|
|
leadership, it may be necessary to get the configuration from a non-leader
|
|
server.
|
|
|
|
- `-event-topic=<allocation,evaluation,job,node,*>:<filter>`: Enable event
|
|
stream capture. Filter by comma delimited list of topic filters or "all".
|
|
Defaults to "none" (disabled). Refer to the [Events API](/nomad/api-docs/events) for
|
|
additional detail.
|
|
|
|
- `-verbose`: Enable verbose output
|
|
|
|
- `-output=path`: Path to the parent directory of the output
|
|
directory. Defaults to the current directory. If specified, no
|
|
archive is built.
|
|
|
|
- `-consul-http-addr=<addr>`: The address and port of the Consul HTTP
|
|
agent. Overrides the `CONSUL_HTTP_ADDR` environment variable.
|
|
|
|
- `-consul-token=<token>`: Token used to query Consul. Overrides the
|
|
`CONSUL_HTTP_TOKEN` environment variable and the Consul token
|
|
file.
|
|
|
|
- `-consul-token-file=<path>`: Path to the Consul token file. Overrides the `CONSUL_HTTP_TOKEN_FILE`
|
|
environment variable.
|
|
|
|
- `-consul-client-cert=<path>`: Path to the Consul client cert file. Overrides the `CONSUL_CLIENT_CERT`
|
|
environment variable.
|
|
|
|
- `-consul-client-key=<path>`: Path to the Consul client key file. Overrides the `CONSUL_CLIENT_KEY`
|
|
environment variable.
|
|
|
|
- `-consul-ca-cert=<path>`: Path to a CA file to use with Consul. Overrides the `CONSUL_CACERT`
|
|
environment variable and the Consul CA path.
|
|
|
|
- `-consul-ca-path=<path>`: Path to a directory of PEM encoded CA cert files to verify the Consul
|
|
certificate. Overrides the `CONSUL_CAPATH` environment variable.
|
|
|
|
- `-vault-address=<addr>`: The address and port of the Vault HTTP agent. Overrides the `VAULT_ADDR`
|
|
environment variable.
|
|
|
|
- `-vault-token=<token>`: Token used to query Vault. Overrides the `VAULT_TOKEN` environment
|
|
variable.
|
|
|
|
- `-vault-client-cert=<path>`: Path to the Vault client cert file. Overrides the `VAULT_CLIENT_CERT`
|
|
environment variable.
|
|
|
|
- `-vault-client-key=<path>`: Path to the Vault client key file. Overrides the `VAULT_CLIENT_KEY`
|
|
environment variable.
|
|
|
|
- `-vault-ca-cert=<path>`: Path to a CA file to use with Vault. Overrides the `VAULT_CACERT`
|
|
environment variable and the Vault CA path.
|
|
|
|
- `-vault-ca-path=<path>`: Path to a directory of PEM encoded CA cert files to verify the Vault
|
|
certificate. Overrides the `VAULT_CAPATH` environment variable.
|
|
|
|
## Output
|
|
|
|
This command prints a summary of the capture and the name of the timestamped
|
|
archive file produced.
|
|
|
|
## Examples
|
|
|
|
```shell-session
|
|
$ nomad operator debug -duration 5s -interval 5s -server-id all -node-id b5,20
|
|
Starting debugger...
|
|
|
|
Servers: (3/3) [server1.global server2.global server3.global]
|
|
Clients: (2/3) [b547cd3a-085f-68c2-55f4-e99beebb0433 20c0964b-72cc-4083-87fe-ec6905b6230a]
|
|
Interval: 5s
|
|
Duration: 5s
|
|
|
|
Capturing cluster data...
|
|
Capture interval 0000
|
|
Capture interval 0001
|
|
Capture interval 0002
|
|
Capture interval 0003
|
|
Created debug archive: nomad-debug-2020-12-08-034455Z.tar.gz
|
|
```
|
|
|
|
```shell-session
|
|
$ nomad operator debug -duration 5s -interval 5s -server-id all -node-id all -max-nodes=1
|
|
Starting debugger...
|
|
|
|
Servers: (3/3) [server1.global server2.global server3.global]
|
|
Clients: (1/3) [b547cd3a-085f-68c2-55f4-e99beebb0433]
|
|
Max node count reached (1)
|
|
Interval: 5s
|
|
Duration: 5s
|
|
|
|
Capturing cluster data...
|
|
Capture interval 0000
|
|
Capture interval 0001
|
|
Capture interval 0002
|
|
Capture interval 0003
|
|
Created debug archive: nomad-debug-2020-12-08-034113Z.tar.gz
|
|
```
|