Commit Graph

49 Commits

Author SHA1 Message Date
Nick Ethier
4b08ef0534 Merge pull request #4765 from jippi/increase-line-scan-limit
fix: increase log rotator line scan limit
2018-10-29 18:46:30 -07:00
Alex Dadgar
7b7cb382dc more test fixes 2018-10-16 16:56:56 -07:00
Nick Ethier
5b14d24bf4 executor v2 (#4656)
* client/executor: refactor client to remove interpolation

* executor: POC libcontainer based executor

* vendor: use hashicorp libcontainer fork

* vendor: add libcontainer/nsenter dep

* executor: updated executor interface to simplify operations

* executor: implement logging pipe

* logmon: new logmon plugin to manage task logs

* driver/executor: use logmon for log management

* executor: fix tests and windows build

* executor: fix logging key names

* executor: fix test failures

* executor: add config field to toggle between using libcontainer and standard executors

* logmon: use discover utility to discover nomad executable

* executor: only call libcontainer-shim on main in linux

* logmon: use seperate path configs for stdout/stderr fifos

* executor: windows fixes

* executor: created reusable pid stats collection utility that can be used in an executor

* executor: update fifo.Open calls

* executor: fix build

* remove executor from docker driver

* executor: Shutdown func to kill and cleanup executor and its children

* executor: move linux specific universal executor funcs to seperate file

* move logmon initialization to a task runner hook

* client: doc fixes and renaming from code review


* taskrunner: use shared config struct for logmon fifo fields

* taskrunner: logmon only needs to be started once per task
2018-10-16 16:53:31 -07:00
Christian Winther
5314bfcd34 fix: increase log rotator line scan limit
In case where gelf/json logging is used, its fairly easy to exceed the 16k limit, resulting in json output being cut up into multiple strings

the result is invalid json lines which can create all kind of badness in the logging server

This fixes https://github.com/hashicorp/nomad/issues/4699

Signed-off-by: Christian Winther <jippignu@gmail.com>
2018-10-09 18:57:18 +02:00
Alex Dadgar
98c7abe541 Tests only use testlog package logger 2018-06-13 15:40:56 -07:00
Alex Dadgar
80ac8c2a37 Add new line test 2018-05-11 10:52:09 -07:00
Alex Dadgar
8aaf074699 Avoid splitting log line across two files
We attempt to avoid splitting a log line between two files by detecting
if we are near the file size limit and scanning for new lines and only
flushing those.

BenchmarkRotator/1KB-8            300000              5613 ns/op
BenchmarkRotator/2KB-8            200000              8384 ns/op
BenchmarkRotator/4KB-8            100000             14604 ns/op
BenchmarkRotator/8KB-8             50000             25002 ns/op
BenchmarkRotator/16KB-8            30000             47572 ns/op
BenchmarkRotator/32KB-8            20000             92080 ns/op
BenchmarkRotator/64KB-8            10000            165883 ns/op
BenchmarkRotator/128KB-8            5000            294405 ns/op
BenchmarkRotator/256KB-8            2000            572374 ns/op
2018-05-10 15:11:01 -07:00
Alex Dadgar
99182736e8 Benchmark for rotator
BenchmarkRotator/1KB-8            200000              5572 ns/op
BenchmarkRotator/2KB-8            200000              8338 ns/op
BenchmarkRotator/4KB-8            100000             14246 ns/op
BenchmarkRotator/8KB-8             50000             25279 ns/op
BenchmarkRotator/16KB-8            30000             48602 ns/op
BenchmarkRotator/32KB-8            20000             92159 ns/op
BenchmarkRotator/64KB-8            10000            154766 ns/op
BenchmarkRotator/128KB-8            5000            296872 ns/op
BenchmarkRotator/256KB-8            3000            551793 ns/op
2018-05-10 14:15:15 -07:00
Alex Dadgar
9966194730 naming of constants 2018-05-09 16:46:52 -07:00
Josh Soref
ef6a98388a spelling: shutdown 2018-03-11 18:55:49 +00:00
Josh Soref
a7476398be spelling: severity 2018-03-11 18:53:52 +00:00
Josh Soref
bd2912978b spelling: isolation 2018-03-11 18:19:02 +00:00
Alex Dadgar
a9e3a41407 Enable more linters 2017-09-26 15:26:33 -07:00
Luke Farnell
7a56971508 fixed all spelling mistakes for goreport 2017-08-07 17:13:05 -04:00
Alex Dadgar
212fed63df Fix leaked plugin files for syslog server
This PR fixes a leaking of the unix socket used when launching a syslog
server for the Docker driver.

Fixes https://github.com/hashicorp/nomad/issues/2844
2017-07-30 17:51:38 -07:00
Alex Dadgar
d4e35815a1 executor and logging pkg 2017-07-21 12:14:54 -07:00
Hynek Schlawack
96c9f9dd02 Fix typos 2017-06-16 16:10:12 +02:00
Ian Eyberg
0d76cd24bb dont throw away errors in log rotation 2017-05-25 11:49:33 -07:00
Michael Schurter
de7351b959 Move chroot building into TaskRunner
* Refactor AllocDir to have a TaskDir struct per task.
* Drivers expose filesystem isolation preference
* Fix lxc mounting of `secrets/`
2017-01-05 16:31:49 -08:00
Michal Wieczorek
78eaeafc15 Enable syslog server and universal collector for windows 2016-09-05 00:26:36 +02:00
Alex Dadgar
ddfae93941 Wrap file rotator tests in wait for 2016-08-11 19:23:03 -07:00
Kenjiro Nakayama
be177a0a3e Add Syslog server start shutdown test 2016-08-06 02:01:33 +09:00
Cameron Davison
c7817127c0 remove the expected leading space, after the colon in syslog 2016-07-06 11:08:24 -05:00
Cameron Davison
df8bd16133 get into the hour minute second part of the time before looking for spaces, and then looking for the : seperator 2016-07-06 11:08:24 -05:00
Alex Dadgar
7cd4866c89 Make line copy to avoid being overriden by subsequent scans 2016-06-20 13:14:43 -07:00
Sean Chittenden
9cb16b5b9e Explicitly enumaret the build targets in _unix
`!windows` was being used as the synonym for `darwin dragonfly freebsd linux netbsd openbsd solaris`.  While I don't imagine `android` will be a prime target for Nomad in the near term, favor explicit build targets.

List of build targets generated by Go 1.7's dist command: `go tool dist list | sort | cut -d '/' -f 1 | sort | uniq`
2016-05-07 10:42:01 -07:00
Sean Chittenden
146c4e8fd6 Rename from posix to unix to parallel x/sys/unix
Use the `_unix` file suffix to denote *NIX-like semantics in order to parallel the designation given by the package `golang.org/x/sys/unix`.
2016-05-07 10:33:43 -07:00
Alex Dadgar
c3c3b66fbf protect writing to closed channel 2016-04-18 11:46:57 -07:00
Diptanu Choudhury
eccebeca9e Fixing race issues in the log rotator 2016-04-06 18:58:34 -07:00
Diptanu Choudhury
fdea9d7905 Logging errors in the file rotator 2016-04-06 12:06:49 -07:00
Diptanu Choudhury
0c9a5dcb8b Fixing comments 2016-03-29 17:30:43 -07:00
Diptanu Choudhury
c0c78f6b36 Not deleting files if the number of files is less than max files 2016-03-29 16:42:18 -07:00
Diptanu Choudhury
c86d38360a Making the calls to exit idempotent 2016-03-22 11:49:56 -07:00
Diptanu Choudhury
0f355e8f87 Introduced a method in executor to launch syslog server 2016-03-18 11:49:10 -07:00
Diptanu Choudhury
8e13eaa522 Using unix domain sockets for docker syslog collector 2016-03-09 23:28:26 -08:00
Diptanu Choudhury
54d7057912 Writing logs to alloc dir 2016-02-24 20:06:43 -08:00
Alex Dadgar
207d544a55 client: standardize error logging format 2016-02-24 15:57:58 -08:00
Diptanu Choudhury
a735ad155d Making the build work for windows 2016-02-23 15:17:07 -08:00
Diptanu Choudhury
e9e3a23e89 Closing the messages channel 2016-02-23 10:14:54 -08:00
Diptanu Choudhury
23cc7820d5 Stoping go-routines in the syslog collector 2016-02-23 09:43:14 -08:00
Diptanu Choudhury
0d9697d955 Buffering writes to a rotated file 2016-02-23 08:52:06 -08:00
Diptanu Choudhury
2a042ff625 Made the syslog server use a buffered chan 2016-02-22 16:10:23 -08:00
Diptanu Choudhury
15082976ed Added more comments 2016-02-19 21:58:44 -08:00
Diptanu Choudhury
657d6a4ff6 Fixed tests 2016-02-19 15:15:59 -08:00
Diptanu Choudhury
56937f6629 Added some docs to the syslog server 2016-02-19 14:20:21 -08:00
Diptanu Choudhury
dfdf22ea4f Added some docs to the FileRotator 2016-02-19 14:11:31 -08:00
Diptanu Choudhury
387fcc36c8 Making the log rotator a writer 2016-02-19 14:01:07 -08:00
Diptanu Choudhury
3ba626df72 Renamed packages 2016-02-19 13:08:25 -08:00
Diptanu Choudhury
dc3f50da27 Implemented file rotator 2016-02-19 12:55:34 -08:00