New Relic's infrastructure agent gathers its own data as well as integrations's logs and consolidates them in a single source. By default, logs appear in standard-output
and are added to a log file. To disable logs in standard output, see the agent's config options.
Logging severity levels
Infrastructure uses a subset of the standard Syslog severity levels:
ERROR
: Error conditions metWARN
: Warning conditions metINFO
: Informational messagesDEBUG
: Contains debug-level messages (useful when troubleshooting)
Important
DEBUG
level is only shown when the verbose mode is enabled.
Log formatting
For infrastructure agent v1.4.9 or higher, log messages are inlined with context values. This offers better grouping and filtering; for example:
containerized agent found in container
containerID: VALUE
By default, Infrastructure logs are formatted as text:
In foreground mode, log output is colored, without a timestamp:
DEBUG Sending deltas divided in blocks component=PatchSender mentityKey=ohaimaci mnumberOfBlocks=1
In background mode, logs are timestamped output, used when running as a service or dumping logs to a file:
time="2019-07-12T09:54:15+02:00" level=info msg="Agent service manager shutdown completed successfully." component=AgentService service=newrelic-infra
Alternatively, logs can be formatted as a JSON file:
{"context":{},"level":"info","msg":"upstart_interval_sec: 0","timestamp":"2019-07-11T18:24:03+02:00"}
{"context":{},"level":"info","msg":"plugin_dir: ","timestamp":"2019-07-11T18:24:03+02:00"}
To change the log format, see the agent configuration settings.
Log rotation
The infrastructure agent does not provide any native log rotation or compression mechanism. Instead, we encourage you to use consolidated log rotation tools, such as the Linux logrotate tool, which is usually installed by default in most Linux distributions.
Logrotate
can be configured as an entry in /etc/logrotate.conf
, or as a file in the /etc/logrotate.d
directory.
Since logrotate
is usually executed automatically as a cron job, verify that there is a logrotate
entry in cron (for example, /etc/cron.daily/logrotate
) similar to:
#!/bin/sh
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit 0
Smart verbose mode
For infrastructure agent versions 1.9.0 or higher, you can enable smart verbose mode for logs.
Smart verbose mode prevents debug messages being logged until an error message is detected. Once an error is detected, debug messages are written to the log file before the error. Note that only the most recent number of configured debug messages are logged. For example, if you have a configured limit of 10, when an error is detected, the 10 most recent debug messages before the error was detected are logged.
For more information on how to enable smart verbose mode and the debug message limit, see Infrastructure configuration settings.
Logging before infrastructure agent v1.4.9
Here is a comparison of functionality for infrastructure agent versions before and after v1.4.9:
Agent v1.4.9 and higher | Before v1.4.9 |
---|---|
Foreground mode logged. | The agent couldn't log some entries in foreground mode because the logging service wasn't able to write data until the agent was completely configured. |
Logs in text and JSON formats. | Logs in text only. |
Logs displayed as inline text. | Logs displayed as static literals in a single, decontextualized line. |
Integration log management
Integrations write JSON payloads into STDOUT and plain-text (JSON structured in the future) logs into STDERR.
The infrastructure agent handles integration STDERR lines and forward this output into the agent one, usually the service log.
Agent handles each STDERR line as follows:
- when agent runs in verbose mode: it just forwards the full STDERR line as a DEBUG agent log entry placing integration line contexts within the `msg` field.
- otherwise: it parses the line against the expected format (see below) and only logs as agent ERROR level, entries produced by integrations with `fatal` or `error` severity levels. In this case fields are extracted and forwarded in structured manner (therefore if JSON output is enabled for the agent fields become queryable.
Integration STDERR expected format
A line is expected to be a list of key-value pairs separated by an equal character. Keys can contain any character, whereas values can have three different formats:
- string: <quote>any character including escaped quotes \"<quote>
- map: &{any character}
- word: any character except spaces
Internally agent used this regex to extract the fields:
([^\s]*?)=(".*?[^\\]"|&{.*?}|[^\s]*)
For instance, this line:
time="2015-03-26T01:27:38-04:00" level=error msg="Foo bar baz" foo=bar
Will generate a structured agent log line with these fields:
- "time": "2015-03-26T01:27:38-04:00"- "level": "error"- "msg": "Foo bar baz"- "foo": "bar"