Skip to content

log: add system_config option to force stacktrace level #5008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

daipom
Copy link
Contributor

@daipom daipom commented Jun 23, 2025

Which issue(s) this PR fixes:
None.

What this PR does / why we need it:

What this PR does

Add a new system_config option: forced_stacktrace_level.

  • Option: system/log/forced_stacktrace_level
  • Type: enum
    • none, trace, debug, info, warn, error, fatal
  • Default: none

By default, the behavior of Fluentd does not change.
So, this does not affect existing users.

If you set this option and change the value, the log levels of stacktraces are forced to that value.

Stacktraces that do not meet the log level are discarded in advance.
After the log level is changed, it is judged again for logging or discarded finally.
For example, by default (log_level info), ...

  • All trace/debug stacktraces are not logged regardless of this option.
  • If you set forced_stacktrace_level debug, then all stacktraces are not logged.

This is because we should avoid logging logs that contradict log_level, and to avoid unexpectedly logging logs that are initially trace/debug level.

Setting example:

<system>
  <log>
    forced_stacktrace_level info
  </log>
</system>

why we need it

To make it easier to exclude stacktraces from monitoring Fluentd's own logs.

For example, there could be a system that raises an alert when an error-level log is detected.
You may want to exclude some error-level logs from the detection.
However, if stacktraces are logged at error level, you need to exclude those stacktraces as well.
With the current specification, that is troublesome.

Ideally, it would be preferable to exclude all stacktraces from the detection.
It is sufficient to have access to stacktraces when investigating the details.

An easy way to improve the current situation is this feature.
If you are detecting error logs as in the previous example, you can easily exclude all stacktraces from the detection by forcing the stacktrace level to info with this feature.

Docs Changes:
TODO

Release Note:
System configuration: Add forced_stacktrace_level to force the log level of stacktraces.

Sample:

<system>
  <log>
    forced_stacktrace_level info
  </log>
</system>

<source>
  @type sample
  @id test
  tag test.fail
</source>

<match test.fail>
  @type null
  never_flush
</match>

Without forced_stacktrace_level info:

2025-06-23 18:16:45 +0900 [info]: #0 fluent/log.rb:371:info: fluentd worker is now running worker=0
2025-06-23 18:16:46 +0900 [warn]: #0 fluent/log.rb:392:warn: emit transaction failed: error_class=RuntimeError error="failed to flush" location="/home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'" tag="test.fail"
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/output.rb:865:in `emit_sync'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:115:in `emit_stream'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:106:in `emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `block in emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `times'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `emit'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:100:in `run'
  2025-06-23 18:16:46 +0900 [warn]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

With forced_stacktrace_level info:

2025-06-23 18:13:59 +0900 [warn]: #0 fluent/log.rb:392:warn: emit transaction failed: error_class=RuntimeError error="failed to flush" location="/home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'" tag="test.fail"
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/out_null.rb:54:in `process'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/output.rb:865:in `emit_sync'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:115:in `emit_stream'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/event_router.rb:106:in `emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `block in emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `times'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:115:in `emit'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin/in_sample.rb:100:in `run'
  2025-06-23 18:13:59 +0900 [info]: #0 fluent/event_router.rb:114:emit_stream: /home/daipom/work/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

TODO:

  • Add tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant