Decoders

There are cases where the log messages being parsed contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.

Consider the following message generated by the application:

{"status": "up and running"}

The Docker log message encapsulates something like this:

{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}

The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string.

Getting Started

Decoders are a built-in feature available through the Parsers file. Each parser definition can optionally set one or more decoders. There are two types of decoders:

• Decode_Field: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message.

• Decode_Field_As: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added.

Our pre-defined Docker parser has the following definition:

[PARSER]
    Name         docker
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    On
    # Command       |  Decoder  | Field | Optional Action   |
    # ==============|===========|=======|===================|
    Decode_Field_As    escaped     log

Each line in the parser with a key Decode_Field instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.

Decoder options

Name	Description
json	Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map.
escaped	Decode an escaped string.
escaped_utf8	Decode a UTF8 escaped string.

Optional Actions

If a decoder fails to decode the field or, you want to try another decoder, you can define an optional action. Available actions are:

Name	Description
try_next	if the decoder failed, apply the next decoder in the list for the same field.
do_next	if the decoder succeeded or failed, apply the next decoder in the list for the same field.

Actions are affected by some restrictions:

• Decode_Field_As: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).

• Decode_Field: If successful, can only be applied once for the same field.Decode_Field` is intended to decode a structured message.

Examples

escaped_utf8

Example input from /path/to/log.log:

{"log":"\u0009Checking indexes...\n","stream":"stdout","time":"2018-02-19T23:25:29.1845444Z"}
{"log":"\u0009\u0009Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary\n","stream":"stdout","time":"2018-02-19T23:25:29.1845536Z"}
{"log":"\u0009Done\n","stream":"stdout","time":"2018-02-19T23:25:29.1845622Z"}

Example output:

[24] tail.0: [1519082729.184544400, {"log"=>"   Checking indexes...
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845444Z"}]
[25] tail.0: [1519082729.184553600, {"log"=>"           Validated: _audit _internal _introspection _telemetry _thefishbucket history main snmp_data summary
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845536Z"}]
[26] tail.0: [1519082729.184562200, {"log"=>"   Done
", "stream"=>"stdout", "time"=>"2018-02-19T23:25:29.1845622Z"}]

Decoder configuration file:

[SERVICE]
    Parsers_File fluent-bit-parsers.conf

[INPUT]
    Name        tail
    Parser      docker
    Path        /path/to/log.log

[OUTPUT]
    Name   stdout
    Match  *

The fluent-bit-parsers.conf file:

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S %z
    Decode_Field_as escaped_utf8 log