Internal logging: A return to JSON

[jordansissel]

Back in the day, logstash used json for its log format. However, non-UTF-8 problems and bad logging styles caused crashes in the logger, so we moved to ruby's Object#inspect.

Now that we have a better handle on UTF-8 problems and we have a better handle on how to do logging, it's probably time to make Cabin go back to emitting json.

[shurane]

Since I'm not so familiar with Logstash in the days of yore, are there some examples that refer to the JSON Logstash logging vs the Ruby logging? At first glance, what I would think is the difference between codec => json vs codec => rubydebug.

[jordansissel]

Ahh, good question!

Your assumption of json vs rubydebug is accurate. Currently logstash's internal logger uses a mechanism similar to the rubydebug codec. Technically rubydebug uses 'awesome_inspect' instead of 'inspect' but the effect is similar. I want to move us back to JSON now that we have less reason to lean so heavily on object inspect output.

[jordansissel]

Folks have asked about the current format, so I'll explain some of the history -

In the beginning, Logstash would log in JSON. A few years into the project, users began reporting all manner of weird string encoding exceptions, and I determined this to be caused by folks including non-UTF-8 text in their log inputs.
Logstash wants to accept all forms of data, including things not encoded in UTF-8, so what happened?

Well, JSON is defined as only valid in UTF-8. No other encoding is acceptable. So if you were sending invalid encoded logs to Logstash, it might want to warn you and say "Hey, bad logs! {here is the log}" - but when it tried to serialize to JSON, it would get an exception!

Summary - Logstash's attempt to log warnings or errors about text encoding while including context about that error (the text that was invalid) would actually crash inside the logger because it would try to serialize to JSON what was not valid as JSON.

Now that Logstash 1.4.0 improves greatly on the UTF-8 validation and text conversion, it is likely safe to move back to JSON as a Logstash log format.

[erik-stephens]

Not sure if there is already a mechanism to protect against an error feedback loop, but thought I'd mention it as something to protect against if this item gets addressed.

[purbon]

It would be really nice as some people ask how to ingest logstash/elasticsearch logs back into the same platform to then be able to monitor it.

👍 is there any work being done on that till now?

[simmel]

I tried this with regex but soon gave up because of FTW's logs. They are just too hard to parse with regex (and regex shouldn't be used for parsing).