Backport/json logging pr 4820 #5276
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JDBC input is a popular plugin and has been stable over the last few releases. Fixes #4198
wip initial Accessors, Event, EventImpl, Path, Timestamp and PathTest wip wip FieldReference and Accessors implementation rename targetCache to lut and set it protected initial Accessors tests todo comment more tests Timestamp implementation Timestamp tests fix method name add Long constructor event initialization, timestamp handling and json serialization add <> type information custom json serializer for Timestamp remove toJson test initial Event test more tests comments debug traces initial jruby Event wrapper and specs added PathCache implemented includes added clone wrap all Event methods Rakefile to build and jar missing getters and implement overwrite support Date conversion proper cast and coercion replace Ruby Event with Java Event test for field reference setter type coercion disable specs timestap setter should also set in map, accept more timestamp types pre cache timestamp and expose isTimestamp constructor from DateTime expose proper Ruby Timestamp object Ruby Timestamp basic specs also load JRuby Timestamp transpose Java<->Ruby Timestamp fix timestamp specs new jar cleanup object construction
This implementation is different from the master version since instead of doing JIT Template compilation on every events we create a data structure of a compiled templates. This template is cached in a concurrent hashmap and used on other requests. fixing conflict
First the case with single fieldref Add support for +%s => epoch Small comments concerning the caching mechanism and the memory usage Make the #sprintf method work with Array and Hash
event clone append in Ruby tag & append add timestamp assignment spec comment out original classes fix at signature implemented del remove from array del test
more event java updates - updated metadata - moved #append to java - other small test fixes commit gradle wrapper jar
…re-event-java impl, relates to #4191 fixed timezone issue extracted logstash-core and reorganized specs extracted logstash-core-event extract java Event into logstash-core-event-java in a proper gem remove obsolete jruby_event bootstrapping fix require path add java code bootstrap use logstash-core-event/logstash-core-event.rb remove obsolete files basic instructions LogStash::Json need to be initialized from event update jruby and gradle versions update compile:logstash-core-event-java rake task WIP tasks refactor fix gem.files skip test if class is not defined fix gem related tasks for new structure add gem spec dirs in core tests bootstrap java implementation when requiring timestamp new Cloner class and Event clone impl fix array fields assignments, see #4140 don't rely on json implementation ordering fix skipped last iterpolation char remove implementation specific unnecessary check also require ruby classes define error class in ruby raise exception on invalid format remove implementation specific tests and extract and put logger related test in pending missing bits for having all core timestamp specs pass run all core specs remove leftover comment regex missing encoding header revert to logstash-core-event by default finished proper gemification useless require dynamically pick specs depending on logstash-core-event-* implementation logstash root package version missing file for proper gemification do not build java event by default always check for root logstash lib dir fix concurrent-ruby version confict fix rebase conflict re-enable specs user vars instead of constants move non core code in bootstrap document version files move version file remove useless code use version in logstash-core fix gem files list put back concurrent-ruby version constrain as in master add dependency on logstash-core-event remove dependency on logstash-core to avoid circular dependency fix rebase conflict remove circular dependency fix specs update README
…idation properly when a private gem server (not rubygems) has been configured. This basically loads the sources so the system is able to perform the validation with the right destionation and not with the default source (rubygems.org) Former commits: add all defined sources to rubygems so verification can talk to all the repositories, even the private ones added a very simple plugin manager install command spec, to check for properly loading sources when validating broke long lines into smaller ones in the install command make sure the update command takes into account all declared sources, plus there is the option to avoid gem validation in case of only having a private gem repo and no internet connection fix wrong conditional in the validation of plugins move the Gem.sources loading to a common place, to sources are loaded all at once and we don't need to worry on each call to be sure we do the right thing for other sources different that rubygems update test to the last changes in the way gemfile is loaded move Gem::SourceList.from to bundler::invoke! so we use the default gemset.sources array fix loading path for Gem::SourceList in bundler, so it's not colliding with other classes Revert "move the Gem.sources loading to a common place, to sources are loaded all at once and we don't need to worry on each call to be sure we do the right thing for other sources different that rubygems" This reverts commit 6e1c809. Revert "update test to the last changes in the way gemfile is loaded" This reverts commit dc5f65c. make the Gem.sources load to happen inside the logstash_plugin? call add the idea of settings to the plugin manager module, like this we can pass throw different stuff usefull to setup dependant components add the install spec back remove the PluginManager::Settings concept change sources for rubygems_sources in the plugin manager options change rubygems_sources to be rubygems_source update comments ammend plugin manager options description in the comments spaces and new lines cleanup merged duplicated plugin manager util_spec introduced during rebase add a check when Gems.versions fail Add the validation exception for the plugin manager add better error handling for situation where the validation is not possible due to a connection issue with the remote server Fixes #3583
…ager as is only relevant for master, not for the version branches Fixes #4212
updates requested by code review changes requested by colin: make workers override from -w arg do not set workers unless user actually specified it via cmdline fix defaults printing add describe block to improve test output readability Closes #4130
including dependencies, and reuse them in an offline installation by providing an package for it. It adds two important commands to the plugin manager, the pack and upack, to handle package creation and installation and adds the --local flag to install and update to pull plugins from the installed local packages. Former commits: add a task to create a bundle, under vendor/cache of the installed gems + their dependencies, this can be used later on to be installed offline add an option to pass the --local flag to the bin/plugin update task, so it fetch plugins from a local cache, by default under vendor/cache rename package:build to package:build-cache as is more meaningfull add a --local flag to bin/plugin install to users can install plugins from the local cache, under the default directory of vendor/cache add a plugin manager command to build the local cache of installed plugins using bundler package alike command It adds code for compressing and extracting in zip and tar formats to the utils module of logstash. The zip module is only jruby dependant as it uses functions provided by java. There is also code add in the plugin manager package command to handle compression of plugins dumping. Cleanup the custom java code to compress and extract zip files as it has been known that using rubyzip is good and it has a more ruby like features so the code is more clean and portable. set of smallish improvement requested during review added several options to handle situation when the package of plugins we want to generate is already there clean up old code applyed feedback from review, mostly changed in documentating behaviour plus better wording for messages relocate the Environment.windows? check to the bootstrap side as it's used also in the plugin manager add an unpack bin/plugin command so users can install packages of plugins throw the plugin manager document override behaviour in the compress helpers (for zip and tar) plus add a fix for the tar extract when reading entries made the unpack filename parameter a required one add a force option to the bin/plugin unpack command add a notice to that if using a local cache the --local flag should be passed with Code cleanup and refactor introduced during review add two wording suggestions comming from review ammend more wording skip the major version validation as in situation where there is no internet connection like when using the local vendor/cache to do the update move compress to the bootstrap environment as being used in the plugin manager means not being loaded with logstash-core Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies Revert "Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies" This reverts commit a9d7f46. patch the Bundler::Source::Rubygems to fetch only gems from a remote source small changes in several parts of the plugin manager and the creation of a common pack command with shared code change compress to read in chuncks fix wrong var name in the bootstrap compress utils module fix namespacing conflicts add basic test for the compress utility module ammend more namespace issues add a comment to the rubygems mockey patch to rebuild the gem cache apply cosmetic changes make the compress module raise CompressError remove vendor_path and pattern_path form environment as they where mix up during rebase change the bin/pack force option to be force-delete-cache rename force_delete_cache to overwrite change force for overwrite in tha pack command remove the override option for bin/plugin unpack revert Gemfile and Genfile.lock wrong committed
When the event was serialized to string using a `to_s` call, mostly happenning when doing logging. Each calls would generate a new template each template was unique because it was containing the date. The accumulation of templates was making logstash goes OOM. We aren't cleaning the templates because they should stabilize to a finite number.
* start logstash with --allow-unsafe-shutdown to force_exit on stalled shutdown * by default --allow-unsafe-shutdown is disabled * stall detection kicks in when SIGTERM/SIGINT is received * check if inflight event count isn't going down and if there are blocked/blocking plugin threads
Fixes #5133
GitHub's rendering is quite ok, just a single callout number is wrong. On https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html the rendered numbering gets quite confusing. Fixes #5146
The example as an incorrect syntax. As per https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field it should be `remove_field` and not `remove` Fixes #5104
Signed-off-by: Stefan Scherer <[email protected]> Fixes #4913
Signed-off-by: Stefan Scherer <[email protected]> Fixes #4913
Signed-off-by: Stefan Scherer <[email protected]> Fixes #4913
Make structural changes to the plugin maintainer guide in 2.x
lut invalidation spec fix lut invalidation extra comment fix lut invalidation, invalid target path, with tests post review fixes
…on characters Fixes #5190
Logstash uses by default in the shell script 1 gig of memory for the JVM. but when we use rake it uses the default settings of JRUBY and start a vm 500megs of memory. The recent changes in JRUBY and JRuby-OpenSSL have increased the base memory and make the installing of plugins with a OOM issues. This PR make sure the CI scripts and the rake jruby launch uses the default of 1gig. Fixes: #5179
When running any logstash's command you can use DEBUG=1 which should give the user a bit more information about what LS is doing. This PR changes the behavior of bundler to make sure its run with the `--verbose` flag. Fixes #5199
add missing overwrite method and preserve @gems_by_name when updating log options preservation discards no constrains constrains rewording
The current logic for printing %JAVA_HOME% inside if ( ) block failes if the path contains a `(` which is the case for 32bit java (`C:\Program Files (x86)\java\jdk`) The current logic for appending to JAVA_OPTS does not work as it will be expanded while interpreting the else block. Use goto instead. Fixes #5214
This is made available by a --log-in-json flag. Default is false. When false, the old behavior [1] is used. When true, JSON logs are emitted. [1] The old behavior is realy two things. First, using Object#inspect to serialize. Second, to color the output if the IO is a tty. For #1569 This is a manual backport of PR #4820 into the 2.x branch.
|
Derp. wrong branch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow Logstash to write its logs in JSON format
This is made available by a --log-in-json flag. Default is false.
When false, the old behavior [1] is used.
When true, JSON logs are emitted.
[1] The old behavior is really two things. First, using Object#inspect to
serialize. Second, to color the output if the IO is a tty.
For #1569
This is a manual backport of PR #4820 into the 2.x branch.