Skip to content

Use optimized grok library #1657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 27, 2014
Merged

Conversation

suyograo
Copy link
Contributor

Tested using %{COMBINEDAPACHELOG} grok format. Almost 3x throughput increase --

With grok version 0.10.12

{
           "@version" => "1",
         "@timestamp" => "2014-08-04T23:19:55.993Z",
            "message" => "Suyogs-MacBook-Pro.local",
       "events.count" => 445142,
     "events.rate_1m" => 8388.367317089305,
     "events.rate_5m" => 7234.09457935761,
    "events.rate_15m" => 6983.090272699621,
               "tags" => [
        [0] "metric"
    ]
}

With grok version 0.11.0 and associated LS changes

{
           "@version" => "1",
         "@timestamp" => "2014-08-05T00:29:43.977Z",
            "message" => "Suyogs-MacBook-Pro.local",
       "events.count" => 2965949,
     "events.rate_1m" => 20580.377595661972,
     "events.rate_5m" => 17600.49616640097,
    "events.rate_15m" => 16340.908759448706,
               "tags" => [
        [0] "metric"
    ]
}

LS Config:

input {
  generator {
    message => '31.184.238.164 - - [24/Jul/2014:05:35:37 +0530] "GET /logs/access.log HTTP/1.0" 200 69849 "http://8rursodiol.enjin.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.12785 YaBrowser/13.12.1599.12785 Safari/537.36" "www.dlwindianrailways.com"'
    #count => 1
  }
}
filter {
  grok {
     match => {"message" => "%{COMBINEDAPACHELOG}"}
  }
  metrics { meter => "events" add_tag => "metric" }
}
output {
  # only emit events with the 'metric' tag
  if "metric" in [tags] {
      stdout {
        codec => "rubydebug"
      }
  }
}

@electrical
Copy link

This is impressive! I love it 👍 :shipit:

@untergeek
Copy link
Member

Rock on!

@jordansissel jordansissel self-assigned this Aug 22, 2014
@jordansissel jordansissel added this to the v1.5.0 milestone Aug 22, 2014
@elasticsearch-release
Copy link

💚 Test passed.
Refer to this link for build results: http://build-eu-1.elasticsearch.org/job/Logstash_PR/79/

@jordansissel
Copy link
Contributor

Tests pass. Manual tests look good. Code looks good. Merge it!

@suyograo suyograo force-pushed the use_optimized_grok branch from 8411eb0 to 0aaf8c6 Compare August 27, 2014 20:21
@suyograo suyograo merged commit 0aaf8c6 into elastic:master Aug 27, 2014
suyograo pushed a commit that referenced this pull request Aug 27, 2014
@elasticsearch-release
Copy link

💚 Test passed.
Refer to this link for build results: http://build-eu-1.elasticsearch.org/job/Logstash_PR/87/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants