Skip to content

nginx logs are not parsed using grok pattern #83

@jatinder10884

Description

@jatinder10884

I am using below grok_pattern for parsing Nginx logs

<source>

  @type tail
  path /data/nginxlogs/access.log
  pos_file /var/log/td-agent/nginx-access.log.pos
  tag es.nginx.acces
  <parse>
   @type multiline_grok
   <grok>
    pattern Started %{IPORHOST:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} \"%{URI:referrer}\" %{QS:user_agent}
#   grok_pattern %{MY_SUPER_PATTERN}
#   custom_pattern_path /etc/td-agent/MY_SUPER_PATTERN
#   multiline_start_regexp /^[^\s]/
  </grok>
  keep_time_key true
  </parse>
</source>
<filter es.nginx.acces>
  @type record_transformer
  <record>
  hostname ${hostname}
  </record>
</filter>
<match es.**>
  type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
 logstash_prefix "ngrokcontainer"
include_tag_key true
  tag_key "@log_name"
  flush_interval 1s
<endpoint>
    url http://testing-logs-ab-east-1.es.amazonaws.com
    region us-east-1
  </endpoint>
</match>

Kibana :

message 3.231.213.28 - - [18/Apr/2020:22:20:16 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
Its not parsing properly, but when I check this grok debugger its working fine.
Please let me know how I can fix this issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions