Skip to content

[system][syslog] add a pattern in filebeat.system module to capture greedy multiline logs with ISO timestamps #13427

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Apr 9, 2025
Merged

[system][syslog] add a pattern in filebeat.system module to capture greedy multiline logs with ISO timestamps #13427

merged 8 commits into from
Apr 9, 2025

Conversation

stefans-elastic
Copy link
Contributor

Proposed commit message

add a pattern in filebeat.system module to capture greedy multiline logs with ISO timestamps

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@stefans-elastic stefans-elastic added the bug Something isn't working, use only for issues label Apr 4, 2025
@stefans-elastic stefans-elastic self-assigned this Apr 4, 2025
@stefans-elastic stefans-elastic requested review from a team as code owners April 4, 2025 12:28
@stefans-elastic stefans-elastic mentioned this pull request Apr 4, 2025
Closed
6 tasks
@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@andrewkroh andrewkroh added Integration:system System Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Apr 4, 2025
@ishleenk17
Copy link
Contributor

Also, can you state details about the second ask as per this issue.

@stefans-elastic
Copy link
Contributor Author

Also, can you state details about the second ask as per this issue.

Sure.

The gh issue contains two points:

Include the greedy grok pattern not only for SYSLOGTIMESTAMP, but for ISO8601 as well.
A process might send logs to /var/log/messages with white space characters and not the white spaces themselves, something like this
So this PR covets point 1. As for point 2: I'm not sure it is possible to have log entries with escaped whitespace characters. The example given in the issue seems to be coming from result event and for those the whitespace characters are escaped (as can be seen here: https://github.com/elastic/beats/blob/main/filebeat/module/system/syslog/test/darwin-syslog.log-expected.json#L31)

ishleenk17
ishleenk17 reviewed Apr 8, 2025
View reviewed changes
...a_stream/syslog/_dev/test/pipeline/test-iso8601-timestamp-multiline-syslog.log-expected.json Outdated
"input": {
"type": "log"
},
"message": "2022-04-21T14:30:00Z\n\t\tGoogleSoftwareUpdateAgent[21412/0x700007399000] [lvl=2] -[KSUpdateEngine updateAllExceptProduct:] KSUpdateEngine updating all\n\t\tinstalled products, except:'com.google.Keystone'.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still matching the 3rd pattern.
If you see we are getting fields as host.hostname, process.name which matches the 3rd pattern .
To match the 4th pattern , one of these fields should be sipped. So that other than timestamp the complete message becomes part of the greedymessage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, what is the "tell" of it (I think I'm missing it)?

@ishleenk17
Copy link
Contributor

Also, can you state details about the second ask as per this issue.

Sure.

The gh issue contains two points:

Include the greedy grok pattern not only for SYSLOGTIMESTAMP, but for ISO8601 as well. A process might send logs to /var/log/messages with white space characters and not the white spaces themselves, something like this So this PR covets point 1. As for point 2: I'm not sure it is possible to have log entries with escaped whitespace characters. The example given in the issue seems to be coming from result event and for those the whitespace characters are escaped (as can be seen here: https://github.com/elastic/beats/blob/main/filebeat/module/system/syslog/test/darwin-syslog.log-expected.json#L31)

Since, we don't observe these logs and this could be a corner case.lets focus on just adding the ISO timestamp for this PR.

ishleenk17
ishleenk17 reviewed Apr 8, 2025
View reviewed changes
packages/system/manifest.yml Outdated
@@ -1,7 +1,7 @@
format_version: 3.0.2
name: system
title: System
version: "1.67.3"
version: "1.67.4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
version: "1.67.4"
version: "1.68.0"

ishleenk17
ishleenk17 approved these changes Apr 8, 2025
View reviewed changes
Copy link
Contributor

@ishleenk17 ishleenk17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make the version change. Otherwise, LGTM.

ishleenk17
ishleenk17 reviewed Apr 8, 2025
View reviewed changes
packages/system/changelog.yml Outdated
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.67.4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- version: "1.67.4"
- version: "1.68.0"

@stefans-elastic
Copy link
Contributor Author

Please make the version change. Otherwise, LGTM.

updated the version

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @stefans-elastic

@stefans-elastic stefans-elastic merged commit 3d765a1 into elastic:main Apr 9, 2025
7 checks passed
@stefans-elastic stefans-elastic deleted the system-syslog-iso-grok-pattern branch April 9, 2025 10:39
@elastic-vault-github-plugin-prod
Copy link

Package system - 1.68.0 containing this change is available at https://epr.elastic.co/package/system/1.68.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ishleenk17 ishleenk17 ishleenk17 approved these changes

Assignees

@stefans-elastic stefans-elastic

Labels
bug Something isn't working, use only for issues Integration:system System Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@stefans-elastic @ishleenk17 @elasticmachine @andrewkroh