Skip to content

o365: fix handling of empty sip IP fields and avoid script allocations #14151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 10, 2025
Merged

o365: fix handling of empty sip IP fields and avoid script allocations #14151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2ea6ff4
o365: fix handling of empty sip IP fields and avoid script allocations
efd6 Jun 5, 2025
a06c7b8
address pr comment
efd6 Jun 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/o365/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# newer versions go on top
- version: "2.18.1"
changes:
- description: Prevent convert processor failures with fields with empty string values.
type: bugfix
link: https://github.com/elastic/integrations/pull/14151
- description: Avoid script parameter allocations.
type: bugfix
link: https://github.com/elastic/integrations/pull/14151
- version: "2.18.0"
changes:
- description: ECS mapping improvements.
Expand Down
38 changes: 37 additions & 1 deletion packages/o365/data_stream/audit/_dev/test/pipeline/test-sec-comp-alerts-events.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,42 @@
"Version": 1,
"Workload": "SecurityComplianceCenter"
}
},
{
"event": {
"original": "{\"Category\":\"ThreatManagement\",\"UserKey\":\"SecurityComplianceAlerts\",\"Operation\":\"AlertEntityGenerated\",\"OrganizationId\":\"aaaaa14f-bbbb-cccc-dddd-eeee5a778630\",\"AlertEntityId\":\"[email protected](external, opens in a new tab or window)\",\"Source\":\"Office 365 Security & Compliance\",\"Name\":\"Email reported by user as malware or phish\",\"AlertType\":\"System\",\"RecordType\":40,\"Version\":1,\"Status\":\"Active\",\"ObjectId\":\"[email protected](external, opens in a new tab or window)\",\"ResultStatus\":\"Succeeded\",\"Comments\":\"New alert\",\"AlertLinks\":[{\"AlertLinkHref\":\"\"}],\"Data\":\"{\\\"etype\\\":\\\"User\\\",\\\"eid\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"tid\\\":\\\"aaaaa14f-bbbb-cccc-dddd-eeee5a778630\\\",\\\"ts\\\":\\\"2025-05-02T05:10:44.5371861Z\\\",\\\"te\\\":\\\"2025-05-02T05:10:44.5371861Z\\\",\\\"op\\\":\\\"UserSubmission\\\",\\\"tdc\\\":\\\"1\\\",\\\"suid\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"ut\\\":\\\"Regular\\\",\\\"ssic\\\":\\\"0\\\",\\\"tsd\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"sip\\\":\\\"\\\",\\\"imsgid\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"srt\\\":\\\"1\\\",\\\"trc\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"ms\\\":\\\"Welkom op My company\\\",\\\"sid\\\":\\\"aaa174f-bbbb-cccc-dddd-eeeea27623b4\\\",\\\"aii\\\":\\\"aaaa109-bbb-cccc-dddd-eeeea1c1dd41\\\",\\\"md\\\":\\\"2025-05-02T10:40:16.9298292Z\\\",\\\"etps\\\":\\\"SubmissionId:aaaae50f-bbbb-4760-cccc-dddda276218e\\\",\\\"lon\\\":\\\"UserSubmission\\\"}\",\"Severity\":\"Low\",\"Workload\":\"SecurityComplianceCenter\",\"EntityType\":\"User\",\"AlertId\":\"aaaa01b-bbbb-cccc-dddd-eeeea276218e\",\"UserId\":\"SecurityComplianceAlerts\",\"CreationTime\":\"2025-06-03T08:10:44\",\"Id\":\"aaaabce60-bbbb-cccc-dddd-eeeea27623da\",\"UserType\":4,\"PolicyId\":\"aaaa5770-bbbb-cccc-dddd-eeee2c27bbb3\"}"
},
"o365audit": {
"Category": "ThreatManagement",
"UserKey": "SecurityComplianceAlerts",
"Operation": "AlertEntityGenerated",
"OrganizationId": "aaaaa14f-bbbb-cccc-dddd-eeee5a778630",
"AlertEntityId": "[email protected](external, opens in a new tab or window)",
"Source": "Office 365 Security & Compliance",
"Name": "Email reported by user as malware or phish",
"AlertType": "System",
"RecordType": 40,
"Version": 1,
"Status": "Active",
"ObjectId": "[email protected](external, opens in a new tab or window)",
"ResultStatus": "Succeeded",
"Comments": "New alert",
"AlertLinks": [
{
"AlertLinkHref": ""
}
],
"Data": "{\"etype\":\"User\",\"eid\":\"[email protected](external, opens in a new tab or window)\",\"tid\":\"aaaaa14f-bbbb-cccc-dddd-eeee5a778630\",\"ts\":\"2025-05-02T05:10:44.5371861Z\",\"te\":\"2025-05-02T05:10:44.5371861Z\",\"op\":\"UserSubmission\",\"tdc\":\"1\",\"suid\":\"[email protected](external, opens in a new tab or window)\",\"ut\":\"Regular\",\"ssic\":\"0\",\"tsd\":\"[email protected](external, opens in a new tab or window)\",\"sip\":\"\",\"imsgid\":\"[email protected](external, opens in a new tab or window)\",\"srt\":\"1\",\"trc\":\"[email protected](external, opens in a new tab or window)\",\"ms\":\"Welkom op My company\",\"sid\":\"aaa174f-bbbb-cccc-dddd-eeeea27623b4\",\"aii\":\"aaaa109-bbb-cccc-dddd-eeeea1c1dd41\",\"md\":\"2025-05-02T10:40:16.9298292Z\",\"etps\":\"SubmissionId:aaaae50f-bbbb-4760-cccc-dddda276218e\",\"lon\":\"UserSubmission\"}",
"Severity": "Low",
"Workload": "SecurityComplianceCenter",
"EntityType": "User",
"AlertId": "aaaa01b-bbbb-cccc-dddd-eeeea276218e",
"UserId": "SecurityComplianceAlerts",
"CreationTime": "2025-06-03T08:10:44",
"Id": "aaaabce60-bbbb-cccc-dddd-eeeea27623da",
"UserType": 4,
"PolicyId": "aaaa5770-bbbb-cccc-dddd-eeee2c27bbb3"
}
}
]
]
}
130 changes: 130 additions & 0 deletions .../o365/data_stream/audit/_dev/test/pipeline/test-sec-comp-alerts-events.json-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,136 @@
"user": {
"id": "SecurityComplianceAlerts"
}
},
{
"@timestamp": "2025-06-03T08:10:44.000Z",
"ecs": {
"version": "8.11.0"
},
"email": {
"local_id": [
"aaaa109-bbb-cccc-dddd-eeeea1c1dd41"
],
"message_id": [
"[email protected](external, opens in a new tab or window)"
],
"sender": {
"address": [
"[email protected](external, opens in a new tab or window)"
]
},
"subject": [
"Welkom op My company"
],
"to": {
"address": [
"[email protected](external, opens in a new tab or window)"
]
}
},
"event": {
"action": "AlertEntityGenerated",
"category": [
"web"
],
"code": "SecurityComplianceAlerts",
"id": "aaaabce60-bbbb-cccc-dddd-eeeea27623da",
"kind": "alert",
"original": "{\"Category\":\"ThreatManagement\",\"UserKey\":\"SecurityComplianceAlerts\",\"Operation\":\"AlertEntityGenerated\",\"OrganizationId\":\"aaaaa14f-bbbb-cccc-dddd-eeee5a778630\",\"AlertEntityId\":\"[email protected](external, opens in a new tab or window)\",\"Source\":\"Office 365 Security & Compliance\",\"Name\":\"Email reported by user as malware or phish\",\"AlertType\":\"System\",\"RecordType\":40,\"Version\":1,\"Status\":\"Active\",\"ObjectId\":\"[email protected](external, opens in a new tab or window)\",\"ResultStatus\":\"Succeeded\",\"Comments\":\"New alert\",\"AlertLinks\":[{\"AlertLinkHref\":\"\"}],\"Data\":\"{\\\"etype\\\":\\\"User\\\",\\\"eid\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"tid\\\":\\\"aaaaa14f-bbbb-cccc-dddd-eeee5a778630\\\",\\\"ts\\\":\\\"2025-05-02T05:10:44.5371861Z\\\",\\\"te\\\":\\\"2025-05-02T05:10:44.5371861Z\\\",\\\"op\\\":\\\"UserSubmission\\\",\\\"tdc\\\":\\\"1\\\",\\\"suid\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"ut\\\":\\\"Regular\\\",\\\"ssic\\\":\\\"0\\\",\\\"tsd\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"sip\\\":\\\"\\\",\\\"imsgid\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"srt\\\":\\\"1\\\",\\\"trc\\\":\\\"[email protected](external, opens in a new tab or window)\\\",\\\"ms\\\":\\\"Welkom op My company\\\",\\\"sid\\\":\\\"aaa174f-bbbb-cccc-dddd-eeeea27623b4\\\",\\\"aii\\\":\\\"aaaa109-bbb-cccc-dddd-eeeea1c1dd41\\\",\\\"md\\\":\\\"2025-05-02T10:40:16.9298292Z\\\",\\\"etps\\\":\\\"SubmissionId:aaaae50f-bbbb-4760-cccc-dddda276218e\\\",\\\"lon\\\":\\\"UserSubmission\\\"}\",\"Severity\":\"Low\",\"Workload\":\"SecurityComplianceCenter\",\"EntityType\":\"User\",\"AlertId\":\"aaaa01b-bbbb-cccc-dddd-eeeea276218e\",\"UserId\":\"SecurityComplianceAlerts\",\"CreationTime\":\"2025-06-03T08:10:44\",\"Id\":\"aaaabce60-bbbb-cccc-dddd-eeeea27623da\",\"UserType\":4,\"PolicyId\":\"aaaa5770-bbbb-cccc-dddd-eeee2c27bbb3\"}",
"outcome": "success",
"provider": "SecurityComplianceCenter",
"type": [
"info"
]
},
"host": {
"id": "aaaaa14f-bbbb-cccc-dddd-eeee5a778630"
},
"message": "Email reported by user as malware or phish",
"o365": {
"audit": {
"AlertId": "aaaa01b-bbbb-cccc-dddd-eeeea276218e",
"AlertType": "System",
"Comments": "New alert",
"CreationTime": "2025-06-03T08:10:44",
"Data": {
"aii": "aaaa109-bbb-cccc-dddd-eeeea1c1dd41",
"eid": "[email protected](external, opens in a new tab or window)",
"etps": "SubmissionId:aaaae50f-bbbb-4760-cccc-dddda276218e",
"etype": "User",
"flattened": {
"aii": "aaaa109-bbb-cccc-dddd-eeeea1c1dd41",
"eid": "[email protected](external, opens in a new tab or window)",
"etps": "SubmissionId:aaaae50f-bbbb-4760-cccc-dddda276218e",
"etype": "User",
"imsgid": "[email protected](external, opens in a new tab or window)",
"lon": "UserSubmission",
"md": "2025-05-02T10:40:16.9298292Z",
"ms": "Welkom op My company",
"op": "UserSubmission",
"sid": "aaa174f-bbbb-cccc-dddd-eeeea27623b4",
"srt": "1",
"ssic": "0",
"suid": "[email protected](external, opens in a new tab or window)",
"tdc": "1",
"te": "2025-05-02T05:10:44.5371861Z",
"tid": "aaaaa14f-bbbb-cccc-dddd-eeee5a778630",
"trc": "[email protected](external, opens in a new tab or window)",
"ts": "2025-05-02T05:10:44.5371861Z",
"tsd": "[email protected](external, opens in a new tab or window)",
"ut": "Regular"
},
"imsgid": "[email protected](external, opens in a new tab or window)",
"lon": "UserSubmission",
"md": "2025-05-02T10:40:16.929Z",
"ms": "Welkom op My company",
"op": "UserSubmission",
"sid": "aaa174f-bbbb-cccc-dddd-eeeea27623b4",
"srt": "1",
"ssic": "0",
"suid": "[email protected](external, opens in a new tab or window)",
"tdc": "1",
"te": "2025-05-02T05:10:44.537Z",
"tid": "aaaaa14f-bbbb-cccc-dddd-eeee5a778630",
"trc": "[email protected](external, opens in a new tab or window)",
"ts": "2025-05-02T05:10:44.537Z",
"tsd": "[email protected](external, opens in a new tab or window)",
"ut": "Regular"
},
"ObjectId": "[email protected](external, opens in a new tab or window)",
"RecordType": "40",
"ResultStatus": "Succeeded",
"Severity": "Low",
"Source": "Office 365 Security & Compliance",
"Status": "Active",
"UserId": "SecurityComplianceAlerts",
"UserKey": "SecurityComplianceAlerts",
"UserType": "4",
"Version": "1"
}
},
"organization": {
"id": "aaaaa14f-bbbb-cccc-dddd-eeee5a778630"
},
"related": {
"user": [
"[email protected](external, opens in a new tab or window)",
"[email protected](external, opens in a new tab or window)"
]
},
"rule": {
"category": "ThreatManagement",
"description": "[email protected](external, opens in a new tab or window)",
"id": "aaaa5770-bbbb-cccc-dddd-eeee2c27bbb3",
"name": "Email reported by user as malware or phish",
"ruleset": "User"
},
"tags": [
"preserve_original_event"
],
"user": {
"id": "SecurityComplianceAlerts"
}
}
]
}
27 changes: 19 additions & 8 deletions packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1383,15 +1383,18 @@ processors:
lang: painless
tag: script_known_Data
if: 'ctx.o365audit?.Data?.flattened instanceof Map'
source: >
def knownKeys = ['ad', 'af', 'aii', 'ail', 'alk', 'als', 'an', 'at',
params:
knownKeys: [
'ad', 'af', 'aii', 'ail', 'alk', 'als', 'an', 'at',
'cid', 'cpid', 'dm', 'dpn', 'eid', 'etps', 'etype', 'f3u', 'fvs',
'imsgid', 'lon', 'mat', 'md', 'ms', 'od', 'op', 'ot', 'plk', 'pud',
'reid', 'rid', 'sev', 'sict', 'sid', 'sip', 'sitmi', 'srt', 'ssic',
'suid', 'tdc', 'te', 'thn', 'tht', 'tid', 'tpid', 'tpt', 'trc', 'ts',
'tsd', 'ttdt', 'ttr', 'upfc', 'upfv', 'ut', 'von', 'wl', 'zfh', 'zfn',
'zmfh', 'zmfn', 'zu'];
for (def key : knownKeys) {
'zmfh', 'zmfn', 'zu'
]
source: >
for (def key : params.knownKeys) {
if (ctx.o365audit.Data.flattened.containsKey(key)) {
ctx.o365audit.Data[key] = ctx.o365audit.Data.flattened[key];
}
Expand All @@ -1400,6 +1403,11 @@ processors:
field: o365audit.Data.sip
type: ip
ignore_missing: true
if: ctx.o365audit?.Data?.sip != ''
on_failure:
- remove:
field: o365audit.Data.sip
ignore_missing: true
- date:
field: o365audit.Data.at
target_field: o365audit.Data.at
Expand Down Expand Up @@ -1491,15 +1499,18 @@ processors:
lang: painless
tag: script_known_Data.Entities
if: ctx.o365audit?.Data?.flattened?.Entities instanceof List
params:
knownEntityKeys: [
'InternetMessageId', 'NetworkMessageId', 'OriginalDeliveryLocation',
'P1Sender', 'P2Sender', 'PhishConfidenceLevel', 'Recipient', 'SenderIP', 'Subject',
'ThreatDetectionMethods', 'Upn'
]
source: >
ctx._tmp = [:];
ctx._tmp.entities = [:];
def knownEntityKeys = ['InternetMessageId', 'NetworkMessageId', 'OriginalDeliveryLocation',
'P1Sender', 'P2Sender', 'PhishConfidenceLevel', 'Recipient', 'SenderIP', 'Subject',
'ThreatDetectionMethods', 'Upn'];
for (def entity: ctx.o365audit.Data.flattened.Entities) {
if (entity instanceof Map) {
for (def key : knownEntityKeys) {
for (def key : params.knownEntityKeys) {
if (! ctx._tmp.entities.containsKey(key)) {
ctx._tmp.entities[key] = [];
}
Expand Down
2 changes: 1 addition & 1 deletion packages/o365/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: o365
title: Microsoft Office 365
version: "2.18.0"
version: "2.18.1"
description: Collect logs from Microsoft Office 365 with Elastic Agent.
type: integration
format_version: "3.2.3"
Expand Down