[o365] Surface credential or subscription errors in Fleet status #14771

chrisberkhout · 2025-08-01T14:48:18Z

Proposed commit message

[o365] Surface credential or subscription errors in Fleet status

Instead of ignoring errors and returning successes, if there are any
errors while creating/checking subscriptions, return an error showing
the problem, wait the interval time, and repeat everything next time.

This is intended to avoid silencing errors such as AF10001 "The
permission set () sent in the request does not include the expected
permission.".

Notes

The changelog.yml entry assumes #14730 will be merged first.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

In the system test data_stream/audit/_dev/test/system/test-cel-bad-creds-config.yml, comment out the skip, run it with --defer-cleanup 10m and inspect the error document produced.

Related issues

Related Allow negative testing in system tests elastic-package#2800

elasticmachine · 2025-08-01T14:48:22Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

andrewkroh

Assuming you have a way to locally test the error case, perhaps we should commit the supporting test files (mocked API setup for elastic/stream and the _dev/test/system/test-bad-creds-config.yml file). However, we should skip the test using skip.link until there is a way to allow the test to run and assert that the Agent goes into a degraded state and produces an error document.

There's a risk that these files could become outdated since they aren't exercised automatically, but at least we capture how it was tested and have a way to replicate the service credential errors locally.

elastic-vault-github-plugin-prod · 2025-08-01T15:17:55Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

chrisberkhout · 2025-08-01T15:57:47Z

Assuming you have a way to locally test the error case, perhaps we should commit the supporting test files (mocked API setup for elastic/stream and the _dev/test/system/test-bad-creds-config.yml file). However, we should skip the test using skip.link until there is a way to allow the test to run and assert that the Agent goes into a degraded state and produces an error document.

There's a risk that these files could become outdated since they aren't exercised automatically, but at least we capture how it was tested and have a way to replicate the service credential errors locally.

Good idea. I added it.

efd6 · 2025-08-04T01:52:38Z

packages/o365/data_stream/audit/agent/stream/cel.yml.hbs

-  		},
-  	}
+  	events_list.collate("events_per_content_type").as(events,
+  	  events.filter(e, e.?error.message.hasValue()).as(error_events,


Suggested change

events.filter(e, e.?error.message.hasValue()).as(error_events,

events.filter(e, has(e.?error.message)).as(error_events,

What's the advantage of the suggested change?

I went with the original because then its just using the optional functionality, not mixing with the different field presence macro.

The .? allocates the optional value, the has macro does not need to do that.

Okay. To avoid the optional value, it'd be has(e.error) && has(e.error.message), right?

I'm not sure this kind of optimization is worth it, but I put it in an additional commit.

No, it can be has(e.?error.message).

andrewkroh

LGTM. I ran the failing test case. It surfaces the error.

Event

{
  "@timestamp": "2025-08-04T10:36:59.276Z",
  "agent": {
    "ephemeral_id": "74451a3d-aa31-408d-b3e6-7213488a53d3",
    "id": "2ae3f72c-87b7-4dbd-82ef-1a67b3fd161d",
    "name": "elastic-agent-22243",
    "type": "filebeat",
    "version": "9.2.0"
  },
  "data_stream": {
    "dataset": "o365.audit",
    "namespace": "65280",
    "type": "logs"
  },
  "ecs": {
    "version": "8.11.0"
  },
  "elastic_agent": {
    "id": "2ae3f72c-87b7-4dbd-82ef-1a67b3fd161d",
    "snapshot": true,
    "version": "9.2.0"
  },
  "error": {
    "message": "Audit.TypeRequiringAdditionalPermissions: AF10001 The permission set (...) sent in the request does not include the expected permission."
  },
  "event": {
    "agent_id_status": "verified",
    "category": [
      "web"
    ],
    "dataset": "o365.audit",
    "ingested": "2025-08-04T10:37:02Z",
    "kind": "pipeline_error",
    "original": "null",
    "outcome": "success",
    "type": [
      "info"
    ]
  },
  "input": {
    "type": "cel"
  },
  "tags": [
    "preserve_original_event",
    "forwarded",
    "o365-cel"
  ]
}

Log Message

{
  "log.level": "error",
  "@timestamp": "2025-08-04T10:36:59.276Z",
  "message": "single event object returned by evaluation",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "cel-default",
    "type": "cel"
  },
  "log": {
    "source": "cel-default"
  },
  "log.origin": {
    "file.line": 433,
    "file.name": "cel/input.go",
    "function": "github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1"
  },
  "id": "cel-o365.audit-f8938c7f-bf6b-42a1-a582-7ad971dc0f98",
  "log.logger": "input.cel",
  "service.name": "filebeat",
  "input_source": "http://svc-o365-cel:8080",
  "input_url": "http://svc-o365-cel:8080",
  "error": {
    "message": "Audit.TypeRequiringAdditionalPermissions: AF10001 The permission set (...) sent in the request does not include the expected permission."
  },
  "ecs.version": "1.6.0"
}

andrewkroh · 2025-08-04T10:47:17Z

packages/o365/data_stream/audit/agent/stream/cel.yml.hbs

  					{
-  						"events_per_content_type": [],
+  						"events_per_content_type": [{
+  						  "error": {


Can you add the HTTP status code and status to the error.code and error.idfields, respectively. I've been finding myself using those fields lot when they are available to filter or group the single event object returned by evaluation log messages.

Okay. Done in a new commit.

If there's a permission problem with just the DLP.All content type (which requires an extra permission) then there will be one response involved, but otherwise it'll likely be one per content type. The messages were already combined. The new code de-duplicates the error.code and error.id values. If there's one value it'll be a scalar (likely), otherwise an array.

The output looks like this for the test:

{ ... "error": { "code": "401", "id": "401 Unauthorized", "message": "Audit.TypeRequiringAdditionalPermissions: AF10001 The permission set (...) sent in the request does not include the expected permission." }, ... }

Skipped until negative system tests are possible.

elasticmachine · 2025-08-06T10:05:07Z

💚 Build Succeeded

Buildkite Build
Commit: 24df591

History

💚 Build #29532 succeeded abc4f19
💚 Build #29497 succeeded 3a33fd4
💚 Build #29436 succeeded 5f7630d
💚 Build #29430 succeeded 0cbd975

cc @chrisberkhout

elastic-sonarqube · 2025-08-06T10:05:10Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-08-06T10:23:58Z

Package o365 - 2.21.0 containing this change is available at https://epr.elastic.co/package/o365/2.21.0/

…stic#14771) Instead of ignoring errors and returning successes, if there are any errors while creating/checking subscriptions, return an error showing the problem, wait the interval time, and repeat everything next time. This is intended to avoid silencing errors such as AF10001 "The permission set () sent in the request does not include the expected permission.".

chrisberkhout self-assigned this Aug 1, 2025

chrisberkhout requested a review from a team as a code owner August 1, 2025 14:48

chrisberkhout added enhancement New feature or request Integration:o365 Microsoft Office 365 Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 1, 2025

andrewkroh reviewed Aug 1, 2025

View reviewed changes

andrewkroh requested a review from a team August 1, 2025 15:10

chrisberkhout mentioned this pull request Aug 1, 2025

Allow negative testing in system tests elastic/elastic-package#2800

Open

efd6 reviewed Aug 4, 2025

View reviewed changes

chrisberkhout requested review from andrewkroh and efd6 August 4, 2025 06:51

chrisberkhout force-pushed the o365-error-fleet-status branch from 5f7630d to 3a33fd4 Compare August 4, 2025 07:05

andrewkroh approved these changes Aug 4, 2025

View reviewed changes

chrisberkhout force-pushed the o365-error-fleet-status branch from 3a33fd4 to 811a328 Compare August 4, 2025 14:29

andrewkroh approved these changes Aug 4, 2025

View reviewed changes

chrisberkhout force-pushed the o365-error-fleet-status branch from 7ea774f to 471b02e Compare August 6, 2025 09:16

chrisberkhout enabled auto-merge (squash) August 6, 2025 09:16

efd6 approved these changes Aug 6, 2025

View reviewed changes

chrisberkhout added 7 commits August 6, 2025 11:42

Surface errors in fleet status.

f02f1dd

Include content type in the error message.

01f5a47

Version bump, changelog entry.

c436425

Add a skipped system test to exercise the new logic.

196f075

Skipped until negative system tests are possible.

Set error.code and error.id.

ed81b93

Avoid allocation of optional value.

e07cea1

Use originally suggested field checking syntax.

24df591

chrisberkhout force-pushed the o365-error-fleet-status branch from 471b02e to 24df591 Compare August 6, 2025 09:42

chrisberkhout merged commit 50b6b4f into elastic:main Aug 6, 2025
7 checks passed

	events.filter(e, e.?error.message.hasValue()).as(error_events,
	events.filter(e, has(e.?error.message)).as(error_events,

[o365] Surface credential or subscription errors in Fleet status #14771

[o365] Surface credential or subscription errors in Fleet status #14771

Uh oh!

Conversation

chrisberkhout commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Notes

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Aug 1, 2025

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 1, 2025

🚀 Benchmarks report

Uh oh!

chrisberkhout commented Aug 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Aug 6, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Aug 6, 2025

Quality Gate passed

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chrisberkhout commented Aug 1, 2025 •

edited

Loading