Skip to content

[O365] Check contentExpiration timestamp before requesting contents #9477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 3, 2024
Merged

[O365] Check contentExpiration timestamp before requesting contents #9477

merged 3 commits into from
Apr 3, 2024

Conversation

@kcreddy
Copy link
Contributor

@kcreddy kcreddy commented Apr 1, 2024 •
edited
Loading

Proposed commit message

  • Check contentExpiration timestamp before requesting contents.
    • Add system tests to verify this scenario
  • Remove outermost reference to now as the minimum kibana.version is already updated to 8.12
  • Avoid state's timestamp being stuck with older than 7 days by checking it against (now - duration("167h55m"))

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v 

--- Test results for package: o365 - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ o365    │ audit       │ system    │ cel       │ PASS   │ 39.764056792s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: o365 - END   ---
Done

@kcreddy kcreddy added Integration:o365 Microsoft Office 365 bugfix Pull request that fixes a bug issue labels Apr 1, 2024
@kcreddy kcreddy marked this pull request as ready for review April 1, 2024 12:17
@kcreddy kcreddy requested a review from a team as a code owner April 1, 2024 12:17
@kcreddy kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Apr 1, 2024
@elasticmachine
Copy link

elasticmachine commented Apr 1, 2024

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6
efd6 reviewed Apr 1, 2024
View reviewed changes
packages/o365/data_stream/audit/agent/stream/cel.yml.hbs Outdated
(
request("GET", l1.contentUri).do_request().as(content_resp,
(has(content_resp.StatusCode) && content_resp.StatusCode == 200) ?
(
Copy link
Contributor

@efd6 efd6 Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indent this block one level.

Copy link
Contributor Author

@kcreddy kcreddy Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indented the block

packages/o365/data_stream/audit/agent/stream/cel.yml.hbs Outdated
)
)
:
{
Copy link
Contributor

@efd6 efd6 Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indent this block one level.

Copy link
Contributor Author

@kcreddy kcreddy Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indented the block

packages/o365/data_stream/audit/agent/stream/cel.yml.hbs Outdated
Comment on lines 190 to 196
"next_page": (
(has(list_contents_resp.Header) && has(list_contents_resp.Header.NextPageUri) ) ?
list_contents_resp.Header.NextPageUri[0]
: (has(list_contents_resp.Header) && has(list_contents_resp.Header.Nextpageuri)) ?
list_contents_resp.Header.Nextpageuri[0]
:
""
Copy link
Contributor

@efd6 efd6 Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are now on 8.12.

Suggested change
"next_page": (
(has(list_contents_resp.Header) && has(list_contents_resp.Header.NextPageUri) ) ?
list_contents_resp.Header.NextPageUri[0]
: (has(list_contents_resp.Header) && has(list_contents_resp.Header.Nextpageuri)) ?
list_contents_resp.Header.Nextpageuri[0]
:
""
"next_page": list_contents_resp.?Header.NextPageUri.orValue(list_contents_resp.?Header.Nextpageuri).orValue(""),

(we can probably get nice cleanup throughout by using these, but lets do that in another PR, rather than in a bugfix)

Copy link
Contributor Author

@kcreddy kcreddy Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I was thinking of the same in another PR.

packages/o365/data_stream/audit/agent/stream/cel.yml.hbs Outdated
"events_per_content_type": [],
"content_type": content_type,
"content_created_at": {"temp": list_contents_resp_body}.collate("temp.contentCreated").max().as(temp_max, temp_max.parse_time(time_layout.RFC3339) > (now - duration("167h55m")) ? temp_max : (now - duration("167h55m")).format(time_layout.RFC3339)),
"next_page": (
Copy link
Contributor

@efd6 efd6 Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment below for use of optional types.

Copy link
Contributor Author

@kcreddy kcreddy Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do the optional types conversion in another PR 👍🏼

@kcreddy kcreddy requested a review from efd6 April 2, 2024 12:28
@elasticmachine
Copy link

elasticmachine commented Apr 2, 2024

💚 Build Succeeded

History

@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Apr 2, 2024

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@kcreddy kcreddy merged commit da6480c into elastic:main Apr 3, 2024
@elasticmachine
Copy link

elasticmachine commented Apr 3, 2024

Package o365 - 2.3.1 containing this change is available at https://epr.elastic.co/search?package=o365

@kcreddy kcreddy deleted the o365-contentexpired branch February 7, 2025 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 approved these changes

Assignees

No one assigned

Labels

bugfix Pull request that fixes a bug issue Integration:o365 Microsoft Office 365 Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kcreddy @elasticmachine @efd6