[first_epss] New integration #10758

clement-fouque · 2024-08-11T15:58:17Z

Proposed commit message

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

Dev

Add PR number in changelog

clement-fouque · 2024-08-12T07:41:58Z

@elastic/security-service-integrations should this integration be owned by your team as it’s linked to vulnerabilities/CVE?

kcreddy · 2024-08-20T06:57:32Z

should this integration be owned by your team as it’s linked to vulnerabilities/CVE?

@jamiehynds @narph can you confirm if this new integration should belong to us?

jamiehynds · 2024-08-20T09:56:55Z

should this integration be owned by your team as it’s linked to vulnerabilities/CVE?

@jamiehynds @narph can you confirm if this new integration should belong to us?

Hey @kcreddy - we're fine to review and publish, but should list the owner type as community.

kcreddy · 2024-08-20T10:03:11Z

@clement-fouque the CI is failing on missing codeowners for this integration.
Can you add this integration to .github/CODEOWNERS file and the owner as @elastic/security-service-integrations.

Please also update your manifest.yml to reflect same change and @jamiehynds comment above:

owner:
  github: elastic/security-service-integrations
  type: community

clement-fouque · 2024-08-20T12:11:42Z

Thank you @jamiehynds and @kcreddy. I'll implement those changes.

Dev

clement-fouque · 2024-08-20T21:24:58Z

@kcreddy please let me know if there are any pending tasks from my side. Thanks.

packages/first/manifest.yml

packages/first/_dev/build/build.yml

packages/first/data_stream/epss/agent/stream/input.yml.hbs

packages/first/data_stream/epss/manifest.yml

kcreddy

Is it possible to add pipeline or system tests?

Also, if there are any conflicting review comments between me and Andrew's review, please apply changes suggested by Andrew. There might be few in titles and descriptions.

packages/first/LICENSE.txt

packages/first/manifest.yml

packages/first/data_stream/epss/elasticsearch/ingest_pipeline/default.yml

packages/first/data_stream/epss/fields/base-fields.yml

packages/first/data_stream/epss/fields/fields.yml

packages/first/data_stream/epss/elasticsearch/ingest_pipeline/default.yml

packages/first/manifest.yml

Co-authored-by: Andrew Kroh <[email protected]>

Dev

elasticmachine · 2024-09-03T21:56:47Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Dev

clement-fouque · 2024-09-04T20:26:53Z

I've implemented all suggestions/remarks. Please let me know if there is anything else.

kcreddy

Some suggestions. Overall LGTM 👍🏼

packages/first_epss/manifest.yml

kcreddy · 2024-09-05T11:19:31Z

packages/first_epss/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

+  - date:
+      field: json.date
+      tag: date_date
+      formats: [yyyy-MM-dd]
+      target_field: first_epss.vulnerability.date


After this date processor you can assign the value to @timestamp so that it gets value from the event.

Reference: https://github.com/elastic/integrations/blob/main/packages/prisma_access/data_stream/event/elasticsearch/ingest_pipeline/default.yml#L4506-L4510

@kcreddy I'm not sure I want to do this as this date corresponds to the time when First EPSS generated this data but not when it was ingested. We might have a gap up to 24 hours between the generated data and the ingestion.

As timestamp is the main date to "control" search, I think it's better to keep it as it is.

Change minimum kibana version and add category

Regenerate sample events and readme files

Bumping the version to pass build tests

elasticmachine · 2024-09-05T20:08:32Z

💚 Build Succeeded

Buildkite Build
Commit: 4545de4

History

💔 Build #15536 failed f3fe7a5
💔 Build #15530 failed d2cf7ea
💚 Build #15481 succeeded 8a15d04
💚 Build #15417 succeeded 0cca1af
💔 Build #15416 failed 04e159e

cc @clement-fouque

elastic-sonarqube · 2024-09-05T20:08:36Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

clement-fouque · 2024-09-05T21:02:55Z

Can I squash and merge or should I wait from someone from the Integration team ?

andrewkroh · 2024-09-05T22:59:51Z

@kreddy Can you take one final look and then please handle the merge with a well formed commit message like we aim for 😄 . Thanks.

elasticmachine · 2024-09-06T06:01:10Z

Package first_epss - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=first_epss

Create new integration package First EPSS. Adds `vulnerability` datastream to collect exploit prediction score data from the First EPSS API. - Adds the ingest pipeline for `vulnerability` data stream. - Maps fields according to the ECS schema and added custom fields. - Adds dashboard. - Adds README. - Adds test for pipeline for `vulnerability` data stream. - Adds system test cases for `vulnerability` data stream.

clement-fouque added 9 commits August 11, 2024 00:10

Initial draft

77cc4e8

Add pretty display parameter to the First API url

00ff7df

Add description properties to fields

2a3eb19

Update documentation

cbef493

chore: Update First EPSS integration documentation

75e6509

Rename parent field from first_epss to first.epss

d22d55a

Merge pull request #1 from clement-fouque/dev

785751c

Dev

Add PR number in changelog

25d449f

Merge pull request #2 from clement-fouque/dev

2d28b16

Add PR number in changelog

clement-fouque marked this pull request as ready for review August 12, 2024 07:40

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Aug 20, 2024

kcreddy added the Community label Aug 20, 2024

clement-fouque added 3 commits August 20, 2024 22:54

chore: Add First package to security-service-integrations

4ffbc89

refactor: Change owner field in manifest.yml

b529cff

Merge pull request #3 from clement-fouque/dev

b7cb83a

Dev

andrewkroh reviewed Aug 21, 2024

View reviewed changes

kcreddy reviewed Aug 21, 2024

View reviewed changes

clement-fouque and others added 6 commits August 25, 2024 22:13

Update packages/first/manifest.yml

d7e04d8

Co-authored-by: Andrew Kroh <[email protected]>

Update packages/first/manifest.yml

f6f8f7d

Co-authored-by: Andrew Kroh <[email protected]>

Update packages/first/manifest.yml

72298e7

Co-authored-by: Andrew Kroh <[email protected]>

Rename package from first to first_epss

64b5f62

Merge branch 'main' into dev

2d9fb74

Delete Makefile

6903663

Merge pull request #6 from clement-fouque/dev

04e159e

Dev

clement-fouque requested a review from a team as a code owner September 3, 2024 21:01

clement-fouque changed the title ~~First EPSS integration creation~~ [first_epss] New integration Sep 3, 2024

Delete Makefile

0cca1af

clement-fouque added 5 commits September 4, 2024 20:52

Generate sample event from system test

9954984

Update logo to generic

1a793d7

Update based on sample event

2a26b4e

Update logo size to 32x32

69f725e

Merge pull request #7 from clement-fouque/dev

8a15d04

Dev

kcreddy approved these changes Sep 5, 2024

View reviewed changes

jsoriano approved these changes Sep 5, 2024

View reviewed changes

clement-fouque added 6 commits September 5, 2024 14:20

Change minimum kibana version

5e2e821

Merge pull request #8 from clement-fouque/dev

d2cf7ea

Change minimum kibana version and add category

Regenerate sample events and readme files

10f334c

Merge pull request #9 from clement-fouque/dev

f3fe7a5

Regenerate sample events and readme files

Bumping the version to pass build tests

40d5e28

Merge pull request #10 from clement-fouque/dev

4545de4

Bumping the version to pass build tests

kcreddy merged commit f62ba53 into elastic:main Sep 6, 2024

andrewkroh added the Integration:first_epss First EPSS (Community supported) label Sep 6, 2024

andrewkroh added maintainer:Community Community supported integration and removed Community labels Mar 21, 2025

[first_epss] New integration #10758

[first_epss] New integration #10758

Uh oh!

Conversation

clement-fouque commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

clement-fouque commented Aug 12, 2024

Uh oh!

kcreddy commented Aug 20, 2024

Uh oh!

jamiehynds commented Aug 20, 2024

Uh oh!

kcreddy commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clement-fouque commented Aug 20, 2024

Uh oh!

clement-fouque commented Aug 20, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Sep 3, 2024

🚀 Benchmarks report

Uh oh!

clement-fouque commented Sep 4, 2024

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kcreddy Sep 5, 2024

Choose a reason for hiding this comment

Uh oh!

clement-fouque Sep 5, 2024

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 5, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Sep 5, 2024

Quality Gate passed

Uh oh!

clement-fouque commented Sep 5, 2024

Uh oh!

andrewkroh commented Sep 5, 2024

Uh oh!

elasticmachine commented Sep 6, 2024

Uh oh!

Reviewers

Assignees

Labels

clement-fouque commented Aug 11, 2024 •

edited

Loading

kcreddy commented Aug 20, 2024 •

edited

Loading

kcreddy left a comment •

edited

Loading