[Prometheus] Add username, password, and SSL related fields for query dataset #13969

mykola-elastic · 2025-05-22T12:27:44Z

Proposed commit message

[Prometheus] Add username, password, and SSL related fields for query dataset

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Closes [Prometheus]: Add username, password and SSL related fields for query dataset #13751

Note

Added fields work well for the case of successful authorization.

When the auth information is wrong, the error messages look the following way:

For the collector dataset:
https://github.com/elastic/beats/blob/main/metricbeat/module/prometheus/collector/collector.go#L154
https://github.com/elastic/beats/blob/main/metricbeat/helper/prometheus/prometheus.go#L107

For the query dataset:
https://github.com/elastic/beats/blob/main/metricbeat/module/prometheus/query/query.go#L86

The error message is misleading and can be fixed by editing the code of Metricbeat

… dataset

ishleenk17 · 2025-05-22T13:15:45Z

@mykola-elastic : Thanks for the PR.
The query dataset is tested with which SSL fields combination ?
Can you please add testing details ?

ishleenk17 · 2025-05-22T13:25:58Z

packages/prometheus/data_stream/query/manifest.yml

+      - name: ssl.verification_mode
+        type: text
+        title: SSL Verification Mode
+        description: SSL verification mode. See [documentation](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-ssl.html#client-verification-mode) for details.


Lets use the corresponding fleet links instead of beats here
https://www.elastic.co/guide/en/fleet/current/elastic-agent-ssl-configuration.html

Thanks, I added the new links

mykola-elastic · 2025-05-22T14:28:24Z

@mykola-elastic : Thanks for the PR. The query dataset is tested with which SSL fields combination ? Can you please add testing details ?

Hello @ishleenk17 . Thanks for the review!

I copied the fields from the collector dataset, where it is already implemented:

https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/collector/agent/stream/stream.yml.hbs#L18-L42
https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/collector/manifest.yml#L62-L105
https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/collector/manifest.yml#L120-L134

I have tested that the username and password work with the query dataset and assumed that all other fields will work too as they are already present in the collector dataset part. For testing I have a locally running Prometheus (in Docker), a locally running Elastic Stack --version 9.0.0-SNAPSHOT, and an elastic-agent. I checked if data is flowing properly when the credentials are right, and if there are errors when the credentials are wrong.

Shall I test the SSL as well? If yes, do we have any easy way to test SSL (a preconfigured Prometheus instance with SSL certs which can be used for testing)?

I think I shall be able to test it using locally generated(self-signed) certificates, I will get back to you when it is done

mykola-elastic · 2025-05-22T15:22:45Z

@ishleenk17 I've tested (successfully) using self-signed certificates. Specifically the fields ssl.certificate_authorities, ssl.certificate, and ssl.key. Do I need to test other scenarios?

mykola-elastic · 2025-05-23T09:44:05Z

I've added an issue for the Metricbeat's error handling elastic/beats#44446 which is related to this PR

ishleenk17 · 2025-05-23T11:30:22Z

@ishleenk17 I've tested (successfully) using self-signed certificates. Specifically the fields ssl.certificate_authorities, ssl.certificate, and ssl.key. Do I need to test other scenarios?

This testing shall suffice @mykola-elastic.
Thanks

ishleenk17

Looks good!
Thanks

elasticmachine · 2025-05-23T12:59:17Z

💚 Build Succeeded

Buildkite Build
Commit: 93bb4f7

History

💚 Build #26267 succeeded f1e06b0
💚 Build #26247 succeeded 90d90fd

elastic-sonarqube · 2025-05-23T12:59:23Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-05-23T13:49:25Z

Package prometheus - 1.24.0 containing this change is available at https://epr.elastic.co/package/prometheus/1.24.0/

* main: (42 commits) [jamf_pro] Fix `flattened` field types for non-object values (elastic#13985) [Netskope Alerts] Add text multi-field to netskope.alerts.breach.description field (elastic#13977) zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams (elastic#13904) apm: Add config for tail-based sampling discard on write (elastic#13950) [CI] Add dev/coverage into backport script (elastic#13987) Update configuration updatecli for 8.x snapshot (elastic#13981) [Prometheus] Add username, password, and SSL related fields for query dataset (elastic#13969) o365: Ignore failures in rename processors for organization fields (elastic#13983) aws.firewall: Document ingested log types of AWS Network Firewall (elastic#13978) mimecast: resolve field data type conflicts between data streams (elastic#13825) [Infoblox NIOS] Handle the parsing of IPv6 address (elastic#13947) [Cribl] Fix handling of metric event type (elastic#13930) zscaler_zpa: fix handling of multiple remote IPs, and event categorisation (elastic#13755) Adding agentless deployment to the sublime security integration (elastic#13963) [integration/system] add use_performance_counters in system integration (elastic#13150) crowdstrike,m365_defender,microsoft_defender_{cloud,endpoint},sentinel_one: normalise severity handling (elastic#13955) [forgerock] Map `forgerock.response.elapsedTime` as a long not a date (elastic#13959) github: squelch errors from pagination ends (elastic#13965) cisco_secure_endpoint: squelch errors from pagination ends (elastic#13964) [Cloud Security] Cloud Asset Inventory: fixed cloud formation URL (elastic#13971) ...

* feature/use-google-secrets: (43 commits) use -ci account [jamf_pro] Fix `flattened` field types for non-object values (#13985) [Netskope Alerts] Add text multi-field to netskope.alerts.breach.description field (#13977) zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams (#13904) apm: Add config for tail-based sampling discard on write (#13950) [CI] Add dev/coverage into backport script (#13987) Update configuration updatecli for 8.x snapshot (#13981) [Prometheus] Add username, password, and SSL related fields for query dataset (#13969) o365: Ignore failures in rename processors for organization fields (#13983) aws.firewall: Document ingested log types of AWS Network Firewall (#13978) mimecast: resolve field data type conflicts between data streams (#13825) [Infoblox NIOS] Handle the parsing of IPv6 address (#13947) [Cribl] Fix handling of metric event type (#13930) zscaler_zpa: fix handling of multiple remote IPs, and event categorisation (#13755) Adding agentless deployment to the sublime security integration (#13963) [integration/system] add use_performance_counters in system integration (#13150) crowdstrike,m365_defender,microsoft_defender_{cloud,endpoint},sentinel_one: normalise severity handling (#13955) [forgerock] Map `forgerock.response.elapsedTime` as a long not a date (#13959) github: squelch errors from pagination ends (#13965) cisco_secure_endpoint: squelch errors from pagination ends (#13964) ...

… dataset (elastic#13969) * [Prometheus] Add username, password, and SSL related fields for query dataset * use corresponding fleet links instead of beats

mykola-elastic requested a review from a team as a code owner May 22, 2025 12:27

[Prometheus] Add username, password, and SSL related fields for query…

90d90fd

… dataset

mykola-elastic force-pushed the prom-query-auth branch from 979d819 to 90d90fd Compare May 22, 2025 12:28

stefans-elastic added the enhancement New feature or request label May 22, 2025

ishleenk17 reviewed May 22, 2025

View reviewed changes

use corresponding fleet links instead of beats

f1e06b0

andrewkroh added Integration:prometheus Prometheus Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels May 22, 2025

mykola-elastic mentioned this pull request May 23, 2025

[Metricbeat] [Prometheus] Add proper error handling for query dataset when the auth/ssl is misconfigured elastic/beats#44446

Closed

mykola-elastic requested a review from ishleenk17 May 23, 2025 09:50

ishleenk17 approved these changes May 23, 2025

View reviewed changes

Merge branch 'main' into prom-query-auth

93bb4f7

ishleenk17 merged commit 47e5a85 into elastic:main May 23, 2025
8 checks passed

mykola-elastic mentioned this pull request May 26, 2025

[Metricbeat] [Prometheus] Add proper error handling for query dataset when the auth/ssl is misconfigured elastic/beats#44493

Merged

6 tasks

mykola-elastic self-assigned this Jun 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Prometheus] Add username, password, and SSL related fields for query dataset #13969

[Prometheus] Add username, password, and SSL related fields for query dataset #13969

Uh oh!

mykola-elastic commented May 22, 2025 •

edited

Loading

Uh oh!

ishleenk17 commented May 22, 2025

Uh oh!

ishleenk17 May 22, 2025

Uh oh!

mykola-elastic May 22, 2025

Uh oh!

mykola-elastic commented May 22, 2025 •

edited

Loading

Uh oh!

mykola-elastic commented May 22, 2025 •

edited

Loading

Uh oh!

mykola-elastic commented May 23, 2025

Uh oh!

ishleenk17 commented May 23, 2025

Uh oh!

ishleenk17 left a comment

Uh oh!

elasticmachine commented May 23, 2025

Uh oh!

elastic-sonarqube bot commented May 23, 2025

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Prometheus] Add username, password, and SSL related fields for query dataset #13969

[Prometheus] Add username, password, and SSL related fields for query dataset #13969

Uh oh!

Conversation

mykola-elastic commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Related issues

Note

Uh oh!

ishleenk17 commented May 22, 2025

Uh oh!

ishleenk17 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

mykola-elastic May 22, 2025

Choose a reason for hiding this comment

Uh oh!

mykola-elastic commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mykola-elastic commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mykola-elastic commented May 23, 2025

Uh oh!

ishleenk17 commented May 23, 2025

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 23, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented May 23, 2025

Quality Gate passed

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mykola-elastic commented May 22, 2025 •

edited

Loading

mykola-elastic commented May 22, 2025 •

edited

Loading

mykola-elastic commented May 22, 2025 •

edited

Loading