apm: Add config for tail-based sampling discard on write

[isaacaflores2]

Proposed commit message

apm: Add policy variable tail_discard_on_write_failure to configure apm-server.sampling.tail.discard_on_write_failure.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
  • Version constraints will be updated to much the below PR for the apm til config
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Merge changes from apm ttl PR apm: Add integration policy variable tail_sampling_ttl to configure apm-server.sampling.tail.ttl #13348 and match version and constraints

How to test this PR locally

1. Validated the config using elastic-packge to install the packaged. Used the Kibana UI to add the APM Integration. Verified the agent policy contained the correct config
2. Installed elastic agent locally. Validated the computed-config.yaml and validated the apm-server behavior when discard_on_write_failure was enabled and disabled

Related issues

• Part of TBS: Document discard_on_write_failure + expose it to the APM Integration apm-server#15330
• Related to TBS: Expose TTL config via integration policy apm-server#13525

Screenshots

Agent Policy

Elastic Agent Logs

discard on write enabled

• computed-config.yaml:

 sampling:
            tail:
                discard_on_write_failure: true
                enabled: true
                interval: 1m
                policies:
                    - sample_rate: 0.1
                storage_limit: 1B
                ttl: 30m

• Traces should be discarded. Verified log shows discarding by default:

{
  "log.level": "info",
  "@timestamp": "2025-05-19T19:29:34.603Z",
  "message": "processing trace failed, discarding by default",
  "component": {
    "binary": "apm-server",
    "dataset": "elastic_agent.apm_server",
    "id": "apm-default",
    "type": "apm"
  },
  "log": {
    "source": "apm-default"
  },
  "log.logger": "sampling",
  "log.origin": {
    "file.line": 151,
    "file.name": "sampling/processor.go",
    "function": "github.com/elastic/apm-server/x-pack/apm-server/sampling.(*Processor).ProcessBatch"
  },
  "service.name": "apm-server",
  "ecs.version": "1.6.0"
}

discard on write disabled

• computed-config.yaml:

 sampling:
            tail:
                discard_on_write_failure: false
                enabled: true
                interval: 1m
                policies:
                    - sample_rate: 0.1
                storage_limit: 1B
                ttl: 30m

• Traces should be sampled. Verified log shows indexing by default

{
  "log.level": "info",
  "@timestamp": "2025-05-19T19:54:16.723Z",
  "message": "processing trace failed, indexing by default",
  "component": {
    "binary": "apm-server",
    "dataset": "elastic_agent.apm_server",
    "id": "apm-default",
    "type": "apm"
  },
  "log": {
    "source": "apm-default"
  },
  "log.logger": "sampling",
  "log.origin": {
    "file.line": 154,
    "file.name": "sampling/processor.go",
    "function": "github.com/elastic/apm-server/x-pack/apm-server/sampling.(*Processor).ProcessBatch"
  },
  "service.name": "apm-server",
  "ecs.version": "1.6.0"
}

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0d19d3e

History

• 💚 Build #26149 succeeded c728162

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[carsonip]

lgtm, thanks

[rubvs]

Tested manually:

• Spin up a deployment in ECH with 9.1.0-SNAPSHOT

• Install a local EA in docker as outlined in Docs

• Send data to EA endpoint: https://localhost:8200/intake/v2/events

• Observe output in Discovery with discard_on_write_failure: false

• Update policy manually via Dev Tools to set discard_on_write_failure: true

PUT kbn:/api/fleet/package_policies/226a181e-c1e9-43ef-a9f1-ba8cfd2a191c

{
  "inputs": [
    {
      "type": "apm",
      "policy_template": "apmserver",
      "enabled": true,
      "config": {
        "apm-server": {
          "value": {
            "rum": {
              "source_mapping": {
                "metadata": [],
                "elasticsearch": {
                  "api_key": "7S8SjpcBBuYedCQP-s1L:8LkMGRCOGI65djTUV7hDOg"
                }
              }
            },
            "agent_config": [],
            "agent": {
              "config": {
                "elasticsearch": {
                  "api_key": "EaISjpcBF4fUUaI19v1i:CQlR906yj6YUccm4gddYpA"
                }
              }
            }
          }
        }
      },
      "streams": [],
      "vars": {
        "host": {
          "value": "localhost:8200",
          "type": "text"
        },
        "url": {
          "value": "http://localhost:8200",
          "type": "text"
        },
        "secret_token": {
          "type": "text"
        },
        "api_key_enabled": {
          "value": false,
          "type": "bool"
        },
        "enable_rum": {
          "value": true,
          "type": "bool"
        },
        "anonymous_enabled": {
          "value": true,
          "type": "bool"
        },
        "anonymous_allow_agent": {
          "value": [
            "rum-js",
            "js-base",
            "iOS/swift"
          ],
          "type": "text"
        },
        "anonymous_allow_service": {
          "value": [],
          "type": "text"
        },
        "anonymous_rate_limit_event_limit": {
          "value": 300,
          "type": "integer"
        },
        "anonymous_rate_limit_ip_limit": {
          "value": 1000,
          "type": "integer"
        },
        "default_service_environment": {
          "type": "text"
        },
        "rum_allow_origins": {
          "value": [
            "\"*\""
          ],
          "type": "text"
        },
        "rum_allow_headers": {
          "value": [],
          "type": "text"
        },
        "rum_response_headers": {
          "type": "yaml"
        },
        "rum_library_pattern": {
          "value": "\"node_modules|bower_components|~\"",
          "type": "text"
        },
        "rum_exclude_from_grouping": {
          "value": "\"^/webpack\"",
          "type": "text"
        },
        "api_key_limit": {
          "value": 100,
          "type": "integer"
        },
        "max_event_bytes": {
          "value": 307200,
          "type": "integer"
        },
        "capture_personal_data": {
          "value": true,
          "type": "bool"
        },
        "max_header_bytes": {
          "value": 1048576,
          "type": "integer"
        },
        "idle_timeout": {
          "value": "45s",
          "type": "text"
        },
        "read_timeout": {
          "value": "3600s",
          "type": "text"
        },
        "shutdown_timeout": {
          "value": "30s",
          "type": "text"
        },
        "write_timeout": {
          "value": "30s",
          "type": "text"
        },
        "max_connections": {
          "value": 0,
          "type": "integer"
        },
        "response_headers": {
          "type": "yaml"
        },
        "expvar_enabled": {
          "value": false,
          "type": "bool"
        },
        "pprof_enabled": {
          "value": false,
          "type": "bool"
        },
        "java_attacher_discovery_rules": {
          "type": "yaml"
        },
        "java_attacher_agent_version": {
          "type": "text"
        },
        "java_attacher_enabled": {
          "value": false,
          "type": "bool"
        },
        "tls_enabled": {
          "value": false,
          "type": "bool"
        },
        "tls_certificate": {
          "type": "text"
        },
        "tls_key": {
          "type": "text"
        },
        "tls_supported_protocols": {
          "value": [
            "TLSv1.2",
            "TLSv1.3"
          ],
          "type": "text"
        },
        "tls_cipher_suites": {
          "value": [],
          "type": "text"
        },
        "tls_curve_types": {
          "value": [],
          "type": "text"
        },
        "tail_sampling_policies": {
          "value": "- sample_rate: 0.1\n",
          "type": "yaml"
        },
        "tail_sampling_interval": {
          "value": "1m",
          "type": "text"
        },
        "tail_sampling_ttl": {
          "value": "30m",
          "type": "text"
        },
        "tail_sampling_enabled": {
          "value": true,
          "type": "bool"
        },
        "tail_sampling_storage_limit": {
          "value": "2B",
          "type": "text"
        },
        "tail_sampling_discard_on_write_failure": {
          "value": true,
          "type": "bool"
        }
      }
    }
  ]
}

• Observe output in Discovery

[rubvs]

UI changed just propagated so tested via UI also in elastic/kibana#224479 (comment)