From b0c58decb76180d38d75069557cbf83d4116267a Mon Sep 17 00:00:00 2001 From: Oleksandr Rudyy Date: Thu, 3 Jul 2025 17:57:34 +0200 Subject: [PATCH 1/3] add flaky test metric docs --- content/en/tests/flaky_management/_index.md | 47 +++++++++++++-------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/content/en/tests/flaky_management/_index.md b/content/en/tests/flaky_management/_index.md index c73ff19cbccfb..ec91ec15154e9 100644 --- a/content/en/tests/flaky_management/_index.md +++ b/content/en/tests/flaky_management/_index.md @@ -42,16 +42,28 @@ Use the status drop-down to change how a flaky test is handled in your CI pipeli
Note: Status actions have minimum version requirements for each programming language's instrumentation library. See Compatibility for details.
+## Track evolution of flaky tests + +Track the evolution of the number of flaky tests with the `test_optimization.test_management.flaky_tests` out-of-the-box metric. The metric is enriched with the tags below to help you investigate the counts in more detail. + +- `repository_id` +- `branch` +- `flaky_status` +- `flaky_category` +- `test_codeowners` + +The `branch` tag only exists when the test has flaked in the default branch of the repository during the last 30 days. This helps you discard flaky tests that only exhibited flakiness in feature branches and may not be relevant. You can configure the default branch of your repositories under [Repository Settings][2]. + ## Investigate a flaky test For more information about a specific flaky test, use these options in the actions menu at the end of each row: - **View Last Failed Test Run**: Open the side panel with the details of the test's most recent failed run. -- **View related test executions**: Open the [Test Optimization Explorer][2] populated with all of the test's recent runs. +- **View related test executions**: Open the [Test Optimization Explorer][3] populated with all of the test's recent runs. ## Create cases for flaky tests -For any flaky test, you can create a case and use [Case Management][3] to track any work toward remediation. Click the **Create Case** button or use the actions menu at the end of the row. +For any flaky test, you can create a case and use [Case Management][4] to track any work toward remediation. Click the **Create Case** button or use the actions menu at the end of the row. ## Confirm fixes for flaky tests @@ -59,7 +71,7 @@ When you fix a flaky test, Test Optimization's remediation flow can confirm the 1. For the test you are fixing, click **Fix this test** in the Flaky Test Management UI. 1. Copy the unique flaky test key that is displayed (for example, `DD_ABC123`). -1. Include the test key in your Git commit title or message for the fix (for example, `git commit -m "DD_ABC123"`). +1. Include the test key in your Git commit title or message for the fix (for example, `git commit -m "DD_ABC123"`). 1. When Datadog detects the test key in your commit, it automatically triggers the remediation flow for that test: - Retries any tests you're attempting to fix 20 times. - Runs tests even if they are marked as `Disabled`. @@ -72,23 +84,24 @@ To use Flaky Test Management features, you must use Datadog's native instrumenta | Language | Quarantine & Disable | Attempt to fix | | --------------- | -------------------- | ---------------- | -| [.NET][4] | 3.13.0+ | 3.17.0+ | -| [Go][5] | 1.73.0+ | Not available | -| [Java][6] | 1.48.0+ | 1.50.0+ | -| [JavaScript][7] | 5.44.0+ | 5.52.0+ | -| [Python][8] | 3.3.0+ | 3.8.0+ | -| [Ruby][9] | 1.13.0+ | 1.17.0+ | +| [.NET][5] | 3.13.0+ | 3.17.0+ | +| [Go][6] | 1.73.0+ | Not available | +| [Java][7] | 1.48.0+ | 1.50.0+ | +| [JavaScript][8] | 5.44.0+ | 5.52.0+ | +| [Python][9] | 3.3.0+ | 3.8.0+ | +| [Ruby][10] | 1.13.0+ | 1.17.0+ | ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: https://app.datadoghq.com/ci/test/flaky -[2]: /tests/explorer -[3]: /service_management/case_management -[4]: /tests/setup/dotnet/ -[5]: /tests/setup/go/ -[6]: /tests/setup/java/ -[7]: /tests/setup/javascript/ -[8]: /tests/setup/python/ -[9]: /tests/setup/ruby/ +[2]: https://app.datadoghq.com/source-code/repositories +[3]: /tests/explorer +[4]: /service_management/case_management +[5]: /tests/setup/dotnet/ +[6]: /tests/setup/go/ +[7]: /tests/setup/java/ +[8]: /tests/setup/javascript/ +[9]: /tests/setup/python/ +[10]: /tests/setup/ruby/ From 49fec253fb15abdbf883f3e135bd1ba1a1f88a41 Mon Sep 17 00:00:00 2001 From: Oleksandr Rudyy Date: Fri, 4 Jul 2025 09:44:55 +0200 Subject: [PATCH 2/3] pr feedback --- content/en/tests/flaky_management/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/tests/flaky_management/_index.md b/content/en/tests/flaky_management/_index.md index ec91ec15154e9..2204a53bbc618 100644 --- a/content/en/tests/flaky_management/_index.md +++ b/content/en/tests/flaky_management/_index.md @@ -52,7 +52,7 @@ Track the evolution of the number of flaky tests with the `test_optimization.tes - `flaky_category` - `test_codeowners` -The `branch` tag only exists when the test has flaked in the default branch of the repository during the last 30 days. This helps you discard flaky tests that only exhibited flakiness in feature branches and may not be relevant. You can configure the default branch of your repositories under [Repository Settings][2]. +The `branch` tag only exists when the test has flaked in the default branch of the repository during the last 30 days. This helps you discard flaky tests that have only exhibited flakiness in feature branches, as these may not be relevant. You can configure the default branch of your repositories under [Repository Settings][2]. ## Investigate a flaky test From 39c13ec2274a009cd3e7adb2ac619e1f9bdccc00 Mon Sep 17 00:00:00 2001 From: Oleksandr Rudyy Date: Fri, 4 Jul 2025 10:45:00 +0200 Subject: [PATCH 3/3] remove flaky_category from docs --- content/en/tests/flaky_management/_index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/tests/flaky_management/_index.md b/content/en/tests/flaky_management/_index.md index 2204a53bbc618..0d51f3496bcb2 100644 --- a/content/en/tests/flaky_management/_index.md +++ b/content/en/tests/flaky_management/_index.md @@ -49,7 +49,6 @@ Track the evolution of the number of flaky tests with the `test_optimization.tes - `repository_id` - `branch` - `flaky_status` -- `flaky_category` - `test_codeowners` The `branch` tag only exists when the test has flaked in the default branch of the repository during the last 30 days. This helps you discard flaky tests that have only exhibited flakiness in feature branches, as these may not be relevant. You can configure the default branch of your repositories under [Repository Settings][2].