You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Standard branch coverage often marks the entire line if a or b or c or d: as "covered" if the condition evaluates to True, regardless of which specific sub-expression (a, b, c, or d) caused it to be true due to short-circuiting. This means we might execute code block A because b was True, but never actually test the cases where a, c, or d would have individually caused the condition to be True.
This makes it difficult to ensure that all logical paths within a complex boolean expression have been exercised by the test suite, particularly in large or legacy codebases.
Describe the solution you'd like
I would like the coverage tool to provide more detailed insight into the evaluation of complex boolean expressions within conditionals. Specifically, it would be beneficial to know which individual sub-expressions within a compound condition were evaluated and which ones were decisive in determining the outcome (True or False) of the condition.
Ideally, the coverage report could indicate if a boolean expression was only "partially" covered, meaning not all potential evaluation paths or short-circuiting scenarios within that specific condition were tested.
Possible ways to report this could include:
Annotated Source View: Highlighting or annotating parts of the boolean expression that were evaluated or determined the outcome during a test run.
# Example annotation showing 'b' was the decisive factor:ifa# F, evaluated # or b # T, decisive # or c # Not evaluated # or d # Not evaluated #:
...
Or perhaps marking the line as "partial" if not all sub-expressions were evaluated across all test runs hitting that line.
Detailed Report/Table: A separate report or section detailing complex conditions and which specific components within them were triggered by which test cases.
Let's consider the expression if a or (b and (c or d)):.
Test Case
Section 1 (a) Evaluation
Section 2 (c or d) Evaluation (if reached)
Section 3 (b and (c or d)) Evaluation
Overall Condition Outcome
test_a_is_true
True
Short-Circuited
Short-Circuited
True
test_a_false_b_true_c_true
False
True (because c is True)
True (because b & Sec 2 are True)
True
test_a_false_b_true_d_true
False
True (because d is True)
True (because b & Sec 2 are True)
True
test_a_false_b_true_c_d_false
False
False
False (because Sec 2 is False)
False
test_a_false_b_false
False
Short-Circuited
False (because b is False)
False
Please note that I'm only advocating for this to be shown on or since the current solution works amazing for and.
Describe alternatives you've considered
The primary alternative is to refactor the complex boolean expressions into simpler conditions, potentially breaking them down into multiple lines, intermediate variables, or helper functions.
However, this presents a catch-22 situation: refactoring risky legacy code requires high confidence in the existing test suite to ensure no behavior is changed unintentionally. Without detailed condition coverage information, it is difficult to know if the current tests are sufficient to cover all the nuances of the existing complex logic before attempting to refactor it. This feature would provide the necessary insight to confidently identify testing gaps and improve coverage prior to or alongside refactoring efforts.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently, when using code coverage with complex boolean expressions in conditional statements (
if
,elif
), the reporting lacks sufficient granularity.For example, consider a condition like:
Standard branch coverage often marks the entire line
if a or b or c or d:
as "covered" if the condition evaluates toTrue
, regardless of which specific sub-expression (a
,b
,c
, ord
) caused it to be true due to short-circuiting. This means we might executecode block A
becauseb
wasTrue
, but never actually test the cases wherea
,c
, ord
would have individually caused the condition to beTrue
.This makes it difficult to ensure that all logical paths within a complex boolean expression have been exercised by the test suite, particularly in large or legacy codebases.
Describe the solution you'd like
I would like the coverage tool to provide more detailed insight into the evaluation of complex boolean expressions within conditionals. Specifically, it would be beneficial to know which individual sub-expressions within a compound condition were evaluated and which ones were decisive in determining the outcome (True or False) of the condition.
Ideally, the coverage report could indicate if a boolean expression was only "partially" covered, meaning not all potential evaluation paths or short-circuiting scenarios within that specific condition were tested.
Possible ways to report this could include:
Annotated Source View: Highlighting or annotating parts of the boolean expression that were evaluated or determined the outcome during a test run.
Or perhaps marking the line as "partial" if not all sub-expressions were evaluated across all test runs hitting that line.
Detailed Report/Table: A separate report or section detailing complex conditions and which specific components within them were triggered by which test cases.
Let's consider the expression
if a or (b and (c or d)):
.a
) Evaluationc or d
) Evaluation (if reached)b and (c or d)
) Evaluationtest_a_is_true
test_a_false_b_true_c_true
c
is True)b
& Sec 2 are True)test_a_false_b_true_d_true
d
is True)b
& Sec 2 are True)test_a_false_b_true_c_d_false
test_a_false_b_false
b
is False)Please note that I'm only advocating for this to be shown on
or
since the current solution works amazing forand
.Describe alternatives you've considered
The primary alternative is to refactor the complex boolean expressions into simpler conditions, potentially breaking them down into multiple lines, intermediate variables, or helper functions.
However, this presents a catch-22 situation: refactoring risky legacy code requires high confidence in the existing test suite to ensure no behavior is changed unintentionally. Without detailed condition coverage information, it is difficult to know if the current tests are sufficient to cover all the nuances of the existing complex logic before attempting to refactor it. This feature would provide the necessary insight to confidently identify testing gaps and improve coverage prior to or alongside refactoring efforts.
The text was updated successfully, but these errors were encountered: