Outcome analysis: detect obsolete exceptions #111

New issue

Open

opened 2025-01-02 04:55:54 -06:00 by gilles-peskine-arm · 0 comments

gilles-peskine-arm commented

2025-01-02 04:55:54 -06:00

(Migrated from github.com)

In outcome_analysis.py (possibly with an impact on analyze_outcomes.py in consuming repositories), add a way to detect obsolete exceptions to coverage checks.

Note that this is different from the existing mechanism that causes a failure if a test case is expected to be skipped, but actually passes. The goal here is to detect exceptions that do not match any test case, presumably because the corresponding test case has disappeared.

This may be costly, and is of low importance, so it can be a special command line option that we run manually, rather than something done at every invocation.

Example: https://github.com/Mbed-TLS/mbedtls/pull/9796/commits/b62279d9608944a15f750581d2c1974e60ca7b7b removed some exceptions that were no longer necessary because we removed (stopped generating) some test cases that were never executed. The goal of this issue is to have a way to detect those exceptions.

In `outcome_analysis.py` (possibly with an impact on `analyze_outcomes.py` in consuming repositories), add a way to detect obsolete exceptions to coverage checks. Note that this is different from the existing mechanism that causes a failure if a test case is expected to be skipped, but actually passes. The goal here is to detect exceptions that do not match any test case, presumably because the corresponding test case has disappeared. This may be costly, and is of low importance, so it can be a special command line option that we run manually, rather than something done at every invocation. Example: https://github.com/Mbed-TLS/mbedtls/pull/9796/commits/b62279d9608944a15f750581d2c1974e60ca7b7b removed some exceptions that were no longer necessary because we removed (stopped generating) some test cases that were never executed. The goal of this issue is to have a way to detect those exceptions.

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

gh-Mbed-TLS/mbedtls-framework#111

No description provided.

Rows
Columns