mirror of
https://github.com/PR0M3TH3AN/SeedPass.git
synced 2026-11-06 04:42:35 +00:00
Run mutation testing weekly in CI, and make it able to fail
The sweep found real gaps but nothing re-runs it, which puts it in exactly the category this branch already has a bad history with: a check nobody runs is not a check. Two changes make it one. The script now exits non-zero. Exit 1 means a mutant survived — a security check whose removal no test notices. That is not automatically a defect (equivalent mutants exist) but it always needs a human to either fix it or annotate it `mutation-equivalent:` with a reason, which is what a red check should mean. Exit 2 means a target generated no mutants at all: not a pass, and it must not share a signal with one. The workflow is scheduled weekly rather than run per-push, deliberately. A full sweep is ~132 mutants at roughly 25s each, and what it measures does not change commit to commit — it drifts as checks and tests are added. Per-push it would be an hour of CI for a signal that moves monthly. It also verifies the tree is clean at the end. The script restores after every mutant, verifies the restore, and traps SIGINT/SIGTERM/SIGHUP — but the entire technique depends on deliberately broken security code never escaping the run, so that gets checked rather than trusted. Also corrects a stale TODO: the fuzzer and cross_impl_check were wired into ts-parity.yml some commits ago and the list still claimed neither ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01Vtx94nXDmNgXEeThUZP4QN
This commit is contained in:
1 parent
f043c95672
commit
d6ffbac312
3 files changed
+92
-4
No files matched your search
@@ -369,6 +369,17 @@ def main() -> int:
|
||||
print(f" was: {mutant.original.strip()[:110]}")
|
||||
print(f" now: {mutant.mutated.strip()[:110]}")
|
||||
print()
|
||||
|
||||
# Exit non-zero so this can gate CI. A survivor is not automatically a
|
||||
# defect -- it may be an equivalent mutant -- but it is always something a
|
||||
# human has to look at and then either fix or annotate with
|
||||
# `mutation-equivalent:`, which is exactly what a red check should mean.
|
||||
# A target that generated nothing gets its own code, because "not
|
||||
# measured" and "measured and clean" must not be the same signal.
|
||||
if overall_survivors:
|
||||
return 1
|
||||
if unmeasured:
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user