A green UI test proves only what its assertions observe. If a purchase test clicks through checkout but never verifies the persisted order, it may keep passing when an important control stops working. I built Vandal to make that blind spot visible through deliberate, bounded mutations.
Start with a passing control run
Run the unchanged test against an authorized staging application and retain its result. Apply one mutation, rerun the same scenario, and compare the observed failure with the injected fault. A browser crash or unavailable environment is an invalid experiment, not evidence that the assertion caught a defect.
These are the strategies described by the current repository. Verify supported selectors and actions against your checkout before integrating them. Start with a disposable account and an action whose effects can be cleaned up.
Classify the experiment
- Killed: the mutation was applied and the test failed for the expected reason.
- Survived: the mutation was applied and the test still passed.
- Invalid: the baseline failed, mutation could not be applied, or infrastructure failed. Keep these out of the detection-rate denominator.
Applied valid mutants: 12
Detected by the tests: 9
Survived: 3
Detection rate for this sample: 9 / 12 = 75%
This is an illustrative experiment, not a Vandal benchmark.What to change when a mutant survives
Inspect the missing assertion before adding more waits. A checkout test may need an API read of the saved order; a form test may need the validation message and the absence of an unwanted write. Rerun the original control and the mutant after changing the test.
