Freeze observables
Define the exact return, revert, state, and event-log behavior that counts as equivalent before generating cases.
Oracle and comparison schema reviewed
Independent portfolio exerciseNot client work · not a new vulnerability claim
QA evidence,not adjectives.
A public test plan, a reproducible historical defect, and an execution matrix that shows how I define, calibrate, test, and report software behavior.
Sample test plan
Make the finish line observable.
The plan starts with behavior and calibration, not tool names. Every phase has a concrete exit condition and an explicit limit.
PhaseControlMethodExit condition
Calibrate the detector
Run a public, historically fixed Vyper defect as a positive control before treating clean campaign results as meaningful.
v0.4.1 fails; v0.4.3 passes
Cross the matrix
Execute each eligible case on REVM and Py-EVM under none, gas, and codesize optimizer modes.
Six observations captured per case
Triage mismatches
Separate harness defects, backend differences, known issues, and compiler candidates before making any claim.
Reproduction survives duplicate review
Report the boundary
Publish exact counts, exclusions, commands, and negative evidence. A clean result is never described as proof of absence.
Result is rerunnable and scoped
Historical positive controlVF-PC-001
A zero-length argument must not erase a side effect.
This publicly fixed Vyper defect is used only to prove the lab can detect a real semantic mismatch. It is not presented as an original finding or bounty submission.
- Expected
- Calling
test()evaluates the side-effectful expression and incrementscounterfrom 0 to 1. - Actual in v0.4.1
- The return value is correct, but
counterremains 0 because the side effect is elided. - Corrected control
- The same check on v0.4.3 returns the value and increments the counter to 1.
- Disposition
- Detector calibrated; campaign results may now be interpreted within the stated scope.
REVMnone
MATCHED
REVMgas
MATCHED
REVMcodesize
MATCHED
Py-EVMnone
MATCHED
Py-EVMgas
MATCHED
Py-EVMcodesize
MATCHED
Scope covered three optimizer modes on two local EVM backends. Experimental code generation and public networks were excluded.
When the spec is thin
Test the decision, not just the screen.
This is the same structure I use for manual product QA: clarify invariants, exercise state transitions, preserve evidence, and make a release recommendation.
Infer intent from invariants
Start with what must remain true for users, permissions, data, and failure recovery—not with the current happy path.
Map state transitions
Test interruption, retry, refresh, empty, boundary, and unusual-order sequences around each important state change.
Make defects reproducible
Record environment, prerequisites, exact steps, expected versus actual behavior, severity, and evidence.
Give a go / no-go
Separate release blockers from improvements and state what remains untested so the decision is honest.
Want a product-facing sample too?
The live CSV cleaner demonstrates conservative transforms, protected identifiers, ambiguous-data review, and an auditable export path—all without uploading data to a server.