Spend the budget

Same 12 traces, two ways to spend an annotation budget

Drag the budget and switch strategy. Hard-case picks the traces ranked highest by (1−conf) + 0.5×flag; random draws a fixed-seed sample instead. The true fail/ok label is revealed only once a trace is picked.

Try it: set the budget to 6 and flip between strategies. Hard-case catches all 4 real failures; random catches about half as many, for the same spend.
Failures found
0/6
density 0.00
Coverage
0.00
of all 4 real failures
Base rate
0.33
4 true failures / 12 traces

The flywheel

Grow a golden set, two hard cases at a time

Each round annotates the next 2 traces by priority. Run all 3 rounds and watch the golden set concentrate on the model's weak spots.

0 of 3 rounds run · golden set is empty
Round New traces Golden set Cum. fail Coverage Gold rate
Click “Run round 1” to annotate the first 2 hard cases.

The golden set's failure rate ends up enriched on purpose: it hunts for where the model breaks. That rate is not the production rate (the true production failure rate here is 4/12 = 0.33). To estimate production health without bias, keep a small random holdout alongside the mined set.