40 tickets, 8 urgent (20%, rare). Only 10 can be hand-labelled for the golden set. Draw it two ways and watch the measured urgent count spread.
Click a button to draw a fresh sample of 10. Simple-random draws from all 40 at once and can miss the rare class; stratified always takes 2 from the 8 urgent and 8 from the 32 not. Draw several times and watch the tally build toward the exact numbers.
Population · 8 urgent (emerald) + 32 not (gray) = 40
simple-random
stratified
Simple-random tally
draws1
mean urgent4.000
std urgent0.000
zero-urgent draws0/1 = 0.000
exact: P(zero)=0.076 · std=1.109 · mean=2.000
Stratified tally
draws1
mean urgent2.000
std urgent0.000
always exactly 2 · std pinned to 0
Simple-random running histogram (urgent count per draw)
your draws exact expected count at this many draws k=2, where stratified always lands
Why measured accuracy lies
Leakage: memorized duplicates inflate the score
4 of these 10 eval tickets are near-duplicates of tickets the model was trained on. It gets all 4 right for free. Toggle them out to see the honest number.
Click "Remove leaked duplicates" to uncheck the 4 memorized items, or toggle any ticket by hand to explore.
Accuracy on included tickets
0.800
8/10 correct (4 of them memorized)
Score with a rubric
Three answers, three criteria, one bar
Each answer is scored 0, 1, or 2 on Correct, Grounded, and Complete (max 6). Click a score to change it, slide the pass bar, and watch who clears it.