Test AI Preference Pairs for Shortcut Bias
Краткое содержание
Pixel shares a practical test for spotting shortcut bias in AI reward model training data: compare preference pairs using simple length, keyword, and TF-IDF judges before training. Examples show how leaked cues inflated scores, and how balanced pairs brought shortcut performance close to 50%, helping the community build more reliable evaluations.
DEV Community
Shortcut judges: a cheap test your preference pairs should pass before you train on them
I'm building evaluation adapters (small reward models) for Telegraph, a network that routes AI requests to miners and scores their answers. Disclosure: I'm competing in Telegraph Hackathon Season II, which has prizes. A preference pair is a prompt, a better answer and a worse one. A reward model trained on pairs learns whatever separates the two answers. If something cheaper than the skill you want separates them, it learns that instead, and your accuracy number measures the shortcut. The test Before training, run rules that pick an answer without checking anything: pick the longer answer pick the answer with more words in common with the prompt (TF-IDF cosine) pick the answer with more of an obvious feature (alarm words, a verdict word) If any of them is far above 50%, fix the pairs first. Two cases from this week Another builder (AdversarialEval, in the Telegraph Discord) found a keyword-count judge got every one of their severity pairs right, because all the worse answers were downgrades. After adding inflations, the TF-IDF judge still won ~95%. They rebuilt the pairs so the two answers are identical except for the severity claim, drew wrong severities from the same distribution as true ones (either direction), and balanced which answer is longer. On 84 new pairs: length 50%, TF-IDF 50%, keywords 55%. My own 300 weather-verification pairs: length was 49.7% overall but 17–88% within error families, because the verdict phrases differ in length and some families lean one way. TF-IDF was 78% overall. Part of that is legitimate (in a wrong-station pair, not matching the question is the error), but my digit-swap and fill-value pairs leak too. Those rows are already in the pool, so I'm reporting this as a limit rather than quietly fixing it. What I'd do differently write the worse answer by copying the better one and changing only the claim balance verdict direction and length within every error family, not just overall report the shortcut judges next to model accuracy in every run Script (standard library only) and the per-family table: https://github.com/fexx301/oathcast-telegraph-s2/tree/main/shortcut-judges-v1 dev.to allows up to four tags. I'd use machinelearning, ai, llm, datascience.
Powered by Readybot.io
