Why 27 AI Models Deliver Only 1.5 Opinions
ملخص
Pixel shares an analysis of model ensemble diversity using 46,807 graded calls across 16,787 fixtures. On ballots with five or more voters, 10.34 models produced just 1.46 effective opinions, and majority voting scored 56.95% versus 57.00% for the average member—showing why model independence matters.
DEV Community
Your 27-model ensemble has 1.5 opinions: measuring diversity with Fleiss' kappa and lineage-weighted voting
An ensemble is a claim about independence. You put k models in a room, they vote, and errors unique to each member cancel out. That only works if members actually disagree. If they all came from the same handful of prompts, voting is theatre. Our platform has 3,784 model records and 46,807 graded calls over 16,787 fixtures. On paper that is a deep committee. I measured how deep with three estimators: stance concentration, pairwise contradiction, and Fleiss' kappa converted to effective ensemble size. On the 2,722 fixtures with five or more voters, the average ballot held 10.34 models and 1.46 effective opinions. A majority vote over those ballots scored 56.95% — slightly worse than the 57.00% average of the members it aggregated. The data Everything below comes from one append-only table. The relevant columns: column meaning match_id / model_id fixture key, model record training_word_id prompt lineage that model was built from prediction_direction home, draw, away, home/draw, away/draw direction_arity single (70.18%) or double (29.76%) confidence 1–10 integer, not a probability hit / actual_result settled once, home / draw / away Two properties matter. First, 29.76% of calls are two-outcome leans, so "agreement" is not string equality — two members can differ in label yet stay logically compatible. Second, training_word_id exposes the pedigree behind each ballot, where the collapse originates. Estimator 1: how concentrated is a ballot? If ten members split 9/1/0, the committee has one opinion and one dissenter. The Herfindahl index over stance shares gives a "number of effective stances" directly: WITH per_match AS (
SELECT match_id, count() AS n,
count() FILTER (WHERE prediction_direction='home') AS h,
count() FILTER (WHERE prediction_direction='draw') AS d,
count() FILTER (WHERE prediction_direction='away') AS a,
count() FILTER (WHERE prediction_direction='home/draw') AS hd,
count() FILTER (WHERE prediction_direction='away/draw') AS ad
FROM ai_predictions WHERE hit IS NOT NULL GROUP BY 1
), scored AS (
SELECT , (hh + dd + aa + hdhd + adad)::numeric / (nn) AS hhi,
greatest(h,d,a,hd,ad)::numeric / n AS top_share
FROM per_match WHERE n >= 5
)
SELECT count() AS fixtures, round(avg(n),2) AS mean_ballot,
round(avg(top_share),4) AS mean_top_share,
round(avg(hhi),4) AS mean_hhi,
round(avg(1.0/hhi),3) AS effective_stances,
round(100.0count() FILTER (WHERE top_share=1)/count(),2) AS pct_unanimous,
round(100.0count() FILTER (WHERE h>0 AND a>0)/count(),2) AS pct_home_and_away
FROM scored;
Read the last two columns together. A third of ballots are unanimous, and only 3.99% of fixtures have at least one member on each side of the home/away divide. The 358-call maximum ballot we once recorded is not a debate; it is a chorus. Mean top share is 0.8119 and mean HHI 0.7486, so 1/hhi = 1.455 effective stances against a 10.34-member roster. Estimator 2: are disagreements real or cosmetic? Label identity is too strict. home and home/draw are compatible; home and away are not. So classify each pair by outcome set, not string: WITH ballots AS (
SELECT match_id, model_id, prediction_direction AS dir
FROM ai_predictions
WHERE hit IS NOT NULL AND match_id IN (
SELECT match_id FROM ai_predictions WHERE hit IS NOT NULL
GROUP BY 1 HAVING count() BETWEEN 2 AND 40)
), pairs AS (
SELECT a.dir AS d1, b.dir AS d2
FROM ballots a JOIN ballots b
ON a.match_id = b.match_id AND a.model_id < b.model_id
)
SELECT count() AS pairs,
round(100.0 * count() FILTER (WHERE d1 = d2) / count(), 2) AS identical_pct,
round(100.0 * count() FILTER (WHERE NOT (
(d1 IN ('home','home/draw') AND d2 IN ('home','home/draw')) OR
(d1 IN ('draw','home/draw','away/draw')
AND d2 IN ('draw','home/draw','away/draw')) OR
(d1 IN ('away','away/draw') AND d2 IN ('away','away/draw'))
...

Powered by Readybot.io*