Why 27 AI Models Deliver Only 1.5 Opinions

Краткое содержание

Pixel shares an analysis of model ensemble diversity using 46,807 graded calls across 16,787 fixtures. On ballots with five or more voters, 10.34 models produced just 1.46 effective opinions, and majority voting scored 56.95% versus 57.00% for the average member—showing why model independence matters.

DEV Community

Your 27-model ensemble has 1.5 opinions: measuring diversity with Fleiss' kappa and lineage-weighted voting

An ensemble is a claim about independence. You put k models in a room, they vote, and errors unique to each member cancel out. That only works if members actually disagree. If they all came from the same handful of prompts, voting is theatre. Our platform has 3,784 model records and 46,807 graded calls over 16,787 fixtures. On paper that is a deep committee. I measured how deep with three estimators: stance concentration, pairwise contradiction, and Fleiss' kappa converted to effective ensemble size. On the 2,722 fixtures with five or more voters, the average ballot held 10.34 models and 1.46 effective opinions. A majority vote over those ballots scored 56.95% — slightly worse than the 57.00% average of the members it aggregated. The data Everything below comes from one append-only table. The relevant columns: column meaning match_id / model_id fixture key, model record training_word_id prompt lineage that model was built from prediction_direction home, draw, away, home/draw, away/draw direction_arity single (70.18%) or double (29.76%) confidence 1–10 integer, not a probability hit / actual_result settled once, home / draw / away Two properties matter. First, 29.76% of calls are two-outcome leans, so "agreement" is not string equality — two members can differ in label yet stay logically compatible. Second, training_word_id exposes the pedigree behind each ballot, where the collapse originates. Estimator 1: how concentrated is a ballot? If ten members split 9/1/0, the committee has one opinion and one dissenter. The Herfindahl index over stance shares gives a "number of effective stances" directly: WITH per_match AS (
SELECT match_id, count() AS n,
count(
) FILTER (WHERE prediction_direction='home') AS h,
count() FILTER (WHERE prediction_direction='draw') AS d,
count(
) FILTER (WHERE prediction_direction='away') AS a,
count() FILTER (WHERE prediction_direction='home/draw') AS hd,
count(
) FILTER (WHERE prediction_direction='away/draw') AS ad
FROM ai_predictions WHERE hit IS NOT NULL GROUP BY 1
), scored AS (
SELECT , (hh + dd + aa + hdhd + adad)::numeric / (nn) AS hhi,
greatest(h,d,a,hd,ad)::numeric / n AS top_share
FROM per_match WHERE n >= 5
)
SELECT count(
) AS fixtures, round(avg(n),2) AS mean_ballot,
round(avg(top_share),4) AS mean_top_share,
round(avg(hhi),4) AS mean_hhi,
round(avg(1.0/hhi),3) AS effective_stances,
round(100.0count() FILTER (WHERE top_share=1)/count(),2) AS pct_unanimous,
round(100.0
count() FILTER (WHERE h>0 AND a>0)/count(),2) AS pct_home_and_away
FROM scored;
Read the last two columns together. A third of ballots are unanimous, and only 3.99% of fixtures have at least one member on each side of the home/away divide. The 358-call maximum ballot we once recorded is not a debate; it is a chorus. Mean top share is 0.8119 and mean HHI 0.7486, so 1/hhi = 1.455 effective stances against a 10.34-member roster. Estimator 2: are disagreements real or cosmetic? Label identity is too strict. home and home/draw are compatible; home and away are not. So classify each pair by outcome set, not string: WITH ballots AS (
SELECT match_id, model_id, prediction_direction AS dir
FROM ai_predictions
WHERE hit IS NOT NULL AND match_id IN (
SELECT match_id FROM ai_predictions WHERE hit IS NOT NULL
GROUP BY 1 HAVING count() BETWEEN 2 AND 40)
), pairs AS (
SELECT a.dir AS d1, b.dir AS d2
FROM ballots a JOIN ballots b
ON a.match_id = b.match_id AND a.model_id < b.model_id
)
SELECT count(
) AS pairs,
round(100.0 * count() FILTER (WHERE d1 = d2) / count(), 2) AS identical_pct,
round(100.0 * count() FILTER (WHERE NOT (
(d1 IN ('home','home/draw') AND d2 IN ('home','home/draw')) OR
(d1 IN ('draw','home/draw','away/draw')
AND d2 IN ('draw','home/draw','away/draw')) OR
(d1 IN ('away','away/draw') AND d2 IN ('away','away/draw'))
...
Why 27 AI Models Deliver Only 1.5 Opinions
Powered by Readybot.io*

Последнее от Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …

Why a Java HPA Got Stuck at 20 Kubernetes Pods

Pixel shares how a Spring Boot pricing service autoscaled to 20 Kubernetes pods and stayed there because the JVM retained committed heap memory after traffic …