Pre-registered behavioral fingerprinting of frontier language models
Two preprints describe the methodology and findings in full. The first measures the biases; the second uses those measurements to improve match-prediction accuracy.
Behavioral fingerprinting of frontier AI models in football talent evaluation and event prediction
Four frontier AI models tested across 12 talent dimensions and ~45,000 trials. Documents League Prestige Discount (Cohen’s h = 1.18–1.41) — unanimous across all models — and demographic evaluation inconsistency with EU AI Act compliance implications.
Two mechanisms for improving LLM football match predictions
979 matches across 18 leagues. A three-bias formula predicts model accuracy with r = 0.997 before any prediction is made. Bias-derived calibration improves Brier score by 4.6–7.3% per model.
Modelball is a pre-registered behavioral fingerprinting study examining systematic biases in five frontier LLMs when evaluating football talent and predicting match outcomes. The study comprises over 45,000 queries across 12 talent evaluation dimensions and a 10-test Prediction Calibration Module (PCM), validated through extensive backtesting.
Weight models by calibration to reduce systematic error in match forecasting
Identify match contexts (host nation, prestige mismatch) where specific models excel
Apply fingerprint-based adjustments to raw model outputs
To validate our behavioral fingerprints against real-world prediction performance, we conducted extensive backtesting across 18 leagues and nearly 1,000 matches.
This breadth of testing across different league contexts — from top European competitions to emerging markets — enables calibration that accounts for regional biases and league-specific model behaviors.
All responses scored by Claude Opus 4.6 without knowledge of which model produced them. Eliminates scorer bias.
10,000 bootstrap resamples for all effect sizes. 95% CIs reported throughout.
Responses that couldn't be cleanly scored are marked ambiguous and excluded from main analysis. Ambiguity rates reported per dimension.
All hypotheses, analysis plans, and stopping rules registered on OSF before data collection began.
Fingerprints validated against 979 historical matches across 18 leagues before deployment. Model weights derived from observed performance patterns.