[QUANT LECTURE] Calibration and scoring under uncertainty
Market Inefficiencies - Information Theoretic Approach
Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.
Calibration and scoring under uncertainty
This chapter turns an inefficiency claim into a measurable probabilistic object. A model earns credibility when its output remains a truthful conditional description of the future under the same decision-time information that defines the claim. Proper scoring rules provide that bridge. They test whether probabilities, quantiles, or densities align with realized outcomes, and they do so inside the exact state regions where the strategy intends to deploy capital.
What’s inside:
Scoring for conditional evidence. The chapter defines prediction as a conditional statement about the future and explains why scoring must reward truthful reporting of the conditional law rather than flattering in-sample behavior.
Proper scoring fundamentals. It establishes the role of strict propriety and separates probabilistic description from downstream action, so learning targets the conditional mechanism itself rather than the trade rule built on top of it.
Log score and likelihood geometry. Full conditional densities are evaluated through log score, which measures how much probability mass the model assigns to realized outcomes and turns distributional mismatch into an empirical cost.
Brier score and probability geometry. Binary event probabilities are scored through squared probability error, which makes calibration visible and tests whether reported event likelihoods match realized frequencies.
Pinball loss and quantile geometry. Conditional quantile boundaries are evaluated through asymmetric loss that checks whether reported boundaries retain the correct crossing behavior under the declared state and horizon.
Calibration under state selection. The chapter shows why a model can look acceptable on pooled data while failing in the narrow states where capital is actually deployed, so calibration must be checked on the same conditional sample that drives execution.
Conditional calibration. Local inefficiency claims require local validation, so probabilities and quantiles must retain their interpretation inside the same state regions where divergence suggests conditional structure.
Miscalibration as false inefficiency. The chapter explains how overconfident outputs can manufacture the illusion of edge when threshold-based decision rules amplify extreme signals that lack stable probabilistic meaning.
Decision-aware scoring. Evaluation is aligned with the architecture of risk, exposure, and constraints, so errors matter most where the portfolio carries the highest operational sensitivity.
