[WITH CODE] RiskOps: Basics of Conformal Prediction applied to risk

Conformal Prediction as your algorithmic trading compass

---

Table of contents:

1. Introduction.

2. What exactly is conformal prediction?

3. The nonconformity scores

4. Theorem: Finite-sample coverage guarantee.

5. Implementation of the conformal prediction framework for RiskOps.

---

Introduction

Before diving into risk and uncertainty, if you're curious about conformal prediction, I'd love for you to check out the book written by Valeriy Manokhin—I'll leave the reference at the end! And here’s his GitHub. He is the ultimate emissary of this framework.

Ready!? Let’s go!

Imagine you’re the captain of a sturdy ship navigating unpredictable waters. Your goal? To maximize your trading system’s PnL while avoiding the pitfalls of unexpected losses. In the world of algorithmic trading, PnL swings can be as unpredictable as sudden storms at sea. Traditional risk management tools are like old maps—helpful, but they can’t always foresee every tempest.

Conformal prediction is a state-of-the-art, mathematically robust tool that not only provides a point forecast but also offers a prediction interval for your trading system’s PnL, allowing you to better understand the uncertainty behind each prediction while enhancing risk management and enabling more adaptive strategy adjustments.

What exactly is conformal prediction?

Before we dive into the deep mathematical seas, let’s break down conformal prediction with a relatable analogy so it will be easier to understand.

Picture yourself running a busy lemonade stand on a scorching summer afternoon. Every day, you forecast your PnL—the profit you’ll make from selling lemonade—based on the weather. Some days, you hit the jackpot; on others, an unexpected downpour means a disappointing profit (or even a loss).

Suppose you want to be 90% sure that your predicted PnL will fall within a safe margin—so you know how much profit (or loss) to expect. Instead of a single number like “$100 profit,” conformal prediction provides you a range: for example, “Between $80 and $120 profit, with 90% confidence.” This range is known as the prediction interval and acts as your shield against unpredictable trading outcomes.

So, why is this useful? In trading, it means maintaining trust in your system—that is, knowing whether it should remain active or be shut down. You have a clear understanding of the risk you're taking and when you're taking it. Moreover, when assuming risk, you know exactly how much you can expect to lose and how much you stand to gain.

A key point here is not to accumulate the returns you get—keep them raw, or you might end up cooking the numbers more than a Michelin-starred chef!

So, do you want to know the secret sauce? Let’s see!

The nonconformity scores

At the heart of conformal prediction lies the concept of nonconformity scores—our “weirdness detector.” When predicting the trading system’s PnL, we compare the actual PnL to the predicted PnL. The absolute difference is our nonconformity score:

\(\text{Nonconformity Score} = \left| \text{Actual PnL} - \text{Predicted PnL} \right| \)

For example, if your system typically makes between $100 and $120 on similar days and you predict $150, the high difference indicates a “weird” prediction. Conversely, a prediction of $110 results in a low nonconformity score—just as you’d expect.

Now that we’ve warmed up with our lemonade analogy, let’s dive into the mathematical framework behind conformal prediction, now applied to the trading system’s PnL.

Imagine you have a treasure trove of historical PnL data from your trading system. We split this data into two parts:

1. Training data:
   Your treasure map—a collection of past observations (e.g., various trading days with features like market volatility, volume, and other signals) used to train your predictive model.

   \(\{(X_1, \text{PnL}_1), (X_2, \text{PnL}_2), \dots, (X_n, \text{PnL}_n)\} \)

2. Calibration data:
   The extra clues you keep aside to measure how “weird” your model’s PnL predictions might be.

\(\{(X_{n+1}, \text{PnL}_{n+1}), \dots, (X_{n+m}, \text{PnL}_{n+m})\}\)

Using a predictive model f (which could be a linear regression, random forest, etc.), you predict the trading system’s PnL from the features X.

Let’s break down the conformal prediction process into digestible steps:

1. Train the model:
   Fit your model f on the training data—your trusted treasure map.

2. Compute residuals:
   For each calibration observation, calculate the residual—the absolute error in your PnL prediction:

   \(R_i = \left| \text{PnL}_i - f(X_i) \right|\)

3. Find the threshold:
   For a desired confidence level 1−α (say, 90%), determine the (1−α)-quantile of the residuals. Suppose the sorted residuals are:

\(R_{(1)} \leq R_{(2)} \leq \dots \leq R_{(m)}\)

Then, select the threshold index

\(k = \lceil (m+1)(1-\alpha) \rceil,\)

and set

\(Q = R_{(k)}.\)

4. Predict with confidence:
   For a new observation X_new, the prediction interval for the trading system’s PnL is given by:

\(\text{Interval} = \left[ f(X_{\text{new}}) - Q, f(X_{\text{new}}) + Q \right]. \)

This interval contains the actual PnL with probability at least 1−α.

Where does the power of conformal prediction come from? It lies in the heart of the following theorem, which encapsulates its mathematical rigor and reliability.