Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

From research question to testable hypothesis

Quantitative trading research begins with market intuition, but it becomes empirically useful only when the researcher defines the instruments, market state, activation trigger, response window, expected price path, and causal mechanism. Strong hypothesis fixes these elements before testing, separates the source of the edge from its execution, and establishes the conditions under which the claim remains valid or expires.

What’s inside:

1. Narrowing the phenomenon: The researcher reduces a broad market observation to a specific inefficiency by defining the relevant instruments, environmental state, and natural time horizon.

2. Selecting instruments, states, and horizons: Assets should be close to the source of the market force, conditions should support the proposed mechanism, and the holding period should match the speed at which the effect unfolds.

3. Defining the expected effect: The hypothesis specifies the direction, shape, magnitude, duration, and decay of the anticipated response, including mean reversion, continuation, spread compression, or ranking changes.

4. Connecting price action to economic meaning: Expected returns require a causal explanation based on information diffusion, liquidity recovery, forced order flow, institutional execution, funding constraints, or another identifiable market process.

5. Defining the trade lifecycle: Entry conditions activate the claim, maintenance conditions verify that its mechanism remains present, and breakdown conditions identify when the original hypothesis has expired.

6. Separating observation from explanation: Similar chart patterns can emerge from different forces. Research must look beyond visual geometry to identify participants, incentives, liquidity constraints, venues, and transmission channels.

7. Separating idea, hypothesis, and execution: A trading idea provides intuition, the research hypothesis identifies the source of predictive value, and the execution method determines how to capture it. Each portfolio component must possess its own causal foundation.

Sample

1.92MB ∙ PDF file

Download

Download

Read more

In this episode of House of Quants, listeners will discover:

1. How quants separate genuine edge from market noise: The episode explains why financial markets are difficult to diagnose and how researchers distinguish persistent information from randomness, temporary anomalies, and misleading patterns.

2. How data problems create false strategies: It examines survivorship bias, timestamp errors, incorrect corporate-action adjustments, data leakage, and excessive data cleaning—problems that can make an impossible strategy appear profitable.

3. How strong trading hypotheses are created: Listeners will learn how quantitative ideas emerge from economics, behavioral finance, market microstructure, and alternative data, as well as why every hypothesis must be specific, measurable, and falsifiable.

4. Why quants try to disprove their own ideas: The discussion explores confirmation bias, narrative fallacy, curve fitting, moving the goalposts, and the importance of defining the methodology and success criteria before examining the results.

5. How strategies are tested under realistic conditions: The episode covers transaction costs, spreads, slippage, financing, liquidity, market impact, position sizing, parameter robustness, and performance across different market regimes.

6. How professional validation methods reduce overfitting: It introduces out-of-sample testing, walk-forward analysis, purged cross-validation, bootstrap methods, multiple-testing corrections, and the Deflated Sharpe Ratio.

7. What makes a quantitative strategy credible: The central lesson is that a strategy becomes credible only after surviving changing regimes, imperfect data, realistic execution costs, statistical scrutiny, and repeated attempts to prove it wrong.

Send me a copy

Table of contents:

1. Introduction.

2. Wrapper-based feature selection.

3. Risks and limitations.

4. Model-conditional optimization.

   1. Sequential Forward Selection (SFS).

   2. Sequential Backward Selection (SBS).

   3. Exhaustive Feature Selection (EFS).

   4. Sequential Floating Selection (SFLS)

   5. Recursive Feature Elimination (RFE).

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Feature selection is often presented as a simple cleanup step… remove the weak variables, keep the useful ones, and move on. In practice, it is much closer to a research decision about what information the model is allowed to trust.

Every feature added to a trading system creates a cost. It may require another data source, increase latency, make the model harder to interpret, or create another opportunity for overfitting.

Wrapper-based methods approach this problem from a practical perspective. Instead of judging features in isolation, they test them through the model that will actually use them. This makes the selection process more realistic, because a feature only matters when it improves the behaviour of the complete model under a credible validation framework.

You can go deeper in warpper methods by checking this two papers:

Wrapper based feature selection

4.29MB ∙ PDF file

Download

Download

Wrapper methods for multi-objective feature selection

811KB ∙ PDF file

Download

Download

Each method follows a different search philosophy, from gradually building a compact feature set to pruning a large one or evaluating every feasible combination. The goal is to understand how each method deals with the trade-off between predictive value, stability, and model complexity. By comparing them under the same structure, it becomes easier to see when a method is useful, where it can fail, and how its selection path affects the final trading model.

Wrapper-based feature selection

Identifying a reliable predictive signal requires more than measuring isolated statistical correlations. Wrapper-based feature selection provides a methodology to evaluate the predictive power of specific feature combinations. This approach integrates the target model into the selection loop. The system wraps a search algorithm around the predictive model and uses empirical validation performance to score candidate feature subsets.

To understand the architecture of wrapper methods, we must examine other selection paradigms:

1. Filter methods operate outside the boundaries of any model. They evaluate features based on intrinsic statistical properties, such as variance or mutual information with the target variable. This standalone structure gives filter methods a lightweight computational profile. They isolate fundamental data characteristics and separate the data from the final algorithmic interpretation. This separation forces filter methods to overlook complex feature interactions.

2. Wrapper methods take a distinct path. They evaluate features by training and validating a defined model in repeated cycles. The wrapper treats the target algorithm as a direct scoring engine. It tests candidate subsets of features and measures their impact on out-of-sample performance. This approach requires retraining the model from a blank state for every candidate subset evaluated, which creates an immense computational cost. Because these methods remain model-dependent, they offer an honest measurement of predictive utility in live trading scenarios.

3. Embedded methods incorporate feature selection into the model training process. Algorithms like Lasso regression penalize lower-impact features during the construction of their mathematical structure. This mechanism merges selection and training into a single phase, which generates a moderate computational cost. The selection process remains locked to the internal mechanics of that chosen algorithm.

A complete wrapper architecture relies on four fundamental components that operate in a continuous loop:

• The search space establishes the boundaries of all available candidate features.

• The search strategy dictates the algorithmic trajectory to navigate the combinations.

• The evaluation metric provides the quantitative objective function to measure model predictions.

• The underlying model defines the algorithm that maps candidate inputs to the target variable.

Wrappers evaluate features in collective groups. This structure gives them the capability to identify joint signal interactions that single-variable statistical filters exclude. For example, order flow dynamics and volatility regimes interact in nuanced patterns and the wrapper asks a specific question “Does the inclusion of this feature improve the predictive accuracy of this specific model?” This collective evaluation makes wrappers suited for discovering market dynamics. The use of the model as a direct scoring engine ties the value of a feature to the tested environment, something that shapes the optimization required for live trading.

Risks and limitations

This sounds trivial from a standard machine learning perspective.

However, in trading, that statement hides a difficult engineering problem because the score itself is unstable, labels contain noise, observations are serially dependent, etc. As a result, a feature subset that looks like pure gold during one sample can become a major source of loses the second market volatility spikes or participant behavior shifts.

The trouble with financial models starts at the foundation with a messy problem called temporal contamination. Standard cross-validation requires independent data points, but market data streams possess a strict order, a clear dependency, and a constant overlap. Breaking this data into random folds causes leakage, because future price action bleeds into past training sets. Fixing this error requires a boundary, which is where purging and embargoing come into play to isolate the training data from the test data.

Once that data leak is under control, a different headache appears, known as the curse of multiple testing. Wrapper algorithms sift through piles of potential features, a process that inflates the top score by pure chance. A huge search space guarantees a spectacular winner born from total noise. Without a strict separation of time horizons during validation, the algorithm flags a historical fluke instead of a true lasting edge.

These flawed features cause more damage due to model dependence, meaning a feature set has zero portability across different setups. A group of features built for a linear model falls apart inside a decision tree. Features picked via Ridge regression turn useless under Lasso regularization. Binary classifier inputs lose their grip when converted into continuous position sizing logic. The whole selection process builds a house of cards that links the model class, the features, the labels, and the validation into one single knot. A change to one thread collapses the entire framework.

And what about research drift? It acts as a hidden form of data snooping. Running a loop through forward selection, backward selection, and recursive elimination stacks up the hidden trial count. Tweaking the metrics, shifting the data splits, or altering the label horizons to chase better historical returns introduces a selection bias. This game ruins the validity of standard significance tests.

Subscribe now

Model-conditional optimization

At its core, a wrapper method selects features by treating the underlying model as a black box and evaluating performance across various candidate subsets. Because the feature set is judged through the lens of a specific model and metric, context dictates everything. The exact same feature can prove useful under one model architecture while remaining useless under another.

To formalize this process, let the full candidate feature pool be defined as F={f₁,f₂,…,f_p}. From this pool, any given candidate subset is represented as S⊆F.

Now, assuming a specific model class M, a training dataset D_train, a validation dataset D_val, and a scoring function J, the wrapper selection process solves the following optimization problem:

Where the fitted model relies entirely on the chosen feature subset:

The path taken to find S* depends on how you define the search space boundaries. An exhaustive feature selection routine evaluates every single possible subset in the space. Conversely, sequential methods restrict the computational burden by following a rigid, step-by-step path. Meanwhile, recursive feature elimination leverages the model’s internal feature rankings to determine the deletion order.

The objective function J requires far more scrutiny than the search algorithm itself. It is easy for a wrapper to optimize a clean statistical metric and lose money. For instance, a model boasting a high Information Coefficient can deliver negative realized returns if the signal exhibits weak ranking power near the critical threshold used for execution. Furthermore, a model with flawless gross performance fails the moment you incorporate exchange fees, slippage, or borrow costs.

To force the wrapper to respect these execution boundaries, the objective function requires a penalty structure:

In this functional, r_t(S) represents the strategy returns generated by the model using feature subset S. The penalty terms translate market fees into the optimization. C(S) captures execution friction and turnover costs, while Ω(S) imposes a penalty on the dimension of the subset. A subset that marginally improves predictive power at the expense of skyrocketing turnover will degrade the net equity curve. Likewise, a subset that performs perfectly in one data fold but collapses in another introduces unstable deployment behavior.

This is why researchers focus less on the final feature list and look instead at the structural stability of the selection process. The result must satisfy several clear operational conditions:

• The selected subset appears persistently across distinct temporal folds.

• The chosen features survive aggressive changes to the underlying scoring metric.

• The subset keeps strategy turnover within the fund’s execution capacity.

• The selected features reduce live decision complexity.

• The subset improves net performance after accounting for all fees and costs.

• The features demonstrate a plausible connection to known market microstructure mechanics.

• The model performance degrades slowly when the market regime shifts.

This brings us to a misconception. The idea that wrapper methods discover an objectively “best” set of features. In reality, they find the optimal subset constrained by a specific search path, a chosen model class, a single metric, a finite sample size, and a defined validation protocol.

Visualizing this behavior clarifies why simpler selection methods fail.

Viewed independently, both x₁ and x₂ demonstrate a flat, weak relationship hovering around zero. Together, however, they may define a significant structural edge. This interaction highlights the exact reason why univariate filters fail to identify effective trading combinations.

To translate this into a concrete trading scenario, suppose x₁ represents short-term price reversal pressure, while x₂ represents prevailing bid-ask liquidity conditions. Reversal strategies often perform well during specific liquidity regimes but fail during others. A univariate filter evaluating x₁ in isolation would reject the feature due to its poor average performance. A wrapper method retains it because the underlying model evaluates the conditional interaction between the two variables.

Send me a copy

Sequential Forward Selection (SFS)

SFS begins with an empty feature set and adds exactly one feature at a time. At each step, the algorithm chooses the candidate that delivers the largest immediate mathematical improvement in the target objective function.

Let the current selected set at step k be denoted as S_k. The algorithm chooses the optimal addition

Then it updates the state space

The algorithm terminates when the score improvement falls below a defined threshold, when a hard target subset size is reached, or when the validation performance begins to show signs of deterioration.

SFS is attractive because it guarantees compact models. Compact models possess immense engineering value. A low-latency execution model with five stable features has operational advantages over a bloated model with fifty fragile inputs. Fewer features reduce data pipeline dependencies, time-synchronization errors, memory footprint, and total inference latency. In execution environments where any individual data feed can drop or delay, a smaller feature set provides a definitive robustness advantage.

Imagine an intraday market-making model evaluating the spread, the trade size, the order book imbalance, the short-term realized volatility, and the last price displacement. Forward selection builds a minimal subset capturing the vast majority of the useful effect while avoiding slow or noisy features.

Forward selection provides another benefit in high-dimensional settings. It starts from zero features, ensuring the early model fits remain numerically stable. Backward procedures begin with the full matrix. The full matrix becomes ill-conditioned when p is large, features exhibit collinearity, or the temporal sample size is constrained. But in this case it avoids the initial full-dimensional matrix inversion problem inherent in linear models.

The primary limitation involves severe path dependence. The algorithm is greedy. It selects the best immediate addition and carries that rigid choice forward forever. A feature that looks weak early could become useful later when paired with another specific feature. Forward selection misses that pair because neither feature produces enough standalone improvement at the very beginning of the search path.

A trading-safe forward step evaluates candidates on temporal validation blocks

The model is trained on a training segment that excludes the validation segment and its required embargo neighborhood. The score averages across the folds and is more credible than a feature selected on a single randomized validation slice.

Many implementations stop at an arbitrary fixed number of features. But in our context, the subset size must be a direct consequence of measured marginal utility versus cost. Let the marginal gain be

A termination rule requires Δ_k>τ where τ includes a “premium”. That premium scales higher when a feature is computationally slow, or statistically unstable

Other rules use a penalized objective

where |S| penalizes dimensionality and C_ops penalizes specific costly inputs.

Forward selection provides a diagnostic output for the researcher. The marginal contribution curve maps the precise geometry of the alpha extraction process. Plotting the sequential step-by-step score creates a direct visual representation of diminishing returns. If the first three features add the vast majority of the predictive value and the next five add microscopic improvements, the model has reached its absolute maximum useful complexity.

This type of system relies on a core directional driver, like a distinct volatility adjustment, and a liquidity filter. Evaluating further correlated technical variations adds zero edge. Forcing the model past this structural plateau increases execution latency and guarantees parameter overfitting in production. Furthermore, if the out-of-sample score keeps rising across dozens of consecutive feature additions, the researcher must investigate the foundational data pipeline.

On the other hand, an upward curve that never flattens proves that severe validation leakage or unchecked selection bias is artificially driving the continuous improvement. The algorithm is memorizing the specific validation sequence through overlapping return horizons, un-lagged data releases, or future price contamination.

Let’s implement it:

def sequential_forward_selection(
    X,
    y,
    splits,
    max_features=None,
    min_improvement=0.0,
    alpha=1.0,
    metric="neg_mse",
    complexity_penalty=0.0,
    feature_costs=None,
    candidate_features=None,
):
    """
    Start with an empty feature set and greedily add the best feature.

    The algorithm stops when:
    - max_features is reached
    - no candidates remain
    - the best marginal improvement is below min_improvement
    """
    X, y = _validate_xy(X, y)
    candidates = _candidate_pool(X.shape[1], candidate_features)

    if max_features is None:
        max_features = candidates.size

    max_features = min(int(max_features), candidates.size)

    selected = []
    remaining = candidates.tolist()

    current_score = evaluate_subset(
        X=X,
        y=y,
        feature_indices=selected,
        splits=splits,
        alpha=alpha,
        metric=metric,
        complexity_penalty=complexity_penalty,
        feature_costs=feature_costs,
    )

    history = []

    while remaining and len(selected) < max_features:
        best_feature = None
        best_score = -np.inf

        for feature in remaining:
            candidate_subset = selected + [feature]

            score = evaluate_subset(
                X=X,
                y=y,
                feature_indices=candidate_subset,
                splits=splits,
                alpha=alpha,
                metric=metric,
                complexity_penalty=complexity_penalty,
                feature_costs=feature_costs,
            )

            if score > best_score:
                best_score = score
                best_feature = feature

        improvement = best_score - current_score

        if improvement <= float(min_improvement):
            break

        selected.append(best_feature)
        remaining.remove(best_feature)
        current_score = best_score

        history.append(
            {
                "step": len(history) + 1,
                "action": "add",
                "feature": int(best_feature),
                "score": float(current_score),
                "improvement": float(improvement),
                "subset": tuple(selected),
            }
        )

    return np.asarray(selected, dtype=np.int64), history

The marginal-gain profile requires intense review across folds. A feature that appears in one single fold is less credible than a feature repeatedly selected across multiple market regimes.

SFS excels under specific trading conditions:

• The candidate feature pool is big.

• The desired model is compact.

• Feature interactions are moderate rather than complex.

• The objective functional penalizes costs and turnover.

• The training procedure uses purged temporal validation.

• The research protocol fixes the search budget before execution.

• The selected path remains stable across time.

The interpretation remains straightforward. Forward selection fails to be a universal feature discovery engine. It represents a strict incremental capital allocation procedure. Each feature receives allocation inside the model exclusively when it proves validated marginal utility. The final selected subset functions as a sparse, highly robust decision layer.

Sequential Backward Selection (SBS)

SBS begins with the complete, full-dimensional feature set and removes one feature at a time. At each specific step, the algorithm discards the feature whose removal causes the absolute smallest loss in target performance.

Let the current feature set at step k be denoted as S_k. The algorithm chooses the optimal removal candidate

Then it updates the state space by shrinking it

It looks pretty similar than the previous one, right? But it represents a different mathematical philosophy. Forward selection asks which feature earns the right to enter. Backward selection asks which feature deserves immediate eviction. That inversion is interesting when features possess joint predictive value.

Market data is saturated with features that could generate alpha—or smart beta— through interaction. Consider a small-cap mean-reversion signal where the overnight gap size matters when the opening liquidity exceeds a specific threshold. Forward selection misses these structures because each individual component looks statistically meaningless alone. Backward selection starts with the complete interaction structure intact and prunes away the dead weight.

The primary advantage of backward selection is its respect for joint information geometry at the start of the process. The disadvantage involves computational cost and severe numerical fragility. Early backward steps require fitting complex models containing almost every single candidate feature. When the feature matrix is wide, collinear, and saturated with noise, those initial model fits become unstable.

A full feature matrix in trading usually contains dozens of redundant transformations of the exact same underlying price path. The matrix contains returns over several horizons, various volatility estimators, moving average distance metrics, breakout flags, range ratios, candlestick wick metrics, and a pretty long list. These variables create a poorly conditioned design matrix.

For a linear model utilizing design matrix X_S, the condition number κ determines the stability of the solution

When features are collinear, the minimum eigenvalue approaches zero, and κ explodes. A condition number means microscopic data perturbations produce coefficient changes. Backward selection ranks features terribly when the fitted model possesses this level of instability.

This approach requires feature clustering or Ridge regularization prior to initiating the backward pass. The goal is to extract redundancy from a verified joint signal structure. The scoring function guiding the backward removal must be calculated net of all frictional costs. Backward selection retains features that improve gross accuracy while increasing turnover if the score ignores that turnover. And here is the issue, the model looks good statistically but fails economically.

Let’s see how this look from coding perspective:

def sequential_backward_selection(
    X,
    y,
    splits,
    target_features=1,
    max_score_loss=None,
    alpha=1.0,
    metric="neg_mse",
    complexity_penalty=0.0,
    feature_costs=None,
    candidate_features=None,
):
    """
    Start with the complete feature set and remove one feature at a time.

    At each iteration, remove the feature whose exclusion produces the
    highest validation score.
    """
    X, y = _validate_xy(X, y)
    candidates = _candidate_pool(X.shape[1], candidate_features)

    target_features = int(target_features)

    if target_features < 1 or target_features > candidates.size:
        raise ValueError(
            "target_features must be between 1 and the candidate pool size."
        )

    selected = candidates.tolist()

    current_score = evaluate_subset(
        X=X,
        y=y,
        feature_indices=selected,
        splits=splits,
        alpha=alpha,
        metric=metric,
        complexity_penalty=complexity_penalty,
        feature_costs=feature_costs,
    )

    history = []

    while len(selected) > target_features:
        best_removed_feature = None
        best_subset = None
        best_score = -np.inf

        for feature in selected:
            candidate_subset = [
                selected_feature
                for selected_feature in selected
                if selected_feature != feature
            ]

            score = evaluate_subset(
                X=X,
                y=y,
                feature_indices=candidate_subset,
                splits=splits,
                alpha=alpha,
                metric=metric,
                complexity_penalty=complexity_penalty,
                feature_costs=feature_costs,
            )

            if score > best_score:
                best_score = score
                best_removed_feature = feature
                best_subset = candidate_subset

        score_loss = current_score - best_score

        if (
            max_score_loss is not None
            and score_loss > float(max_score_loss)
        ):
            break

        selected = best_subset
        current_score = best_score

        history.append(
            {
                "step": len(history) + 1,
                "action": "remove",
                "feature": int(best_removed_feature),
                "score": float(current_score),
                "score_loss": float(score_loss),
                "subset": tuple(selected),
            }
        )

    return np.asarray(selected, dtype=np.int64), history

The backward perspective proves valuable when the researcher suspects redundancy. Suppose the candidate matrix contains the 5-minute return, the 15-minute return, the 30-minute return, the 1-hour return, the distance to daily VWAP, the realized volatility, the ATR ratio, the spread percentile, and the opening range position. Several of these variables overlap geometrically. The model leverages them as correlated versions of the exact same latent state. Backward selection identifies and removes features whose absence changes the predictive distribution. The final result is a streamlined representation of an initial state space.

The defining statistic here is the conditional performance loss

The microscopic value for L(f | S) proves the feature contributes nothing after accounting for the presence of all other features. This implies the feature adds zero conditional value to the current set. It is redundant.

“Backward selection plays a role in post-research simplification. A quantitative researcher begins with a broad, complex model, utilizes backward selection to prune all redundant inputs, and then retrains the model from scratch on the radically reduced set. The final model boasts lower operational complexity, execution speed, and improved interpretability.”

Backward selection also demands safeguards:

• The full feature model requires enforced numerical stability through regularization.

• The validation protocol requires purged temporal blocking.

• The entire removal path requires evaluation across multiple folds.

• The final subset requires total retraining from a blank state.

• The selected subset requires testing on untouched holdout data.

• The pruning decision requires metrics based entirely on net trading performance.

A useful diagnostic is the removal curve, where if removing features barely alters the net performance for twenty consecutive steps, the original feature set contained redundancy. If the net performance collapses instantly after removing one specific feature, that feature represents a structural pillar of the model. And if performance actually improves as features are removed, the original set was saturated with noise.

Exhaustive Feature Selection (EFS)

EFS evaluates every possible subset of the candidate feature set. If the environment contains p features, the number of non-empty subsets is governed by the relation

This combinatorial explosion grows with terrifying speed. With 10 features, the engine tests 1,023 subsets. With 20 features, the engine tests 1,048,575 subsets. With 30 features, the engine tests 1,073,741,823 subsets. With 40 features, the engine evaluates more than one trillion distinct subsets.

Exhaustive search guarantees finding the absolute “best” subset within the defined search space. Greedy methods frequently miss the global optimum because of the restricted trajectory. Exhaustive search evaluates all paths simultaneously. For small feature sets, this capability is pretty valuable.

Every single subset requires training, validation, and scoring. If the model requires recalibration across rolling windows, the total computational cost multiplies by the number of windows. If the strategy spans multiple assets, the total cost multiplies again. If the score utilizes purged cross-validation, the cost multiplies by the total number of folds.

A simplified cost expression defines the boundary

where K is the validation folds, W is the rolling windows, A is the asset count, and C_fit is the hardware time required to fit one model.

For low latency infrastructure, this cost becomes something impossible. This kind of models require continuous recalibration, with micro or even nanosecond latency budgets. Exhaustive search across a large feature matrix conflicts with that operating environment.

On the other hand, choosing three factors—tiny feature set—from a universe of eight candidates is a reasonable exhaustive problem. Selecting a compact portfolio allocation model from a small universe of carry, value, momentum, inflation, liquidity, and growth factors justifies exhaustive evaluation. Here the researcher runs the exhaustive engine and compares the results of forward, backward, and floating methods against the known exhaustive optimum. If the greedy methods lock onto near-optimal subsets at a fraction of the computational cost, the greedy methods become verified production tools.

The researcher reduces the search space by enforcing subset size caps. Define the bounded space as

The number of evaluated subsets collapses to

For p = 40 and q = 3, the engine evaluates a manageable 10.700 subsets. Full exhaustive search over 40 features is nonsense. There is another issue here, this kind of search exponentially increases overfitting risk because it tests an astronomical number of alternatives. The more subsets tested, the higher the expected maximum score under pure noise.

The simulation generates pure Gaussian noise scores for thousands of subsets. The mean score of the population remains firmly anchored at zero. The selected maximum score rises as the number of tested subsets grows. The selected winner looks impressive despite every single candidate possessing zero true statistical edge.

Let Z₁, Z₂, …, Z_M be validation scores for M candidate subsets under a null setting where each score is independent standard normal noise. The selected maximum score is

Even though E[Z_i] = 0, extreme value theory proves that the expected value of the maximum grows proportionally to √(ln M). The larger M becomes, the larger the expected illusion of edge becomes.

This represents the defining danger of exhaustive search. The selected subset from an exhaustive search operates as a hypothesis generated by a machine. It identifies the best subset in-sample. The best in-sample subset frequently represents the absolute best historical accident. Due to that, this approach needs nested validation protocols and synthetic data validation for independent verification.

Let’s see how to implement it:

Read more

Table of contents:

1. Introduction.

2. The transition of United States monetary policy.

3. Japanese fiscal expansion and economic strategy (2026–2041).

4. The economics of AI infrastructure.

5. Evolution of digital banking and financial markets.

6. Asset allocation and portfolio reallocation.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The global financial architecture enters a phase of profound structural realignment. In the United States, the ascension of Kevin Warsh to the chairmanship of the Federal Reserve establishes a new paradigm for monetary policy. This regime prioritizes strategic ambiguity, concise communication, and an insulation of policy actions from market expectations.

Cross-border economic dynamics mirror this institutional transformation. Japan embarks on an expansive macroeconomic doctrine under Prime Minister Sanae Takaichi. The multi-decade investment framework combines state capital with structural reforms to foster domestic technology hubs and automation systems, accepting sovereign balance sheet expansion as a necessary trade-off for long-term production growth.

The catalyst for global capital movement remains the physical construction of artificial intelligence infrastructure. Corporate entities allocate immense capital budgets toward advanced silicon accelerators, specialized liquid-cooled data facilities, and direct power grid access. This investment cycle transitions technical software capabilities from passive digital applications into autonomous agentic systems, altering corporate expenditure structures.

Parallel disruptions redefine the financial services sector. Cloud-native digital banks secure full regulatory charters, dismantling legacy moats through multi-engine platforms. Diversified streams of interest income, asset management, and subscription products yield sustainable corporate profitability, transforming consumer habits and global market shares.

Go deeper by checking:

Mind On The Markets Quaterly

1.05MB ∙ PDF file

Download

Download

Tracking Trillions

2.25MB ∙ PDF file

Download

Download

These combined shifts mandate a thorough restructuring of investment strategies. Effective asset allocation demands precise duration management in fixed income portfolios, active regional strategies within emerging markets, and a balance between technology growth assets and cash-generating cyclical industries. The subsequent sections detail these structural microeconomic changes and outline concrete frameworks for institutional capital reallocation.

Subscribe now

The transition of United States monetary policy

The transition of leadership at the Federal Reserve in mid-2026 represents one of the most volatile, legally contested, and philosophically disruptive chapters in the history of modern central banking. Following the swearing-in of Kevin Warsh as the 17th Chair of the Federal Reserve on May 22, 2026, the institution has embarked on a fundamental regime shift. Confronted by sticky inflation driven by Middle East energy supply shocks and a vocal, growth-oriented executive branch, Warsh has initiated an aggressive overhaul of monetary policy communications and structural frameworks. This transition of the Federal Reserve chairmanship was defined by unprecedented institutional friction, cutting across all three branches of the federal government and challenging long-standing norms of monetary independence.

The political battle to confirm Kevin Warsh began in earnest within a polarized Senate. On May 13, 2026, the United States Senate confirmed Warsh as Chairman of the Board of Governors by a razor-thin margin of 54 votes in favor to 45 against. This vote established a record for the narrowest confirmation margin for a Federal Reserve Chair in U.S. history, eclipsing the previous record set by Ben Bernanke’s 70 - 30 confirmation vote in 2010. The division was partisan, reflecting deep ideological splits over the future of monetary policy and central bank oversight. The Republican majority voted as a unified block in favor of Warsh, viewing him as a reformer capable of bringing accountability and market discipline to an institution they argued had overstepped its mandate. Democratic Senator John Fetterman of Pennsylvania crossed party lines to join the Republican majority, citing Warsh’s explicit commitment during hearings to increase congressional responsiveness and dismantle what he termed the “incomprehensible jargon” of Fed communications. Conversely, prominent progressive critics, led by Senator Elizabeth Warren, fiercely opposed Warsh. They labeled him a “sock puppet” for the executive branch and Wall Street, citing his regulatory positions during the 2008 global financial crisis and his public alignment with President Donald Trump’s pro-growth, low-interest-rate rhetoric in the lead-up to his nomination.

The nomination path was logjammed for months within the Senate Banking Committee due to a high-stakes standoff involving outgoing Chair Jerome Powell. Senator Thom Tillis placed a temporary hold on Warsh’s nomination, demanding that the Department of Justice terminate its criminal investigation into Powell regarding a controversial, multi-million-dollar renovation of the Federal Reserve’s Washington headquarters. Tillis and other committee members argued that the Department of Justice probe was a “frivolous, politically motivated prosecution” designed to compromise central bank independence and coerce the Fed into lowering interest rates. On April 24, 2026, the Department of Justice announced it would drop the criminal probe and refer the matter to the Federal Reserve’s independent Inspector General. This concession broke the legislative deadlock, allowing the Senate Banking Committee to advance Warsh’s nomination to the Senate floor in a tense, party-line 13-11 vote.

Simultaneously, the Federal Reserve became the battleground for one of the most critical executive power disputes in American history. In August 2025, President Donald Trump attempted to terminate Board Governor Lisa Cook—originally appointed by President Joe Biden to a 14-year term ending in 2038—alleging mortgage fraud on her historical loan agreements. Cook unequivocally denied the allegations, characterizing them as a bad-faith pretext to clear a seat on the Board for a monetary dove. Cook sued to halt her removal, arguing that under 12 U.S.C., Federal Reserve Governors may only be removed by the President “for cause” and are entitled to robust due process. On June 29, 2026, the Supreme Court ruled 5 - 4 in Trump v. Cook (No. 25A312) to uphold a lower court’s preliminary injunction, blocking her removal.

Chief Justice John Roberts delivered the majority opinion, joined by Associate Justice Brett Kavanaugh and the three liberal justices (Justices Sotomayor, Kagan, and Jackson). The Court held that while the executive branch possesses broad authority over executive departments, the Federal Reserve’s unique role in maintaining economic stability requires protection from arbitrary political interference. The Court ruled that Cook was entitled to basic due process—including an explanation of the evidence, a formal notice, and a meaningful opportunity to respond—and that because the administration failed to provide this, the attempted removal was “erroneous and void.” The dissent, led by Justice Clarence Thomas and joined by Justices Alito, Gorsuch, and Barrett, argued that the President’s Article II removal power should extend unconditionally to all independent agency heads.

Jerome Powell’s official chairmanship expired on May 15, 2026, but his 14-year term as a Fed Governor is legally protected until January 2028. To defend institutional continuity and preserve the Board’s legal defenses amid the ongoing Inspector General probe, Powell took the rare step of choosing to remain on the Board of Governors as a regular member. This decision triggered an immediate structural shift. To make room for Warsh’s confirmation to the Board, temporary Trump appointee Stephen Miran—a vocal monetary dove who had registered formal dissents in favor of interest rate cuts at six consecutive meetings—was displaced. With Miran off the Board and Powell remaining alongside Cook, Warsh must now build consensus within a deeply divided, legally embattled seven-member Board of Governors, navigating factions that are highly protective of Powell’s policy legacy.

Do you need to make better decisions when buying or selling stocks? Take a look at QuantX.es

The Federal Open Market Committee convened on June 16–17, 2026, for its inaugural session under Chairman Warsh. While the committee’s decision to maintain the federal funds target range at 3.50% - 3.75% was anticipated by Wall Street, the shift in operational style, post-meeting statement architecture, and communications philosophy took market participants by surprise. In a calculated break from two decades of hyper-detailed central bank communication, Warsh slashed the post-meeting statement to a mere 114 words. This was a reduction from the 244 words issued in April and historical averages of over 300 words. The radical brevity of the June statement accomplished three key objectives:

1. Omission of forward guidance: The statement removed all explicit qualitative and quantitative guidance regarding future interest rate trajectories, replacing references to “additional firming” or “gradual easing” with a blank slate.

2. Elimination of the participant roster: The customary list of meeting attendees and their individual voting records was removed from the primary statement, consolidating the policy decision under the corporate identity of the Committee.

3. Macroeconomic condensation: Rather than providing granular details on consumer spending, employment metrics, and business fixed investment, the document condensed the economic landscape into a single sentence, stating that economic activity is “expanding at a robust pace, and productivity and investment remain strong.”

Warsh has long been a vocal academic and professional critic of forward guidance, arguing that the practice has transformed the Fed from a reactive monetary authority into a captive of market expectations. In his view, the Fed’s 2021 - 2022 failure to contain inflation stemmed directly from its self-imposed forward-guidance trap, which made the central bank hesitant to raise rates for fear of triggering market volatility. By introducing “strategic ambiguity,” Warsh seeks to reintroduce two-way risk to financial markets. Under this philosophy, the Fed treats market pricing as an organic, independent source of information rather than a metric to be managed and massaged. The central bank communicates only its core objective—price stability—and leaves the precise path of interest rates intentionally vague, forcing market participants to conduct independent fundamental analysis. The policy statement is no longer used as a monetary tool in itself, but rather as a direct, unvarnished summary of current actions.

The macroeconomic backdrop in mid-2026 is challenging, exhibiting classic signs of stagflationary stress. Geopolitical escalations in the Middle East have severely disrupted global shipping lanes and energy infrastructure, pushing domestic Brent crude prices back above $100 per barrel. Consequently, the consumer price index climbed to a three-year peak of 4.2% in May 2026. The June Summary of Economic Projections reflected these mounting pressures. The core Personal Consumption Expenditures inflation forecast for 2026 was marked up significantly from 2.7% in March to 3.6% in June, while real GDP growth expectations were revised downward by 20 basis points to 2.2%.

In a symbolic protest against the forward-guidance utility of the Summary of Economic Projections, Chairman Warsh declined to submit his own interest rate projection for the Dot Plot. Warsh argued that the Dot Plot creates a false sense of certainty and encourages herd behavior among governors. By leaving his “dot” missing, he signaled that the Chairman of the Federal Reserve is not bound by a pre-committed path, thereby maximizing his tactical flexibility in upcoming meetings. Despite the Chairman’s absent dot, the remaining 17 participating officials revealed a deeply fractured committee struggling to find consensus. Nine of the eighteen participating officials now project a minimum of one 25 basis point rate hike before the end of the year, bringing the median year-end target to 3.75% - 4.00%. This is a sharp hawkish shift from the March 2026 projections, which had anticipated a median of 25 basis points of rate cuts by December. The remaining members are split between keeping rates unchanged and advocating for immediate cuts to protect the labor market from the compounding effects of high energy costs.

To reshape both the operational mechanics and the theoretical foundation of the Federal Reserve, Warsh announced the creation of five specialized internal task forces. Each group, which includes independent outside experts alongside Fed insiders to ensure better cognitive diversity, is mandated to deliver concrete, actionable policy proposals by December 2026:

1. Communications Reform Task Force: This group is tasked with formalizing the reduction of post-meeting press conferences from eight per year to four, limiting the volume of public speeches delivered by regional Fed presidents, and designing a framework to phase out the Dot Plot. The objective is to quiet the “noise” of public central bank commentary, allowing the markets to focus exclusively on formal policy decisions and get central bankers off the front page.

2. Balance Sheet Management (Quantitative Tightening) Task Force: This task force is exploring an acceleration of the Fed’s asset runoff program. The Federal Reserve’s balance sheet has declined from its pandemic-era peak of nearly 9 trillion to approximately 6.7 trillion (roughly 21% of GDP) by mid-2026. Warsh is an advocate of a “lean” balance sheet, arguing that the massive footprint of central bank holdings distorts market pricing, crowds out private capital, and disproportionately benefits owners of financial assets at the expense of broader society. The task force is evaluating plans to sell agency mortgage-backed securities outright to hasten the return to an all-Treasury portfolio.

3. Data Quality Metrics Task Force: Charged with evaluating the “data-dependent” framework of the Powell era, this group seeks to transition the Fed away from backward-looking, lagging indicators (such as the monthly establishment survey for employment) and toward real-time market signals. This includes the integration of high-frequency transactional data, corporate credit spreads, and market-implied inflation expectations into the Fed’s forecasting models.

4. AI and Productivity Dynamics Task Force: This task force is studying the supply-side impact of artificial intelligence integration across the economy. Warsh holds a structurally bullish outlook on technology, arguing that AI-driven productivity gains can increase the non-inflationary capacity of the economy. Under this theory, an elevated growth rate does not necessarily require a restrictive interest rate stance, as productivity gains naturally suppress unit labor costs. Understanding this dynamic is central to Warsh’s long-term plan to lower rates safely without triggering wage-price spirals.

5. Inflation Framework Review Task Force: This group is reviewing alternative inflation metrics to replace or supplement standard core PCE, which Warsh argues has been distorted by housing models and transient geopolitical spikes. The task force is focused on “trimmed mean” metrics, such as the Dallas Fed Trimmed Mean PCE and the Cleveland Fed 16% Trimmed Mean CPI. Because these metrics strip out extreme price outliers on both sides, they currently track below standard core PCE, providing what Warsh argues is a more stable measure of baseline inflation.

The introduction of “strategic ambiguity,” combined with the hawkish shift in the June Summary of Economic Projections, triggered immediate repricing across global financial markets. Following Warsh’s June press conference—during which he referenced the central bank’s commitment to “price stability” 12 times—investors adjusted their expectations for the interest rate path. The 2-year Treasury yield surged by 13 basis points, reflecting immediate expectations of a more restrictive short-term policy path and a potential rate hike by November. Meanwhile, the 10-year Treasury yield rose by just 3 basis points. This classic bear-flattening move indicates that while markets expect higher policy rates in the near term, they also expect these rates to successfully cool long-term growth and inflation. The US Dollar Index appreciated by 0.75%, reversing its losses from the prior week, while the S&P 500 retracted by 0.5%. The bond futures market now prices in an 85% probability of a rate hike by the close of 2026.

This hawkish pivot has created a direct conflict with the executive branch. President Donald Trump, who nominated Warsh with the explicit expectation that he would deliver lower interest rates to support his tariff and tax-cutting economic agenda, has expressed strong dissatisfaction with the Fed’s restrictive stance. This friction represents a critical test of central bank independence. In the 1970s, political pressure from the Nixon administration coerced Fed Chair Arthur Burns into maintaining artificially low interest rates, triggering a decade of stagflation. Academic economists and former Fed officials, including those represented by organizations like the Washington Center for Equitable Growth, have pointed to this period as a warning.

Some market commentators and progressive critics have proposed a more cynical interpretation of Warsh’s strategic shift. They suggest that the elimination of forward guidance and the heavy reliance on “strategic ambiguity” serve as a personal and political shield for the new Chairman. By refusing to commit to an explicit interest rate path, Warsh avoids direct, early confrontations. The Trump administration is permitted to believe he is a structural dove who is simply nudging a reluctant committee, while Wall Street can interpret his silence as orthodox, hawkish commitment to price stability. So long as he preserves this ambiguity, Warsh insulates himself from immediate political fallout while solidifying his control over the FOMC. However, this approach risks injecting significant term premiums and volatility into the fixed-income markets as investors lose their traditional policy guideposts and are forced to discover prices entirely on their own.

Japanese fiscal expansion and economic strategy (2026–2041)

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

From market observation to research question

The process begins with visible market behavior, but it becomes useful only when the researcher defines the phenomenon, classifies the market setting, identifies the mechanism, specifies the scope, and formulates a testable claim. Strong research moves from intuition toward measurable, comparable, and falsifiable market processes.

What’s inside:

1. Market observation. Trading ideas begin with repeated contact between price, volume, liquidity, time, events, and relative asset behavior.

2. Phenomenon classification. The researcher separates observations into single-instrument responses, delayed adjustments, event-driven reactions, and cross-sectional or relative-value behavior.

3. Reversal and continuation patterns. Price responses are studied through their lifecycle: initial pressure, dislocation, liquidity return, continuation path, or movement back toward balance.

4. Delayed adjustment. Some opportunities arise when one instrument reacts first and another follows later because of market clocks, venues, liquidity pools, or transmission delays.

5. Event-driven research. Earnings, macro releases, order-flow shocks, scheduled events, and surprise events create clean windows for studying price discovery, liquidity withdrawal, drift, dispersion, and repricing.

6. Cross-sectional and relative-value behavior. We study how dispersion, peer-group misalignment, and linked-instrument spreads reveal hidden capital migration and market friction beneath index-level returns.

7. Research question formation. A raw idea becomes research-ready when it defines the trigger, asset universe, market state, response window, expected path, and comparison benchmark.

8. Existence, conditional, and mechanism questions. Research asks whether the effect appears, when it strengthens or weakens, and why it exists through identifiable forces, transmission channels, and sustaining agents.

Sample

1.9MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. What is feature selection?

3. Risks and limitations of feature selection.

4. Feature selection as a research problem.

5. Filter-based feature selection.

   1. Information Gain.

   2. Chi-Square test (X²).

   3. Fisher score.

   4. ANOVA F-Test.

   5. Correlation with target.

   6. ReliefF.

   7. Minimum Redundancy Maximum Relevance.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Financial markets produce mountains of data, spanning simple price movements to limit order book dynamics. A common misconception assumes a larger dataset guarantees superior predictions. Reality proves the opposite. Excessive variables introduce noise, and invite to overfitting.

Every input added to a model represents a specific market hypothesis. Including a volatility metric implies price instability shapes future returns. Adding a spread feature connects liquidity to transaction costs. The selection process goes further than basic data preprocessing to become the core architectural decision of a trading system. The target variable dictates the necessary inputs, and aligning these inputs with actual market execution determines strategy success.

High-dimensional environments carry distinct dangers, exposing models to feature leakage, dimensional overreach, and selection instability. A feature might improve a statistical score in a backtest but erode capital in live trading due to increased turnover or delayed entries.

Go deeper in this topic by checking this presentation:

Feature Selection

16.7MB ∙ PDF file

Download

Download

Let’s outline the mechanics and limitations of distinct statistical filters, including Information Gain, Chi-Square tests, Fisher Scores, ANOVA, and Mutual Information. Each method offers a unique lens for evaluating data, from measuring entropy reduction to capturing non-linear dependencies.

What is feature selection?

Feature selection is the process of deciding which variables deserve to enter a model and which variables should stay outside it. For example:

1. A return feature embeds the claim that recent price movement contains information about future outcomes.

2. A volatility feature treats the magnitude and instability of price movement as relevant to the future distribution of returns.

3. A volume feature assumes that participation carries informational value.

4. A spread feature connects liquidity conditions to execution quality, transaction costs, and the practical usefulness of a signal.

5. A regime feature formalizes the idea that market behavior changes across states.

Therefore, feature selection gives structure to the model. It separates useful information from noise, redundancy, leakage, and unnecessary complexity. A model can receive hundreds of possible inputs, while only a smaller group contributes to consistent performance. Its purpose is to reduce the model’s information set to the variables that improve prediction, reduce instability, and improve the final performance.

The point here is to know which variables remain useful when the model trades under live conditions. A selected feature must contribute beyond other similar variables and survive different market periods. Let us start with the feature set X = {x₁, x₂, …, x_p}.

The problem can be stated simply. The original dataset contains (p) candidate features, each one representing a possible source of information for the model. Feature selection looks for a smaller subset S ⊂ X.

The objective is to choose (S) so the model performs better, generalizes better, and remains easier to monitor. In fact, performance must be measured through the full chain:

Then, a feature becomes valuable when it improves that chain. Conversely, a feature loses value when it improves a statistical score while weakening the final PnL. For example, a feature may improve directional accuracy from 51 percent to 52 percent. At first, that looks useful. Yet if the feature doubles turnover, increases transaction costs, and creates late entries, the strategy becomes weaker. Another feature may show only moderate predictive power, but it may reduce false entries during high-volatility periods. That feature becomes more valuable because it improves the strategy where losses usually appear.

This is why feature selection in follows a higher standard than ordinary data science. In many machine learning problems, the main question is predictive accuracy (that includes recall and F1). But in trading, the question is broader. The model needs useful predictions, stable signals, controlled turnover, and realistic execution across time.

Feature selection also protects the model from redundancy. Many variables express the same idea in different forms. A 5-day return, a 10-day momentum score, a moving-average slope, and a breakout distance may all describe directional pressure. Adding all of them can make the dataset look richer, while the model receives repeated exposure to the same underlying signal.

There are several common ways to select features. Low-variance filters remove variables that barely change. Univariate methods rank each feature by its individual relationship with the target. Recursive feature elimination repeatedly removes weaker variables from a model. L1 regularization pushes weak coefficients toward zero. Tree-based methods estimate which variables help nonlinear decisions. Sequential feature selection adds or removes variables based on their contribution to validation performance.

Feature selection is therefore both a modeling technique and a research control mechanism.

Subscribe now

Risks and limitations of feature selection

As stated earlier, a broader feature set gives a model a richer pool for describing market structure. It expands the number of ways in which the model can represent volatility, direction, liquidity, regime behavior, cross-asset transmission, and conditional response. However, every additional feature gives the model another opportunity to attach meaning to patterns that may belong only to the accidental geometry of the sample.

Each feature therefore carries a hypothesis, and a failure mode. Let’s see these costs:

1. The first risk is dimensional overreach. A model with too many variables can produce excellent in-sample performance while increasing out-of-sample overfitting. This risk grows when the research process tests many transformations of the same underlying idea. A dataset may contain RSI 7, RSI 14, RSI 21, stochastic oscillators, z-scored returns, rolling quantiles, rolling ranks, and several normalized momentum measures. On the research dataframe, this appears diversified. But in market terms, it may express the same directional claim several times under different names.

   This is one of the central traps of high-dimensional research. The number of variables grows faster than the number of independent market states. As the feature space expands, the probability of discovering a convincing but empty relationship increases.

2. The second risk is feature leakage. Leakage appears when a feature contains information that was unavailable at the moment of decision. This problem often appears in subtle forms. For example, a rolling statistic may include the current bar close while the simulated trade is placed at the open; or a volatility estimate may rely on high and low values that occur only after the entry decision.

   Feature selection magnifies this problem because selection methods reward the variables that appear most predictive. It improves the validation score, stabilizes the equity curve, and gives the model an artificial sense of intelligence. The problem is that this intelligence is simply overfitting.

3. The third risk is selection instability. A feature can rank highly in one period, lose relevance in another, and return later. This behavior reflects regime dependence, sample noise, and structural decay. In fact, it sits close to the boundary between usefulness and randomness.

   A feature that wins inside a backtest could lose practical value during live conditions. The apparent ranking of features can shift when the research process moves from statistical prediction to executable trading. A variable that improves a model score may still damage the strategy when placed inside the full execution chain.

4. The fourth risk is economic irrelevance. A feature can improve a statistical metric while weakening the actual trading system. It can raise accuracy from 51.0 percent to 51.5 percent and still reduce profitability when it increases turnover, or concentrates trades. In trading, every feature must be evaluated through its final contribution to the PnL because predictive usefulness alone is insufficient.

Feature selection is designed to reduce the model to useful information, yet the act of selecting features can itself become a source of bleeding money. The researcher uses feature selection to control complexity, but the selection process also needs its own control mechanism. Therefore, it requires more than ranking variables by historical usefulness. The feature set must remain coherent across time, robust under small perturbations, and aligned with the strategy’s market hypothesis.

Feature selection as a research problem

Feature selection can appear mechanical when it is presented as a catalogue of techniques. It creates the impression that a researcher can pass a dataset through a set of filters and obtain, at the end of the process, a smaller and more useful information set.

This impression is attractive because it transforms a difficult research decision into an apparently technical routine. However, the problem is a little bit more complex than that. Feature selection is a research problem because every feature represents a claim about future market behavior.

A lagged return claims that recent price movement contains useful information, while a volatility estimate claims that the current distribution of movement changes the next decision. There are tons of assumptions related to that, like those related to liquidity, cross-asset variable, regime variable and so on.

This changes the role of feature selection completely. The researcher is reducing the number of columns in a dataset and deciding what the model is allowed to observe. That choice defines the structure of the model before the model begins to learn. A feature set with many versions of the same idea gives the model a narrow view expressed through many columns.

The first major challenge is temporal order. Trading data arrives through time. The researcher stands at time t, observes only the information available at time t, makes a decision for time t+1 or later, and receives the result only after the market has moved and after trading costs have entered the calculation. This structure changes the entire selection problem.

A feature used at time t must belong to the information set available at time t. The structure can be written as:

The position then follows from the selected information:

The future outcome appears only after the decision:

where q_t​ is the position, r_t,t+h is the future return over the holding horizon, and C_t​ represents commissions, spread, slippage, latency, etc. This small structure captures the essence of the trading problem. The model receives information, takes exposure, and earns a net result. Feature selection must improve that entire chain. This is why time alignment becomes central. The method may look correct while the experiment becomes false. The correctness lives in the timing of information.

The second major challenge comes from the target. In many ordinary prediction problems, the target has a direct definition. In trading, the target is a design decision: next-bar return, next-day return, probability of a positive close, volatility-adjusted return, maximum favorable excursion, maximum adverse excursion, triple-barrier outcome, trend continuation, mean-reversion response, drawdown-conditioned payoff, or expected return after costs. And a pretty long list where each target asks the features a different question.

For this reason, feature selection has to start with the trading objective, not with the dataset. Predicting next-day direction requires one type of information. Sizing exposure from volatility-adjusted expected return requires another. Trend continuation depends on evidence of persistence, while mean reversion depends on evidence of displacement and exhaustion. The same column can carry alpha in one objective and become almost useless in another.

For example, realized volatility may rank highly when the target is absolute return y_t=|r_t+1|. The same feature may rank much lower when the target is signed return y_t=r_t+1

Feature correlation is also a major challenge. Different selection methods react to correlation in different ways. This creates a practical problem because the selected feature list can change across time even when the underlying market idea remains stable. One month the model selects a five-day return and another month it selects a moving-average slope. The names change, while the concept remains close to the same directional signal. A shallow reading calls the model unstable. But it is just feature-level instability—from concept-level stability.

Due to all of that financial datasets can look large while containing limited independent information. Intraday data may contain millions of rows, yet many of those rows repeat the same session structure. Daily data may cover decades, yet it still contains a limited number of cycles, liquidity contractions, and structural breaks. The number of rows can exaggerate the amount of independent evidence.

The researcher therefore needs two forms of data sufficiency:

• The first is observation count. The feature has to appear often enough for its relationship with the target to be estimated.

• The second is regime coverage. The feature has to experience enough different market states for its survival to be tested.

A feature that performs across calm sessions, volatile sessions, gap sessions, trend days, reversal days… carries stronger evidence than a feature that performs in one narrow environment.

Feature selection does not work in isolation. Its value depends on the model that reads the features and the trading objective they are meant to serve. Each method reveals a different part of the picture, and the decision becomes stronger when those signals are interpreted together rather than treated as separate answers.

A prediction eventually has to become an order. That transition changes the evaluation because a feature can improve a prediction metric while weakening the final strategy. For this reason, feature selection must be judged by its effect on realized trading outcomes.

Send me a copy

Filter-based feature selection

The curse of dimensionality presents a constant threat to model generalization. When designing trading models, researchers typically construct hundreds or thousands of candidate features from raw limit order book data, transaction flows, technical indicators, and macroeconomic series. Including redundant or uninformative variables accelerate overfitting, and degrades out-of-sample performance.

This is compounded by the low signal-to-noise ratio characteristic of financial markets, where genuine predictive signals are often buried under overwhelming market noise. If a model with high capacity (such as a deep neural network or a gradient-boosted tree) is fed an excessive number of noisy features, it will exploit spurious in-sample correlations, leading to capital loss during live deployment.

Filter-based methods are computationally efficient and model-agnostic. They evaluate the intrinsic statistical properties of individual candidate features relative to a target label—independent of the downstream training phase. This decoupling prevents the leakage of model-specific assumptions and acts as the first line of defense against data-snooping biases.

Information Gain

Read more

Table of contents:

1. Introduction.

2. Dilution analysis.

3. Income.

4. Balance.

5. Cash flow.

6. Ratios.

7. Segment.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

We are opening a new subsection inside MarketOps dedicated to small-cap opportunities. The goal of this series is to analyze companies that may be overlooked by the broader market but still present interesting financial, operational, or strategic characteristics for investors. This first report marks the beginning of that new series!!!

Rush Street Interactive, Inc. (ticker: RSI) is an online gambling and sports betting company operating within the Consumer Cyclical sector, specifically in the Gambling industry. RSI should be analyzed as a digital gaming operator whose value depends on its ability to grow revenue, retain users, improve margins, and manage dilution in a highly competitive and regulated market.

The company’s business is centered on online casino gaming, sports betting, and related digital wagering products. Unlike traditional casino operators, RSI does not depend primarily on physical casino properties, hotel capacity, or destination traffic. Its business model is more closely tied to digital platform performance, market access, customer acquisition, regulatory approvals, payment infrastructure, product experience, and player retention. This gives RSI a different investment profile from land-based gambling businesses.

The main attraction of RSI is its exposure to the continued legalization and expansion of online gambling. As more jurisdictions allow regulated online betting and iGaming, companies with existing platforms, licenses, and operating experience can expand their addressable market. Growth in this industry requires significant spending on marketing, promotions, compliance, technology, and customer support.

RSI’s investment case depends on four main areas:

1. Revenue quality.

2. Margin expansion.

3. Balance sheet strength

4. Dilution control.

Strong revenue growth is valuable only if it leads to better operating leverage and cash generation. Improving EBITDA is useful, but investors should also examine net income, operating cash flow, free cash flow, and the difference between adjusted and reported profitability.

Overall, RSI can be introduced as a small-cap digital gambling company with exposure to a structurally growing industry, but also with meaningful execution and capital structure risks. The company operates in an attractive market, but its investment value depends on whether management can convert market growth into sustainable profitability and improved per-share value. For that reason, RSI deserves a detailed analysis across dilution, income statement quality, balance sheet position, cash flow generation, operating ratios, and segment performance.

All information and data presented in this article have been sourced from the QuantX platform.

Dilution analysis

Rush Street Interactive’s dilution profile is a relevant part of the equity analysis because the company has a meaningful public share base and a visible increase in shares outstanding over time. The dilution panel shows 104.15 million shares outstanding and a float of 98.29 million shares, which means that most of the company’s share base is already available for public trading.

The first important observation is that RSI is classified with medium overall dilution risk. The share-count chart shows a clear upward trend from roughly 58 million shares in 2020 to 104.15 million shares currently. That increase is material from a shareholder perspective because the ownership base has expanded over the period shown.

The chart shows that RSI’s share count increases in a gradual way. A company can increase revenue, improve its operating footprint, and expand into new markets, but if the share count also rises materially, the economic benefit must be measured on a per-share basis.

At the same time, the dilution risk is partly offset by the company’s cash position. RSI’s data shows net cash per share of $5.68, estimated cash of $576.94 million, and quarterly burn of $20.11 million. These figures confirm that RSI is not under immediate financing pressure. The cash need indicator is marked as low, which is consistent with the idea that the company has enough liquidity to reduce the near-term probability of raising capital through additional equity issuance.

The cash position chart shows a sharp increase around 2020, followed by a multi-year decline and then a recovery into the most recent period. This pattern shows that RSI’s balance sheet has experienced both cash drawdown and cash rebuilding. The decline from the earlier peak indicates that cash has been consumed or redeployed over time, while the recent recovery suggests improved liquidity.

The quarterly burn of $20.11 million is also important. Even with a large estimated cash balance, burn rate must be monitored because persistent cash consumption can gradually change the dilution profile. At the current level shown, the company appears to have enough liquidity to absorb ongoing burn. If burn decreases as revenue scales and margins improve, dilution risk declines further. If burn remains persistent or increases, the company’s future need for capital becomes more relevant.

Offering ability is marked low, which indicates that the screener does not currently identify strong near-term offering pressure. Overhead supply is also marked low, suggesting that visible selling pressure from prior issuance or overhang is not considered severe by the screener. Historical dilution is marked low, although this should be interpreted carefully because the chart still shows a meaningful increase in shares outstanding over time.

The institutional ownership figure is also notable. Data shows 98.2% institutional ownership, which implies that RSI’s shareholder base is concentrated among institutions. This can be interpreted in two ways. On one hand, high institutional ownership can indicate that professional investors are involved in the stock. On the other hand, it can also make the stock more sensitive to institutional portfolio adjustments, block sales, or changes in sentiment. When institutional ownership is this high, investors should monitor ownership changes, insider activity, secondary offerings, and large holder transactions.

Short interest is shown at 9.2%. A short interest level of this size can indicate skepticism or hedging activity, but dilution risk depends more directly on the company’s share issuance behavior, cash burn, capital needs, and financing structure. For RSI, the short interest figure should be viewed as a secondary market signal rather than the main determinant of shareholder risk.

Overall, RSI’s dilution profile is mixed but positive. The company has a rising share count, which creates a real per-share issue. However, the current risk indicators are mostly low, and the cash position reduces immediate financing pressure.

RSI must show that growth translates into better per-share economics. The most relevant confirmation would come from stable or declining share count, improving net income, stronger operating cash flow, lower cash burn, and consistent growth in cash flow per share. If the company continues to grow while limiting new issuance, the dilution risk becomes manageable. If the share count continues to rise faster than profitability and cash generation, shareholder

Subscribe now

Income

The income statement shows a company with strong top-line growth and improving operating leverage. Total revenue increased from $262.4M in Q1 2025 to $370.4M in Q1 2026, representing approximately 41% year-over-year growth. Sequentially, revenue also increased from $324.9M in Q4 2025 to $370.4M in Q1 2026, showing that the company continued to expand beyond the previous quarter.

Operating income also improved significantly. RSI generated $42.8M in Q1 2026, compared with $14.6M in Q1 2025. That represents an increase of almost 193% year over year. This is a positive signal because operating income grew much faster than revenue, which indicates that the company is gaining operating leverage. Basically, RSI is converting a larger portion of its revenue into operating profit.

Cost of revenue increased from $170.9M in Q1 2025 to $238.2M in Q1 2026. This increase is expected given the higher revenue base, but it grew slightly slower than revenue. As a result, gross profit improved from $91.5M in Q1 2025 to approximately $132.2M in Q1 2026. Gross margin also improved slightly, from around 34.9% to 35.7%. This margin stability is important because it shows that RSI’s revenue growth is not being achieved through a major deterioration in direct profitability.

Marketing and advertising expense is especially important for an online gambling company because customer acquisition is one of the main drivers of both growth and risk. RSI spent $47.4M on marketing and advertising in Q1 2026, compared with $42.1M in Q1 2025. In absolute terms, marketing expense increased. However, as a percentage of revenue, it declined from 16.0% in Q1 2025 to 12.8% in Q1 2026. It indicates that RSI generated more revenue without increasing marketing spend at the same pace. From an investor’s perspective, this suggests improved marketing efficiency and better customer monetization.

General and administrative expense increased from $25.3M in Q1 2025 to $31.3M in Q1 2026. The increase is material, but it remained controlled relative to revenue growth. As a percentage of revenue, G&A declined from approximately 9.6% to 8.5%. This supports the same operating leverage argument. The company is scaling revenue faster than some of its operating cost categories.

Total costs and expenses increased from $247.8M in Q1 2025 to $327.6M in Q1 2026, an increase of 32%. Since revenue increased by 41%, the income statement shows a favorable spread between revenue growth and expense growth. This spread is what drives the improvement in operating income. This shows that the company’s growth is translating into better profitability.

Net income also improved. RSI reported $26.2M of net income in Q1 2026, compared with $11.2M in Q1 2025. That represents growth of 134% year over year. Sequentially, net income increased from $19.1M in Q4 2025 to $26.2M in Q1 2026. This confirms that the improvement is visible also at the bottom line.

However, the data also shows that not all net income belongs directly to common shareholders. Net income attributable to the parent was $9.1M in Q1 2026, while net income attributable to noncontrolling interest was $17.1M. RSI’s structure means that investors should continue separating consolidated net income from income available to common shareholders.

The Q2 2025 column contains unusual items that distort comparability. Other nonoperating income was negative $110.5M, while income tax expense showed a benefit of $115.0M. These two large items heavily affected pre-tax income and tax expense, making Q2 2025 less useful as a clean operating comparison. For that reason, Q1 2026 should be compared more directly with Q1 2025, Q3 2025, and Q4 2025 when analyzing recurring income statement performance.

Earnings per share is shown as $0 across the quarters. This means EPS is not very informative, likely because of rounding, share structure, or the relationship between consolidated income and income attributable to common stockholders.

Basic shares outstanding increased from 93.9M in Q1 2025 to 102.2M in Q1 2026, an increase of approximately 8.8%. This is relevant because the company’s profitability improved, but shareholders also experienced dilution. The key investor question is whether earnings and cash flow are growing faster than the share count. In Q1 2026, profit growth exceeded share count growth, but the share count still needs to be monitored.

Overall, RSI’s income statement shows a stronger operating profile than the prior year. Revenue growth is strong, operating income is expanding faster than revenue, marketing expense is becoming more efficient as a percentage of sales, and net income is improving. The main investor watchpoints are the split between consolidated net income and income attributable to common shareholders, the unusual nonoperating and tax effects in Q2 2025, and the continued increase in basic shares outstanding.

Balance

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Quality standards of a strong trading hypothesis

A strong trading hypothesis turns a market idea into an research object. It defines what is claimed, where the claim applies, how the effect appears in data, and how evidence improves the hypothesis over time. Precision, measurability, and falsifiability give the hypothesis structure and make the research process coherent, testable, and repeatable.

What’s inside:

1. Precision. A strong hypothesis defines the exact claim, the market setting, and the expected response.

2. Precise wording. Clear wording transforms broad ideas such as pressure, stress, momentum, or dislocation into specific market processes.

3. Precise scope. The hypothesis identifies the asset universe, market state, event type, time horizon, and environment where the effect belongs.

4. Precise outcome definition. The research defines the empirical object used to judge the effect, such as return, spread compression, rank shift, abnormal movement, path behavior, or persistence.

5. Measurability. The hypothesis enters the data through observable variables, classifiable states, and outcomes that can be scored consistently.

6. Observable variables and classifiable states. Market concepts such as crowding, hidden demand, liquidity pressure, or absorption become defensible proxies and coherent state definitions.

7. Falsifiability. The hypothesis exposes the claim to evidence by defining the conditions that weaken, contradict, or challenge the proposed mechanism.

8. Revision after contradiction. Evidence improves the hypothesis by narrowing the trigger, refining the state filter, adjusting the horizon, improving the outcome, or replacing the mechanism when needed.

Sample

1.92MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Where to find data?

3. Risks and web-scraping failure modes.

4. Dependency envelope and system boundary definition.

5. WebSocket transport.

6. TradingView message, session state, and symbols.

7. OHLCV and structured NumPy arrays.

8. Symbol acquisition loop.

9. Metadata normalization, and universe construction.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Algorithmic trading begins with a data question before it becomes a model question. Every signal, backtest, risk estimate, execution assumption, and portfolio rule depends on the historical record that feeds the system. A strategy is a function of its data. When the data source changes, the object under study changes with it.

Independent researchers face this problem under capital constraints. Market data subscriptions, platform limits, and research time compete for the same budget. Everybody knows how hard is to find a profitable strategy so traders may need data before premium infrastructure becomes affordable. However, the researcher needs enough historical evidence to test an idea before expanding spending.

Free data has value. It gives the researcher access, speed, breadth, and experimentation capacity. Its risk profile comes from the conditions attached to that access. Endpoint changes, rate limits, missing fields, symbol inconsistencies, session conventions, adjustment modes, and partial responses can enter the research process as hidden assumptions.

The practical answer is to stop treating acquisition as a casual download. This builds a small research stack around that idea. One part retrieves OHLCV bars from TradingView. The other builds the symbol universe that feeds those requests. The code stays close to the source, so the researcher can see how raw access becomes a structured object for testing.

From there, the workflow becomes research-ready. The researcher can trace the source path, preserve the request logic, validate the output, and understand which data conventions shape the sample before the first model decision occurs.

Where to find data?

The first question in algorithmic trading concerns data origin. The question comes before model choice. A trading strategy is a function over a historical record. When that record contain unstable fields, adjustment errors, or hidden provider changes, the strategy test becomes unstable.

Independent traders enter the problem from a cash constraint. They seek cheap data because the platform account, broker margin, hosting, and research time consume capital. Paying hundreds per month for a data subscription before a strategy proves itself creates an uncomfortable sequence. The market still imposes the same standard. A weak data feed can make a bad strategy look acceptable, make a decent strategy look broken, and hide a good idea under avoidable noise. Free data can be true. The core difficulty is that free data comes with weak evidence about error conditions.

Premium data reduces many operational risks and keeps responsibility on the researcher. A paid feed may provide service-level agreements, corporate-action files, delisting history, explicit licensing, historical corrections, and support channels. Those advantages have economic value because they reduce unknowns. The premium label still has to match the strategy. A daily equity strategy needs corporate-action logic. A futures strategy needs roll methodology. A foreign-exchange strategy needs session boundaries. An intraday strategy needs timezone, holiday, daylight-saving, and microstructure details. A paid vendor can deliver observations that are correct in format and wrong in use.

The trader needs research data before institutional data becomes affordable. The correct response is to treat free data as a constrained research source with documented guarantees and documented gaps. The trader gets data from an available source, then records what that source can guarantee and what it leaves uncertain.

Algorithmic trading contains many arguments about alpha built on casual data pipelines. Alpha is a small signal estimated inside a noisy environment. A two-basis-point edge can vanish through timestamp leakage, holiday mismatch, stale universe selection, or a split adjustment imported after the backtest export. The data layer defines the sample and which trades could have existed. When the sample is wrong, the alpha estimate describes a different object.

Risks and web-scraping failure modes

The uncomfortable truth is that a data pipeline is one of the sources of model risk. In algorithmic trading, the market data loader is part of the strategy, even when it looks like infrastructure. A bad entry signal is easy to blame. A silent data defect is harder.

Free data carries a different risk profile. The visible risks are rate limits, endpoint changes, missing values, duplicated bars, inconsistent symbol formats, unavailable historical depth, and provider-side throttling. Indeed, a free source can return adjusted close with raw open, split-adjusted prices with incomplete dividend treatment, or bars whose volume semantics differ across markets. It can omit delisted names, creating a survivorship-biased universe that looks stronger than the universe available at the time. It can make calendars look aligned because the stored sample contains successful requests and much more.

Let’s look at some of these risks in more detail:

1. The first limitation is protocol fragility. The script handles the WebSocket layer: socket creation, TLS wrapping, HTTP upgrade, WebSocket masking, ping/pong responses, frame parsing, TradingView packet packing, and ~m~<length>~m~<payload> message extraction. This is technically strong because it avoids third-party dependencies, but it also means the script depends on message conventions that may change without warning.

   A premium data vendor normally gives an official API contract: documented fields, deprecation policy, status pages, versioned endpoints, and support. A scraped or reverse-engineered feed gives none of that. If TradingView changes a header requirement, modifies the session protocol, renames a method, changes the packet envelope, throttles unauthenticated requests, or alters how heartbeats behave, the script may stop working.

2. The second limitation is rate limits, throttling and sampling bias. The ticker scanner already includes a pause argument between paginated requests. That is a design decision that reduces aggressive request behavior and lowers the probability of rate-limit problems. In fact, it can create sampling bias.

3. The final limitation is the most important, web-scraped or unofficially acquired data is usually not execution-grade by default. Execution-grade data needs stronger guarantees: timestamp precision, exchange-specific calendars, correction handling, corporate-action methodology, symbol mapping, delisting coverage, latency expectations, outage reporting, and licensing clarity. The current scripts are useful for acquisition and research, but they do not yet implement a full institutional data stack.

Subscribe now

Dependency envelope and system boundary definition

In a research stack, every dependency is implicit. A wrapper library may save time, but it can also hide protocol assumptions, retry behavior, schema changes, rate-limit responses, and timestamp conversions. In this case, we chose a narrow dependency envelope so the researcher can see what happens between the socket and the structured array.

The purpose of a minimal acquisition layer is to reduce the number of unknown transformations between the remote data source. In market data, unknown transformations are expensive:

• Did the wrapper adjust for splits?

• Did it include dividends?

• Did it localize timestamps?

• Did it align sessions?

• Did it forward-fill missing volume?

• Did it retry after a failed request and mix partial data into a final table?

The less visible the path, the more forensic work is required when a strategy behaves too well.

The header of the OHLCV script names the supported output. A dictionary mapping TradingView symbols such as NASDAQ:AAPL or TVC:VIX to structured NumPy arrays with datetime, open, high, low, close, and volume fields. That output is the first useful data model in the project. This makes the next stage of research straightforward because every downstream function can rely on named fields and stable numeric dtypes.

# SCRIPT 1 - OHLCV
TV_HOST = "data.tradingview.com"
TV_PATH = "/socket.io/websocket"
TV_ORIGIN = "https://www.tradingview.com"

Portability across vendors should be built at the interface, not faked inside the source adapter. This script is a source adapter. It should speak TradingView’s language and then emit a local object with a stable schema. The right boundary is therefore a small function that returns arrays in a controlled dtype. A separate layer can later convert those arrays into pandas, Polars, Arrow, Parquet, a feature store, or a database. Mixing the vendor protocol and the research table abstraction too early would make the system harder to test.

When a library wrapper fails, you often see only the wrapper exception. Here, the possible failures are closer to the actual layers—DNS or socket creation, TLS wrapping, WebSocket upgrade, frame serialization or a TradingView message. That is more work, but it gives a better map where debugging becomes easier.

The second script uses the same philosophy but applies it to ticker extraction. It imports json, time, urllib.request, and urllib.error. The scanner client therefore depends on the standard library rather than requests. The script wants to POST JSON to a known scanner endpoint, receive JSON back, and build dictionaries. Nothing about that requires a large HTTP stack. If later the user needs sessions, proxies, rotating headers, backoff middleware, caching, or async dispatch, those can be added.

# SCRIPT 2 - Tickers
import json
import time
import urllib.request
import urllib.error

TV_SCANNER_BASE_URL = "https://scanner.tradingview.com"

The data script starts when the universe is defined.

WebSocket transport

The WebSocket portion is the most technically interesting part of the OHLCV script. It opens a TCP connection, wraps it with TLS, creates a Sec-WebSocket-Key, writes the HTTP upgrade request, reads the upgrade response, and checks for the 101 status. Many researchers never see this layer because websocket-client or a vendor SDK does it for them. Here it is visible, and that visibility is useful because the data feed is a stream with a handshake, framing rules, liveness behavior, and failure states.

The function _ws_connect is the gatekeeper. If it fails, there is no chart session, no symbol resolution, and no bars. In production, this function is part of the availability budget. If a research script runs once per day, failure is annoying. If a scanner-plus-fetcher runs across thousands of symbols, even a small connection failure probability becomes a measurable drag on data completeness. A proper acquisition engine would count connection attempts, upgrade failures, elapsed time, remote closes, and timeout frequency. The current function raises clear errors, which is a good starting point.

def _ws_connect(timeout: int = 20):
    raw_sock = socket.create_connection((TV_HOST, 443), timeout=timeout)
    ctx = ssl.create_default_context()
    sock = ctx.wrap_socket(raw_sock, server_hostname=TV_HOST)

    key = base64.b64encode(os.urandom(16)).decode()
    request = (
        f"GET {TV_PATH} HTTP/1.1\r\n"
        f"Host: {TV_HOST}\r\n"
        "Upgrade: websocket\r\n"
        "Connection: Upgrade\r\n"
        f"Sec-WebSocket-Key: {key}\r\n"
        "Sec-WebSocket-Version: 13\r\n"
        f"Origin: {TV_ORIGIN}\r\n"
        "User-Agent: Mozilla/5.0\r\n"
        "\r\n"
    )
    sock.sendall(request.encode())

That makes the TradingView dependency obvious and exposes the browser-like headers. If TradingView changes accepted headers, this is where the break will happen.

The next important design point is frame masking. WebSocket clients must mask frames sent to the server. The _ws_send_text function constructs the frame header, chooses the correct payload-length encoding, generates a random four-byte mask, applies it byte by byte, and sends the final frame. This is the kind of code that looks low-level for a research workflow, but it is exactly the level at which many unofficial data clients break when protocol assumptions drift.

This implementation is compact and correct for basic text frames. It implements the WebSocket behavior needed for this research task. For a batch OHLCV downloader, that boundary is appropriate. The researcher understands that the code targets the data-download workflow rather than the full surface area of a general-purpose WebSocket library, including fragmentation, compression extensions, advanced close-code handling, and reconnect state.

def _ws_send_text(sock, text: str):
    payload = text.encode("utf-8")
    length = len(payload)

    header = bytearray()
    header.append(0x81)  # FIN + text frame

    if length <= 125:
        header.append(0x80 | length)
    elif length < 65536:
        header.append(0x80 | 126)
        header += struct.pack(">H", length)
    else:
        header.append(0x80 | 127)
        header += struct.pack(">Q", length)

    mask = os.urandom(4)
    masked_payload = bytes(payload[i] ^ mask[i % 4] for i in range(length))
    sock.sendall(header + mask + masked_payload)

The mask is required for client-to-server frames. If this function is wrong, the failure can look like a remote-side refusal even though the symbol, interval, and TradingView method are all valid.

The receive side is equally important. The function _ws_recv_text reads two bytes, extracts the opcode, checks the payload length, optionally unmasks the payload, handles close frames, responds to pings with pongs, ignores pongs, and returns text frames.

The script’s ping-pong handling is a strength because many quick data clients ignore it. In a cross-sectional downloader, slow symbols, large n_bars values, and network jitter make liveness important. The code treats the stream as a conversation rather than a one-way file download.

def _ws_recv_text(sock) -> str:
    while True:
        b1, b2 = _recv_exact(sock, 2)
        opcode = b1 & 0x0F
        masked = b2 & 0x80
        length = b2 & 0x7F

        if length == 126:
            length = struct.unpack(">H", _recv_exact(sock, 2))[0]
        elif length == 127:
            length = struct.unpack(">Q", _recv_exact(sock, 8))[0]

        mask = _recv_exact(sock, 4) if masked else None
        payload = _recv_exact(sock, length) if length else b""

        if opcode == 0x8:
            raise ConnectionError("TradingView closed the WebSocket.")
        if opcode == 0x9:
            _ws_send_pong(sock, payload)
            continue
        if opcode == 0x1:
            return payload.decode("utf-8", errors="ignore")

It suppresses control-frame noise and returns only text payloads to the TradingView parser. This separation keeps the higher-level OHLCV function focused on messages such as timescale update and series completed.

The current script raises or prints errors in sensible places, but a production research engine should turn those errors into events. A CSV or JSON log with symbol, interval, n_bars, error class, elapsed seconds, attempt number, and returned bar count would make data acquisition auditable.

TradingView message, session state, and symbols

Once the WebSocket is alive, the helper _tv_pack wraps a JSON payload inside the TradingView envelope format (a marker, the payload length, another marker, and the JSON body). The length prefix is key because TradingView can send multiple logical messages inside a single WebSocket text frame, and the client needs a deterministic way to separate them.

The _extract_tv_packets function is therefore a core parser. It scans the accumulated buffer for the envelope marker, reads the declared payload length, waits if the current buffer is incomplete, and returns complete packets with a residual buffer. This is the kind of logic that prevents subtle parse corruption. If the client assumes one socket frame equals one TradingView message, it will work until it does not. When it fails, the error will look random. A truncated JSON object, a missing series update, or a swallowed heartbeat. The script avoids that by treating the TradingView envelope as the true unit of protocol state.

def _tv_pack(method: str, params: list) -> str:
    payload = json.dumps({"m": method, "p": params}, separators=(",", ":"))
    return f"~m~{len(payload)}~m~{payload}"


def _extract_tv_packets(buffer: str):
    packets = []
    pos = 0
    while True:
        start = buffer.find("~m~", pos)
        if start == -1:
            return packets, ""
        len_start = start + 3
        len_end = buffer.find("~m~", len_start)
        if len_end == -1:
            return packets, buffer[start:]
        length_str = buffer[len_start:len_end]
        if not length_str.isdigit():
            pos = len_end + 3
            continue
        payload_len = int(length_str)
        payload_start = len_end + 3
        payload_end = payload_start + payload_len
        if len(buffer) < payload_end:
            return packets, buffer[start:]
        packets.append(buffer[payload_start:payload_end])
        pos = payload_end

The session layer begins with _random_session, which generates identifiers such as cs_xxxxxxxxxxxx. This is more than a convenience. TradingView chart sessions multiplex state. The downloader creates a chart session, resolves a symbol within that session, and creates a series attached to that symbol. The random suffix makes a session identifier collision unlikely, but the key point is conceptual. The requested bars are a stateful resource. The client builds a stateful context and then asks the server to stream a series update.

The main get_tradingview_ohlcv function sends messages in a sequence that makes sense, like set_auth_token, chart_create_session, resolve_symbol, create_series.

_tv_send(sock, "set_auth_token", ["unauthorized_user_token"])
_tv_send(sock, "chart_create_session", [chart_session, ""])

resolved_symbol = _symbol_payload(
    symbol=symbol,
    session=session,
    adjustment=adjustment,
)

_tv_send(sock, "resolve_symbol", [chart_session, "symbol_1", resolved_symbol])
_tv_send(sock, "create_series", [
    chart_session,
    series_id,
    series_id,
    "symbol_1",
    interval,
    int(n_bars),
])

The complete request goes beyond a ticker string. It includes a session mode, an adjustment mode, an interval, and a bar count. In our research, those fields define the dataset and belong with the downloaded array.

The symbol payload function deserves special attention because it exposes two parameters that directly affect conclusions—session and adjustment. Session defines whether the data uses regular or extended trading hours. Adjustment defines whether prices are adjusted for splits, dividends, or left raw. These choices are research-defining inputs. A moving-average strategy, volatility estimate, overnight return study, breakout rule, or volume filter can change materially depending on regular-session versus extended-session bars and split-adjusted versus raw prices.

The script defaults to regular session and splits adjustment. That is a defensible default for many equity research tasks because it avoids some discontinuities caused by stock splits while keeping pre-market and after-hours activity separate from regular bars. The convention still needs to be documented. When a strategy shows edge, the first reviewer should ask “edge under which session and adjustment convention?” A reproducible backtest records that answer clearly.

def _symbol_payload(symbol: str, session: str = "regular", adjustment: str = "splits") -> str:
    payload = {
        "symbol": symbol,
        "adjustment": adjustment,
        "session": session,
    }
    return "=" + json.dumps(payload, separators=(",", ":"))

Adjustment mode defines the price process being modeled. Let C_t​ be the close at time t. A split-adjusted close C_t^adj​ is a different stochastic process from a raw close C_t^raw​. A return model usually requires continuity through corporate actions, so it often uses adjusted data. An execution simulator uses raw tradable prices and handles corporate actions. The script can request different adjustment modes, so the research layer records the adjustment convention as metadata attached to each array.

The equation is simple, but it is enough to invalidate a backtest when ignored. A split in raw data creates a large negative return that does not represent an economic loss. A split-adjusted series removes that discontinuity for return research. Dividend adjustment introduces another explicit convention. The script’s flexibility is valuable because it supports these different conventions, and the metadata makes that flexibility reproducible. A useful extension returns both the structured array and a small metadata object containing source, symbol, interval, n_bars, session, adjustment, request_time_utc, and downloader_version.

Send me a copy

OHLCV and structured NumPy arrays

The parser _parse_timescale_update turns TradingView's timescale_update payload into a list of numeric tuples. The function expects bars with a vector v containing timestamp, open, high, low, close, and possibly volume. It skips bars with fewer than five values and sets volume to np.nan if volume is absent. This is one of the best practical decisions in the script. Missing volume remains missing rather than being converted to zero. Zero volume is a statement, missing volume is uncertainty. Preserving that distinction avoids a common retail-data mistake.

A zero-volume bar can mean no volume, depending on the instrument and venue. A missing volume field means the source did not provide the value in that message. Those are different observations. In algorithmic trading, volume feeds liquidity filters, execution-cost models, volatility scaling, breakout confirmation, and regime classification. Treating missing volume as zero can falsely classify a liquid symbol as untradable or distort turnover constraints. Using np.nan preserves the distinction and forces downstream code to decide how to handle the missing value.

def _parse_timescale_update(msg: dict, series_id: str):
    try:
        payload = msg["p"][1]
        series = payload.get(series_id, {})
        bars = series.get("s", [])
    except Exception:
        return []

    out = []
    for bar in bars:
        values = bar.get("v", [])
        if len(values) < 5:
            continue

        timestamp = float(values[0])
        open_ = float(values[1])
        high = float(values[2])
        low = float(values[3])
        close = float(values[4])
        if len(values) >= 6 and values[5] is not None:
            volume = float(values[5])
        else:
            volume = np.nan

The next function, _bars_to_array, defines the local schema. It creates a structured NumPy dtype with datetime64 seconds and float64 fields for open, high, low, close, and volume. The timestamp conversion uses integer seconds and casts to datetime64[s]. This is compact and efficient. It also avoids the overhead of a DataFrame at the source boundary.

A structured array is serializable, typed, compact, and easy to validate. DataFrames are excellent for research manipulation, but they can hide dtype drift. A structured array forces this schema. In a larger system, one could convert to a DataFrame after validation.

dtype = [
    ("datetime", "datetime64[s]"),
    ("open", "f8"),
    ("high", "f8"),
    ("low", "f8"),
    ("close", "f8"),
    ("volume", "f8"),
]

arr = np.zeros(len(bars), dtype=dtype)
arr["datetime"] = timestamps.astype("datetime64[s]")
arr["open"] = [b[1] for b in bars]
arr["high"] = [b[2] for b in bars]
arr["low"] = [b[3] for b in bars]
arr["close"] = [b[4] for b in bars]
arr["volume"] = [b[5] for b in bars]

The minimum validation layer around this array should enforce OHLC inequalities. For each bar, low should not exceed open, high, or close. High should not be below open, low, or close. Prices should be finite and positive for standard equities, indices, and spot FX rates. Timestamps should increase after sorting, unless the source returns reverse chronological data and the script normalizes it. The current code does not perform these checks, but it is okay for the moment.

A violated OHLC bar can break volatility estimates, range features, candlestick rules, stop-loss simulations, and intrabar execution assumptions. One corrupted high can create a false breakout. One negative close can explode logarithmic returns. One non-monotonic timestamp can produce lookahead when arrays are joined across assets. The array function should remain simple, but the caller should immediately pass the output through a validator.

The validation snippet looks like this:

Read more

Table of contents:

1. Introduction.

2. Inflation reopens.

3. Energy and chokepoints.

4. Global trade.

5. U.S.–China.

6. Europe’s energy decline continues.

7. East mediterranean gas and the new energy geography.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Growth remains alive, yet price formation has moved from demand cycles into shipping lanes, fuel markets, mineral chains, defence budgets, and regulatory systems. The global economy still expands, but each point of growth now carries a higher security cost.

Today we review the pressure points that define the current environment. Inflation has reopened through energy, fertilizers, freight, and expectations. Hormuz has become a macro variable because oil, LNG, refined products, and insurance prices move through the same corridor. Global trade has entered a selective phase, where AI infrastructure expands while tariffs, standards, and compliance rules reshape market access.

To go deeper you can check this:

EIA

4.39MB ∙ PDF file

Download

Download

Global Trade Update

11.1MB ∙ PDF file

Download

Download

Gas Market Report, Q2 2026

2.19MB ∙ PDF file

Download

Download

The U.S.–China relationship sits at the centre of this structure. Both powers use the same channels for commerce and leverage: rare earths, chips, agriculture, Treasury holdings, industrial capacity, and maritime routes. Europe faces a harsher version of the same problem. Its energy model has shifted away from Russia, but the replacement system depends on LNG, external suppliers, cloud platforms, AI hardware, and fiscal room that bond markets continue to test.

The East Mediterranean has gained strategic weight within this context. Gas fields, pipelines, LNG terminals, electricity cables, hydrogen plans, and maritime claims now form part of a larger energy map shaped by Gulf disruption and European supply risk. The region offers optionality, but also concentrates infrastructure, diplomacy, and security inside one contested space.

Subscribe now

Inflation reopens

The central feature of this macro regime is the coexistence of positive growth with a renewed inflation impulse. The IMF places global growth at 3.1% in 2026 and 3.2% in 2027, below the recent pace of about 3.4% seen in 2024 and 2025 and below the 3.7% average recorded during 2000–2019. Global headline inflation is expected to rise to 4.4% in 2026 before easing to 3.7% in 2027, which turns the 2026 cycle into a test of disinflation credibility rather than a clean continuation of the previous normalization path.

The Middle East war has changed the macro baseline because the shock enters through commodity markets, inflation expectations, transport costs, and financial conditions at the same time. The IMF states that, absent the war, its 2026 growth forecast would have received a slight upward revision to 3.4%. The downward move to 3.1% therefore comes from the conflict shock, partly offset by recent strong data and lower tariff rates. The current slowdown reflects an external constraint on supply and confidence rather than a broad collapse in private demand.

The baseline also hides a wide dispersion across economies. Advanced economies show a more contained revision, while commodity-importing emerging and developing economies carry the largest pressure. These countries face a triple squeeze from higher energy bills, weaker external balances, and tighter financing conditions. The IMF’s downside scenarios show the scale of that vulnerability. Under an adverse path with larger and more persistent energy-price increases, global growth falls to 2.5% in 2026 and inflation reaches 5.4%. Under a more severe path with greater damage to regional energy infrastructure, global growth falls near 2%, while headline inflation moves above 6% by 2027.

The OECD reinforces this picture with a slightly weaker growth profile. It projects global GDP growth at 2.9% in 2026 and 3.0% in 2027. The same outlook shows that the energy-price surge and the Middle East conflict raise costs and lower demand, while strong technology-related investment, lower effective tariff rates, and the momentum inherited from 2025 provide support. This creates a two-engine global economy. One engine is constrained by fuel, freight, insurance, and import costs.

Early 2026 activity data show that the world entered the shock with strength across both services and manufacturing. The OECD notes that global growth remained solid in 2025, expanding at an annualized pace of about 3.25% in the second half of the year. Private consumption and investment were major drivers across many economies, while technology-related business investment carried the U.S. expansion through tariff headwinds. In Asia and the United States, technology-related industrial production continued to rise in the opening months of 2026, supported by stronger capital-expenditure intentions among large listed technology firms.

This detail changes the interpretation of the global cycle. The world is dealing with a supply shock layered on top of an investment upswing. The AI capital cycle supports manufacturing, electricity demand, high-end components, and corporate spending. The energy shock pushes in the opposite direction by compressing margins, lowering household purchasing power, increasing transport costs, and raising the hurdle rate for investment in energy-intensive sectors. The result is a macro regime where the same economy can show strong capex in strategic technology and weak real-income momentum in consumer-facing sectors.

Inflation has reopened through the energy channel. The OECD expects G20 inflation to rise from 3.4% in 2025 to 4.0% in 2026, then ease to 2.7% in 2027. In advanced G20 economies, headline inflation rises from 2.5% to 3.5% in 2026, while core inflation stays around 2.6% before easing in 2027. In the United States, higher energy prices more than offset the reduction in effective tariff rates, lifting projected headline inflation from 2.6% in 2025 to 4.2% in 2026.

The World Bank projects overall commodity prices to rise 16% in 2026, driven by energy, fertilizer, and several key metals. Its energy price index is forecast to rise 24% in 2026, reaching its highest level since the 2022 energy shock. The same forecast describes the 2026 energy move as a near 40% shock relative to January expectations. This means the macro forecast now depends on physical routing, spare production capacity, strategic reserves, and conflict duration.

The WTO projects a slowdown in global trade growth during 2026, while AI-related goods create an offsetting source of demand. Continued strength in AI-enabling goods trade could add 0.5 percentage points to world merchandise trade growth in 2026. This turns trade into a split system. Strategic technology flows remain strong, while energy-linked freight costs, security risks, insurance premia, and compliance barriers weigh on broader merchandise movement.

UNCTAD reports that non-tariff measures now impose higher export costs than tariffs for 88% of countries. In 2025, tariffs on exports rose 10% for developed economies, 16% for developing economies, and 18% for least-developed economies. For smaller exporters, technical regulations, certification requirements, sanitary rules, and product standards have become a larger obstacle than headline tariff rates.

Fiscal policy sits under pressure from the same shock structure. Finances entered 2026 with weak debt dynamics, while the Middle East war added a new source of pressure through energy disruption, tighter financial conditions, and demands for household support. Governments now face a narrow balance between cushioning the energy shock and preserving fiscal credibility. This is one reason the macro regime has become more state-driven. Defence spending, energy subsidies, industrial policy, reconstruction finance, and debt service all compete for the same fiscal space.

The result is a global economy that still expands, but expands through a narrower and more political channel. Growth depends on AI investment, defence production, energy substitution, and public-sector intervention. Inflation depends on chokepoints, commodity flows, and expectations. Trade depends on standards and strategic access. Financial stability depends on the capacity of bond markets to absorb larger public-sector needs. May 2026 therefore marks a shift from a classic business-cycle framework toward a security-constrained macro framework, where physical infrastructure, military risk, energy routing, and fiscal credibility determine the path of output and prices.

Energy and chokepoints

The Strait of Hormuz handled around 20 million barrels per day of crude oil and oil products in 2025, close to 25% of world seaborne oil trade. The route also carried around 110 bcm of LNG, equal to almost one-fifth of global LNG trade. Qatar and the UAE sit at the centre of this exposure because around 93% of Qatar’s LNG exports and 96% of UAE LNG exports passed through Hormuz in 2025. This turns a local maritime disruption into a world pricing event because Asian utilities, European importers, petrochemical firms, airlines, refiners, and shipping operators compete for the same displaced molecules and cargoes.

Saudi Arabia and the UAE hold the main operational bypass capacity through pipelines that can redirect crude flows toward the Red Sea and Fujairah. The available rerouting capacity is estimated between 3.5 million and 5.5 million barrels per day, which covers only a fraction of normal Gulf export flows. Iran, Iraq, Kuwait, Qatar, and Bahrain remain tied to the Strait for most oil exports. The shock therefore prices the difference between gross energy supply and deliverable energy supply. A barrel inside the Gulf has a market value only when it can reach a buyer, pass through insurance, secure shipping, clear port risk, and arrive inside a refinery schedule.

The gas shock has a separate macro channel because LNG markets have thinner buffers than oil markets. The IEA estimates that transit disruption through Hormuz has reduced LNG supplies from Qatar and the UAE by more than 300 million cubic metres per day since 1 March, equal to more than 2 bcm per week. Ras Laffan, the world’s largest liquefaction facility, has been offline since the 2 March attack, while associated gas output has also been affected by oil-field shut-ins. This creates pressure on Asian spot LNG prices, European gas benchmarks, power prices, fertilizer costs, and industrial input prices.

Asia carries the largest direct import exposure. About 80% of oil and oil products moving through Hormuz in 2025 went to Asia, and almost 90% of LNG volumes exiting the Persian Gulf through the Strait also went to Asia. China, India, and Japan are among the main crude importers exposed to the route, with China and India together receiving 44% of crude exports passing through Hormuz. This gives the conflict an Asian growth channel. Energy-importing economies face higher import bills, weaker terms of trade, margin compression, and pressure on administered fuel systems.

Europe faces a different exposure. Its direct share of Gulf crude flows is smaller, with around 4% of the region’s crude routed into Europe, yet the price effect reaches Europe through global Brent pricing, LNG competition, electricity markets, aviation fuel, fertilizer, and industrial chemicals. The ECB describes the shock as a negative supply shock for the euro area because Hormuz affects LNG, refined products, aluminium, helium, sulphur, and fertilizers. These commodities enter the cost base of electricity, aircraft fuel, food production, manufacturing, construction, and strategic industries.

The inflation channel is visible in Europe’s April data. Euro area headline inflation rose to 3% in April, driven by a 10.9% increase in energy prices, while inflation excluding energy fell to 2.2%. This composition matters because the shock comes from imported energy and critical inputs rather than from broad domestic overheating. The ECB also reported longer delivery times, rising input costs, and supply shortages, with the latest observations for April 2026. That combination creates a policy dilemma because monetary policy can anchor expectations, while the physical shortage originates outside the domestic credit cycle.

The oil-price path remains the central market gauge. Brent averaged was $103 per barrel in March and expects a peak near $115 per barrel in the second quarter of 2026. The same outlook keeps a risk premium through the forecast period because supply disruption uncertainty keeps prices above pre-conflict levels. The forecast sees Brent moving below $90 in the fourth quarter of 2026 and averaging $76 in 2027, based on a gradual return of flows and lower shut-ins. This makes the forecast path contingent on shipping access, infrastructure damage, spare capacity, inventories, and diplomatic sequencing.

The Brent-WTI spread shows the geographic nature of the shock. The EIA states that the spread averaged $12 per barrel in March and peaks near $15 per barrel in April because Brent carries greater exposure to Middle East flows, shipping costs, and Asian consuming markets. This spread turns the conflict into a regional balance-sheet issue. U.S. refiners, European buyers, Asian utilities, Gulf exporters, and tanker owners experience the same conflict through different price curves.

The World Bank projects energy prices to rise 24% to their highest level since 2022. Overall commodity prices are forecast to rise 16%. Fertilizer prices are projected to increase 31%, driven by a 60% jump in urea prices. This links the Hormuz shock to food affordability because natural gas is a key feedstock for nitrogen fertilizers, while transport fuel affects every stage of agricultural logistics. The World Bank warns that prolonged conflict pressure could push up to 45 million more people into acute food insecurity.

The latest monthly commodity data reinforces the same direction. The World Bank’s May commodity update shows the energy price index rising 12.1% in April, driven by crude oil gains of 8.7%. Food prices edged up 1.5%, fertilizer prices jumped 14%, raw materials rose 2.5%, and metals gained 1.4%. These moves show that the energy shock has started to move across adjacent commodity baskets rather than staying isolated inside crude oil.

Strategic reserves can smooth the shock, but they cannot recreate Gulf export capacity. The market now prices the speed at which governments release reserves, the capacity of refineries to adapt crude slates, the availability of tankers, the cost of war-risk insurance, the integrity of pipelines, and the ability of Gulf producers to sustain production under threat. Saudi Aramco is warning that the oil market could lose around 100 million barrels per week if the disruptions continue at the current rate and Hormuz remains closed. This figure gives the shock a weekly depletion rhythm that matters for inventories, refinery planning, and political pressure.

Global trade

The global trade regime has moved into a split structure. One side carries the weight of tariffs, energy shocks, insurance costs, route risk, sanctions screening, and regulatory barriers. The other side carries the force of the AI capital cycle, where chips, servers, networking equipment, electrical systems, cooling units, data-centre components, and advanced manufacturing inputs keep crossing borders because artificial intelligence requires a large physical supply chain.

Merchandise trade volume is projected to fall from 4.6% growth in 2025 to 1.9% in 2026, then recover to 2.6% in 2027. Services trade volume is projected to ease from 5.3% in 2025 to 4.8% in 2026, then rise to 5.1% in 2027. Goods and services trade grew around 4.7% in 2025, above world GDP growth of 2.9%, while 2026 places trade growth near GDP growth. This means trade has shifted from an expansion phase to a filtering phase, where strategic sectors expand while broad merchandise flows face macro and policy friction.

The trade rebound came from two sources with different political meanings. First, companies brought imports forward ahead of expected tariff changes. Second, AI-related demand created a hardware boom across semiconductor supply chains, server systems, and electronic components. That combination lifted trade before the full impact of new policy barriers entered the system. The frontloading effect fades, while AI hardware demand remains a support factor. The result is a trade cycle where last year’s strength creates this year’s comparison problem, and where the quality of trade growth matters more than the headline number.

AI has become the strongest structural trade engine because it transforms digital demand into physical shipments. AI-enabling goods trade reached $4.18 trillion after 21.9% annual growth. These goods accounted for 42% of global trade growth, despite representing a smaller share of total merchandise trade. Asia remained the global hub with 62% of AI-related goods trade, while North America became the fastest-growing market and accounted for 16% of the total. This structure gives AI a geopolitical footprint because the model layer depends on the chip layer, the chip layer depends on capital equipment, and the equipment layer depends on minerals, power systems, logistics, and standards.

The earlier trade model rewarded low-cost production and efficient container movement. The current model rewards secure access to compute inputs, energy, advanced components, trusted suppliers, and export licenses. A server shipment now reflects a chain of permissions. It requires chips, memory, networking systems, cooling equipment, energy infrastructure, cybersecurity rules, and destination controls. Trade remains open in aggregate, while the strategic layer becomes more selective.

The tariff layer adds uncertainty to that selective structure. The OECD estimates that changes in U.S. bilateral tariff rates lowered the effective tariff rate on U.S. merchandise imports to 9.9%, compared with the 14% assumption used in its outlook. The same assessment describes a new 10 percentage point U.S. tariff applied across countries, with exclusions for sector-specific goods and exempted products, including many pharmaceuticals and semiconductor-related goods. This gives companies a changing tariff map where country exposure, product classification, exemption status, and origin rules shape sourcing choices.

The tariff map has a second effect, it extends planning risk. Firms can reroute supply chains around a tariff, but repeated rule changes alter inventory strategy, contract length, supplier selection, customs classification, and hedging costs. Trade policy now affects the working-capital cycle of companies. Importers carry more inventory when they expect border costs to rise. Exporters diversify buyers when access terms become unstable. Multinationals split production when rules of origin become strategic.

Regulatory barriers have become the deeper trade constraint. Tariffs rose by 10% for developed economies, 16% for developing economies, and 18% for least-developed economies. Yet regulatory trade measures now impose higher export costs than tariffs for 88% of countries. These measures include technical regulations, health and safety requirements, certification procedures, administrative rules, and product standards. They determine which firms can access a market, which goods qualify, and which producers can absorb the compliance cost.

This shifts the centre of trade policy from border taxes to rule architecture. A tariff is visible at the border. A product standard sits inside the production process. A certification rule affects testing capacity. A cybersecurity requirement affects software design. A carbon rule affects energy sourcing. A data rule affects cloud architecture. A rule-of-origin test affects supplier geography. For large firms, these rules become manageable costs. For smaller exporters, they become market-entry barriers.

Data shows the development impact of this rule-based trade system. Least-developed economies lose around 10% of their exports to G20 markets because firms lack the capacity to meet regulatory requirements. Smaller exporters face higher costs when local testing and certification facilities are scarce, since products require external verification before market entry. This creates a trade hierarchy based on compliance capacity. The firm with legal teams, testing access, documentation systems, and certification budgets enters the market. The firm with production capacity alone faces exclusion from high-value markets.

Regulatory convergence offers one of the largest trade gains inside the current system. Rules or recognizing standards can reduce regulatory-measure costs by 15% to 30%. In Africa, limited cooperation could reduce costs by 30% to 40% in sectors such as agriculture and manufacturing. This shows that trade expansion depends less on classical tariff cuts and more on mutual recognition, shared testing systems, transparent documentation, and regional standards architecture.

A country or bloc that defines safety rules, digital rules, carbon rules, product rules, payment rules, and certification rules can shape the direction of trade. The largest markets gain leverage because exporters adapt to their requirements. This creates a form of regulatory gravity. Producers build around the rulebook of the market they want to enter. Trade flows follow demand, but production systems follow standards.

The AI trade boom intensifies this regulatory gravity. AI hardware crosses borders, but each shipment sits inside controls on chips, cloud access, cybersecurity, data governance, model safety, export licensing, and critical minerals. The value of trade has moved toward goods that carry strategic meaning. A chip is a commercial product, a defence input, a data-centre component, a productivity asset, and a geopolitical lever. A server rack is infrastructure for commerce, surveillance, science, finance, and military logistics.

China’s role in this structure is central because manufacturing depth, mineral processing, component scale, and export capacity give it leverage across the trade stack. China uses rare earths and bilateral commercial agreements as tools of bargaining, while stressing that China’s financial strength still depends on exports and dollar inflows. That combination creates a system of conflict inside dependence. China gains leverage from supply-chain concentration. The United States gains leverage from demand, finance, technology standards, and security alliances. The trade regime becomes a negotiation over who controls the bottleneck.

For Europe, the same regime creates a strategic dilemma. European firms need AI hardware, cloud infrastructure, energy inputs, and export access. European regulators want sovereignty through rules, but the physical supply chain for AI and the energy shock sit outside Europe’s full control. This gives Europe strong influence over market access and weaker influence over upstream technology capacity. The result is a continent that can write standards, enforce compliance, and shape consumer-market rules, while depending on external actors for several inputs that define the next productivity cycle.

The investment implication is that trade exposure must be read through bottlenecks rather than through country labels alone. The relevant question is where a company sits inside the chain of controlled inputs, certification rules, tariff exposure, energy intensity, and AI infrastructure demand. A firm tied to AI hardware can benefit from the capex boom while facing export-license risk. A firm tied to consumer imports can face margin pressure from tariff shifts and freight costs. A firm selling into regulated markets can face growth limits from certification costs. Trade beta has become rule beta, energy beta, and bottleneck beta.

U.S.–China

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Internal anatomy of a good hypothesis

A genuine hypothesis has an internal structure. It defines when the claim begins, what market response should appear, and why that response should emerge. Its value comes from connecting activation, response, mechanism, and evidence into one coherent research object that can be tested, challenged, refined, and extended.

What’s inside:

1. Internal anatomy of a good hypothesis. The lecture presents a hypothesis as an organized structure whose parts must fit together.

2. The condition that activates the claim. A hypothesis begins by identifying the market state in which the effect should appear.

3. Trigger variable. The trigger translates pressure, dislocation, surprise, imbalance, or regime transition into a variable that can select relevant episodes.

4. State filter. The state filter places the trigger inside the right market environment.

5. Eligibility rule. The eligibility rule protects the empirical sample and decides which observations belong to the same research object by controlling for trading session, liquidity, data quality, structural breaks, and special distortions.

6. The response that defines the effect. A hypothesis must specify what should happen after activation.

7. Sign, magnitude, and timing. Sign defines direction, such as continuation, reversal, convergence, or divergence. Magnitude defines economic substance after costs and frictions. Timing defines the internal path of adjustment after the trigger appears.

8. The mechanism that explains the effect. The final part explains why the trigger should lead to the response.

Sample

1.96MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Risks and transformation limitations.

3. Processing text and features related to text.

4. Lexical canonicalization as a trading signal operator.

5. Token boundary specification and market microstructure semantics.

6. Stopword removal, negation retention, and signal integrity.

7. Lemmatization, stemming, and contextual LLM preprocessing.

8. Sparse and dense feature construction for event-driven alpha.

9. Alignment, and log-template parsing.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Between a market event and a model-generated trading signal exists a sequence of transformations that determines what the model is permitted to observe. Standard literature describes this sequence as text preprocessing. Tokenization partitions a character stream into discrete units. Normalization maps heterogeneous strings into canonical forms. Stopword removal deletes tokens according to frequency-based exclusion lists. Stemming and lemmatization compress morphological variants into common representations. Multiword grouping binds adjacent tokens into higher-order semantic objects. In conventional natural language processing, these operations are often treated as preliminary cleaning procedures. In algorithmic trading, they are active signal-shaping mechanisms.

Trading model receives the version of language produced by the preprocessing layer. A headline, filing paragraph, policy statement, earnings transcript, or execution log become usable information only after a pipeline decides which words survive, which strings are merged, which entities are protected, which symbols are erased, and which boundaries define a tradable event. Every rule in that pipeline determines what the classifier can see and what it is forced to ignore.

The dilemma materializes when quantitative researchers treat these transformations as benign standardization rather than active parameterizations of the feature space. A text feature is realized when a preprocessing policy determines the semantic boundary under which that word becomes measurable. Splitting, merging, deleting, lowercasing, masking, or embedding a token changes the geometry of the model input. It alters the sparsity of the design matrix, changes the relationships among features, shifts the timing of event triggers, and reweights the probability of a future market move conditional on the processed text.

Consider the ingestion of a Federal Open Market Committee press release. The raw document is a non-stationary sequence of characters containing policy language, forward guidance, inflation references, balance-sheet terminology, and deliberately controlled ambiguity. If the tokenizer splits on hyphens, the phrase “mortgage-backed” becomes two separate objects rather than one economically coherent instrument descriptor. If a stopword filter removes “not” or “without,” a restrictive sentence may be pushed toward the same representation as an accommodative sentence. If a normalization rule lowercases all symbols, a ticker, acronym, institution, and ordinary noun may collapse into a single token.

The same problem appears in higher-frequency environments. A headline, order-routing message, or execution log may be transformed within milliseconds of arrival. A small parsing decision can determine whether “USD/JPY” remains a currency pair or becomes two unrelated tokens, whether “5.2%” remains a magnitude or becomes an unanchored digit, whether “Order rejected: Code 404” remains distinct from “Order routed: Latency 404ms.” When the preprocessing layer destroys these boundaries, the downstream model may still behave correctly according to its training objective, but it is acting on a distorted version of the event.

Do you want to know more about this? Check that:

How does a text preprocessing pipeline affect ontology matching?

3.08MB ∙ PDF file

Download

Download

Preprocessing is all you need

324KB ∙ PDF file

Download

Download

Today we argues that text preprocessing in trading systems must be evaluated as part of the signal-generation process rather than as a detachable data-cleaning stage. The relevant question is whether it preserves economically meaningful distinctions under the latency, auditability, and chronological constraints of live trading. A transformation that improves linguistic compactness can still generate negative alpha if it deletes directional modifiers, collapses distinct issuers, introduces future information, or destabilizes the feature basis across time.

Risks and transformation limitations

When preprocessing is decoupled from the objective function of the trading strategy, four structural failure modes emerge. The first risk is semantic inversion. This occurs when a transformation rule reverses the directional polarity of the information. The deletion of negation markers or contrast conjunctions maps risk-reducing statements to risk-escalating feature vectors. In credit trading, the semantic distance between “will default” and “will not default” is the difference between a short and a long position. A generic stopword filter that removes “not” projects both text inputs onto the exact same coordinate in the feature space, forcing the model to calculate an expected return based on a corrupted prior.

The second risk is unstable representation. A fixed concept receives disparate numerical encodings across time intervals or data sources because the preprocessing logic lacks invariance to input formatting. Data vendors frequently update their schema. A news feed might transition from capitalizing standard tickers to formatting them in lowercase within brackets. If the canonicalization logic relies on strict case-matching without entity recognition fallback, the historical feature vector for that asset decays to zero, while a spurious new feature dimension begins accumulating frequency. The model observes this as a sudden regime shift in the underlying asset when it is a mechanical artifact of the data handler.

The third risk is false alignment. Two distinct economic entities or states are collapsed into identical token representations. Stemming protocols force independent market concepts to share an etymological root, inflating feature counts and degrading the precision of the classifier. Truncating “organization” and “organic” to the root “organ” collapses corporate structural news into agricultural commodities data. The classifier inherits a dense, overlapping feature representation that dilutes the predictive power of both original terms.

The fourth risk is operational opacity. Algorithmic systems generate execution and state logs, often communicating via standard protocols such as FIX. Preprocessing frameworks convert these continuous text streams into discrete templates for anomaly detection and latency monitoring. Lossy transformations mask critical dynamic variables, rendering the reconstructed event templates insufficient for post-trade attribution or incident resolution. If a regular expression intended to mask order quantities accidentally masks routing destination tags, the quantitative team loses the ability to diagnose venue-specific slippage.

The key problem for redefining text transformations is a live trading drawdown linked to an information extraction failure. A sentiment classifier parameterized to trade sovereign policy headlines initiated long positions following restrictive policy announcements and short positions following neutral macroeconomic updates. Post-trade analysis isolated the failure to the data transformation layer. A static text processing rule removed contrast terms, reduced specific sovereign entities to generic geographic tokens, and applied morphological stemming that equated distinct central bank actions.

During a critical trading session, a headline reading “Central Bank pauses rate hikes, despite inflation pressures” was ingested. The preprocessing layer stripped “despite” and stemmed “hikes” and “pauses”. The resulting token array fed to the support vector machine lacked the logical dependency structure of the original sentence. The model output a high-confidence positive sentiment score, triggering a large, unhedged long position in the sovereign bond market seconds before a massive sell-off.

The classification algorithm executed correctly given the input vector. The input vector misrepresented the market event because the text transformation policy was optimized for corpus reduction rather than economic fidelity. The post-diagnostics proved that the strategy’s negative alpha was generated within the first twenty milliseconds of the text handling pipeline. This event forces the next question. Must text processing remain rigid, deterministic, and auditable, or should it become contextual, adaptive, and reliant on large language models?

Subscribe now

Processing text and features related to text

The prevailing workflow in quantitative text analysis isolates natural language processing from financial modeling. Text is ingested, cleaned, and vectorized using generic linguistic conventions before the quantitative researcher trains a predictive model. This separation is flawed. Let the raw text at time t be x_t, and let the preprocessing policy be P. The prediction model f does not observe x_t but z_t = P(x_t). The trading signal is s_t = f(P(x_t)). Therefore, P is a functional operator embedded within the trading strategy. The parameters of P modulate the conditional expectation of the forward return,

We can formalize the optimization problem. The strategy seeks to minimize a risk-adjusted loss function J(θ, P), where θ represents the continuous weights of the classification model and P represents the discrete parameters of the preprocessing policy. Because P consists of non-differentiable string operations, gradient-based optimization fails. Researchers bypass this computational bottleneck by freezing P at generic heuristic defaults and optimizing over θ. This guarantees a suboptimal solution because the feature space itself remains unoptimized for the specific financial task.

Recent advancements in large language models present an alternative to static rules. Neural architectures can resolve linguistic ambiguity conditional on the surrounding text—the regular LLMs everybody knows. They can preserve negation, identify domain-specific entities, and differentiate identical surface strings based on usage. However, replacing deterministic algorithms with generative models introduces latency, variable execution costs, and non-deterministic outputs. High-frequency and mid-frequency statistical arbitrage strategies operate under temporal bounds. Executing a transformer network forward pass introduces latency measured in milliseconds, violating the execution constraints of a strategy engineered for microsecond-level reactions. The quantitative challenge is to engineer a text processing pipeline that extracts the semantic precision of contextual models while maintaining the execution speed and exact reproducibility of static rules.

The first obstacle is the absence of economic loss functions for text operations. Standard linguistic tasks measure success using metrics such as classification accuracy or F₁ scores on static text corpora, optimizing for average case performance. The cost of a false positive classification is asymmetric to the cost of a false negative. A transformation rule that improves overall text categorization accuracy by two percent but simultaneously degrades precision on severe, fat-tailed drawdown events produces negative alpha. The preprocessing layer must be calibrated against actual capital deployment metrics, such as maximum drawdown or turnover-adjusted return.

The second obstacle is source heterogeneity. Market text originates from diverse distributions with distinct generative processes. Regulatory filings, such as SEC 10-K and 10-Q documents, are dense, structured, and rely on formal accounting lexicons. Social media feeds are sparse, adversarial, non-standard, and populated with cashtags. Execution logs are machine-generated deterministic strings with dynamic alphanumeric variables. Applying a uniform, global preprocessing operator across these distinct domains guarantees information loss. The pipeline requires domain-conditioned transformation paths.

The third obstacle is look-ahead bias induced by contextual processing. If a large language model relies on weights trained on data generated after time t to process text observed at time t, the resulting feature vector z_t contains future information. Pre-trained language representations, whether produced by Word2Vec-style embeddings, BERT-like encoders, GPT-style transformers, or modern embedding models, inherit the temporal context of their training corpus. If a model was pre-trained on a corpus containing 2020 macroeconomic data, validating a 2018 trading strategy using those embeddings constitutes a forward-looking leak. Validating complex text transformations requires chronological segregation of vocabulary sets, embedding spaces, and rule dictionaries. Every artifact must be timestamped and generated only from data available prior to the simulation step.

Lexical canonicalization as a trading signal operator

Lexical canonicalization standardizes heterogeneous text strings into a controlled vocabulary. Market data feeds transmit text with arbitrary capitalization, markup artifacts, and non-standard character encodings. The standard approach applies universal lowercasing and punctuation stripping to reduce the vocabulary dimension and force text convergence.

This reduction changes the basis of the feature space. Consider a text sequence mapped to a sparse vector c(x) ∈ N^|V|, where |V| is the dimension of the vocabulary and c_j(x) represents the occurrence count of token j. A generic canonicalization policy P_a produces vocabulary V_a. A domain policy P_b produces vocabulary V_b. The linear combination of features

The transition from a high-dimensional raw text space to a lower-dimensional canonical space is an explicit projection operator.

Canonicalization must be evaluated as an information-theoretic operator. If a token carries distinct economic meaning based on its capitalization—such as the ticker symbol for a listed equity versus a common noun—lowercasing acts as a destructive operator. The string “APPLE” extracted from a financial data vendor carries a probability mass concentrated entirely on a specific technology equity. The string “Apple” might refer to the equity, or it might initiate a sentence. The string “apple” refers to an agricultural commodity. A universal case-folding mapping function collapses these three distinct nodes into a single coordinate. The mutual information between the raw text feature and the target variable is truncated, reducing the upper bound of the classifier’s predictive capability.

To quantify this instability, we define feature turnover when a canonicalization policy is updated or when the underlying data feed shifts formats from P_old to P_new at time t. Let V_old and V_new be the active vocabularies observed over a trailing window. Feature turnover is the Jaccard distance between the active feature sets:

High feature turnover indicates that the canonicalization operator is shifting the representation of the market. If τ_t spikes without a corresponding macroeconomic regime shift or market microstructure event, the canonicalization operator is malfunctioning and injecting mechanical noise into the feature vectors.

import re
from dataclasses import dataclass
from typing import List, Dict, Set

NEGATION = {"not", "no", "never", "without", "neither", "nor"}
UNCERTAINTY = {"may", "might", "could", "expects", "guides", "sees"}

@dataclass(frozen=True)
class ProcessedText:
    tokens: List[str]
    entities: List[str]
    flags: Dict[str, bool]

def normalize_market_text(text: str, known_entities: Set[str]) -> ProcessedText:
    """
    Standardizes text while preserving case-sensitive market entities and 
    extracting logical flags before irreversible lowercasing.
    """
    # Remove basic HTML markup
    text = re.sub(r"<[^>]+>", " ", text)
    # Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    
    # Extract entities preserving case. This prevents "APPLE" from becoming "apple"
    entities = [e for e in known_entities if e in text]
    
    # Tokenize broadly, capturing percentages and punctuation
    raw_tokens = re.findall(r"[A-Za-z][A-Za-z.'\-]*|\$?\d+(?:\.\d+)?%?|[!?]", text)
    tokens = []
    
    for tok in raw_tokens:
        if tok in entities:
            # Preserve exact case for known entities
            tokens.append(tok)
        elif tok.lower() in NEGATION or tok.lower() in UNCERTAINTY:
            # Standardize logical operators
            tokens.append(tok.lower())
        else:
            # Fallback canonicalization
            tokens.append(tok.lower())
            
    # Extract boolean flags representing the logical geometry of the sequence
    flags = {
        "has_negation": any(t in NEGATION for t in tokens),
        "has_uncertainty": any(t in UNCERTAINTY for t in tokens),
        "has_percent": any(t.endswith("%") for t in tokens),
    }
    return ProcessedText(tokens=tokens, entities=entities, flags=flags)

The next plot visualizes the baseline accuracy variances across disparate datasets under different canonicalization regimes, demonstrating that baseline controls are necessary. A robust canonicalization policy avoids global destructiveness. It extracts protected entities using deterministic gazetteers and named entity recognition modules before applying generic, dimension-reducing transformations to the residual text.

Token boundary specification and market microstructure semantics

Token boundary specification determines the minimum atomic unit of information available to the prediction model. Generic tokenizers partition text using whitespace and standard punctuation delimiters. Financial text consistently violates the core assumptions of generic tokenization, relying on special characters to denote specialized meaning.

Consider market identifiers and quantitative formats. Ticker symbols contain periods to denote share classes (BRK.B). Currency pairs rely on forward slashes (USD/JPY). ISINs and CUSIPs contain structured alphanumeric sequences without spacing. Magnitudes are similarly complex. Interest rates combine digits, decimal points, and percentage signs without whitespace, while financial statements combine currency symbols, numerical digits, and alphabetical magnitude modifiers (e.g., $1.5B). A generic tokenizer fractures these sequences blindly based on static delimiter rules. The string “USD/JPY” becomes

The explicit, hard-coded relationship between the base currency and the quote currency is severed.

When token boundaries are misspecified, the classifier must relearn the fractured relationship through computationally expensive sequence modeling or n-gram concatenation. This increases the data requirement to reach statistical significance and reduces the overall statistical power of the model. Token boundary specification in algorithmic trading must construct typed semantic spans.

Let x be the raw text string. A financial tokenizer first applies an ordered set of regular expressions R to identify protected spans E(x). Conflicts and overlaps are resolved using explicit priority queues, ensuring that a longer, more specific match supersedes a generic match. The tokenizer then partitions the complement sequence x\E(x) using standard whitespace rules. The output is an ordered sequence of discrete tokens interleaved with immutable typed entities.

The probability distribution of market events is conditional on these typed spans. An earnings surprise is a function of a specific reporting entity combined with a numerical magnitude measured relative to a predefined consensus estimate. If the tokenizer fractures the magnitude, the model receives unanchored digits. The feature vector contains noise rather than signal. Let’s implement it.

def protected_tokenize(text: str) -> List[str]:
    """
    Applies regex to extract specific market boundaries (money, percentages, ratings)
    before standard tokenization shatters them into unanchored digits.
    """
    patterns = {
        "MONEY": r"\$\d+(?:\.\d+)?(?:\s?-\s?\$?\d+(?:\.\d+)?)?",
        "PCT": r"[-+]?\d+(?:\.\d+)?%",
        "RATING": r"\b(?:AAA|AA|A|BBB|BB|B|CCC|CC|C|D)[+-]?\b",
        "FX": r"\b[A-Z]{3}/[A-Z]{3}\b",
        "FILING": r"\b(?:10-K|10-Q|8-K|S-1)\b"}
    
    spans = []
    for label, pat in patterns.items():
        for m in re.finditer(pat, text):
            spans.append((m.start(), m.end(), label, m.group()))
            
    # Sort by start index, resolve overlaps by taking the longest match
    spans = sorted(spans, key=lambda s: (s[0], -(s[1]-s[0])))
    
    tokens = []
    i = 0
    for start, end, label, raw in spans:
        if start < i:
            continue # Skip overlapping spans
        # Tokenize text BEFORE the protected span
        tokens.extend(re.findall(r"[A-Za-z][A-Za-z'\-]*|\d+|[!?]", text[i:start]))
        # Append the protected span as a single, typed token
        tokens.append(f"<{label}:{raw}>")
        i = end
        
    # Tokenize any remaining text after the last span
    tokens.extend(re.findall(r"[A-Za-z][A-Za-z'\-]*|\d+|[!?]", text[i:]))
    return tokens

The impact of tokenization extends into the temporal precision of market microstructure events. Execution logs and limit order book updates record exact latencies, side, and sequence numbers. A message reading “ADD order 12345 100@150.50” must not be parsed into arbitrary digits. Misspecifying the boundary of a timestamp, an order identifier, or a price-quantity tuple corrupts the sequence alignment. If the text pipeline fails to generate <QTY>@<PRICE>, the parser cannot reconstruct the state of the limit order book accurately, destroying the integrity of the downstream order flow imbalance calculations.

Send me a copy

Stopword removal, negation retention, and signal integrity

Read more

Table of contents:

1. Introduction.

2. Global macro regime after the February rupture.

3. Trade, tariffs, and the AI capital cycle.

4. Financial conditions, emerging markets, and the new debt geometry.

5. Europe between inflation, industry, and rearmament.

6. Chokepoints, rare earths, and the structure of strategic rivalry.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The central feature of this moment is that macroeconomics can no longer be separated from geopolitics. Oil routes, LNG flows, shipping insurance, semiconductor supply, rare-earth processing, defence procurement, and AI infrastructure now sit inside the same transmission system. A military clashes can move inflation expectations. A tariff decision can alter capital spending. A mineral export license can affect vehicle production, renewable deployment, and defence readiness. The channels that once belonged to separate policy worlds now reinforce one another.

The AI capital cycle is still creating demand for chips, servers, data centres, power systems, and advanced equipment. Fiscal support and defence spending are cushioning parts of the cycle. Services trade remains more resilient than goods trade. But the composition of growth has changed. Expansion increasingly depends on investment linked to security, technology, energy resilience, and state-backed industrial strategy.

That shift creates a new hierarchy among economies. Countries with energy supply, fiscal space, reserve buffers, advanced industrial capacity, or control over strategic inputs gain room to maneuver. Countries dependent on imported fuel, external financing, weak currencies, or narrow export bases face a tighter constraint. Financial markets are beginning to price that difference. The old distinction between advanced and emerging markets matters less than exposure to chokepoints, debt rollover, energy costs, and industrial depth.

Europe sits at the center of this adjustment because it faces all of these pressures at once. Imported energy inflation, weak industrial momentum, rising defence commitments, and the need to rebuild competitiveness without abandoning decarbonisation, all at the same time. The United States sits in a different position, supported by technology investment and capital inflows but still exposed to inflation persistence and strategic overextension. China retains industrial scale and export strength, while also relying on external demand and commanding key processing stages in critical supply chains.

You can go deeper by reviewing this:

OECD Economic Outlook, Interim Report March 2026

1.66MB ∙ PDF file

Download

Download

World Economic Outlook, April 2026

1.1MB ∙ PDF file

Download

Download

Global Trade Outlook And Statistics March 2026

1.44MB ∙ PDF file

Download

Download

The result is a world economy still moving forward, but no longer under the assumptions that defined the previous era. Efficiency is giving way to resilience. Open access is giving way to managed access. Supply chains are being judged not only by cost, but by survivability. The rest of 2026 will therefore depend less on whether growth exists and more on what kind of growth remains possible when war risk, tariffs, energy security, AI investment, debt pressure, and strategic bottlenecks all operate at the same time.

Subscribe now

Global macro regime after the February rupture

The macro setting in April carries a dual character with unusual clarity. Growth continues, yet the safety margin around that growth has narrowed. Productive capacity still expands, yet the cost of keeping that expansion on course has risen. The world economy entered the year with stronger momentum than many institutions had expected in late 2025, then moved into a new phase when the Middle East conflict. That event arrived while the international trade system was still adjusting to the earlier shift in United States trade policy, which means the present environment combines war risk, tariff uncertainty, supply-chain rewiring, and a tighter premium on energy security inside a single macro regime. The result!? The economy still moves forward…

Even under that path, the global outlook loses altitude. World growth is projected at 3.1 percent in 2026 and 3.2 percent in 2027, while global headline inflation rises to 4.4 percent in 2026 before easing to 3.7 percent in 2027. It confirms continued global expansion, yet it also confirms a return of the energy channel as a first-order driver of inflation, monetary timing, household purchasing power, and corporate margins. The downside scenarios reinforce the message. In the adverse case, growth falls to 2.5 percent and inflation rises to 5.4 percent in 2026. In the severe case, growth falls to 2 percent in both 2026 and 2027 while inflation rises above 6 percent. That spread between the reference path and the tail scenarios defines the new regime.

This chart gives the best opening frame for this because it shows that the February rupture belongs to a longer rise in geopolitical tension across regions. The 2020s stand above the 2000s and 2010s in every major region shown. The Middle East and Africa carry the highest level, which fits the immediate conflict geography. Europe and Asia-Pacific also register a clear step-up, which speaks to the wider strategic setting that already included the war in Ukraine, technology rivalry, sanctions, and maritime frictions before the latest shock. North America and South America show a lower level than the conflict core, yet they also sit above earlier decades.

That structural shift affects the transmission channels because it moves through the entire macro system. There are three central channels. The first is the direct commodity-price shock, which raises energy and input costs, disrupts transport, and weakens real incomes. The second is the expectations channel, where firms and workers seek to recover losses and thereby give price persistence a stronger foothold. The third is the financial channel, where higher risk premia, weaker asset valuations, tighter credit conditions, and capital outflows reduce demand and complicate policy. These channels reinforce one another. Oil and gas feed into fertilizer, freight, aviation fuel, and industrial inputs. Higher shipping and insurance costs pass into goods prices. Exchange-rate depreciation amplifies imported inflation, especially in commodity-importing emerging economies. In that setting, macroeconomics and statecraft operate in one field of transmission. Growth, inflation, shipping security, energy access, sanctions, and industrial policy act together.

The regional pattern adds another layer of meaning because the large macro blocs entered this point with different structures. The activity in China and the United States had been stronger than expected, yet that strength carried an uneven internal composition. In China, exports ran ahead while domestic activity, especially housing, remained softer. In the United States, strong activity coincided with low employment growth and slower labor-force expansion. At the same time, the trade regime itself remained in motion. United States statutory tariff rate stood about 5.3 percentage points below the level assumed after court rulings and executive actions, while other countries advanced trade negotiations and new partnerships of their own.

The United States enters this year with firmer demand conditions than most advanced peers and with a continuing technology-investment wave that still supports capital spending. AI-related technology investment added about 0.5 percentage point to US GDP growth, with sizeable spillovers into Asia through import demand for equipment and components. At the same time, inflation pressure in the United States remains higher than expected. Core personal consumption expenditure inflation at 3.1 percent year over year, and the wider WEO discussion presents a path in which US core inflation returns to target during 2027 rather than through an immediate or automatic easing sequence. In the case of China, export performance and industrial scale still carry the expansion. China is projected to grow 4.4 percent in 2026 and 4.0 percent in 2027, with external demand and some fiscal support offsetting a weaker domestic side where consumption remains soft. The euro area enters the year with modest growth and a higher sensitivity to imported energy and industrial input costs. The WEO places euro area growth at 1.2 percent in 2026 and 1.5 percent in 2027, while the euro area headline inflation rises above 2 percent in 2026 and remains above target in 2027.

In the United States, the issue centers on second-round price pressure because domestic demand has held up and inflation remains elevated. In the euro area, the issue centers on imported-price exposure because energy and other tradable inputs still form a key external channel into headline inflation. In China, the policy mix carries a different balance in which low inflation, export competitiveness, the exchange rate, and fiscal support weigh more heavily than overheating fears. The IMF press briefing therefore recommends patience from central banks under the reference case, together with clear communication and readiness to act if inflation expectations shift. On the fiscal side, the Fund emphasizes targeted and temporary support over broad subsidies or price caps, since public debt burdens have already reduced fiscal room.

The world economy still retains pillars of support, and these pillars explain why the baseline remains one of slower expansion rather than a broad contraction. The first pillar is the AI capital cycle. Technology investment remains a real source of demand for semiconductors, network equipment, power systems, and digital infrastructure, while the import intensity of that wave supports trade and investment spillovers across Asia. The second pillar is fiscal support. Fiscal support in various economies and the role of public spending in cushioning economic activity. The third pillar is adaptation through trade and inventories. Global trade remained robust because technology-related exports offset slower momentum in other product categories, and the IMF expects world trade volume growth to slow from 5.1 percent in 2025 to 2.8 percent in 2026 before improving to 3.8 percent in 2027. Goods trade faces a steeper deceleration, while services trade shows greater resilience. That distinction favors economies with stronger positions in digital, financial, and business services and gives the global system a partial buffer against fragmentation in goods.

This other chart deepens the argument because it shows that the present shock lands on an international system already shaped by long-running external asymmetries. Global imbalances may decline only modestly over the medium term. Some cyclical narrowing comes from expansionary fiscal packages in surplus economies, yet that narrowing faces countervailing forces. A technology-driven investment surge continues to attract capital toward the United States, and is expected stronger US productivity and high returns in technology-related sectors to keep capital inflows elevated. The same idea states that positive wealth effects and sustained inflows keep the US current account deficit wider than the level seen during the decade before the pandemic. At the same time, China’s continued reliance on export-led growth, together with limited rebalancing toward domestic consumption, sustains another major source of external imbalance.

That is important for geopolitics due to external imbalances influence power, and vulnerability. Current-account surpluses provide room for reserve accumulation, outward finance, and shock absorption. Large debtor positions sharpen sensitivity to financing conditions and exchange-rate moves. In the presence of higher geopolitical risk, those balance-sheet realities shape coalition behavior, sanctions exposure, reserve management, and the capacity to absorb commodity shocks.

Here the estimation is one-standard-deviation. Geopolitical risk leads to a decline in real GDP of about 0.8 percent one year after the initial shock. The main channels run through weaker private consumption and weaker investment, while consumer prices rise and the nominal effective exchange rate falls. Besides average geopolitical risk in Europe rose by about 1.2 standard deviations after Russia’s invasion of Ukraine and remained elevated at about 0.5 standard deviation now. This chart shows that geopolitical shocks reduce output through spending decisions made by households and firms, lift inflation through price channels, and weaken currencies through financial and external adjustment.

The broader implication is that supply security now carries a direct macro price. Maritime corridors, tanker insurance, refinery access, data infrastructure, and cross-border finance sit much closer to the center of the business cycle than they did under the earlier assumption of a largely rules-based and low-friction global system. Fertilizer markets, food security, reversals in capital flows, and abrupt adjustments in asset prices are all part of the chain of downside risks, especially for economies with weaker fiscal frameworks and lower external reserves.

Trade, tariffs, and the AI capital cycle

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

What a trading hypothesis is not

The lecture explains that charts, indicators, and backtests can all be useful, but none of them are complete hypotheses on their own. Their value appears only when they are connected to mechanism, scope, timing, variable design, and evidence, so the researcher can move from surface regularity to a claim that can be tested, challenged, and refined.

What’s inside:

1. What a trading hypothesis is not. The lecture begins by separating a genuine hypothesis from three incomplete objects (a chart pattern, an indicator, and a backtest result).

2. A chart pattern without explanatory content. A visual formation may reveal tension, compression, rupture, or recovery, but it remains only an empirical clue until the researcher explains what market process could have produced it.

3. Visual regularity without mechanism. Repeated shapes on a chart can create the impression of structure, but research begins only when those shapes are translated into participants, frictions, information, liquidity, or structural relations.

4. Repetition without causal interpretation. A pattern that appears many times does not automatically become a hypothesis. Repetition gains research value only when it is linked to a stable cause and a clear condition under which it should persist or disappear.

5. Description without research value. Statements such as “price tends to bounce after strong declines” are useful starting observations, but they need scope, variables, comparison design, and evidence criteria before they become researchable.

6. An indicator without market logic. Indicators compress data into signals, thresholds, scores, or ratios, but they matter scientifically only when they represent a real market condition such as crowding, strain, dislocation, liquidity withdrawal, or delayed adjustment.

7. A backtest result without a prior claim. A good historical result is not enough. The lecture emphasizes that a backtest should examine a claim stated in advance, rather than becoming a machine for discovering attractive performance and inventing explanations afterward.

8. The importance of forward logic. Strong research moves from observation to mechanism, from mechanism to formal claim, and from claim to evidence. This order gives meaning to both success and failure, allowing the researcher to learn from the market rather than simply collect historical artifacts.

Sample

1.9MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Data-Shape transformation.

3. Identifying the risks and limitations.

4. Data shape transformations for predictive features.

   1. State-space approach.

   2. Dynamic benchmark neutralization.

   3. Stochastic trend removal via cointegration.

   4. Multiscale isolation approach.

   5. Bounded state encoding.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Imagine that a team downloads a price series, defines a target, applies a transformation, and moves on to signal design, model fitting, validation, and execution. That sequence looks efficient. However, the transformation of the data is is the first act of model construction.

That is why data-shape transformation sits at the true front line of feature engineering. The problem is whether the operator used to create that series preserves the economic structure that generates predictive edge. A feature can look impeccable under statistical diagnostics and still be useless in live trading. It can pass every formal test of stability while removing the component that contained the alpha.

This article examines that hidden fault line. It argues that data-shape transformation must be treated as a structural modeling choice rather than a preprocessing convenience. We will identify the main risks introduced by transformation operators, explain how they distort the topology of predictive information, and then study five targeted transformations designed for distinct market nuisances: state-space innovations, rolling benchmark residuals, cointegration spreads, multiscale Haar details, and rolling range positions. The objective is not to make features look cleaner. The objective is to ensure that the shape of the data remains faithful to the market mechanism the model is supposed to learn.

Every transformation enforces a decision about relevance. It selects one part of the market path and suppresses another. It may emphasize short-horizon variation, eliminate common drift, compress local extremes, or redefine the position of the current price relative to its recent history. Before any model is trained, the researcher has already imposed a theory of what matters.

By the way, you can go deeper in this topic by exploring this paper.

Canonical time-series characteristics

1.3MB ∙ PDF file

Download

Download

That choice is decisive because financial data are layered. A single price series contains slow structural movement, temporary dislocations, benchmark comovement, volatility clustering, and random noise at the same time. A predictive model is never interested in all these layers equally. It depends on a narrow component that matches the strategy horizon and economic logic. The transformation either isolates that component or removes it.

Data-Shape transformation

The starting phase of feature engineering contains a severe trap because it masquerades as a trivial data processing step. A quant team downloads a time series of prices, computes a target variable, and faces the decision of whether the predictive model should ingest the raw level, a discrete difference, a standardized statistical score, a regression residual, an equilibrium spread, or a bounded state variable. The standard academic and industry reflex is to treat this decision as a technical prerequisite, often labeled preprocessing or data normalization. That classification is a mistake.

In many pipelines, this step is outsourced to data engineers or automated via standard library defaults before the alpha researchers even begin their core modeling. This assumes that data transformation is a lossless translation of information. It is not. Once a specific transformation is applied to the raw time series, the researcher has declared what specific variation in the market path constitutes actionable information and what variation constitutes noise. If the chosen transformation is misaligned with the economic mechanism the strategy intends to exploit, the resulting model can exhibit perfect statistical behavior while remaining economically blind.

In our domain, this situation is unforgiving because the price is a cumulative, high-memory object. It aggregates historical microstructure shocks, macroeconomic regime shifts, discrete splits in volatility, slow reallocations of institutional capital, structural risk premia, and continuous benchmark movements.

A raw price level carries a massive amount of historical baggage that the short-term or medium-term target variable is invariant to. Assume the core research problem is to predict the magnitude and direction of a short-horizon log return y_t+h. The actual feature matrix the learning algorithm receives is rarely the raw price p_t. Instead, the model ingests a transformed object x_t = T(F_t), where F_t represents the filtration generated by the history of prices and exogenous covariates up to time t. The selection of the transformation operator T is the actual front line of the research design. This operator dictates whether the vector x_t represents a local surprise, a relative displacement, a structurally bounded state, etc.

Identifying the risks and limitations

The risk surface introduced by the operator T is broader and more destructive than standard statistical literature admits. Researchers frequently inherit standard transformation recipes without interrogating their geometric implications or their effect on the data topology. They compute discrete log returns r_t = \ln(p_t) - \ln(p_t-1) because financial libraries expect stationary inputs. They build rolling z-scores because the resulting chart oscillates symmetrically around zero and fits neatly into the activation functions of deep neural networks. These choices impose heavy structural assumptions. A standard return calculation deletes the absolute price level and the cumulative memory of the path. A residual calculation deletes the variance explained by the specified reference model.

The first specific risk is false statistical comfort. A transformed time series can be manipulated to exhibit perfect weak-sense stationarity, zero mean, and constant variance, while simultaneously failing to express any tradable market state. Any researcher can run an Augmented Dickey-Fuller or KPSS test, achieve a spectacular p-value, and incorrectly assume the feature is ready for predictive modeling. Extracting white noise from a time series guarantees stability, but white noise contains zero predictive alpha. The researcher has converted the price into a stable sequence that is devoid of economic value.

The second risk is structural signal “amputation”. A transformation can neutralize the exact slow-moving component that carried the predictive edge. This failure mode occurs when researchers impose rigid discrete differencing, rolling normalization, or arbitrary short lookback windows without first establishing the exact holding period the systematic strategy is designed to exploit. If a macro trend-following strategy relies on three-month momentum, applying a 5-day rolling z-score to the input features will delete the low-frequency eigenvalue that the model requires to identify the trend.

The third risk is benchmark confusion and specification transfer. When a transformation operator incorporates a secondary series—such as a benchmark ETF, a sector index, or a paired asset—the resulting feature inherits the specification risk of that external asset. A regression residual is only valid if the variable against which it is residualized represents a stable economic factor. If a researcher isolates a tech stock by residualizing it against the QQQ index, and the index undergoes a massive constituent rebalance or suffers a liquidity shock, the resulting feature vector will exhibit severe volatility that has nothing to do with the target stock’s idiosyncratic state. The feature becomes contaminated by the reference frame.

The fourth risk is causality leakage. This is the most common fatal error in systematic trading. When the transformation relies on parameters estimated using information outside the filtration F_t, the feature becomes predictive in the research environment and false in the production environment. Using centered rolling windows, applying Principal Component Analysis eigenvectors derived from the full target matrix, computing full-sample mean vectors for standardization, or extracting post-hoc equilibrium coefficients creates a matrix that cannot exist in live trading. Forward-looking bias is adept at hiding within the scaling constants of data-shape transformations.

The fifth risk is geometric distortion. This transformation alters local curvature, variance persistence, variance clustering, and the relative distance metric of excursions. Consider a neural network optimizer relying on Euclidean distance to compute gradients. Mapping a log-normal price path into a bounded state space between zero and one forces the distance metric near the boundaries to behave differently from the distance metric in the center of the distribution. A raw price move that represents a massive four-standard-deviation tail event might only translate to a 0.01 shift in the bounded feature space if the state was already sitting at 0.98. Extracting a state space model collapses a continuous smooth drift into a localized sequence of forecast errors clustered around zero. This alters the fundamental definition of what a market burst represents to the downstream algorithm.

The conflict between raw data and transformed geometry culminates in a specific, observable failure in the research lifecycle. A model is built on a transformed feature set. The initial teardown looks immaculate. The in-sample backtest “demonstrates” it works. The cross-validation metrics are stable across distinct market regimes. The feature importance vectors are dense and logical. The Sharpe ratio appears robust across multiple chronological subsamples, and the drawdowns are well contained.

Then… the model is advanced to paper trading or live execution, and the predictive relationship collapses. The alpha decay instantaneous. The post-mortem analysis reveals that the algorithm never learned the underlying market mechanism. The optimization process learned the geometry introduced by the transformation operator T. It fit itself to sample-specific trend removal artifacts, spurious benchmark dependencies, bound-rejection behavior dictated by rolling window mechanics rather than market participants, and local scaling normalization patterns that possessed no forward-looking stability.

Subscribe now

Data shape transformations for predictive features

If the strategy is engineered to trade benchmark-relative mispricing, then the absolute market benchmark exposure is the primary nuisance structure. If the strategy capitalizes on statistical arbitrage and pair reversion, the common stochastic trend is the primary nuisance structure. If the strategy dictates short-horizon momentum reactions, slow multi-month macroeconomic drift is the primary nuisance structure. If the strategy operates as a local-state mean-reversion classifier, the absolute nominal price level is the primary nuisance structure.

This perspective enforces a precise operational taxonomy of data-shape transformations. The state space algorithm eliminates predictable local trends governed by state-space transition matrices, leaving the prediction surprise. The rolling_regression_residual algorithm eliminates time-varying benchmark exposure, yielding a relative-value deviation vector. The cointegration_spread algorithm eliminates common stochastic drift based on equilibrium vector autoregression logic, resulting in a stationary spread conducive to error-correction modeling. The haar_detail_same_length algorithm eliminates low-frequency power at a selected scale, leaving a targeted multiscale shock component. The rolling_range_position algorithm eliminates absolute nominal levels and local variance scaling, yielding a bounded coordinate variable.

These five methods represent interventions designed to blind the model to specific, targeted market nuisances. However without the right measures, the same mistakes could continue in some of them. Now that you’re aware of them, we can move forward.

State-space approach

The first targeted transformation is the state-space. In technical literature, this is defined as the one-step-ahead prediction error derived from a local linear trend state-space model. This transformation alters the time-series geometry by removing the exact variation that the internal state estimate already expects the system to produce at time t. This isolation mechanism is required when the raw log-price is cumulative, exhibiting a smooth drift that masks the actual high-frequency surprise component necessary for short-horizon prediction.

The formulation relies on setting up the observed log-price y_t as the output of a local linear trend model. The system requires an observation equation and state transition equations:

Here, the unobserved latent state consists of the level μ_t and the slope β_t. The disturbances ε_t, η_t, and ζ_t are mutually independent Gaussian white noise processes governing the measurement error, the level shift, and the slope shift. The state space model operates recursively. Given the filtration up to t-1, it produces the optimal linear one-step-ahead prediction E[y_t | F_t-1]. The feature we extract is the sequence v_t:

It is crucial to understand that v_t represents the precise segment of the current price movement that could not be predicted by the prior dynamic state. In the context of algorithmic trading, this means the feature measures orthogonal surprise rather than raw directional motion.

The rationale for this transformation is embedded in market microstructure. A high-frequency algorithm processing order flow must differentiate between a market that ticks upward at a steady, anticipated velocity and a market that prints an identical upward tick against a declining latent trend. The nominal difference y_t -y_t-1 will record an identical positive value in both scenarios. The state space v_t will output a minor value for the first scenario and a massive positive value for the second. Innovation-based feature engineering forces the learning algorithm to respect the conditional context of the regime.

def state_space(price_series: pd.Series) -> pd.Series:
    """
    Extracts the one-step-ahead forecast error from a local linear trend
    state-space model.
    """
    lp = price_series.rename("log_price")

    model = UnobservedComponents(lp, level="local linear trend")
    result = model.fit(disp=False)

    space = pd.Series(
        result.filter_results.forecasts_error[0],
        index=lp.index,
        name="state_space",)

    space.iloc[0] = np.nan
    return space

However, the researcher must decide if local drift should be neutralized. For execution timing, short-term liquidity provision, and algorithmic burst-detection, neutralizing drift is necessary. Utilizing v_t ensures the resulting feature matrix is robust across shifting volatility regimes because the state-space covariance matrices adapt to changing latent slopes. The downstream classifier is thus relieved of the computational burden of discovering that adaptation matrix from raw inputs.

The primary failure mode of this transformation is specification error, which occurs when the assumed dynamics of the state-space model fail to map onto the conditions of the asset’s microstructure. The standard local linear trend model relies on continuous Gaussian diffusion. If the true data generation process is instead dominated by discrete Poisson jumps—such as overnight gaps, scheduled FOMC macroeconomic releases, or sudden liquidity vacuums—the linear Gaussian assumptions break down. When a structural price jump occurs, the linear filter cannot snap to the new regime. Instead, it lags, requiring multiple timesteps for the state space gain matrix to fully correct the latent state components. During this structural catch-up phase, the sequence v_t will output an artificial cluster of large values. These values no longer represent an orthogonal market surprises but a deterministic filter correction error.

Researchers are tempted by the aesthetic stability of the a posteriori residual, often extracted via two-sided smoothing algorithms provided by default in many statistical libraries. Because these smoothers utilize future price information from t+1 to T to optimize the state estimate at time t, utilizing them injects terminal forward-looking leakage into the feature matrix. The resulting in-sample backtest will generate flawless, profitable metrics, while the live production system will bleed money.

Dynamic benchmark neutralization

The second transformation points to a different geometric annoyance known as time-varying external beta. The label is the rolling Ordinary Least Squares residual. In systematic portfolio management, this is recognized as the dynamic hedge residual. The core logic dictates that the variance of the target asset y_t is contaminated by the variance of an external factor x_t, which could represent a sector ETF, a broad index future, or a principal component basket. To model the asset’s idiosyncratic state, the researcher must regress the asset against the reference factor and isolate the residual error.

The model specifies:

Because the relationship between financial assets is never static, α_t and β_t cannot be estimated over the full sample. They must be approximated over a rolling window of length w. The parameter vector is computed utilizing the local design matrix X_t-w+1:t. The resulting transformed feature is the out-of-sample or end-of-sample deviation:

This specific subtraction is geometrically distinct from subtracting simple returns or computing a ratio. The dynamic residual projects the asset into the null space of the benchmark over the specified lookback window. The output geometry abandons the absolute coordinate system and adopts a relative, benchmark-neutral coordinate system.

def rolling_regression_residual(
    price_series: pd.Series,
    benchmark_series: pd.Series,
    window: int = 80,) -> pd.Series:
    """
    Computes the dynamic rolling residual of the asset relative to the benchmark.
    """
    y = price_series.rename("log_price")
    x = benchmark_series.rename("log_benchmark")

    X = sm.add_constant(x)
    result = RollingOLS(y, X, window=window).fit()
    params = result.params

    fitted = params["const"] + params["log_benchmark"] * x
    residual = (y - fitted).rename(f"rolling_regression_residual_{window}")

    return residual

In algorithmic trading, raw price vectors deceive predictive models by presenting sector-wide beta expansion as idiosyncratic momentum. A single equity instrument may display severe overbought technical geometry when the underlying condition is uniform sector strength. A cryptocurrency token may exhibit severe downward drift when the entire beta complex is repricing. Feeding raw prices into a deep neural network forces the network to allocate hidden layers to the task of inferring the benchmark matrix and computing the subtraction. By passing û_t, the transformation executes the subtraction analytically, presenting the network with the pure, actionable relative-value deviation.

The geometric shift transforms a trend-dominated price path into a sequence that continually crosses zero, widening when the beta relationship degrades and compressing when the hedge is tight. This sequence describes relative-value dislocations that are primed for statistical arbitrage.

The implementation contains a critical stability trade-off governed by the window parameter w. A minimal window length allows the matrix inversion (X^TX)^-1 to adapt to microstructural shifts in beta, but exposes the parameters to severe estimation noise and matrix ill-conditioning. A maximum window length stabilizes the eigenvalue structure of the covariance matrix but renders the hedge ratio unresponsive to macroeconomic events.

Stochastic trend removal via cointegration

Read more

Table of contents:

1. Introduction.

2. Global macro after the energy shock.

3. The United States, the axis of growth, finance, and military escalation.

4. The main price-maker in the global economy.

5. China and Asia are the first receivers of the Hormuz shock.

6. The dollar, bond markets, and the return of geopolitical duration risk.

7. NATO, military burden-sharing, and the remapping of Western power.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The global backdrop has entered a harder regime. Energy, shipping, fiscal capacity, military posture, and capital allocation now trade through the same set of constraints. Variables that once sat in separate analytical buckets now belong to a single pricing framework. Oil routes shape inflation. Alliance commitments shape issuance. Industrial policy shapes defence readiness. Currency strength shapes import costs and external adjustment. The market is pricing all of it together.

The central question concerns the kind of growth that can endure when energy friction rises, logistics tighten, and states absorb a larger strategic burden. That shift changes the hierarchy of variables. Security of supply carries more weight. Duration risk carries more weight. Fiscal credibility carries more weight. The ability to convert spending into usable capacity also carries more weight, whether in semiconductors, power systems, refining, shipping, or defence production.

This piece follows that chain from the macro layer into the strategic layer and back again. It begins with the global economy after the energy shock, where an active investment cycle now meets a harsher cost structure. It then turns to the United States, the core balance sheet in the system, where growth leadership, Treasury funding, monetary policy, and force projection converge. From there, the focus shifts to the conflict itself as the marginal driver of price formation across oil, gas, freight, and risk premia.

The analysis then moves to Asia, where direct exposure to Gulf energy flows makes the region the first receiver of the shock in physical, financial, and policy terms. It then examines the dollar and sovereign debt markets, where the length of the shock has become a key variable for yields, term premium, and global liquidity demand. The final section studies NATO and the wider Western security architecture, where rising military spending points to a deeper redistribution of industrial capacity, operational responsibility, and strategic weight.

Check more about all of this here.

US employment situation

491KB ∙ PDF file

Download

Download

Economic bulletin 2026

3.17MB ∙ PDF file

Download

Download

Shorter energy outlook

6.84MB ∙ PDF file

Download

Download

The aim is to frame the current environment the way markets are beginning to price it, as one integrated system. Growth, inflation, war, energy, debt, and alliance structure now move through the same circuit. The edge comes from seeing the connections early, before consensus fully absorbs their implications.

Subscribe now

Global macro after the energy shock

We can say we have two engines that pull in opposite directions. One engine comes from the capital cycle that formed through 2025 and carried into 2026 through technology investment, private-sector adaptation, and supportive demand. The other engine comes from the war-driven energy shock, the tightening of transport routes, and the rise in geopolitical risk premia across commodities, shipping, rates, and equities. The world economy is treated as a system where energy access, market confidence, military escalation, and asset pricing move together. The IMF and OECD now describe the same architecture in institutional language. The IMF’s update still presented a world economy with resilient growth, projecting global output growth at 3.3% in 2026 and 3.2% in 2027, driven in part by technology investment, supportive financial conditions, and private-sector adaptability. By late March, the OECD had redrawn that picture. It projected global growth at 2.9% in 2026 and 3.0% in 2027, with the Middle East conflict, higher energy prices, and supply-chain disruption pushing costs higher and demand lower.

The tension between those two engines defines the whole macro regime. The first engine still matters because technology investment remains large, especially in sectors tied to artificial intelligence, semiconductors, cloud infrastructure, industrial automation, and the digital build-out of the energy and security complex. The IMF highlighted strong performance in exports of semiconductors and related equipment across Asian economies and linked that strength to the rise in information and technology investment. That matters because a technology-led capex cycle creates growth through many channels at once. It supports equipment orders, lifts productivity expectations, raises corporate spending, improves cash-flow visibility for firms near the center of the build-out, and attracts capital toward the economies that lead those value chains. This is one reason the United States has strong relative support. It is also one reason parts of East Asia carried stronger industrial momentum into the year than many analysts expected at the end of 2025.

At the same time, the second engine has become the dominant marginal force in pricing. The OECD says the conflict in the Middle East is testing the resilience of a global economy that had been supported by financial and fiscal conditions and growing demand for artificial intelligence technologies. It states that the halt in shipments through the Strait of Hormuz and the closure or damage of energy infrastructure generated a surge in energy prices and disrupted the global supply of energy and other key commodities, including fertilisers. This is central because it shifts the macro discussion away from the classic debate about whether demand remains firm and toward a more strategic debate about how long the energy shock lasts, how broad the shipping disruption becomes, and how much of the rise in commodity prices passes into wages, services, industrial margins, and inflation expectations.

This leaves the world economy in a narrower corridor than the one investors saw at the beginning of the year. In January, the baseline still looked like a world of steady growth with disinflation in progress. Now that baseline became conditional on the moderation of energy-market disruption from mid-2026 onward. The OECD states that its projections rely on that technical assumption. The economic meaning is clear. A shorter disruption preserves a large part of the prior macro script. A longer disruption transforms the outlook into one shaped by persistent cost pressure, weaker real income growth, broader financial repricing, and stronger dependence on fiscal triage and central-bank credibility.

The inflation story reflects this shift with unusual clarity. The IMF’s update expected global inflation to fall, projecting headline inflation at 3.8% in 2026 and 3.4% in 2027. That forecast came from a world in which supply conditions improved, commodity pricing eased from earlier extremes, and private adaptation softened the impact of trade-policy friction. The OECD’s update already incorporates the war shock. It projects G20 inflation at 4.0% in 2026, which is 1.2 percentage points higher than it expected before the conflict escalated, before easing to 2.7% in 2027 under the assumption that energy pressures fade. This difference between the January IMF view and the March OECD view is one of the cleanest expressions of the regime change. It shows how quickly a global disinflation narrative can bend when energy, logistics, and risk premia move together.

The inflation process has also split into layers. Headline inflation is the first recipient of the shock because oil, gas, fuel transport, chemicals, and fertilisers transmit rapidly into measured prices. Core inflation then becomes the field of contest. If households and firms treat the energy shock as a passing event, core inflation stabilises with less secondary impact. If firms push higher input costs through broader price categories and workers seek compensation for weaker real incomes, core inflation becomes more persistent. The OECD says medium-term inflation expectations already rose following the energy-price spike. That point matters because expectations serve as the bridge between a commodity event and a more durable inflation process. The shape of that bridge has become one of the decisive macro variables.

Growth now rests on dispersion rather than on one shared global rhythm. The United States still benefits from strong AI-related investment, and the OECD projects U.S. growth at 2.0% in 2026 before easing to 1.7% in 2027. China remains a large growth pole at 4.4% in 2026 and 4.3% in 2027 in the OECD baseline, with state support and industrial strength still carrying weight. The euro area carries a weaker profile. The OECD projects euro area growth at 0.8% in 2026 before a move to 1.2% in 2027, helped in part by stronger defence spending. These numbers describe more than regional variation. They describe the way energy exposure, trade composition, industrial structure, and policy room now shape macro outcomes. Economies close to the technology capex wave retain momentum. Economies more exposed to imported energy and external demand face a tighter balance.

That regional dispersion also changes the meaning of policy credibility. In a calmer cycle, policy credibility often refers to a central bank’s ability to anchor inflation and a finance ministry’s ability to sustain debt. In the current cycle, the idea becomes wider. Credibility now includes the ability to secure energy, preserve transport flows, design temporary support measures without weakening the fiscal base, and maintain enough strategic coherence that firms can plan capital expenditure with confidence. The OECD writes that government measures to cushion higher energy prices should be targeted, preserve incentives to lower energy use, and carry clear expiry mechanisms. It also stresses limited fiscal space and the need for credible medium-term adjustment paths. This means macro policy has become an exercise in war-era selectivity rather than a broad stimulus exercise.

The capital-market layer follows the same logic. In the technology engine, markets still reward growth visibility, margin durability, and long-duration cash flows linked to infrastructure, semiconductors, power systems, software, automation, and data architecture. In the energy-shock engine, markets demand compensation for transport friction, raw-material scarcity, rate volatility, and policy uncertainty. The OECD says financial conditions tightened after the outbreak of hostilities, with volatility rising and the previous degree of accommodation shrinking. The important point lies in the combination rather than in a single asset class. Equities, bonds, currencies, commodities, and credit spreads are all trying to price the same unknown at once, namely the duration and reach of the conflict shock. That creates a macro system in which every bond curve carries an energy assumption, every central-bank path carries a shipping assumption, and every equity multiple carries an embedded judgment about geopolitical durability.

The structural dimension runs deeper than the current quarter. The OECD highlights that persistent disruptions to Middle East exports could aggravate shortages of key commodities and produce a broader repricing in financial markets. It also notes that countries in the Gulf serve as an important source of aluminium, helium, bromine, fertiliser inputs, and logistics services. This widens the macro significance of the current conflict. The issue extends beyond oil and gas. It reaches into food systems through fertilisers, into industrial production through metals and specialty materials, and into advanced manufacturing through chemical and memory-chip supply chains. A prolonged disruption therefore reshapes more than headline inflation. It reshapes the cost map of the industrial economy.

The United States, the axis of growth, finance, and military escalation.

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Why a trading hypothesis?

A trading hypothesis is presented as the first scientific form of a market idea, one that transforms intuition into a structured claim with an activating condition, an expected response, a time horizon, and a proposed mechanism. The lecture develops this framework as the bridge between market logic and empirical evidence, and explains how restriction, precision, and coherence turn a vague observation into a research object that can be tested, challenged, and refined.

What’s inside:

1. Why a trading hypothesis matters. A trading hypothesis is the first formal research object in quantitative trading because it gives direction, causal meaning, and empirical consequences to an otherwise informal market observation.

2. Hypothesis as a conditional claim about market behavior. The lecture defines a genuine hypothesis as a structured conditional statement that specifies when a market condition appears, what response should follow, over what horizon, and because of what underlying process.

3. The activating condition. A hypothesis begins with a trigger that identifies when the market enters a meaningful state, such as a volatility expansion, spread dislocation, order flow imbalance, or liquidity shock, so the sample becomes selective in a principled way.

4. The expected market response. The claim must state what should happen after activation by defining the object of movement, the direction of adjustment, and the scale or relevance of that movement, so the idea acquires a clear empirical identity.

5. The horizon over which the response should emerge. Time is treated as part of the hypothesis itself because different mechanisms operate on different clocks, and the validity of the research depends on matching the response window to the real rhythm of the market process.

6. A bridge between market logic and empirical evidence. The lecture shows that a good hypothesis connects two domains at once, the logic of participants, frictions, and structural forces on one side, and the language of variables, samples, outcomes, and comparisons on the other.

7. Empirical signature and supporting evidence. A strong hypothesis does not predict profitability alone, but a specific patterned trace in the data, including direction, structure, timing, conditional strength, and comparative contrast, so support comes from alignment rather than from one backtest statistic.

8. The hypothesis as a restricted object. The lecture explains that every valid hypothesis must be restricted to a coherent asset universe, a defined market state, and a proper time horizon, because restriction is what gives the claim scientific shape and protects it from vague breadth.

Sample

1.87MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Why transform data?

3. The risks of data transformation.

4. The memory dilemma and distributional shifts.

   1. The fractional differencing paradigm.

   2. Higher moments and the Cornish-Fisher expansion.

5. Time and variance.

   1. Time deformations and information clocks.

   2. Structural volatility standardization.

6. Cross-sectional geometry and noise mitigation.

   1. Entropy-based outlier mitigation without information loss.

   2. Cross-sectional orthogonalization.

7. Non-linear embedding and phase-space reconstruction.

8. The unified pipeline.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Raw market data is the closest thing we have to the market’s observable dynamics, so the instinct to leave it untouched feels sensible. But predictive models consume numerical representations. Between the tape and the optimizer there is always a translation step, whether explicit or hidden, and that translation determines which structures remain visible, which ones are attenuated, and which ones disappear. In financial machine learning, representation is part of the model itself. Prices, returns, clocks, scales, and cross-sectional coordinates all define different statistical objects, and each one presents a different geometry to the learning algorithm.

The literature on volume clocks showed that market activity is better understood in an event-driven metric than in a chronological one. The literature on fractional differencing showed that differencing need not be an all-or-nothing choice, but can instead vary continuously to preserve long-range structure while moving toward stationarity. The literature on conditional heteroskedasticity showed that variance itself has memory and must be modeled as a dynamic object rather than treated as a fixed nuisance parameter.

Here an old one about volume clock:

The volume clock

661KB ∙ PDF file

Download

Download

Yet every transformation carries a cost. A bad transformation can alter the topology of the series, flatten its tails, suppress its regime structure, and leak future information into the past. The goal here is to show that each transformation answers a specific failure mode of financial data. Some methods regularize the clock. Others preserve memory while controlling non-stationarity. Others stabilize conditional variance, correct tail asymmetry, reduce cross-sectional noise, or reconstruct hidden structure from a single observable. When these operations are applied in the right order, transformation stops being a destructive preprocessing ritual and becomes a way of making market structure legible to statistical and machine-learning models.

This is the real question is “how to make data look cleaner, but how to modify it without amputating the very dependencies we hope to trade.”

Why transform data?

If raw data represents the perfect economic reality of the market, why do we need to transform it at all? Why not feed the raw sequence of events, the actual execution prices and sizes, directly into our predictive algorithms?

The answer lies in the rigid constraints of linear algebra and numerical optimization. Machine learning algorithms, from basic Ordinary Least Squares to stochastic gradient descent solvers, don’t understand economics, market microstructure, or the concept of fiat currency. They only understand geometry and topology. They evaluate the distance between points in a high-dimensional space.

Let X∈R^TxN be a feature matrix of raw asset prices over T time steps for N assets. Raw prices are strictly non-stationary I(1) processes, their mean and variance scale continually with time. If we attempt to fit a linear model to minimize the empirical risk, we must compute the inverse covariance matrix (X^TX)^-1 to solve for the coefficient vector. Because the raw prices share a common, unbounded macro trend, the vectors in X are collinear. The matrix X^TX becomes nearly singular. Its condition number explodes, meaning the matrix inversion becomes numerically unstable. The optimizer can’t find a unique global minimum. Instead, it yields spurious regressions. The model will report a massive R²X value and highly significant t-statistics, suggesting it has found a relationship, when in reality, it has correlated two independent random walks that happen to drift in the same general direction.

Furthermore, deep learning architectures—specifically Recurrent Neural Networks and Long Short-Term Memory networks—require bounded, zero-centered inputs to propagate gradients efficiently through time. If we feed raw, un-scaled prices into a network utilizing sigmoid or hyperbolic tangent (tanh) activation functions, the architecture breaks down immediately. If an asset’s price has drifted from $10 to $400 over a decade, feeding the value $400 into a tanh function yields an output that is computationally indistinguishable from $1.0. The local derivative of the activation function at this asymptote is zero. When the backpropagation algorithm attempts to calculate the chain rule to update the network’s weights, it multiplies by this zero derivative, causing the vanishing gradient problem. The network stops learning on the very first epoch. The weights freeze.

Subscribe now

The risks of data transformation

For some models transformation is a necessity, however it is the highest-risk component of the entire pipeline. Every operation applied to a temporal series acts as a filter. Applying an operator T(·) to a raw series X_t involves a manipulation of entropy. You are altering the properties of the dataset. If the transformation isn’t an exact bijection tailored to the specific distributional properties of the underlying asset, it is a destructive process that deletes alpha.

1. The primary risk is geometric distortion. Quants attempt to force non-Gaussian market data into Gaussian topologies using standardized scalers. Let μ_t and σ_t be the rolling mean and standard deviation. The standard Z-score transformation is Z_t = (X_t - μ_t) / σ_t. This operation is ubiquitous in basic data science, but it is catastrophic in finance. As established by empirical market data, equity returns exhibit severe negative skewness and massive excess kurtosis. By applying this linear transformation, we assume the data is symmetric around the mean. We geometrically compress the left tail and stretch the right tail. When the optimizer processes Z_t, it assigns incorrect probabilistic weights to the downside risk. The transformation has lied to the algorithm about the probability of ruin, mapping a 6σ market crash into a normalized space that makes it look like a mild, acceptable deviation.

2. The second critical risk is information destruction via arbitrary numerical thresholds. Common quantitative practices like hard-clipping outliers or dropping low-volume trading hours manually alter the sequence of the time series to make the data cleaner for the optimizer. If a quant runs a standard Winsorization protocol and drops an outlier return of -12% down to a hard cap of -3% simply because the extreme value “ruins the scale of the loss function,” they have committed a severe theoretical error. They have deleted the exact moment of maximum market inefficiency. That -12% print is the precise data point where algorithmic alpha is generated. It represents a moment where traders capitulated and forced liquidations occurred. Transforming the data to make it look smooth and continuous for an optimizer strips out the true signal, leaving behind highly stationary, useless white noise.

3. The third, and most fatal, risk is structural data leakage, also known as look-ahead bias. This occurs when a transformation operator inadvertently uses information from the future to scale or center the data of the past. If a quant runs a Principal Component Analysis (PCA) on the entire matrix X spanning from 2010 to 2024 to find orthogonal risk factors, and then trains a predictive model sequentially on the data from 2015, the model is compromised. The eigenvectors calculated by the global PCA contain variance data from the 2020 pandemic market crash. The transformation has leaked future volatility structures into the historical training set.

This leakage happens with basic functions like global mean subtraction or min-max scaling if the boundaries aren’t rolling. The model will appear profitable during out-of-sample backtesting because the transformed features implicitly “know” the future basis vectors and maximum bounds of the market. In live trading, where the production operator only has access to historical data up to time t, this false performance collapses instantly.

The memory dilemma and distributional shifts.

The fundamental dilemma in quantitative modeling revolves around the requirement for stationarity and the predictive requirement for memory. To train any robust inferential model, whether a vector autoregression or a neural architecture, the input data must be stationary. The statistical properties of the series—mean, variance, and autocorrelation structure—can’t vary over time. If they do, the model learns a regime that ceases to exist the moment it is deployed in live trading.

When you feed non-stationary price levels into a learning algorithm, you violate the core assumptions of the optimizer. As previously mentioned, in linear models like Ordinary Least Squares, non-stationarity leads to spurious regressions where the model identifies high R² values and significant t-statistics between independent random walks. In deep learning architectures like Recurrent Neural Networks, non-stationary variance causes the gradient descent process to either vanish into zero or explode to infinity, preventing convergence.

However, the standard mathematical transformations used to enforce this required stationarity destroy the predictive power of the dataset. Financial time series, specifically price levels, are mostly I(1) processes, meaning they possess a unit root. The conventional approach to solving this is applying an integer differentiation, calculating the first-order difference or the standard log-return. This transformation achieves stationarity, yielding a series with a constant mean and finite variance. Yet, it erases the long-term memory of the process. Prices retain the entire history of market shocks, but returns only remember the shock of the immediately preceding period. By integer differencing, we obtain stationary noise that is impossible to predict.

Furthermore, integer differencing often leads to over-differencing. When you subtract X_t-1 from X_t to create a return series, you inject a moving average MA(1) component with a coefficient of -1 into the resulting noise. This artificially creates a strong negative autocorrelation at the first lag. Algorithms trained on this over-differenced data will discover short-term mean reversion that doesn’t exist in the market microstructure, leading to systematic trading losses.

The risks of relying on models built atop these conventional transformations are severe. We force non-stationary data through linear, memoryless filters, creating an illusion of predictability. We evaluate stationarity using binary heuristics like the Augmented Dickey-Fuller test, treating it as an absolute state rather than a continuous spectrum. Models trained on first-differenced data often exhibit high backtest performance due to noise-fitting but fail rapidly out-of-sample because the underlying structural dependencies of the market have been severed.

Do you rememenber the Quant Meltdown? Statistical arbitrage funds, operating with massive leverage, relied on mean-reversion models trained on orthogonalized, first-differenced equity returns. These models assumed that the residuals of their transformations were stationary, memoryless white noise, safely confined to narrow bands. When sudden, massive liquidations hit the market due to subprime mortgage exposure, the assumed independence of these time series broke down.

The residuals exhibited extreme, long-term memory. A portfolio liquidation creates a directional order flow imbalance that persists across days, resurrecting the unit root that the quants thought they had differenced away. The standard transformations had hidden it under normal, high-liquidity market conditions. When the regime shifted, the models failed because the foundation of their data transformation was inadequate. The algorithms kept doubling down on mean-reverting bets while the market displayed a persistent, memory-driven structural break.

The fractional differencing paradigm

We must abandon the misconception that differentiation is an integer operator. In the context of trading, forcing a time series into either a raw price format (where d=0) or a returns format (where d=1) is an arbitrary constraint. The correct approach is to treat differentiation as a continuous domain operator, allowing us to find the exact fractional value of d that achieves stationarity while preserving the maximum possible amount of memory.

We define the backshift operator B such that BX_t = X_t-1. For any real number d, the fractional difference operator (1-B)^d can be expanded using the infinite binomial series:

This expansion yields an infinite sequence of weights ω_k that we apply to past observations.

Notice the recursive property of these weights:

When d=1, the math is trivial: ω₀=1, ω₁=-1, and all subsequent weights are zero. This describes the standard first-order difference. The memory of the series is truncated at one period.

But consider what happens when d is a fraction, say 0.5. The sequence of weights becomes ω₀=1, ω₁=-0.5, ω₂=-0.125, ω₃=-0.0625, and so on. The weights decay asymptotically to zero rather than abruptly terminating. This means X_t is transformed using an exponentially decaying window of its entire history. The current value of the transformed series is heavily influenced by yesterday’s price, moderately influenced by last week’s price, and influenced by the price a year ago. This try to mirror the market information dissemination.

In practice, financial time series like equity indices or major fiat currencies achieve ADF stationarity at d values ranging between 0.3 and 0.4. This is an empirical fact. It proves that by using standard log-returns (d=1), the quantitative finance industry is discarding 60% to 70% of the useful memory contained in the price data.

To implement algorithm, we calculate the weights iteratively and apply a fixed-width window to prevent data leakage and memory overflow. We establish a tolerance threshold τ (often set to 10^-4 or 10^-5) and drop weights where |ω_k| < τ. This generates a fixed lookback window, typically spanning a few thousand observations depending on the chosen threshold.

We must use a fixed-width window rather than an expanding window. If we apply fractional differencing using the entire available history of the series starting from t=0, the number of weights applied to the current observation grows as time progresses. This causes the variance of the transformed series to drift, violating the finite variance requirement of stationarity. The fixed-width window ensures that the exact same mathematical operation is applied to every observation, maintaining a constant variance profile.

Let’s implement this.

import numpy as np
import pandas as pd
from statsmodels.tsa.stattools import adfuller

def get_weights_ffd(d, length, threshold=1e-5):
    """
    Calculate weights for fractional differentiation using a fixed-width window.
    Weights iteratively decay, preserving continuous memory constraints.
    """
    weights = [1.0]
    k = 1
    while True:
        weight_k = -weights[-1] * (d - k + 1) / k
        if abs(weight_k) < threshold and k >= length:
            break
        weights.append(weight_k)
        k += 1
        if k > 10000:  # Safety break for massive arrays
            break
    return np.array(weights[::-1])

def fractional_diff(series, d, threshold=1e-5):
    """
    Applies fractional differencing to a pandas Series to maintain memory 
    while achieving strict mathematical stationarity.
    """
    weights = get_weights_ffd(d, length=1, threshold=threshold)
    width = len(weights) - 1
    
    df = pd.Series(index=series.index, dtype=float)
    
    # Apply the dot product across the fixed-width rolling window
    for i in range(width, len(series)):
        window = series.iloc[i - width : i + 1]
        df.iloc[i] = np.dot(weights, window)
        
    return df.dropna()

def optimize_fractional_d(series, d_range=np.arange(0.1, 1.0, 0.1), p_value_threshold=0.05):
    """
    Finds the minimum fractional d that successfully passes the Augmented 
    Dickey-Fuller (ADF) test for stationarity, maximizing retained memory.
    """
    for d in d_range:
        diffed = fractional_diff(series, d)
        if len(diffed) > 10:
            adf_stat = adfuller(diffed, maxlag=1, regression='c', autolag=None)
            if adf_stat[1] < p_value_threshold:
                return d
    return 1.0

This gives us the orange series you see below. Do you notice the difference compared to a simple differentiation?

Higher moments and the Cornish-Fisher expansion

Quants frequently take log-returns, subtract the rolling mean, and divide by the rolling standard deviation, assuming the resulting series z follows a standard normal distribution N(0,1). Sound familiar? This is the foundation of the Z-score, a metric that is built into the base functionality of almost every data science library.

Financial returns don’t follow a Gaussian distribution, they exhibit significant skewness and excess kurtosis. Equity markets, for example, display structural negative skew, prices drop much faster than they rise due to the mechanics of margin calls, stop-loss triggering, and fear-driven liquidity vacuums. Besides, the phenomenon of volatility clustering generates excess kurtosis, creating a distribution with a sharp central peak and fat tails.

Standardizing without accounting for these higher moments leads to severe underestimation of tail risk and generates false trading signals. Consider a standard normal distribution. A 5-standard-deviation event has a probability of approximately 2.8x10^-7, which translates to an expected occurrence of once every 13,800 years of daily trading. Yet, in live equity markets, 5-sigma moves happen every few years. If an algorithm feeds a raw 5-sigma Z-score into a neural network, the model interprets it as an impossible anomaly and the assigned weights will skew, breaking the loss function.

To transform a time series into a Gaussian-equivalent state, we must incorporate its empirical skewness (S) and excess kurtosis (K). We achieve this via the Cornish-Fisher expansion, which provides a framework to estimate the quantiles of a non-normal distribution based on its specific cumulants.

If Z_α is the α-quantile of the standard normal distribution, the corresponding quantile W_α of our financial time series can be approximated as:

However, in feature engineering for predictive modeling, we often need the inverse transformation. We have an observed standardized return w coming from the live market, and we want to map it back to a normal variable z so that our downstream linear models and gradient descent algorithms function correctly.

We compute the rolling skewness and rolling excess kurtosis of the time series. Estimating the third and fourth moments requires significant data to achieve statistical stability, as outliers heavily distort these metrics. Therefore, the lookback window for S and K must be longer than the window used for the mean and variance—often spanning two to three years of daily data.

Once we establish stable estimates for the cumulants, we apply the inverse Cornish-Fisher expansion to normalize the data. We map the empirical quantiles of the fat-tailed data to the theoretical quantiles of the standard normal distribution.

This adjustment guarantees that a severe market shock is processed correctly by the predictive architecture. A drop that registers as a 5-sigma event in raw standard deviations might map to a much more manageable 2.5-sigma event in the Cornish-Fisher adjusted space, reflecting its true probabilistic reality in a fat-tailed environment.

import scipy.stats as stats
import pandas as pd

def cornish_fisher_transform(returns):
    """
    Normalizes higher moments (skewness and kurtosis) mapping an empirically 
    fat-tailed financial series back to a strictly Gaussian space.
    """
    # Standardize the data first based on initial 2 moments
    z = (returns - returns.mean()) / returns.std()
    
    # In a strict ML feature engineering pipeline, we map the empirical quantiles 
    # of the fat-tailed data directly to the theoretical quantiles of the standard normal distribution
    
    ranks = z.rank(pct=True)
    
    # Map back to standard normal quantiles. We use the PPF (Percent Point Function)
    normalized = stats.norm.ppf(ranks)
    
    return pd.Series(normalized, index=returns.index)

I admit I like this change in silhouette. In the next post, we’ll talk about changes in shapes, I have some ideas that might be quite interesting.

As can be seen, our distribution now looks more like a normal distribution than the classic silhouette of financial returns.

Time and variance

Read more

Table of contents:

1. Introduction.

2. Escalation of war in the Gulf.

3. Hormuz as the strategic hinge.

4. Kharg island is the energy chokepoint.

5. Energy shock and the macro transmission channel.

6. Second-order economic effect of war.

7. Egypt external war pressure.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The Gulf war now sits at the center of the global economic system. What once looked like a regional military confrontation now shapes the price of oil, the cost of maritime insurance, the movement of LNG, the availability of fertilizers, the outlook for inflation, and the fiscal stability of import-dependent states. The conflict has crossed the boundary between battlefield dynamics and macroeconomic reality. Every strike on infrastructure, every disruption in transit, and every shift in force posture now feeds directly into trade flows, industrial costs, sovereign risk, and household pressure across regions far from the Gulf.

At the center of that system stands the Strait of Hormuz. This narrow maritime corridor connects armed confrontation with global energy supply in a direct and immediate way. It is the hinge between missiles and inflation, between naval pressure and industrial costs, between regional violence and worldwide repricing across oil, gas, freight, food, and sovereign debt. Around it, Kharg Island emerges as a second critical node, a concentrated export point whose strategic value lies in its ability to convert Iranian production into global market pressure and fiscal survival.

What follows traces that chain with a clear line of sight. It begins with the expansion of war across the Gulf export system and then moves through Hormuz as the decisive chokepoint where military risk becomes macroeconomic shock. From there, the focus shifts to Kharg Island and the wider energy channel, where the conflict enters inflation, trade, growth, and monetary policy. The later sections turn to the deeper economic consequences of prolonged confrontation, including alliance strain, defence spending, fiscal burdens, and the exposure of vulnerable import-dependent economies such as Egypt.

You can go deeper by checking this report:

Gobal trade outlook

1.44MB ∙ PDF file

Download

Download

This moment also carries a strategic message for markets and states alike. The most important risks sit in the infrastructure that keeps economies supplied, in the narrow corridors that keep trade alive and domestic stability. Once those channels come under pressure, the effects spread with speed from ports to prices, from commodities to currencies, and from regional confrontation to global adjustment.

Subscribe now

Escalation of war in the Gulf

The war now sits inside the core of the global economy. The opening phase concentrated on Iranian leadership nodes, missile capacity, nuclear assets, and military infrastructure. The present phase covers the Gulf export system, maritime transit, shipping insurance, alliance bargaining, and market pricing across oil, gas, freight, food, and sovereign risk. Attacks hit major oil and gas sites across Iran, Saudi Arabia, Kuwait, Qatar, the United Arab Emirates, Bahrain, and Iraq. Bahrain declared force majeure after damage at the Sitra refinery, Iraq cut output as Hormuz disruption blocked crude flows, and Qatar reported the loss of around one-sixth of LNG capacity after damage to Ras Laffan. That sequence transformed the conflict into a regional energy and trade shock with global reach.

Persian Gulf oil shipments has lifted energy prices. Its adjusted high-energy scenario lowers 2026 world merchandise trade volume growth to 1.4% from 1.9% and trims global GDP growth to 2.5% from 2.8%. The IMF added that a persistent 10% increase in oil prices can raise global headline inflation by around 40 basis points and reduce global output by 0.1 to 0.2 percentage points. Together, those estimates show that the conflict now acts through the central channels of inflation, trade, and growth rather than through a narrow regional risk premium.

The Strait of Hormuz sits at the center of this transmission mechanism. UNCTAD estimated that the strait carries around one quarter of global seaborne oil trade along with major LNG and fertilizer flows. Its briefing shows ship transits through the strait falling by 97% from the February average, while oil and gas prices surged between 27 February and 9 March. In that setting, Hormuz becomes the hinge that connects military escalation with industrial input costs, European gas pricing, Asian energy security, fertilizer markets, freight rates, bunker fuel, and food pressure across import-dependent economies.

Washington is sending thousands of additional Marines and sailors to the region, adding to an existing U.S. military presence of about 50,000 personnel. At the same time, the White House has pressed allies to help secure passage through Hormuz, while European partners have adopted a posture centered on safe passage, inflation control, and de-escalation. This combination places deterrence, burden sharing, maritime security, and macro stabilization inside the same strategic field.

Outside powers have entered the same equation. China warned that the war threatens global energy supplies, trade routes, and economic stability. Russia declared political support for Tehran through a Kremlin message that described Russia as a loyal friend and reliable partner. These positions matter because they influence shipping expectations, sanctions politics, commodity diplomacy, and the pricing of geopolitical persistence across global markets.

Financial transmission has also accelerated. A U.S.-backed maritime reinsurance plan led by Chubb seeks to restore commercial shipping through Hormuz, which shows how finance and logistics now function as strategic infrastructure alongside naval and air power. UNCTAD’s assessment also tracks sharp rises in tanker costs, bunker fuel, and bond yields across exposed Gulf economies, confirming that the shock now moves through transport costs, refinancing conditions, and imported inflation as well as through crude and LNG benchmarks.

The confrontation therefore belongs to the category of wider-system conflict. Geography, energy, trade, shipping, insurance, alliance politics, inflation, and growth forecasts now move inside one connected field. Each strike on infrastructure or transit routes now feeds into industrial costs, consumer prices, sovereign funding conditions, and policy choices across regions far from the battlefield.

By the way, before we move to next section, a new TACO just happned today because of:

Hormuz as the strategic hinge

The closure of the strait has stopped the passage of about 20% of the world’s oil and liquefied natural gas since 28 February. The International Maritime Organization said around 20,000 seafarers on nearly 2,000 ships west of the strait are caught in the disruption, and Iraq has already declared force majeure because Hormuz disruption halted most crude exports and cut Basra output from 3.3 million barrels per day to 900,000. The narrow channel now links missile risk and naval pressure with global energy supply, shipping capacity, and state revenue.

The U.S. Energy Information Administration reported that Hormuz carried about 20 million barrels per day in 2024, equal to about 20% of global petroleum liquids consumption and more than one-quarter of seaborne oil trade. Around one-fifth of global LNG trade also moved through the strait. EIA estimated bypass capacity in Saudi and Emirati pipelines at about 2.6 million barrels per day, so rerouting covers only a small share of the volume at risk. UNCTAD’s adds that, one week before the conflict, crude oil represented 38% of the relevant seaborne trade categories moving through Hormuz, LPG 29%, LNG 19%, refined oil products 19%, and chemicals, including fertilizers, 13%. Asia receives 84% of Hormuz crude and 83% of Hormuz LNG, which gives the chokepoint immediate reach across the main import centers of the global economy.

Traffic data show how fast military pressure becomes market shock. UNCTAD recorded an average of 141 daily ship transits during 1–27 February, followed by a 97% collapse. Brent had risen 27% and Dutch TTF gas 74% from then. Chubb joined a $20 billion maritime reinsurance plan because shipowners, cargo interests, and financiers need war-risk cover before normal commercial flows can return. Passage through Hormuz therefore sets the marginal cost of energy movement, insurance, and cargo finance for economies far beyond the Gulf.

The macroeconomic spillover now reaches trade volumes, food costs, inflation expectations, and sovereign financing. Severe curtailment of Persian Gulf oil shipments could cut world merchandise trade volume growth in 2026 to 1.4% from 1.9% and lower world GDP growth to 2.5% from 2.8% under its high-energy-price scenario. This connects higher Gulf energy prices with fertilizer stress and broader food-system pressure. Besides, government bond yields in the United States and Europe jumped as investors priced a more persistent inflation shock, with the UK 10-year yield rising above 5% and markets shifting from rate-cut expectations toward rate-hike expectations. Hormuz therefore acts through central-bank reaction functions and sovereign borrowing costs as well as through the oil curve.

Kharg island is the energy chokepoint

Kharg Island now stands as one of the clearest points where battlefield logic and macroeconomic leverage meet. Kharg handles 90% of Iran’s oil exports, and some reports showed that Iran continued to ship between 1.1 million and 1.5 million barrels per day during the war. Iran exported about 1.7 million barrels per day so far in 2026, with about 1.55 million barrels per day moving through Kharg, after exports had reached about 2.17 million barrels per day in February. That concentration gives the island a dual role. It functions as Iran’s main export valve and as a target whose condition shapes global expectations for supply, escalation, and endurance.

The strategic value of Kharg comes from concentration of flow, storage, and loading capacity in a single hub. The island holds storage capacity of roughly 30 million barrels and held about 18 million barrels of crude. Its offshore position also matters. Kharg as sitting 16 miles from Iran’s coast in waters deep enough for very large tankers that cannot approach the shallow mainland coast. That gives Kharg a special role in converting upstream production into export revenue at scale. A strike, blockade, or sustained interdiction at that one node would reach revenue, shipping schedules, tanker availability, buyer confidence, and physical balances at the same time. Market analysis are pointing to a potential loss of around 2 million barrels per day from the market if Kharg infrastructure were taken out.

Kharg also carries the fiscal weight of Iran’s external energy system. The U.S. Energy Information Administration’s 2025 SHIP Act report estimated Iran’s crude oil and condensate export revenue at $43 billion in 2024, with exports at 1.483 million barrels per day. The same report estimated exports to China at 1.444 million barrels per day in 2024, which shows how strongly Iran’s export realization depends on a narrow buyer corridor linked to Asian demand. Iranian oil accounted for 11.6% of China’s seaborne imports so far in 2026. Kharg therefore sits at the intersection of Iranian fiscal resilience and Chinese energy procurement. The island matters because it converts production into cash flow, and it does so through a commercial route that remains central to a major importing power.

U.S. forces struck more than 90 Iranian military targets on Kharg while preserving the oil infrastructure. That decision revealed the hierarchy of escalation inside the war. Military assets on the island formed the first target set, while the export system itself remained the larger coercive lever. Trump administration was considering plans to occupy or blockade Kharg in order to pressure Iran to reopen the Strait of Hormuz. Kharg thus moved from vulnerable infrastructure into active strategic bargaining space. In this setting, the island serves as a pressure point whose control, impairment, or threatened seizure can alter both the battlefield and the oil market.

Energy shock and the macro transmission channel

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Sufficient statistics and minimal signals

This lecture defines a trading signal through sufficiency rather than through a rule, threshold, or pattern. A signal is a compression of the decision-time information set that preserves the conditional law of the declared outcome, so predictive meaning remains tied to the future law rather than to a single backtest metric. The lecture then develops minimality, bottleneck design, and representation stability as the conditions that make that compressed state usable under finite data, drift, and noise.

What’s inside:

1. Sufficiency as the definition of signal. A signal is a representation that preserves the conditional distribution of the outcome, which separates signal from strategy and makes feature relevance a property of conditional-law preservation rather than raw correlation.

2. Signal as compression with law preservation. Compression becomes signal construction when the reduced state carries the same predictive content as the full decision-time information set, which lets the system operate on a smaller and more stable representation without changing the law it conditions on.

3. Approximate sufficiency in practice. Exact sufficiency appears as the ideal case, while practical sufficiency is defined through a declared tolerance using divergence, proper-score regret, and the residual predictive content that remains outside the bottleneck.

4. Minimal sufficient sets. The lecture shows that a minimal signal is the smallest representation that still preserves the full conditional law, removes nuisance variation, lowers selection pressure, and keeps the signal portable across operators and model classes.

5. Non-uniqueness of sufficient representations. Many different representations can preserve the same predictive law, so the scientific question is whether preservation exists, while the engineering question is which representation remains most stable under noise, drift, and deployment constraints.

6. Bottleneck trade-offs. Bottleneck design becomes an information-budget problem that balances relevance against representational capacity, because too little compression increases variance and selection pressure, while too much compression removes predictive fragments the outcome still needs.

7. Overcompression and undercompression. The lecture explains that overcompression distorts the conditional law by blending distinct regimes, while undercompression creates a high-capacity overfitting surface where sharp in-sample structure often fails under recurrence and proper scoring.

8. Representation stability. A minimal signal must retain stable meaning under measurement noise, drift in state occupancy, and regime mixtures, so conditional beliefs, calibration, and divergence structure remain coherent when the signal moves from research into live deployment.

Sample

1.93MB ∙ PDF file

Download

Download

Read more