Table of contents:

1. Introduction.

2. Global macro regime after the February rupture.

3. Trade, tariffs, and the AI capital cycle.

4. Financial conditions, emerging markets, and the new debt geometry.

5. Europe between inflation, industry, and rearmament.

6. Chokepoints, rare earths, and the structure of strategic rivalry.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

The central feature of this moment is that macroeconomics can no longer be separated from geopolitics. Oil routes, LNG flows, shipping insurance, semiconductor supply, rare-earth processing, defence procurement, and AI infrastructure now sit inside the same transmission system. A military clashes can move inflation expectations. A tariff decision can alter capital spending. A mineral export license can affect vehicle production, renewable deployment, and defence readiness. The channels that once belonged to separate policy worlds now reinforce one another.

The AI capital cycle is still creating demand for chips, servers, data centres, power systems, and advanced equipment. Fiscal support and defence spending are cushioning parts of the cycle. Services trade remains more resilient than goods trade. But the composition of growth has changed. Expansion increasingly depends on investment linked to security, technology, energy resilience, and state-backed industrial strategy.

That shift creates a new hierarchy among economies. Countries with energy supply, fiscal space, reserve buffers, advanced industrial capacity, or control over strategic inputs gain room to maneuver. Countries dependent on imported fuel, external financing, weak currencies, or narrow export bases face a tighter constraint. Financial markets are beginning to price that difference. The old distinction between advanced and emerging markets matters less than exposure to chokepoints, debt rollover, energy costs, and industrial depth.

Europe sits at the center of this adjustment because it faces all of these pressures at once. Imported energy inflation, weak industrial momentum, rising defence commitments, and the need to rebuild competitiveness without abandoning decarbonisation, all at the same time. The United States sits in a different position, supported by technology investment and capital inflows but still exposed to inflation persistence and strategic overextension. China retains industrial scale and export strength, while also relying on external demand and commanding key processing stages in critical supply chains.

You can go deeper by reviewing this:

OECD Economic Outlook, Interim Report March 2026

1.66MB ∙ PDF file

Download

Download

World Economic Outlook, April 2026

1.1MB ∙ PDF file

Download

Download

Global Trade Outlook And Statistics March 2026

1.44MB ∙ PDF file

Download

Download

The result is a world economy still moving forward, but no longer under the assumptions that defined the previous era. Efficiency is giving way to resilience. Open access is giving way to managed access. Supply chains are being judged not only by cost, but by survivability. The rest of 2026 will therefore depend less on whether growth exists and more on what kind of growth remains possible when war risk, tariffs, energy security, AI investment, debt pressure, and strategic bottlenecks all operate at the same time.

Subscribe now

Global macro regime after the February rupture

The macro setting in April carries a dual character with unusual clarity. Growth continues, yet the safety margin around that growth has narrowed. Productive capacity still expands, yet the cost of keeping that expansion on course has risen. The world economy entered the year with stronger momentum than many institutions had expected in late 2025, then moved into a new phase when the Middle East conflict. That event arrived while the international trade system was still adjusting to the earlier shift in United States trade policy, which means the present environment combines war risk, tariff uncertainty, supply-chain rewiring, and a tighter premium on energy security inside a single macro regime. The result!? The economy still moves forward…

Even under that path, the global outlook loses altitude. World growth is projected at 3.1 percent in 2026 and 3.2 percent in 2027, while global headline inflation rises to 4.4 percent in 2026 before easing to 3.7 percent in 2027. It confirms continued global expansion, yet it also confirms a return of the energy channel as a first-order driver of inflation, monetary timing, household purchasing power, and corporate margins. The downside scenarios reinforce the message. In the adverse case, growth falls to 2.5 percent and inflation rises to 5.4 percent in 2026. In the severe case, growth falls to 2 percent in both 2026 and 2027 while inflation rises above 6 percent. That spread between the reference path and the tail scenarios defines the new regime.

This chart gives the best opening frame for this because it shows that the February rupture belongs to a longer rise in geopolitical tension across regions. The 2020s stand above the 2000s and 2010s in every major region shown. The Middle East and Africa carry the highest level, which fits the immediate conflict geography. Europe and Asia-Pacific also register a clear step-up, which speaks to the wider strategic setting that already included the war in Ukraine, technology rivalry, sanctions, and maritime frictions before the latest shock. North America and South America show a lower level than the conflict core, yet they also sit above earlier decades.

That structural shift affects the transmission channels because it moves through the entire macro system. There are three central channels. The first is the direct commodity-price shock, which raises energy and input costs, disrupts transport, and weakens real incomes. The second is the expectations channel, where firms and workers seek to recover losses and thereby give price persistence a stronger foothold. The third is the financial channel, where higher risk premia, weaker asset valuations, tighter credit conditions, and capital outflows reduce demand and complicate policy. These channels reinforce one another. Oil and gas feed into fertilizer, freight, aviation fuel, and industrial inputs. Higher shipping and insurance costs pass into goods prices. Exchange-rate depreciation amplifies imported inflation, especially in commodity-importing emerging economies. In that setting, macroeconomics and statecraft operate in one field of transmission. Growth, inflation, shipping security, energy access, sanctions, and industrial policy act together.

The regional pattern adds another layer of meaning because the large macro blocs entered this point with different structures. The activity in China and the United States had been stronger than expected, yet that strength carried an uneven internal composition. In China, exports ran ahead while domestic activity, especially housing, remained softer. In the United States, strong activity coincided with low employment growth and slower labor-force expansion. At the same time, the trade regime itself remained in motion. United States statutory tariff rate stood about 5.3 percentage points below the level assumed after court rulings and executive actions, while other countries advanced trade negotiations and new partnerships of their own.

The United States enters this year with firmer demand conditions than most advanced peers and with a continuing technology-investment wave that still supports capital spending. AI-related technology investment added about 0.5 percentage point to US GDP growth, with sizeable spillovers into Asia through import demand for equipment and components. At the same time, inflation pressure in the United States remains higher than expected. Core personal consumption expenditure inflation at 3.1 percent year over year, and the wider WEO discussion presents a path in which US core inflation returns to target during 2027 rather than through an immediate or automatic easing sequence. In the case of China, export performance and industrial scale still carry the expansion. China is projected to grow 4.4 percent in 2026 and 4.0 percent in 2027, with external demand and some fiscal support offsetting a weaker domestic side where consumption remains soft. The euro area enters the year with modest growth and a higher sensitivity to imported energy and industrial input costs. The WEO places euro area growth at 1.2 percent in 2026 and 1.5 percent in 2027, while the euro area headline inflation rises above 2 percent in 2026 and remains above target in 2027.

In the United States, the issue centers on second-round price pressure because domestic demand has held up and inflation remains elevated. In the euro area, the issue centers on imported-price exposure because energy and other tradable inputs still form a key external channel into headline inflation. In China, the policy mix carries a different balance in which low inflation, export competitiveness, the exchange rate, and fiscal support weigh more heavily than overheating fears. The IMF press briefing therefore recommends patience from central banks under the reference case, together with clear communication and readiness to act if inflation expectations shift. On the fiscal side, the Fund emphasizes targeted and temporary support over broad subsidies or price caps, since public debt burdens have already reduced fiscal room.

The world economy still retains pillars of support, and these pillars explain why the baseline remains one of slower expansion rather than a broad contraction. The first pillar is the AI capital cycle. Technology investment remains a real source of demand for semiconductors, network equipment, power systems, and digital infrastructure, while the import intensity of that wave supports trade and investment spillovers across Asia. The second pillar is fiscal support. Fiscal support in various economies and the role of public spending in cushioning economic activity. The third pillar is adaptation through trade and inventories. Global trade remained robust because technology-related exports offset slower momentum in other product categories, and the IMF expects world trade volume growth to slow from 5.1 percent in 2025 to 2.8 percent in 2026 before improving to 3.8 percent in 2027. Goods trade faces a steeper deceleration, while services trade shows greater resilience. That distinction favors economies with stronger positions in digital, financial, and business services and gives the global system a partial buffer against fragmentation in goods.

This other chart deepens the argument because it shows that the present shock lands on an international system already shaped by long-running external asymmetries. Global imbalances may decline only modestly over the medium term. Some cyclical narrowing comes from expansionary fiscal packages in surplus economies, yet that narrowing faces countervailing forces. A technology-driven investment surge continues to attract capital toward the United States, and is expected stronger US productivity and high returns in technology-related sectors to keep capital inflows elevated. The same idea states that positive wealth effects and sustained inflows keep the US current account deficit wider than the level seen during the decade before the pandemic. At the same time, China’s continued reliance on export-led growth, together with limited rebalancing toward domestic consumption, sustains another major source of external imbalance.

That is important for geopolitics due to external imbalances influence power, and vulnerability. Current-account surpluses provide room for reserve accumulation, outward finance, and shock absorption. Large debtor positions sharpen sensitivity to financing conditions and exchange-rate moves. In the presence of higher geopolitical risk, those balance-sheet realities shape coalition behavior, sanctions exposure, reserve management, and the capacity to absorb commodity shocks.

Here the estimation is one-standard-deviation. Geopolitical risk leads to a decline in real GDP of about 0.8 percent one year after the initial shock. The main channels run through weaker private consumption and weaker investment, while consumer prices rise and the nominal effective exchange rate falls. Besides average geopolitical risk in Europe rose by about 1.2 standard deviations after Russia’s invasion of Ukraine and remained elevated at about 0.5 standard deviation now. This chart shows that geopolitical shocks reduce output through spending decisions made by households and firms, lift inflation through price channels, and weaken currencies through financial and external adjustment.

The broader implication is that supply security now carries a direct macro price. Maritime corridors, tanker insurance, refinery access, data infrastructure, and cross-border finance sit much closer to the center of the business cycle than they did under the earlier assumption of a largely rules-based and low-friction global system. Fertilizer markets, food security, reversals in capital flows, and abrupt adjustments in asset prices are all part of the chain of downside risks, especially for economies with weaker fiscal frameworks and lower external reserves.

Trade, tariffs, and the AI capital cycle

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

What a trading hypothesis is not

The lecture explains that charts, indicators, and backtests can all be useful, but none of them are complete hypotheses on their own. Their value appears only when they are connected to mechanism, scope, timing, variable design, and evidence, so the researcher can move from surface regularity to a claim that can be tested, challenged, and refined.

What’s inside:

1. What a trading hypothesis is not. The lecture begins by separating a genuine hypothesis from three incomplete objects (a chart pattern, an indicator, and a backtest result).

2. A chart pattern without explanatory content. A visual formation may reveal tension, compression, rupture, or recovery, but it remains only an empirical clue until the researcher explains what market process could have produced it.

3. Visual regularity without mechanism. Repeated shapes on a chart can create the impression of structure, but research begins only when those shapes are translated into participants, frictions, information, liquidity, or structural relations.

4. Repetition without causal interpretation. A pattern that appears many times does not automatically become a hypothesis. Repetition gains research value only when it is linked to a stable cause and a clear condition under which it should persist or disappear.

5. Description without research value. Statements such as “price tends to bounce after strong declines” are useful starting observations, but they need scope, variables, comparison design, and evidence criteria before they become researchable.

6. An indicator without market logic. Indicators compress data into signals, thresholds, scores, or ratios, but they matter scientifically only when they represent a real market condition such as crowding, strain, dislocation, liquidity withdrawal, or delayed adjustment.

7. A backtest result without a prior claim. A good historical result is not enough. The lecture emphasizes that a backtest should examine a claim stated in advance, rather than becoming a machine for discovering attractive performance and inventing explanations afterward.

8. The importance of forward logic. Strong research moves from observation to mechanism, from mechanism to formal claim, and from claim to evidence. This order gives meaning to both success and failure, allowing the researcher to learn from the market rather than simply collect historical artifacts.

Sample

1.87MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Data-Shape transformation.

3. Identifying the risks and limitations.

4. Data shape transformations for predictive features.

   1. State-space approach.

   2. Dynamic benchmark neutralization.

   3. Stochastic trend removal via cointegration.

   4. Multiscale isolation approach.

   5. Bounded state encoding.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Imagine that a team downloads a price series, defines a target, applies a transformation, and moves on to signal design, model fitting, validation, and execution. That sequence looks efficient. However, the transformation of the data is is the first act of model construction.

That is why data-shape transformation sits at the true front line of feature engineering. The problem is whether the operator used to create that series preserves the economic structure that generates predictive edge. A feature can look impeccable under statistical diagnostics and still be useless in live trading. It can pass every formal test of stability while removing the component that contained the alpha.

This article examines that hidden fault line. It argues that data-shape transformation must be treated as a structural modeling choice rather than a preprocessing convenience. We will identify the main risks introduced by transformation operators, explain how they distort the topology of predictive information, and then study five targeted transformations designed for distinct market nuisances: state-space innovations, rolling benchmark residuals, cointegration spreads, multiscale Haar details, and rolling range positions. The objective is not to make features look cleaner. The objective is to ensure that the shape of the data remains faithful to the market mechanism the model is supposed to learn.

Every transformation enforces a decision about relevance. It selects one part of the market path and suppresses another. It may emphasize short-horizon variation, eliminate common drift, compress local extremes, or redefine the position of the current price relative to its recent history. Before any model is trained, the researcher has already imposed a theory of what matters.

By the way, you can go deeper in this topic by exploring this paper.

Canonical time-series characteristics

1.3MB ∙ PDF file

Download

Download

That choice is decisive because financial data are layered. A single price series contains slow structural movement, temporary dislocations, benchmark comovement, volatility clustering, and random noise at the same time. A predictive model is never interested in all these layers equally. It depends on a narrow component that matches the strategy horizon and economic logic. The transformation either isolates that component or removes it.

Data-Shape transformation

The starting phase of feature engineering contains a severe trap because it masquerades as a trivial data processing step. A quant team downloads a time series of prices, computes a target variable, and faces the decision of whether the predictive model should ingest the raw level, a discrete difference, a standardized statistical score, a regression residual, an equilibrium spread, or a bounded state variable. The standard academic and industry reflex is to treat this decision as a technical prerequisite, often labeled preprocessing or data normalization. That classification is a mistake.

In many pipelines, this step is outsourced to data engineers or automated via standard library defaults before the alpha researchers even begin their core modeling. This assumes that data transformation is a lossless translation of information. It is not. Once a specific transformation is applied to the raw time series, the researcher has declared what specific variation in the market path constitutes actionable information and what variation constitutes noise. If the chosen transformation is misaligned with the economic mechanism the strategy intends to exploit, the resulting model can exhibit perfect statistical behavior while remaining economically blind.

In our domain, this situation is unforgiving because the price is a cumulative, high-memory object. It aggregates historical microstructure shocks, macroeconomic regime shifts, discrete splits in volatility, slow reallocations of institutional capital, structural risk premia, and continuous benchmark movements.

A raw price level carries a massive amount of historical baggage that the short-term or medium-term target variable is invariant to. Assume the core research problem is to predict the magnitude and direction of a short-horizon log return y_t+h. The actual feature matrix the learning algorithm receives is rarely the raw price p_t. Instead, the model ingests a transformed object x_t = T(F_t), where F_t represents the filtration generated by the history of prices and exogenous covariates up to time t. The selection of the transformation operator T is the actual front line of the research design. This operator dictates whether the vector x_t represents a local surprise, a relative displacement, a structurally bounded state, etc.

Identifying the risks and limitations

The risk surface introduced by the operator T is broader and more destructive than standard statistical literature admits. Researchers frequently inherit standard transformation recipes without interrogating their geometric implications or their effect on the data topology. They compute discrete log returns r_t = \ln(p_t) - \ln(p_t-1) because financial libraries expect stationary inputs. They build rolling z-scores because the resulting chart oscillates symmetrically around zero and fits neatly into the activation functions of deep neural networks. These choices impose heavy structural assumptions. A standard return calculation deletes the absolute price level and the cumulative memory of the path. A residual calculation deletes the variance explained by the specified reference model.

The first specific risk is false statistical comfort. A transformed time series can be manipulated to exhibit perfect weak-sense stationarity, zero mean, and constant variance, while simultaneously failing to express any tradable market state. Any researcher can run an Augmented Dickey-Fuller or KPSS test, achieve a spectacular p-value, and incorrectly assume the feature is ready for predictive modeling. Extracting white noise from a time series guarantees stability, but white noise contains zero predictive alpha. The researcher has converted the price into a stable sequence that is devoid of economic value.

The second risk is structural signal “amputation”. A transformation can neutralize the exact slow-moving component that carried the predictive edge. This failure mode occurs when researchers impose rigid discrete differencing, rolling normalization, or arbitrary short lookback windows without first establishing the exact holding period the systematic strategy is designed to exploit. If a macro trend-following strategy relies on three-month momentum, applying a 5-day rolling z-score to the input features will delete the low-frequency eigenvalue that the model requires to identify the trend.

The third risk is benchmark confusion and specification transfer. When a transformation operator incorporates a secondary series—such as a benchmark ETF, a sector index, or a paired asset—the resulting feature inherits the specification risk of that external asset. A regression residual is only valid if the variable against which it is residualized represents a stable economic factor. If a researcher isolates a tech stock by residualizing it against the QQQ index, and the index undergoes a massive constituent rebalance or suffers a liquidity shock, the resulting feature vector will exhibit severe volatility that has nothing to do with the target stock’s idiosyncratic state. The feature becomes contaminated by the reference frame.

The fourth risk is causality leakage. This is the most common fatal error in systematic trading. When the transformation relies on parameters estimated using information outside the filtration F_t, the feature becomes predictive in the research environment and false in the production environment. Using centered rolling windows, applying Principal Component Analysis eigenvectors derived from the full target matrix, computing full-sample mean vectors for standardization, or extracting post-hoc equilibrium coefficients creates a matrix that cannot exist in live trading. Forward-looking bias is adept at hiding within the scaling constants of data-shape transformations.

The fifth risk is geometric distortion. This transformation alters local curvature, variance persistence, variance clustering, and the relative distance metric of excursions. Consider a neural network optimizer relying on Euclidean distance to compute gradients. Mapping a log-normal price path into a bounded state space between zero and one forces the distance metric near the boundaries to behave differently from the distance metric in the center of the distribution. A raw price move that represents a massive four-standard-deviation tail event might only translate to a 0.01 shift in the bounded feature space if the state was already sitting at 0.98. Extracting a state space model collapses a continuous smooth drift into a localized sequence of forecast errors clustered around zero. This alters the fundamental definition of what a market burst represents to the downstream algorithm.

The conflict between raw data and transformed geometry culminates in a specific, observable failure in the research lifecycle. A model is built on a transformed feature set. The initial teardown looks immaculate. The in-sample backtest “demonstrates” it works. The cross-validation metrics are stable across distinct market regimes. The feature importance vectors are dense and logical. The Sharpe ratio appears robust across multiple chronological subsamples, and the drawdowns are well contained.

Then… the model is advanced to paper trading or live execution, and the predictive relationship collapses. The alpha decay instantaneous. The post-mortem analysis reveals that the algorithm never learned the underlying market mechanism. The optimization process learned the geometry introduced by the transformation operator T. It fit itself to sample-specific trend removal artifacts, spurious benchmark dependencies, bound-rejection behavior dictated by rolling window mechanics rather than market participants, and local scaling normalization patterns that possessed no forward-looking stability.

Subscribe now

Data shape transformations for predictive features

If the strategy is engineered to trade benchmark-relative mispricing, then the absolute market benchmark exposure is the primary nuisance structure. If the strategy capitalizes on statistical arbitrage and pair reversion, the common stochastic trend is the primary nuisance structure. If the strategy dictates short-horizon momentum reactions, slow multi-month macroeconomic drift is the primary nuisance structure. If the strategy operates as a local-state mean-reversion classifier, the absolute nominal price level is the primary nuisance structure.

This perspective enforces a precise operational taxonomy of data-shape transformations. The state space algorithm eliminates predictable local trends governed by state-space transition matrices, leaving the prediction surprise. The rolling_regression_residual algorithm eliminates time-varying benchmark exposure, yielding a relative-value deviation vector. The cointegration_spread algorithm eliminates common stochastic drift based on equilibrium vector autoregression logic, resulting in a stationary spread conducive to error-correction modeling. The haar_detail_same_length algorithm eliminates low-frequency power at a selected scale, leaving a targeted multiscale shock component. The rolling_range_position algorithm eliminates absolute nominal levels and local variance scaling, yielding a bounded coordinate variable.

These five methods represent interventions designed to blind the model to specific, targeted market nuisances. However without the right measures, the same mistakes could continue in some of them. Now that you’re aware of them, we can move forward.

State-space approach

The first targeted transformation is the state-space. In technical literature, this is defined as the one-step-ahead prediction error derived from a local linear trend state-space model. This transformation alters the time-series geometry by removing the exact variation that the internal state estimate already expects the system to produce at time t. This isolation mechanism is required when the raw log-price is cumulative, exhibiting a smooth drift that masks the actual high-frequency surprise component necessary for short-horizon prediction.

The formulation relies on setting up the observed log-price y_t as the output of a local linear trend model. The system requires an observation equation and state transition equations:

Here, the unobserved latent state consists of the level μ_t and the slope β_t. The disturbances ε_t, η_t, and ζ_t are mutually independent Gaussian white noise processes governing the measurement error, the level shift, and the slope shift. The state space model operates recursively. Given the filtration up to t-1, it produces the optimal linear one-step-ahead prediction E[y_t | F_t-1]. The feature we extract is the sequence v_t:

It is crucial to understand that v_t represents the precise segment of the current price movement that could not be predicted by the prior dynamic state. In the context of algorithmic trading, this means the feature measures orthogonal surprise rather than raw directional motion.

The rationale for this transformation is embedded in market microstructure. A high-frequency algorithm processing order flow must differentiate between a market that ticks upward at a steady, anticipated velocity and a market that prints an identical upward tick against a declining latent trend. The nominal difference y_t -y_t-1 will record an identical positive value in both scenarios. The state space v_t will output a minor value for the first scenario and a massive positive value for the second. Innovation-based feature engineering forces the learning algorithm to respect the conditional context of the regime.

def state_space(price_series: pd.Series) -> pd.Series:
    """
    Extracts the one-step-ahead forecast error from a local linear trend
    state-space model.
    """
    lp = price_series.rename("log_price")

    model = UnobservedComponents(lp, level="local linear trend")
    result = model.fit(disp=False)

    space = pd.Series(
        result.filter_results.forecasts_error[0],
        index=lp.index,
        name="state_space",)

    space.iloc[0] = np.nan
    return space

However, the researcher must decide if local drift should be neutralized. For execution timing, short-term liquidity provision, and algorithmic burst-detection, neutralizing drift is necessary. Utilizing v_t ensures the resulting feature matrix is robust across shifting volatility regimes because the state-space covariance matrices adapt to changing latent slopes. The downstream classifier is thus relieved of the computational burden of discovering that adaptation matrix from raw inputs.

The primary failure mode of this transformation is specification error, which occurs when the assumed dynamics of the state-space model fail to map onto the conditions of the asset’s microstructure. The standard local linear trend model relies on continuous Gaussian diffusion. If the true data generation process is instead dominated by discrete Poisson jumps—such as overnight gaps, scheduled FOMC macroeconomic releases, or sudden liquidity vacuums—the linear Gaussian assumptions break down. When a structural price jump occurs, the linear filter cannot snap to the new regime. Instead, it lags, requiring multiple timesteps for the state space gain matrix to fully correct the latent state components. During this structural catch-up phase, the sequence v_t will output an artificial cluster of large values. These values no longer represent an orthogonal market surprises but a deterministic filter correction error.

Researchers are tempted by the aesthetic stability of the a posteriori residual, often extracted via two-sided smoothing algorithms provided by default in many statistical libraries. Because these smoothers utilize future price information from t+1 to T to optimize the state estimate at time t, utilizing them injects terminal forward-looking leakage into the feature matrix. The resulting in-sample backtest will generate flawless, profitable metrics, while the live production system will bleed money.

Dynamic benchmark neutralization

The second transformation points to a different geometric annoyance known as time-varying external beta. The label is the rolling Ordinary Least Squares residual. In systematic portfolio management, this is recognized as the dynamic hedge residual. The core logic dictates that the variance of the target asset y_t is contaminated by the variance of an external factor x_t, which could represent a sector ETF, a broad index future, or a principal component basket. To model the asset’s idiosyncratic state, the researcher must regress the asset against the reference factor and isolate the residual error.

The model specifies:

Because the relationship between financial assets is never static, α_t and β_t cannot be estimated over the full sample. They must be approximated over a rolling window of length w. The parameter vector is computed utilizing the local design matrix X_t-w+1:t. The resulting transformed feature is the out-of-sample or end-of-sample deviation:

This specific subtraction is geometrically distinct from subtracting simple returns or computing a ratio. The dynamic residual projects the asset into the null space of the benchmark over the specified lookback window. The output geometry abandons the absolute coordinate system and adopts a relative, benchmark-neutral coordinate system.

def rolling_regression_residual(
    price_series: pd.Series,
    benchmark_series: pd.Series,
    window: int = 80,) -> pd.Series:
    """
    Computes the dynamic rolling residual of the asset relative to the benchmark.
    """
    y = price_series.rename("log_price")
    x = benchmark_series.rename("log_benchmark")

    X = sm.add_constant(x)
    result = RollingOLS(y, X, window=window).fit()
    params = result.params

    fitted = params["const"] + params["log_benchmark"] * x
    residual = (y - fitted).rename(f"rolling_regression_residual_{window}")

    return residual

In algorithmic trading, raw price vectors deceive predictive models by presenting sector-wide beta expansion as idiosyncratic momentum. A single equity instrument may display severe overbought technical geometry when the underlying condition is uniform sector strength. A cryptocurrency token may exhibit severe downward drift when the entire beta complex is repricing. Feeding raw prices into a deep neural network forces the network to allocate hidden layers to the task of inferring the benchmark matrix and computing the subtraction. By passing û_t, the transformation executes the subtraction analytically, presenting the network with the pure, actionable relative-value deviation.

The geometric shift transforms a trend-dominated price path into a sequence that continually crosses zero, widening when the beta relationship degrades and compressing when the hedge is tight. This sequence describes relative-value dislocations that are primed for statistical arbitrage.

The implementation contains a critical stability trade-off governed by the window parameter w. A minimal window length allows the matrix inversion (X^TX)^-1 to adapt to microstructural shifts in beta, but exposes the parameters to severe estimation noise and matrix ill-conditioning. A maximum window length stabilizes the eigenvalue structure of the covariance matrix but renders the hedge ratio unresponsive to macroeconomic events.

Stochastic trend removal via cointegration

The third data-shape transformation addresses the phenomenon of common stochastic trends. Referred to as the Engle-Granger residual or the cointegration spread, this method supersedes local benchmark neutralization by enforcing an equilibrium hypothesis. It relies on the proof that while two distinct log-price series y_t and x_t may both possess unit roots, a specific linear combination of the two series may be integrated of order 0, yielding a stationary process.

If there exists a parameter vector [α, β] such that the linear combination:

is a stationary process, the variables are defined as cointegrated, and s_t represents the equilibrium spread. The Engle-Granger two-step procedure first estimates the scalar β via ordinary least squares on a causally valid training set, and applies an Augmented Dickey-Fuller test to the residual sequence s_t to reject the null hypothesis of a unit root.

The concept of the cointegration spread over a simple rolling OLS residual lies in its equilibrium framework and the physical market mechanics that enforce it. A dynamic rolling residual states that a local, temporary variance explanation exists. It makes no claims about the terminal behavior of the asset pair. A cointegration differential spread that the components of the two assets are structurally linked to each other by an exogenous economic force. This force could be identical underlying cash flows (in the case of dual-listed equities), arbitrage boundaries enforced by Exchange Traded Fund creation and redemption mechanisms, or capitalized statistical arbitrage desks trading the structural mean. The raw log-prices are permitted to drift into uncharted coordinate space, but the distance metric between them, s_t, is constrained to revert to its historical mean.

The standard Error Correction Model representation models the short-term dynamics of the system as a direct response to the current magnitude of the equilibrium deviation, while controlling for localized high-frequency shocks:

In this structural equation, λ represents the speed-of-adjustment coefficient, and it is the most critical parameter in the relative-value strategy. The predictive alpha is not contained in the absolute nominal level of y_t or x_t. The actionable alpha is isolated within s_t-1, the exact geometric distance from equilibrium. By executing this transformation, the researcher compresses the shared macroeconomic drift and aligns the input feature with the logic of convergence.

The magnitude of λ dictates the interactions of the trading book. A high absolute value of λ implies rapid mean reversion. This creates a high-turnover strategy that requires minimal capital lockup but demands extreme execution efficiency, as the profit margin per trade is small and sensitive to latency. Conversely, a low absolute value of λ implies slow, structural reversion. This dictates a low-turnover, capacity-rich strategy that suffers from capital lockup and extended periods of mark-to-market drawdown when the spread temporarily widens beyond historical norms.

def cointegration_spread(
    price_series: pd.Series,
    partner_series: pd.Series,) -> pd.Series:
    """
    Estimates the Engle-Granger equilibrium spread.
    """
    y = price_series.rename("log_price")
    x = partner_series.rename("log_partner")

    X = sm.add_constant(x)
    fit = sm.OLS(y, X, missing="drop").fit()

    spread = (y - fit.predict(X)).rename("cointegration_spread")
    return spread

Transforming non-stationary prices into a stationary spread state variable provides relief to machine learning algorithms. If a random forest or a gradient boosting machine is fed two raw, drifting I(1) sequences, it will attempt to construct arbitrary, overfit split points in coordinate spaces that the asset has never historically visited. When the asset prices drift beyond the maximum values observed in the training set, the tree algorithm can only output a constant terminal leaf value, breaking the model. Similarly, feeding unbounded I(1) sequences into deep neural networks causes severe gradient instability and forces activation functions into extreme saturation zones. By feeding the optimization algorithm the bounded sequence s_t, the algorithm is restricted to evaluating the magnitude and velocity of the equilibrium error. The model focuses its entire computational capacity on timing the reversion and sizing the risk optimally, rather than wasting matrix operations attempting to subtract non-stationary trends.

Remember that correlation does not imply cointegration. Two assets can exhibit a Pearson correlation coefficient of 0.95 over a multi-year horizon while failing unit root tests on their residuals. Trading this spurious pattern leads to the classic relative-value failure where the spread widens, the model increases the position size assuming mean reversion, and the error variance expands until you face a margin call.

Furthermore, a statistically verified stationary spread is not guaranteed to be economically tradable. The theoretical spread s_t ignores market friction. If the half-life of mean reversion exceeds the margin constraints of the portfolio, the position will be forcibly liquidated before the equilibrium error corrects. Additionally, if the borrow costs on the short leg of the spread exceed the expected mathematical yield of the reversion, the statistical feature is invalid in production.

Multiscale isolation approach

The fourth transformation steps away from reference benchmarks and equilibrium partners to address internal scale contamination. The algorithm extracts the Haar wavelet detail coefficient. In signal processing, this operator isolates a multiscale high-frequency component while decimating low-frequency drift. This shape transformation is deployed when the researcher has established that the predictive alpha resides within a very specific time scale, and is masked by the simultaneous presence of microstructural noise at lower scales and macro-level cumulative drift at higher scales.

Financial order books process an amalgamation of participants operating at different frequencies. Tick-by-tick is dominated by market makers canceling and replacing quotes. Intraday minutes to hours is dominated by algorithmic execution engines slicing large institutional blocks via VWAP algorithms. Days to weeks is dominated by fundamental portfolio reallocation. A standard differencing operator (calculating a basic log return) compresses all of these distinct horizons into an identical, localized logic. It forces the learning algorithm to separate these overlapping frequency bands using raw function approximation.

The discrete Haar wavelet transform operates as an orthogonal filter bank designed to decouple these scales. It applies a low-pass filter to compute local averages, pushing the slow trend information into a lower-frequency approximation space. Simultaneously, it applies a high-pass filter to compute local differences, capturing the exact detail variation occurring uniquely at that specific scale. For adjacent discrete observations a and b, the fundamental operations are:

Applied recursively in a cascaded architecture, the algorithm generates multiscale approximation coefficients and detail coefficients. A detail coefficient extracted at Level 2 represents the difference of the local averages computed at Level 1. This distinction allows to target a specific frequency band—a precise temporal horizon—rather than defaulting to the single-step increment forced by standard return calculations.

The selection of the Haar wavelet over continuous alternatives (such as the Morlet or Daubechies wavelets) is intentional in financial time series analysis. Continuous wavelets are optimized for smooth signals like acoustics or electromagnetics. Market prices are noisy. The Haar basis function is defined as a step function. It matches the market dislocation, preserving the exact magnitude of the discrete jump without introducing the artificial ringing artifacts or smoothed boundaries common in continuous wavelet convolutions.

To utilize this transformed output in a predictive feature matrix alongside standard, daily-sampled covariates, the implementation requires a length-preserving operation. The standard decimated wavelet transform halves the array length at each level. To align the sequence with the original time index t, the system must upsample the coefficients. In live trading systems, this upsampling must be causal. Utilizing standard linear interpolation imports future values into the current state. The correct engineering solution relies on zero-order hold upsampling, repeating the most recent coefficient forward to prevent any look-ahead bias.

The resulting geometric intervention transforms a drifting, multi-layered log-price path into a sequence clustered around zero, where the amplitude of the oscillations represents the magnitude of the targeted scale-specific burst. The lower-frequency trend components are annihilated.

def haar_detail_same_length(price_series: pd.Series, level: int = 2) -> pd.Series:
    """
    Extracts the multiscale high-frequency detail coefficient using a discrete
    Haar wavelet filter bank and returns it with the same length as the input.
    """
    x = np.asarray(price_series, dtype=float)
    n = len(x)
    coeffs = x.copy()
    current_len = n
    detail = None
    block = 1

    for _ in range(level):
        if current_len < 2:
            break

        used_len = current_len if current_len % 2 == 0 else current_len - 1
        current = coeffs[:used_len]

        avg = (current[0:used_len:2] + current[1:used_len:2]) / np.sqrt(2)
        det = (current[0:used_len:2] - current[1:used_len:2]) / np.sqrt(2)

        coeffs[: len(avg)] = avg
        current_len = len(avg)
        detail = det
        block *= 2

    if detail is None:
        return pd.Series(
            np.nan,
            index=price_series.index,
            name=f"haar_detail_L{level}",)

    upsampled = np.repeat(detail, block)
    if len(upsampled) < n:
        upsampled = np.r_[upsampled, np.repeat(upsampled[-1], n - len(upsampled))]

    return pd.Series(
        upsampled[:n],
        index=price_series.index,
        name=f"haar_detail_L{level}",)

This transformation is critical for horizon-specific execution logic. Consider a temporary order book imbalance generated by digesting a scheduled macroeconomic data release versus the slow absorption of a multi-day block trade. The macro release generates a visible, high-amplitude signature. However, this precise signature remains undetectable within the raw log-price sequence due to the overwhelming magnitude of the concurrent multi-month macro trend drift. By passing the multiscale detail coefficient D_t^(j) to the predictive model, the signal-to-noise ratio is maximized at the exact frequency the strategy is designed to hold the position. The optimization algorithm can recognize the structural footprint of the transient shock.

The limitations of wavelet transformations revolve around boundary conditions, asynchronous sampling, and noise. When applied to raw tick data characterized by bid-ask bounce and irregular timestamps, fine-scale detail coefficients will isolate and amplify the microstructural noise rather than the intended economic event. Wavelet transforms assume a uniform sampling grid, violating this assumption distorts the frequency targeting.

Furthermore, handling the right-hand boundary (the actual live time t) is hazardous. Wavelet filters require data points that straddle the center point of the convolution. At the live edge of the time series, the filter lacks the future data points necessary to complete the calculation. Utilizing symmetric boundary extension—a common default in open-source Python libraries—folds the array back on itself, utilizing data points from t-k as proxies for t+k. While elegant, this corrupts the state variable at the exact moment the trading system requires it. Researchers must enforce causal padding or utilize specialized boundary wavelets to ensure the feature matrix remains uncontaminated.

Bounded state encoding

The fifth and final transformation algorithm in this framework is the rolling range position. While computationally less complex than state-space estimators or orthogonal wavelets, it introduces non-linear geometric modifications. In fact, it converts a non-stationary price path into a bounded state occupancy variable.

Let P_(1),t denote the minimum order statistic and P_(n),t denote the maximum order statistic of the price sequence over a causal trailing window of length w. The rolling range position r_t is defined as:

Assuming the local variance is positive, the output space of r_t is bounded within the closed interval [0, 1]. (The addition of a microscopic constant ε in the denominator is a production requirement to prevent zero-division errors during market halts, extended limit-up/limit-down states, or extreme microstructural illiquidity where the local maximum exactly equals the local minimum).

A feature vector evaluating to 0.05 indicates that the current market price is compressed against its recent lower extreme. A value of 0.98 indicates extreme upper-bound occupancy. This transformation kills the nominal price leveles by using the local variance.

Breakout momentum models, dynamic stop-loss placement, and urgency-based execution routing evaluate expected return based on whether the local boundary conditions are being tested or rejected. Furthermore, bounded interval features are immune to cross-sectional scaling issues. If a portfolio must evaluate a penny stock trading at 1.50 and a mega-cap equity trading at 3,500, passing the raw levels or even standard deviations will warp the loss function. The variable r_t normalizes the topological space, making the states comparable across disparate asset classes.

def rolling_range_position(series: pd.Series, window: int = 60) -> pd.Series:
    """
    Maps the series into [0, 1] based on the rolling min-max range.
    """
    lo = series.rolling(window).min()
    hi = series.rolling(window).max()

    pos = (series - lo) / (hi - lo).replace(0, np.nan)
    return pos.rename(f"range_position_{window}")

The interaction between bounded variables and tree-based learning algorithms (Random Forests, Gradient Boosted Trees) is pretty interesting. These algorithms show great performance when provided with bounded state features. Trees partition coordinate space via orthogonal hyperplanes. If the raw price continues to trend upwards into uncharted numerical area, the tree algorithm can only output the constant value of its terminal leaf node. By mapping the price into r_t, the price drift is recursively folded back into the [0, 1] interval, allowing the decision tree to apply its historically learned logic regarding bound-rejection.

The danger of the rolling range position lies in the non-differentiability at the boundaries and the assumption of local support. The trailing window w enforces a boundary that the market does not acknowledge. The range is an artifact of local sample memory. If w is excessively short, standard Brownian noise will trigger extreme readings of 0 or 1, causing the classifier to generate false breakout signals. If w is excessively long, the range minimum and maximum become anchored to stale macro-regimes, rendering the current fluctuations invisible as the variable hovers near 0.5. The parameter w must be mapped to the temporal memory of the specific market participants the strategy is attempting to exploit.

Okay! Great job today, guys! Solid work. And remember, the full code is in the appendix, sitting there for you to dissect and torture as much as you like. Time to say goodbye. Stay sharp, stay bold, stay unstoppable 📈

PS: What would you do if you found a product with a drawdown of <1% and more than 1 year performance?

Appendix

Full code

import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.tsa.statespace.structural import UnobservedComponents
from statsmodels.regression.rolling import RollingOLS
from matplotlib.ticker import MaxNLocator


# Data
def generate_synthetic_market_data(n_samples: int = 1000) -> pd.DataFrame:
    """
    Generates realistic log-price data exhibiting stochastic trend,
    benchmark beta exposure, and pair cointegration.
    """
    np.random.seed(42)
    dates = pd.date_range(start="2020-01-01", periods=n_samples, freq="D")

    benchmark_returns = np.random.normal(loc=0.0002, scale=0.01, size=n_samples)
    benchmark_log_price = np.cumsum(benchmark_returns)

    beta_true = 1.2
    idiosyncratic_shocks = np.random.normal(loc=0.0, scale=0.015, size=n_samples)
    idiosyncratic_shocks[400:450] += np.random.normal(loc=0.02, scale=0.05, size=50)

    latent_trend = np.cumsum(
        np.random.normal(loc=-0.0001, scale=0.005, size=n_samples)
    )

    asset1_log_price = (
        beta_true * benchmark_log_price
        + latent_trend
        + idiosyncratic_shocks
    )

    equilibrium_error = np.zeros(n_samples)
    for i in range(1, n_samples):
        equilibrium_error[i] = (
            0.85 * equilibrium_error[i - 1] + np.random.normal(0, 0.008)
        )

    asset2_log_price = asset1_log_price - 0.5 - equilibrium_error

    df = pd.DataFrame(
        {
            "log_benchmark": benchmark_log_price,
            "log_price": asset1_log_price,
            "log_partner": asset2_log_price,
        },
        index=dates,)

    return df



# Transformation algorithms
def state_space(price_series: pd.Series) -> pd.Series:
    """
    Extracts the one-step-ahead forecast error from a local linear trend
    state-space model.
    """
    lp = price_series.rename("log_price")

    model = UnobservedComponents(lp, level="local linear trend")
    result = model.fit(disp=False)

    space = pd.Series(
        result.filter_results.forecasts_error[0],
        index=lp.index,
        name="state_space",)

    space.iloc[0] = np.nan
    return space


def rolling_regression_residual(
    price_series: pd.Series,
    benchmark_series: pd.Series,
    window: int = 80,) -> pd.Series:
    """
    Computes the dynamic rolling residual of the asset relative to the benchmark.
    """
    y = price_series.rename("log_price")
    x = benchmark_series.rename("log_benchmark")

    X = sm.add_constant(x)
    result = RollingOLS(y, X, window=window).fit()
    params = result.params

    fitted = params["const"] + params["log_benchmark"] * x
    residual = (y - fitted).rename(f"rolling_regression_residual_{window}")

    return residual


def cointegration_spread(
    price_series: pd.Series,
    partner_series: pd.Series,) -> pd.Series:
    """
    Estimates the Engle-Granger equilibrium spread.
    """
    y = price_series.rename("log_price")
    x = partner_series.rename("log_partner")

    X = sm.add_constant(x)
    fit = sm.OLS(y, X, missing="drop").fit()

    spread = (y - fit.predict(X)).rename("cointegration_spread")
    return spread


def haar_detail_same_length(price_series: pd.Series, level: int = 2) -> pd.Series:
    """
    Extracts the multiscale high-frequency detail coefficient using a discrete
    Haar wavelet filter bank and returns it with the same length as the input.
    """
    x = np.asarray(price_series, dtype=float)
    n = len(x)
    coeffs = x.copy()
    current_len = n
    detail = None
    block = 1

    for _ in range(level):
        if current_len < 2:
            break

        used_len = current_len if current_len % 2 == 0 else current_len - 1
        current = coeffs[:used_len]

        avg = (current[0:used_len:2] + current[1:used_len:2]) / np.sqrt(2)
        det = (current[0:used_len:2] - current[1:used_len:2]) / np.sqrt(2)

        coeffs[: len(avg)] = avg
        current_len = len(avg)
        detail = det
        block *= 2

    if detail is None:
        return pd.Series(
            np.nan,
            index=price_series.index,
            name=f"haar_detail_L{level}",
        )

    upsampled = np.repeat(detail, block)
    if len(upsampled) < n:
        upsampled = np.r_[upsampled, np.repeat(upsampled[-1], n - len(upsampled))]

    return pd.Series(
        upsampled[:n],
        index=price_series.index,
        name=f"haar_detail_L{level}",)


def rolling_range_position(series: pd.Series, window: int = 60) -> pd.Series:
    """
    Maps the series into [0, 1] based on the rolling min-max range.
    """
    lo = series.rolling(window).min()
    hi = series.rolling(window).max()

    pos = (series - lo) / (hi - lo).replace(0, np.nan)
    return pos.rename(f"range_position_{window}")

Table of contents:

1. Introduction.

2. Global macro after the energy shock.

3. The United States, the axis of growth, finance, and military escalation.

4. The main price-maker in the global economy.

5. China and Asia are the first receivers of the Hormuz shock.

6. The dollar, bond markets, and the return of geopolitical duration risk.

7. NATO, military burden-sharing, and the remapping of Western power.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

The global backdrop has entered a harder regime. Energy, shipping, fiscal capacity, military posture, and capital allocation now trade through the same set of constraints. Variables that once sat in separate analytical buckets now belong to a single pricing framework. Oil routes shape inflation. Alliance commitments shape issuance. Industrial policy shapes defence readiness. Currency strength shapes import costs and external adjustment. The market is pricing all of it together.

The central question concerns the kind of growth that can endure when energy friction rises, logistics tighten, and states absorb a larger strategic burden. That shift changes the hierarchy of variables. Security of supply carries more weight. Duration risk carries more weight. Fiscal credibility carries more weight. The ability to convert spending into usable capacity also carries more weight, whether in semiconductors, power systems, refining, shipping, or defence production.

This piece follows that chain from the macro layer into the strategic layer and back again. It begins with the global economy after the energy shock, where an active investment cycle now meets a harsher cost structure. It then turns to the United States, the core balance sheet in the system, where growth leadership, Treasury funding, monetary policy, and force projection converge. From there, the focus shifts to the conflict itself as the marginal driver of price formation across oil, gas, freight, and risk premia.

The analysis then moves to Asia, where direct exposure to Gulf energy flows makes the region the first receiver of the shock in physical, financial, and policy terms. It then examines the dollar and sovereign debt markets, where the length of the shock has become a key variable for yields, term premium, and global liquidity demand. The final section studies NATO and the wider Western security architecture, where rising military spending points to a deeper redistribution of industrial capacity, operational responsibility, and strategic weight.

Check more about all of this here.

US employment situation

491KB ∙ PDF file

Download

Download

Economic bulletin 2026

3.17MB ∙ PDF file

Download

Download

Shorter energy outlook

6.84MB ∙ PDF file

Download

Download

The aim is to frame the current environment the way markets are beginning to price it, as one integrated system. Growth, inflation, war, energy, debt, and alliance structure now move through the same circuit. The edge comes from seeing the connections early, before consensus fully absorbs their implications.

Subscribe now

Global macro after the energy shock

We can say we have two engines that pull in opposite directions. One engine comes from the capital cycle that formed through 2025 and carried into 2026 through technology investment, private-sector adaptation, and supportive demand. The other engine comes from the war-driven energy shock, the tightening of transport routes, and the rise in geopolitical risk premia across commodities, shipping, rates, and equities. The world economy is treated as a system where energy access, market confidence, military escalation, and asset pricing move together. The IMF and OECD now describe the same architecture in institutional language. The IMF’s update still presented a world economy with resilient growth, projecting global output growth at 3.3% in 2026 and 3.2% in 2027, driven in part by technology investment, supportive financial conditions, and private-sector adaptability. By late March, the OECD had redrawn that picture. It projected global growth at 2.9% in 2026 and 3.0% in 2027, with the Middle East conflict, higher energy prices, and supply-chain disruption pushing costs higher and demand lower.

The tension between those two engines defines the whole macro regime. The first engine still matters because technology investment remains large, especially in sectors tied to artificial intelligence, semiconductors, cloud infrastructure, industrial automation, and the digital build-out of the energy and security complex. The IMF highlighted strong performance in exports of semiconductors and related equipment across Asian economies and linked that strength to the rise in information and technology investment. That matters because a technology-led capex cycle creates growth through many channels at once. It supports equipment orders, lifts productivity expectations, raises corporate spending, improves cash-flow visibility for firms near the center of the build-out, and attracts capital toward the economies that lead those value chains. This is one reason the United States has strong relative support. It is also one reason parts of East Asia carried stronger industrial momentum into the year than many analysts expected at the end of 2025.

At the same time, the second engine has become the dominant marginal force in pricing. The OECD says the conflict in the Middle East is testing the resilience of a global economy that had been supported by financial and fiscal conditions and growing demand for artificial intelligence technologies. It states that the halt in shipments through the Strait of Hormuz and the closure or damage of energy infrastructure generated a surge in energy prices and disrupted the global supply of energy and other key commodities, including fertilisers. This is central because it shifts the macro discussion away from the classic debate about whether demand remains firm and toward a more strategic debate about how long the energy shock lasts, how broad the shipping disruption becomes, and how much of the rise in commodity prices passes into wages, services, industrial margins, and inflation expectations.

This leaves the world economy in a narrower corridor than the one investors saw at the beginning of the year. In January, the baseline still looked like a world of steady growth with disinflation in progress. Now that baseline became conditional on the moderation of energy-market disruption from mid-2026 onward. The OECD states that its projections rely on that technical assumption. The economic meaning is clear. A shorter disruption preserves a large part of the prior macro script. A longer disruption transforms the outlook into one shaped by persistent cost pressure, weaker real income growth, broader financial repricing, and stronger dependence on fiscal triage and central-bank credibility.

The inflation story reflects this shift with unusual clarity. The IMF’s update expected global inflation to fall, projecting headline inflation at 3.8% in 2026 and 3.4% in 2027. That forecast came from a world in which supply conditions improved, commodity pricing eased from earlier extremes, and private adaptation softened the impact of trade-policy friction. The OECD’s update already incorporates the war shock. It projects G20 inflation at 4.0% in 2026, which is 1.2 percentage points higher than it expected before the conflict escalated, before easing to 2.7% in 2027 under the assumption that energy pressures fade. This difference between the January IMF view and the March OECD view is one of the cleanest expressions of the regime change. It shows how quickly a global disinflation narrative can bend when energy, logistics, and risk premia move together.

The inflation process has also split into layers. Headline inflation is the first recipient of the shock because oil, gas, fuel transport, chemicals, and fertilisers transmit rapidly into measured prices. Core inflation then becomes the field of contest. If households and firms treat the energy shock as a passing event, core inflation stabilises with less secondary impact. If firms push higher input costs through broader price categories and workers seek compensation for weaker real incomes, core inflation becomes more persistent. The OECD says medium-term inflation expectations already rose following the energy-price spike. That point matters because expectations serve as the bridge between a commodity event and a more durable inflation process. The shape of that bridge has become one of the decisive macro variables.

Growth now rests on dispersion rather than on one shared global rhythm. The United States still benefits from strong AI-related investment, and the OECD projects U.S. growth at 2.0% in 2026 before easing to 1.7% in 2027. China remains a large growth pole at 4.4% in 2026 and 4.3% in 2027 in the OECD baseline, with state support and industrial strength still carrying weight. The euro area carries a weaker profile. The OECD projects euro area growth at 0.8% in 2026 before a move to 1.2% in 2027, helped in part by stronger defence spending. These numbers describe more than regional variation. They describe the way energy exposure, trade composition, industrial structure, and policy room now shape macro outcomes. Economies close to the technology capex wave retain momentum. Economies more exposed to imported energy and external demand face a tighter balance.

That regional dispersion also changes the meaning of policy credibility. In a calmer cycle, policy credibility often refers to a central bank’s ability to anchor inflation and a finance ministry’s ability to sustain debt. In the current cycle, the idea becomes wider. Credibility now includes the ability to secure energy, preserve transport flows, design temporary support measures without weakening the fiscal base, and maintain enough strategic coherence that firms can plan capital expenditure with confidence. The OECD writes that government measures to cushion higher energy prices should be targeted, preserve incentives to lower energy use, and carry clear expiry mechanisms. It also stresses limited fiscal space and the need for credible medium-term adjustment paths. This means macro policy has become an exercise in war-era selectivity rather than a broad stimulus exercise.

The capital-market layer follows the same logic. In the technology engine, markets still reward growth visibility, margin durability, and long-duration cash flows linked to infrastructure, semiconductors, power systems, software, automation, and data architecture. In the energy-shock engine, markets demand compensation for transport friction, raw-material scarcity, rate volatility, and policy uncertainty. The OECD says financial conditions tightened after the outbreak of hostilities, with volatility rising and the previous degree of accommodation shrinking. The important point lies in the combination rather than in a single asset class. Equities, bonds, currencies, commodities, and credit spreads are all trying to price the same unknown at once, namely the duration and reach of the conflict shock. That creates a macro system in which every bond curve carries an energy assumption, every central-bank path carries a shipping assumption, and every equity multiple carries an embedded judgment about geopolitical durability.

The structural dimension runs deeper than the current quarter. The OECD highlights that persistent disruptions to Middle East exports could aggravate shortages of key commodities and produce a broader repricing in financial markets. It also notes that countries in the Gulf serve as an important source of aluminium, helium, bromine, fertiliser inputs, and logistics services. This widens the macro significance of the current conflict. The issue extends beyond oil and gas. It reaches into food systems through fertilisers, into industrial production through metals and specialty materials, and into advanced manufacturing through chemical and memory-chip supply chains. A prolonged disruption therefore reshapes more than headline inflation. It reshapes the cost map of the industrial economy.

The United States, the axis of growth, finance, and military escalation.

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Why a trading hypothesis?

A trading hypothesis is presented as the first scientific form of a market idea, one that transforms intuition into a structured claim with an activating condition, an expected response, a time horizon, and a proposed mechanism. The lecture develops this framework as the bridge between market logic and empirical evidence, and explains how restriction, precision, and coherence turn a vague observation into a research object that can be tested, challenged, and refined.

What’s inside:

1. Why a trading hypothesis matters. A trading hypothesis is the first formal research object in quantitative trading because it gives direction, causal meaning, and empirical consequences to an otherwise informal market observation.

2. Hypothesis as a conditional claim about market behavior. The lecture defines a genuine hypothesis as a structured conditional statement that specifies when a market condition appears, what response should follow, over what horizon, and because of what underlying process.

3. The activating condition. A hypothesis begins with a trigger that identifies when the market enters a meaningful state, such as a volatility expansion, spread dislocation, order flow imbalance, or liquidity shock, so the sample becomes selective in a principled way.

4. The expected market response. The claim must state what should happen after activation by defining the object of movement, the direction of adjustment, and the scale or relevance of that movement, so the idea acquires a clear empirical identity.

5. The horizon over which the response should emerge. Time is treated as part of the hypothesis itself because different mechanisms operate on different clocks, and the validity of the research depends on matching the response window to the real rhythm of the market process.

6. A bridge between market logic and empirical evidence. The lecture shows that a good hypothesis connects two domains at once, the logic of participants, frictions, and structural forces on one side, and the language of variables, samples, outcomes, and comparisons on the other.

7. Empirical signature and supporting evidence. A strong hypothesis does not predict profitability alone, but a specific patterned trace in the data, including direction, structure, timing, conditional strength, and comparative contrast, so support comes from alignment rather than from one backtest statistic.

8. The hypothesis as a restricted object. The lecture explains that every valid hypothesis must be restricted to a coherent asset universe, a defined market state, and a proper time horizon, because restriction is what gives the claim scientific shape and protects it from vague breadth.

Sample

1.89MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Why transform data?

3. The risks of data transformation.

4. The memory dilemma and distributional shifts.

   1. The fractional differencing paradigm.

   2. Higher moments and the Cornish-Fisher expansion.

5. Time and variance.

   1. Time deformations and information clocks.

   2. Structural volatility standardization.

6. Cross-sectional geometry and noise mitigation.

   1. Entropy-based outlier mitigation without information loss.

   2. Cross-sectional orthogonalization.

7. Non-linear embedding and phase-space reconstruction.

8. The unified pipeline.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Raw market data is the closest thing we have to the market’s observable dynamics, so the instinct to leave it untouched feels sensible. But predictive models consume numerical representations. Between the tape and the optimizer there is always a translation step, whether explicit or hidden, and that translation determines which structures remain visible, which ones are attenuated, and which ones disappear. In financial machine learning, representation is part of the model itself. Prices, returns, clocks, scales, and cross-sectional coordinates all define different statistical objects, and each one presents a different geometry to the learning algorithm.

The literature on volume clocks showed that market activity is better understood in an event-driven metric than in a chronological one. The literature on fractional differencing showed that differencing need not be an all-or-nothing choice, but can instead vary continuously to preserve long-range structure while moving toward stationarity. The literature on conditional heteroskedasticity showed that variance itself has memory and must be modeled as a dynamic object rather than treated as a fixed nuisance parameter.

Here an old one about volume clock:

The volume clock

661KB ∙ PDF file

Download

Download

Yet every transformation carries a cost. A bad transformation can alter the topology of the series, flatten its tails, suppress its regime structure, and leak future information into the past. The goal here is to show that each transformation answers a specific failure mode of financial data. Some methods regularize the clock. Others preserve memory while controlling non-stationarity. Others stabilize conditional variance, correct tail asymmetry, reduce cross-sectional noise, or reconstruct hidden structure from a single observable. When these operations are applied in the right order, transformation stops being a destructive preprocessing ritual and becomes a way of making market structure legible to statistical and machine-learning models.

This is the real question is “how to make data look cleaner, but how to modify it without amputating the very dependencies we hope to trade.”

Why transform data?

If raw data represents the perfect economic reality of the market, why do we need to transform it at all? Why not feed the raw sequence of events, the actual execution prices and sizes, directly into our predictive algorithms?

The answer lies in the rigid constraints of linear algebra and numerical optimization. Machine learning algorithms, from basic Ordinary Least Squares to stochastic gradient descent solvers, don’t understand economics, market microstructure, or the concept of fiat currency. They only understand geometry and topology. They evaluate the distance between points in a high-dimensional space.

Let X∈R^TxN be a feature matrix of raw asset prices over T time steps for N assets. Raw prices are strictly non-stationary I(1) processes, their mean and variance scale continually with time. If we attempt to fit a linear model to minimize the empirical risk, we must compute the inverse covariance matrix (X^TX)^-1 to solve for the coefficient vector. Because the raw prices share a common, unbounded macro trend, the vectors in X are collinear. The matrix X^TX becomes nearly singular. Its condition number explodes, meaning the matrix inversion becomes numerically unstable. The optimizer can’t find a unique global minimum. Instead, it yields spurious regressions. The model will report a massive R²X value and highly significant t-statistics, suggesting it has found a relationship, when in reality, it has correlated two independent random walks that happen to drift in the same general direction.

Furthermore, deep learning architectures—specifically Recurrent Neural Networks and Long Short-Term Memory networks—require bounded, zero-centered inputs to propagate gradients efficiently through time. If we feed raw, un-scaled prices into a network utilizing sigmoid or hyperbolic tangent (tanh) activation functions, the architecture breaks down immediately. If an asset’s price has drifted from $10 to $400 over a decade, feeding the value $400 into a tanh function yields an output that is computationally indistinguishable from $1.0. The local derivative of the activation function at this asymptote is zero. When the backpropagation algorithm attempts to calculate the chain rule to update the network’s weights, it multiplies by this zero derivative, causing the vanishing gradient problem. The network stops learning on the very first epoch. The weights freeze.

Subscribe now

The risks of data transformation

For some models transformation is a necessity, however it is the highest-risk component of the entire pipeline. Every operation applied to a temporal series acts as a filter. Applying an operator T(·) to a raw series X_t involves a manipulation of entropy. You are altering the properties of the dataset. If the transformation isn’t an exact bijection tailored to the specific distributional properties of the underlying asset, it is a destructive process that deletes alpha.

1. The primary risk is geometric distortion. Quants attempt to force non-Gaussian market data into Gaussian topologies using standardized scalers. Let μ_t and σ_t be the rolling mean and standard deviation. The standard Z-score transformation is Z_t = (X_t - μ_t) / σ_t. This operation is ubiquitous in basic data science, but it is catastrophic in finance. As established by empirical market data, equity returns exhibit severe negative skewness and massive excess kurtosis. By applying this linear transformation, we assume the data is symmetric around the mean. We geometrically compress the left tail and stretch the right tail. When the optimizer processes Z_t, it assigns incorrect probabilistic weights to the downside risk. The transformation has lied to the algorithm about the probability of ruin, mapping a 6σ market crash into a normalized space that makes it look like a mild, acceptable deviation.

2. The second critical risk is information destruction via arbitrary numerical thresholds. Common quantitative practices like hard-clipping outliers or dropping low-volume trading hours manually alter the sequence of the time series to make the data cleaner for the optimizer. If a quant runs a standard Winsorization protocol and drops an outlier return of -12% down to a hard cap of -3% simply because the extreme value “ruins the scale of the loss function,” they have committed a severe theoretical error. They have deleted the exact moment of maximum market inefficiency. That -12% print is the precise data point where algorithmic alpha is generated. It represents a moment where traders capitulated and forced liquidations occurred. Transforming the data to make it look smooth and continuous for an optimizer strips out the true signal, leaving behind highly stationary, useless white noise.

3. The third, and most fatal, risk is structural data leakage, also known as look-ahead bias. This occurs when a transformation operator inadvertently uses information from the future to scale or center the data of the past. If a quant runs a Principal Component Analysis (PCA) on the entire matrix X spanning from 2010 to 2024 to find orthogonal risk factors, and then trains a predictive model sequentially on the data from 2015, the model is compromised. The eigenvectors calculated by the global PCA contain variance data from the 2020 pandemic market crash. The transformation has leaked future volatility structures into the historical training set.

This leakage happens with basic functions like global mean subtraction or min-max scaling if the boundaries aren’t rolling. The model will appear profitable during out-of-sample backtesting because the transformed features implicitly “know” the future basis vectors and maximum bounds of the market. In live trading, where the production operator only has access to historical data up to time t, this false performance collapses instantly.

The memory dilemma and distributional shifts.

The fundamental dilemma in quantitative modeling revolves around the requirement for stationarity and the predictive requirement for memory. To train any robust inferential model, whether a vector autoregression or a neural architecture, the input data must be stationary. The statistical properties of the series—mean, variance, and autocorrelation structure—can’t vary over time. If they do, the model learns a regime that ceases to exist the moment it is deployed in live trading.

When you feed non-stationary price levels into a learning algorithm, you violate the core assumptions of the optimizer. As previously mentioned, in linear models like Ordinary Least Squares, non-stationarity leads to spurious regressions where the model identifies high R² values and significant t-statistics between independent random walks. In deep learning architectures like Recurrent Neural Networks, non-stationary variance causes the gradient descent process to either vanish into zero or explode to infinity, preventing convergence.

However, the standard mathematical transformations used to enforce this required stationarity destroy the predictive power of the dataset. Financial time series, specifically price levels, are mostly I(1) processes, meaning they possess a unit root. The conventional approach to solving this is applying an integer differentiation, calculating the first-order difference or the standard log-return. This transformation achieves stationarity, yielding a series with a constant mean and finite variance. Yet, it erases the long-term memory of the process. Prices retain the entire history of market shocks, but returns only remember the shock of the immediately preceding period. By integer differencing, we obtain stationary noise that is impossible to predict.

Furthermore, integer differencing often leads to over-differencing. When you subtract X_t-1 from X_t to create a return series, you inject a moving average MA(1) component with a coefficient of -1 into the resulting noise. This artificially creates a strong negative autocorrelation at the first lag. Algorithms trained on this over-differenced data will discover short-term mean reversion that doesn’t exist in the market microstructure, leading to systematic trading losses.

The risks of relying on models built atop these conventional transformations are severe. We force non-stationary data through linear, memoryless filters, creating an illusion of predictability. We evaluate stationarity using binary heuristics like the Augmented Dickey-Fuller test, treating it as an absolute state rather than a continuous spectrum. Models trained on first-differenced data often exhibit high backtest performance due to noise-fitting but fail rapidly out-of-sample because the underlying structural dependencies of the market have been severed.

Do you rememenber the Quant Meltdown? Statistical arbitrage funds, operating with massive leverage, relied on mean-reversion models trained on orthogonalized, first-differenced equity returns. These models assumed that the residuals of their transformations were stationary, memoryless white noise, safely confined to narrow bands. When sudden, massive liquidations hit the market due to subprime mortgage exposure, the assumed independence of these time series broke down.

The residuals exhibited extreme, long-term memory. A portfolio liquidation creates a directional order flow imbalance that persists across days, resurrecting the unit root that the quants thought they had differenced away. The standard transformations had hidden it under normal, high-liquidity market conditions. When the regime shifted, the models failed because the foundation of their data transformation was inadequate. The algorithms kept doubling down on mean-reverting bets while the market displayed a persistent, memory-driven structural break.

The fractional differencing paradigm

We must abandon the misconception that differentiation is an integer operator. In the context of trading, forcing a time series into either a raw price format (where d=0) or a returns format (where d=1) is an arbitrary constraint. The correct approach is to treat differentiation as a continuous domain operator, allowing us to find the exact fractional value of d that achieves stationarity while preserving the maximum possible amount of memory.

We define the backshift operator B such that BX_t = X_t-1. For any real number d, the fractional difference operator (1-B)^d can be expanded using the infinite binomial series:

This expansion yields an infinite sequence of weights ω_k that we apply to past observations.

Notice the recursive property of these weights:

When d=1, the math is trivial: ω₀=1, ω₁=-1, and all subsequent weights are zero. This describes the standard first-order difference. The memory of the series is truncated at one period.

But consider what happens when d is a fraction, say 0.5. The sequence of weights becomes ω₀=1, ω₁=-0.5, ω₂=-0.125, ω₃=-0.0625, and so on. The weights decay asymptotically to zero rather than abruptly terminating. This means X_t is transformed using an exponentially decaying window of its entire history. The current value of the transformed series is heavily influenced by yesterday’s price, moderately influenced by last week’s price, and influenced by the price a year ago. This try to mirror the market information dissemination.

In practice, financial time series like equity indices or major fiat currencies achieve ADF stationarity at d values ranging between 0.3 and 0.4. This is an empirical fact. It proves that by using standard log-returns (d=1), the quantitative finance industry is discarding 60% to 70% of the useful memory contained in the price data.

To implement algorithm, we calculate the weights iteratively and apply a fixed-width window to prevent data leakage and memory overflow. We establish a tolerance threshold τ (often set to 10^-4 or 10^-5) and drop weights where |ω_k| < τ. This generates a fixed lookback window, typically spanning a few thousand observations depending on the chosen threshold.

We must use a fixed-width window rather than an expanding window. If we apply fractional differencing using the entire available history of the series starting from t=0, the number of weights applied to the current observation grows as time progresses. This causes the variance of the transformed series to drift, violating the finite variance requirement of stationarity. The fixed-width window ensures that the exact same mathematical operation is applied to every observation, maintaining a constant variance profile.

Let’s implement this.

import numpy as np
import pandas as pd
from statsmodels.tsa.stattools import adfuller

def get_weights_ffd(d, length, threshold=1e-5):
    """
    Calculate weights for fractional differentiation using a fixed-width window.
    Weights iteratively decay, preserving continuous memory constraints.
    """
    weights = [1.0]
    k = 1
    while True:
        weight_k = -weights[-1] * (d - k + 1) / k
        if abs(weight_k) < threshold and k >= length:
            break
        weights.append(weight_k)
        k += 1
        if k > 10000:  # Safety break for massive arrays
            break
    return np.array(weights[::-1])

def fractional_diff(series, d, threshold=1e-5):
    """
    Applies fractional differencing to a pandas Series to maintain memory 
    while achieving strict mathematical stationarity.
    """
    weights = get_weights_ffd(d, length=1, threshold=threshold)
    width = len(weights) - 1
    
    df = pd.Series(index=series.index, dtype=float)
    
    # Apply the dot product across the fixed-width rolling window
    for i in range(width, len(series)):
        window = series.iloc[i - width : i + 1]
        df.iloc[i] = np.dot(weights, window)
        
    return df.dropna()

def optimize_fractional_d(series, d_range=np.arange(0.1, 1.0, 0.1), p_value_threshold=0.05):
    """
    Finds the minimum fractional d that successfully passes the Augmented 
    Dickey-Fuller (ADF) test for stationarity, maximizing retained memory.
    """
    for d in d_range:
        diffed = fractional_diff(series, d)
        if len(diffed) > 10:
            adf_stat = adfuller(diffed, maxlag=1, regression='c', autolag=None)
            if adf_stat[1] < p_value_threshold:
                return d
    return 1.0

This gives us the orange series you see below. Do you notice the difference compared to a simple differentiation?

Higher moments and the Cornish-Fisher expansion

Quants frequently take log-returns, subtract the rolling mean, and divide by the rolling standard deviation, assuming the resulting series z follows a standard normal distribution N(0,1). Sound familiar? This is the foundation of the Z-score, a metric that is built into the base functionality of almost every data science library.

Financial returns don’t follow a Gaussian distribution, they exhibit significant skewness and excess kurtosis. Equity markets, for example, display structural negative skew, prices drop much faster than they rise due to the mechanics of margin calls, stop-loss triggering, and fear-driven liquidity vacuums. Besides, the phenomenon of volatility clustering generates excess kurtosis, creating a distribution with a sharp central peak and fat tails.

Standardizing without accounting for these higher moments leads to severe underestimation of tail risk and generates false trading signals. Consider a standard normal distribution. A 5-standard-deviation event has a probability of approximately 2.8x10^-7, which translates to an expected occurrence of once every 13,800 years of daily trading. Yet, in live equity markets, 5-sigma moves happen every few years. If an algorithm feeds a raw 5-sigma Z-score into a neural network, the model interprets it as an impossible anomaly and the assigned weights will skew, breaking the loss function.

To transform a time series into a Gaussian-equivalent state, we must incorporate its empirical skewness (S) and excess kurtosis (K). We achieve this via the Cornish-Fisher expansion, which provides a framework to estimate the quantiles of a non-normal distribution based on its specific cumulants.

If Z_α is the α-quantile of the standard normal distribution, the corresponding quantile W_α of our financial time series can be approximated as:

However, in feature engineering for predictive modeling, we often need the inverse transformation. We have an observed standardized return w coming from the live market, and we want to map it back to a normal variable z so that our downstream linear models and gradient descent algorithms function correctly.

We compute the rolling skewness and rolling excess kurtosis of the time series. Estimating the third and fourth moments requires significant data to achieve statistical stability, as outliers heavily distort these metrics. Therefore, the lookback window for S and K must be longer than the window used for the mean and variance—often spanning two to three years of daily data.

Once we establish stable estimates for the cumulants, we apply the inverse Cornish-Fisher expansion to normalize the data. We map the empirical quantiles of the fat-tailed data to the theoretical quantiles of the standard normal distribution.

This adjustment guarantees that a severe market shock is processed correctly by the predictive architecture. A drop that registers as a 5-sigma event in raw standard deviations might map to a much more manageable 2.5-sigma event in the Cornish-Fisher adjusted space, reflecting its true probabilistic reality in a fat-tailed environment.

import scipy.stats as stats
import pandas as pd

def cornish_fisher_transform(returns):
    """
    Normalizes higher moments (skewness and kurtosis) mapping an empirically 
    fat-tailed financial series back to a strictly Gaussian space.
    """
    # Standardize the data first based on initial 2 moments
    z = (returns - returns.mean()) / returns.std()
    
    # In a strict ML feature engineering pipeline, we map the empirical quantiles 
    # of the fat-tailed data directly to the theoretical quantiles of the standard normal distribution
    
    ranks = z.rank(pct=True)
    
    # Map back to standard normal quantiles. We use the PPF (Percent Point Function)
    normalized = stats.norm.ppf(ranks)
    
    return pd.Series(normalized, index=returns.index)

I admit I like this change in silhouette. In the next post, we’ll talk about changes in shapes, I have some ideas that might be quite interesting.

As can be seen, our distribution now looks more like a normal distribution than the classic silhouette of financial returns.

Time and variance

Data transformation involves addressing how we sample the data itself. The most pervasive obstacle in modeling financial markets is the unquestioned reliance on chronological time. Across the industry, the default behavior is to extract open, high, low, and close prices at fixed chronological intervals—minute bars, hourly bars, daily bars. This framework is a legacy artifact of print newspapers and early charting software, and it is disjointed from the reality of market micro-structure.

Market information doesn’t arrive in a continuous, uniform flow. It arrives in distinct bursts driven by human reaction and algorithmic execution. The central limit order book operates on an event-driven basis. The matching engine processes incoming FIX messages—adds, cancels, and modifications—sequentially. It has no internal concept of a “minute.” During the opening auction, or in the milliseconds following a critical macroeconomic data release like the Non-Farm Payrolls, thousands of trades occur per second. These specific moments carry massive amounts of market-moving information, representing true price discovery.

During the mid-day lull or an overnight electronic session, hours can pass with negligible volume and absolute zero new information entering the system. When we sample data chronologically, we force these disparate states into sized computational intervals. This creates a severe statistical anomaly known as a zero-inflated distribution. If you sample an illiquid asset using one-second bars, 90% of your dataset will consist of zero-returns because no trades occurred in those specific seconds. When you feed a neural network a continuous string of zero-returns, the gradient vectors collapse. The algorithm determines that the optimal prediction is always zero, stalling the learning process.

The obstacle this creates for data transformation is systemic. Chronological sampling guarantees heteroskedasticity—the variance of the data is unstable. The standard variance of a one-minute bar at 09:30 AM is an order of magnitude larger than the variance of a one-minute bar at 12:30 PM. This phenomenon creates the well-documented U-shaped intraday volatility curve. The beginning and end of the trading session contain massive variance, while the middle contains almost none.

Furthermore, chronological sampling leads to massive oversampling of low-information periods and dangerous undersampling of high-information periods. If we attempt to apply technical indicators like moving averages, momentum oscillators, or machine learning models to chronological bars, we are feeding the algorithms a distorted geometry of the market.

Time deformations and information clocks

To resolve the obstacle of chronological heteroskedasticity, we must change the domain of the temporal series. We must transition from chronological time to information time. Information can be approximated by the volume of shares traded or the total fiat value exchanged. We implement this through Time Deformations, specifically by constructing Volume Clocks or Dollar Clocks.

While some practitioners use Tick Bars (sampling every N transactions) or Volume Bars (sampling every V shares), these are suboptimal over long horizons. Tick bars are vulnerable to order fragmentation. A large institutional order sliced into 100 smaller algorithmic executions via a Time-Weighted Average Price algorithm will accelerate a tick clock without actually injecting new economic information.

You can go deeper about this kind of methods here:

Volume bars are better but remain vulnerable to corporate actions like stock splits or long-term price appreciation. Consider a multi-year backtest on a high-growth technology equity. A 10,000-share bar of a stock trading at $10 in year one represents $100,000 of market participation. If that same stock appreciates to $200 by year five, a 10,000-share bar now represents $2,000,000 of market participation. By using volume bars, the implicit information threshold has unknowingly shifted by a factor of twenty. The statistical properties of the bars from year one are incomparable to the bars from year five.

Dollar Bars correct these deficiencies, instead of sampling a new data point every time t seconds pass, we sample a new data point every time a cumulative fiat threshold θ is reached. Let v_i be the volume of the i-th tick and p_i be its price. The cumulative dollar volume traded at tick t is

We define the sampling times T_k such that:

When we sample the time series at these intervals T_k, we generate Dollar Bars. The threshold θ is often calculated dynamically, set to a fraction of the 50-day moving average of the daily dollar volume, ensuring the clock ticks at a consistent daily frequency even across decades of data.

The transformation from chronological space to dollar-volume space has statistical implications that directly improve model performance. Empirical studies demonstrate that the returns of dollar bars exhibit lower serial correlation. Indeed, they reduce the excess kurtosis found in chronological data. By sampling based on fiat flow, we eliminate the zero-return intervals of the overnight session and parse the massive volatility spikes of the open into multiple, manageable bars. They recover the properties of an independent and identically distributed.

By defining the series through the Dollar Clock, we adjust for both intraday seasonality and macro-regime shifts. A single Dollar Bar might span exactly 1.2 seconds during the market open, capturing the rapid influx of high-density information with perfect granularity. The subsequent Dollar Bar might span 45 minutes during the lunch hour. The variance of the returns across these bars becomes more stable.

import pandas as pd
import numpy as np

def generate_dollar_bars(ticks_df, dollar_threshold):
    """
    Transforms chronological tick data into Dollar-Volume bars to eliminate 
    chronological heteroskedasticity.
    """
    # Calculate exact fiat commitment per tick
    ticks_df['dollar_value'] = ticks_df['price'] * ticks_df['volume']
    ticks_df['cumulative_dollars'] = ticks_df['dollar_value'].cumsum()
    
    # Determine discrete bar groupings based on cumulative flow
    ticks_df['bar_id'] = (ticks_df['cumulative_dollars'] // dollar_threshold).astype(int)
    
    # Aggregate raw ticks into clean OHLCV structural geometry
    bars = ticks_df.groupby('bar_id').agg(
        open=('price', 'first'),
        high=('price', 'max'),
        low=('price', 'min'),
        close=('price', 'last'),
        volume=('volume', 'sum'),
        dollars=('dollar_value', 'sum')
    )
    
    return bars

Let’s take an image from one of the previous articles. If you look closely, you’ll see how sampling results in fewer bars, but also less noise.

Structural volatility standardization

Once the time series is sampled correctly using an information clock, we face the next critical transformation hurdle: volatility clustering. Even within time-deformed series, prolonged macroeconomic regimes—such as quantitative tightening cycles, global liquidity crises, or prolonged zero-interest-rate environments—cause the baseline variance of the asset to expand and contract over weeks and months. The standard quantitative practice to normalize this behavior is to standardize returns by dividing them by a simple rolling standard deviation.

This naive approach is flawed and introduces dangerous artifacts into the data matrix. A rolling standard deviation applies an equal, flat weight to all observations within the lookback window and absolute zero weight to everything outside it. Consider the exact mechanics of this operation when a massive exogenous shock enters the window. Suppose we use a 30-period rolling window. On period 1, the market drops by 10%. The estimated volatility spikes and remains elevated at a constant plateau for the next 29 periods. During this time, the true market volatility may have completely subsided, but our denominator remains massive, compressing the transformed returns and suppressing valid trading signals.

Then, exactly 30 periods later, when that specific 10% shock falls out of the lookback window, the estimated volatility drops sharply and discontinuously. This creates a mathematically induced ghost effect. The algorithm perceives a sudden collapse in market volatility, generating spurious trading signals and artificial regime shifts. A normal 0.5% return on period 31 is suddenly divided by a tiny denominator, creating a massive standardized value. In reality, absolutely nothing happened in the live market on that specific day. The anomaly was purely a structural failure of the flat rolling window equation.

The classical transformation for volatility standardization relies on Generalized Autoregressive Conditional Heteroskedasticity ak GARCH models. Rather than relying on a flat, memoryless rolling window, we use a structural probabilistic model to estimate the conditional variance σ_t² at each specific time step. For a standard GARCH(1,1) specification, the conditional variance is modeled as:

Where ε is the squared residual of the previous period (the ARCH term, representing immediate short-term market panic), and σ is the variance of the previous period (the GARCH term, representing long-term structural persistence). The constant ω defines the baseline variance level.

The power of this formulation lies in the interplay between α and β. A high α indicates a market that reacts to new information, while a high β indicates a market where volatility shocks persist for extended periods. To transform the temporal series, we fit these parameters ω, α, and β using maximum likelihood estimation over an expanding or long-term rolling window. We then filter the raw returns r_t by dividing them by the modeled conditional volatility:

This GARCH-standardized series reacts to volatility because the GARCH model incorporates an exponential autoregressive decay component (β), the influence of a massive price shock diminishes over time. It never drops off a cliff. The denominator decays according to the empirical persistence of the market, eliminating the ghost effects caused by naive rolling windows.

This transformation contextualizes price action for the downstream inferential models. In a quiet market with low σ_t, a 2% directional move results in a value that correctly identifies an anomaly. On the other hand, in a volatile market where VIX is trading above 40, a 2% move is divided by σ_t.

The transformation strips out the predictable component of the volatility clustering, leaving behind a standardized, stationary residual series. It forces the predictive algorithm to focus on directional alpha of the price movement. By factoring out the structural volatility, the features fed into the final machine learning layer have a linear relationship with the future expected returns, improving the optimizer’s convergence rate.

from arch import arch_model
import warnings

def garch_standardize(returns):
    """
    Standardizes returns using structural conditional volatility estimated 
    from a GARCH(1,1) model, completely bypassing the 'ghost effects' 
    of naive flat rolling windows.
    """
    # Rescale returns to avoid numerical precision errors in the MLE optimizer
    scaled_returns = returns * 100 
    
    # Define a pure volatility model assuming a zero-mean return 
    am = arch_model(scaled_returns, vol='Garch', p=1, q=1, mean='Zero')
    
    # Fit parameters via Maximum Likelihood Estimation
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        res = am.fit(disp='off')
    
    # Extract conditional volatility, scale back down, and apply standardizer
    conditional_volatility = res.conditional_volatility / 100
    standardized_returns = returns / conditional_volatility
    
    return standardized_returns

Remember that these are just simple examples to give you a complete picture of how and why to transform data. There are many more approaches and techniques for transforming data. In this case, if we continued with the example and used toy data, we would see something like this, and if we wanted market data, we would see something like the image above.

Cross-sectional geometry and noise mitigation

Up to this point, our transformations have focused on the isolated temporal dynamics of a single series—adjusting its memory, clock, and variance. However, institutional algorithmic trading doesn’t operate on single assets in isolation. We evaluate massive, dense feature matrices spanning hundreds of correlated instruments, exposing our optimizers to severe geometric hazards. The two most destructive issues in this cross-sectional space are the gradient-exploding impact of extreme microstructure shocks and the matrix-destabilizing curse of multicollinearity. To prepare our data for complex inferential models without erasing the underlying market signal, we must implement noise mitigation. In the following sections, we introduce two key approaches to solve outlier compression that preserves topological rank without triggering numerical overflow. Then, we apply Random Matrix Theory to orthogonally decouple and denoise the cross-sectional feature space, ensuring our covariance matrices remain mathematically stable.

Entropy-based outlier mitigation without information loss

Financial time series contain severe outliers. Broken trades, massive exogenous shocks, and unexpected central bank interventions create data points that lie far beyond the expected bounds of any standard probability distribution. Leaving these unaddressed destroys the gradient descent processes. Most optimization functions, particularly those relying on Mean Squared Error, penalize large deviations. A 10σ price shock creates a gradient update that is a hundred times larger than a standard 1σ move. If this raw data is fed into a neural network, the weights will update so violently that the model diverges, forgetting previously learned market structures.

The traditional quantitative response to this problem is Winsorization: setting a static threshold (e.g., the 1st and 99th percentiles) and hard-capping any values beyond those limits. If a return exceeds the 99th percentile, it is rewritten as the 99th percentile value.

This approach is destructive and theoretically unsound. In algorithmic trading, extreme events contain the most critical alpha. Hard-capping an extreme move flattens the local gradient of the price action, deleting information about market capitulation, liquidity voids, and directional conviction. If we cap the data, a +5% move and a +15% move become identical to the algorithm. The algorithm loses the ability to distinguish between high volatility and a fundamental structural break. We need a transformation that mitigates the numerical impact of the outlier on the model’s loss function without erasing the topological information and rank order it provides.

We achieve this via trimming based on Shannon Entropy. Instead of an arbitrary, static percentile, we compress the tails of the distribution using a soft-clipping function parameterized by the entropy of the rolling lookback window.

Let H(X) be the Shannon Entropy of the time series X over a recent window. We calculate this by discretizing the continuous return space into a set of B fixed bins and computing the empirical probability p(x_i) of a return falling into the i-th bin:

Shannon Entropy measures the unpredictability or noise of the local market regime. If prices are bouncing randomly within a tight, mean-reverting range, the probability mass is spread across many bins, resulting in high entropy. If the market is experiencing a persistent, directional trend with low noise, the probability mass concentrates in specific bins, resulting in low entropy.

We define a threshold τ that scales inversely with this local entropy. When entropy is high (noise is elevated), the threshold tightens to compress spurious microstructure spikes. When entropy is low (a clear, persistent trend is forming), the threshold widens to capture the full magnitude of the directional move without interference.

We then apply a non-linear squashing function, the hyperbolic tangent, to the data points that exceed this dynamic τ_t.

Where λ controls the asymptotic limit of the compression. This entropy-based transformation acts as a differential shock absorber. The tanh function guarantees monotonicity. A +15% move will always compute to a higher transformed value than a +5% move, preserving the exact sequential logic and rank order of the time series. However, the numerical value is compressed into a safe bound (bounded by τ_t + λ). We preserve the exact directional information of the capitulation while neutralizing the numerical hazard that causes gradient explosion.

import numpy as np
import pandas as pd

def shannon_entropy(series, bins=50):
    """Calculates the Shannon Entropy of a time series chunk."""
    hist, _ = np.histogram(series, bins=bins, density=True)
    hist = hist[hist > 0] # Avoid log(0)
    return -np.sum(hist * np.log2(hist))

def entropy_winsorization(series, window=100, base_threshold=3.0, lambda_val=1.5):
    """
    Soft-clips outliers based on dynamic rolling entropy.
    Preserves topological information while guarding against gradient explosion.
    """
    cleaned_series = series.copy()
    
    for i in range(window, len(series)):
        chunk = series.iloc[i-window:i]
        entropy = shannon_entropy(chunk)
        
        dynamic_tau = base_threshold * (1.0 / (entropy + 0.1)) 
        current_val = series.iloc[i]
        
        if abs(current_val) > dynamic_tau:
            # Apply tanh soft clipping
            sign = np.sign(current_val)
            clipped = dynamic_tau + lambda_val * np.tanh((abs(current_val) - dynamic_tau) / lambda_val)
            cleaned_series.iloc[i] = clipped * sign
            
    return cleaned_series

This is starting to look more like a series that you can trade on.

Cross-sectional orthogonalization

Quants don’t usually model a single time series in isolation. It’s pretty common to evaluate dense feature matrices spanning dozens of correlated assets, hundreds of technical indicators, or entire statistical arbitrage baskets. When we transform these multiple temporal series into a unified dataset, we immediately encounter the curse of multicollinearity.

If we feed highly correlated time series—for example, the minute-by-minute returns of SPY, QQQ, and IWM—into a linear predictive model, the covariance matrix X^TX becomes nearly singular. Because the vectors are almost parallel in geometric space, the determinant of X^TX approaches zero. When the algorithm attempts to invert this matrix to calculate coefficients via (X^TX)^-1, the condition number explodes. The resulting feature weights become hyper-sensitive to microscopic noise. The model might assign a weight of +1500 to SPY and -1499 to QQQ just to fit a tiny basis point of random variance. This leads to catastrophic overfitting and immediate out-of-sample failure.

The required transformation is Cross-Sectional Orthogonalization, executed via PCA or Gram-Schmidt orthonormalization across the time domain. By transforming the raw, correlated feature space into an orthogonal basis, we geometrically decouple the variance.

Let X be our T ∈ N matrix of standardized time series. We compute the empirical covariance matrix

Because Σ is symmetric and positive semi-definite, we can perform an eigendecomposition to find the orthogonal eigenvectors W and their corresponding eigenvalues λ:

The eigenvectors represent the orthogonal directions of maximum variance in the market. The time series are then transformed into Principal Components Z:

The resulting matrix Z consists of N uncorrelated time series. The dot product of any two distinct columns in Z is zero. In quantitative finance, these components have direct, tradable economic interpretations. The first principal component (Z₁), associated with the largest eigenvalue, universally represents the underlying market beta—the systemic risk factor moving all assets. The second and third components represent sector rotations, value-growth spreads, or distinct macroeconomic risk premia.

By applying this transformation, we solve the multicollinearity problem. The optimizer can now assign weights to independent risk factors. Furthermore, we leverage this orthogonalization to denoise the entire system using Random Matrix Theory, applying the Marchenko-Pastur distribution.

Not all computed eigenvectors contain actual signal. Eigenvectors associated with eigenvalues near zero represent idiosyncratic microstructure bounce—pure noise. According to the Marchenko-Pastur theorem, we can analytically calculate a threshold eigenvalue below which the variance is statistically indistinguishable from a random matrix.

We discard these low-variance principal components. We set their eigenvalues to zero and reconstruct the time series using only the top K components. This generates a synthetic, orthogonalized data matrix that captures the true underlying drivers of the market while deleting the highly correlated, overfitted noise. Algorithms trained on this subspace converge rapidly and display superior out-of-sample Sharpe ratios.

from sklearn.decomposition import PCA
import pandas as pd
import numpy as np

def pca_orthogonalize(df_features, variance_threshold=0.95):
    """
    Transforms cross-sectional time series into orthogonal Principal Components,
    filtering out multicollinearity and idiosyncratic noise.
    """
    # Standardize features to ensure fair variance extraction
    standardized = (df_features - df_features.mean()) / df_features.std()
    
    # Initial fit to evaluate the variance explained by each orthogonal factor
    pca = PCA()
    pca.fit(standardized)
    
    # Determine the number of principal components to keep (Marchenko-Pastur proxy)
    cumulative_variance = np.cumsum(pca.explained_variance_ratio_)
    n_components = np.argmax(cumulative_variance >= variance_threshold) + 1
    
    # Re-fit and transform the data using only the strictly valid components
    pca_reduced = PCA(n_components=n_components)
    components = pca_reduced.fit_transform(standardized)
    
    # Reconstruct the feature matrix in the new, uncorrelated topological space
    columns = [f'PC_{i+1}' for i in range(n_components)]
    return pd.DataFrame(components, index=df_features.index, columns=columns)

In this plot we can see how there are a few components that explain the variance of the data we have used.

Non-linear embedding and phase-space reconstruction

The preceding six transformations focus on structural adjustment, stationarity, and linear geometry. However, financial markets also display complex nonlinear dynamics driven by interacting algorithmic agents, endogenous liquidity feedback loops, and other factors.

In dynamical-systems terms, this behavior arises from a deterministic system with extreme sensitivity to initial conditions, a property that can be quantified through a positive maximal Lyapunov exponent. To model this kind of temporal behavior, analyzing a one-dimensional sequence x₁, x₂, …, x_t is insufficient. The time series obtained from an exchange API is only a single observable projection of a far more complex multidimensional system.

When we track the price of an asset, we observe only one coordinate of a much larger state space. The hidden variables remain unobserved: aggregate market maker inventory, institutional limit orders resting off-book, options-driven delta-hedging pressure, and the intraday collateral constraints of leveraged funds. Since explicit measurement of all these variables is impossible, we must transform the 1D time series into a multidimensional geometric representation.

This is precisely the role of Takens’ Delay Embedding Theorem. Under broad regularity conditions, the theorem shows that a single observed time series can be used to reconstruct a geometry that is diffeomorphic, or smoothly equivalent, to the underlying hidden attractor of the full dynamical system. We don’t need direct access to the latent variables. Their influence is already encoded in the sequential history of the observable series.

We transform the scalar time series x_t into an m-dimensional vector x_t using a time delay τ:

This mapping lifts the time series into phase space. The choice of the delay τ and the embedding dimension m is critical because the validity of the reconstructed geometry depends on both.

If τ is too small, the coordinates x_t and x_t-τ remain highly correlated, so the embedded points collapse near a diagonal and provide almost no new structural information. If τ is too large, the coordinates become nearly independent, the deterministic linkage weakens, and the reconstructed geometry loses coherence. The standard way to select τ is to compute the mutual information between x_t and x_t-τ across candidate lags. Unlike Pearson correlation, mutual information is entropy-based and captures nonlinear dependence. We choose the first local minimum, which identifies the delay where the next coordinate contributes maximum new structural information while still preserving dynamical dependence.

We determine the embedding dimension m through the False Nearest Neighbors algorithm. If a genuinely high-dimensional system is projected into a space of insufficient dimension, trajectories that are actually far apart can appear artificially close. The FNN procedure increases the dimension step by step. For each point, it identifies its nearest neighbor in dimension m, then measures how that same pair separates when lifted into dimension m+1. If the distance expands beyond a prescribed threshold, the neighbor was false. We stop increasing m when the proportion of false nearest neighbors approaches zero. At that point, the attractor has been properly unfolded and the geometric trajectories no longer overlap because of projection error.

Once we plot the transformed matrix X in two or three dimensions, we no longer observe a jagged price trace indexed by time. We observe the geometry of market regimes. A mean-reverting regime can generate a torus-like structure. A persistent directional regime can form a stretched manifold. A volatility expansion can appear as an outward spiral.

By transforming the time series into phase space, we allow models such as convolutional neural networks and support vector machines to analyze the data as a spatial and topological object rather than as a raw sequence of values. We can further map the embedded trajectories into Gramian angular fields. A CNN can then scan the resulting matrix as it would scan an image, learning the geometric signature of a volatil regime instead of memorizing a noisy run of negative returns. The task shifts from forecasting a scalar value to recognizing a geometric configuration.

import numpy as np

def takens_embedding(series, m, tau):
    """
    Reconstructs the multi-dimensional phase space of a chaotic dynamical system 
    from a single 1D temporal series using Takens' Delay Embedding Theorem.
    """
    N = len(series)
    
    # Mathematical guardrail: ensure we have enough data points to populate the geometry
    if N < (m - 1) * tau + 1:
        raise ValueError("Time series is mathematically too short for these embedding parameters.")
        
    # Initialize the phase-space matrix
    embedded = np.zeros((N - (m - 1) * tau, m))
    
    # Vectorized mapping of the delayed coordinates into the topological space
    for i in range(m):
        # We slice the array with the necessary delays and assign to the specific dimension
        embedded[:, i] = series.iloc[i * tau : N - (m - 1) * tau + i * tau]
        
    return embedded

The reconstructed phase-space projection forms a dense cloud centered near the origin, with no clear low-dimensional manifold, closed orbit, or recurrent geometric skeleton. This suggests that, at the selected delay, the observable is dominated by noise or weak dependence rather than by a visually recoverable deterministic structure.

The unified pipeline

The core conflict of data transformation in quantitative finance lies in the attempt to apply singular, naive adjustments to a complex system. The quantitative finance industry is littered with the remnants of funds that treated data transformation as a menu of independent choices, or worse, engaged in kitchen-sink feature engineering by feeding 500 un-transformed indicators into a gradient boosting model. If a quant applies a standard scaler without adjusting for non-normality, or differentiates integer-wise without considering memory, the resulting models are doomed to fail in production.

The correct chronological execution of this pipeline resolves the fundamental obstacles:

1. Deformation: We first eliminate chronological heteroskedasticity by sampling the raw tick data using a Dollar-Volume clock. The series moves into information-time.

2. Stationarity: We apply Fractional Differencing to the dollar bars. We optimize the d parameter to achieve stationarity while maintaining the maximum theoretical memory. This step requires the stable variance profile created by the time deformation in step one.

3. Outlier mitigation: We apply the entropy-based soft-clipping function to neutralize extreme prints without destroying the topological footprint of the shock. We do this before volatility modeling to prevent a single extreme outlier from biasing the GARCH parameter estimation. If a extreme volatility hits a GARCH model without prior entropy clipping, the conditional variance explodes and suppresses the transformed returns for the next N periods.

4. Volatility standardization: We fit a GARCH(1,1) model and divide the fractionally differenced, clipped series by the conditional volatility to remove autoregressive variance clustering.

5. Distributional adjustment: We calculate the empirical skewness and kurtosis and apply the inverse Cornish-Fisher expansion to normalize the higher moments of the data. This must occur after GARCH standardization. GARCH removes the conditional, short-term volatility bursts, leaving behind the true, unconditional structural non-normality of the asset for the Cornish-Fisher expansion to correct.

6. Orthogonalization: Across multiple parallel assets, we apply PCA to extract the uncorrelated risk factors, dropping the noise-heavy eigenvectors via Marchenko-Pastur thresholding. We can only compute valid covariance matrices for PCA because the prior five steps guaranteed that our individual series are stationary, outlier-free, and distributionally stable. If you run PCA on raw returns, the principal components align with the noise.

7. Embedding: Finally, if required by the predictive architecture, we map the clean, stationary, orthogonalized sequence into an m-dimensional phase space using Takens’ Theorem. This topological mapping requires pristine data. Otherwise, Takens’ theorem simply maps the microstructure noise instead of the system’s actual attractor.

In a live trading environment, these seven steps must be computed sequentially on streaming tick data with sub-millisecond latency. This demands optimized C++ implementations or JAX-compiled Python arrays, bypassing the overhead of standard pandas dataframes.

Cool guys! Good job today! Remember that you have the full code in the appendix so you can torture it. Time to say good bye! Stay sharp, stay bold, stay unshakeable 📈

PS: What do you prefer to trade, big caps or small caps?

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full code:

import warnings
import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt

from scipy.optimize import minimize
from statsmodels.tsa.stattools import adfuller
from sklearn.decomposition import PCA

warnings.filterwarnings("ignore")

class TimeSeriesTransformer:
    def get_weights_ffd(self, d, length, threshold=1e-5):
        weights = [1.0]
        k = 1
        while True:
            weight_k = -weights[-1] * (d - k + 1) / k
            if abs(weight_k) < threshold and k >= length:
                break
            weights.append(weight_k)
            k += 1
            if k > 10000:
                break
        return np.array(weights[::-1])

    def fractional_diff(self, series, d, threshold=1e-5):
        weights = self.get_weights_ffd(d, length=1, threshold=threshold)
        width = len(weights) - 1
        out = pd.Series(index=series.index, dtype=float)
        vals = series.to_numpy(dtype=float)
        for i in range(width, len(series)):
            out.iloc[i] = np.dot(weights, vals[i - width:i + 1])
        return out.dropna()

    def optimize_fractional_d(self, series, d_range=np.arange(0.1, 1.0, 0.1), p_value_threshold=0.05):
        best = None
        rows = []
        for d in d_range:
            diffed = self.fractional_diff(series, d)
            if len(diffed) > 10:
                adf_stat = adfuller(diffed, maxlag=1, regression="c", autolag=None)
                rows.append((d, adf_stat[0], adf_stat[1], len(diffed)))
                if best is None and adf_stat[1] < p_value_threshold:
                    best = d
        return (1.0 if best is None else best), pd.DataFrame(rows, columns=["d", "adf_stat", "p_value", "n_obs"])

    def cornish_fisher_transform(self, returns):
        z = (returns - returns.mean()) / returns.std(ddof=0)
        ranks = z.rank(pct=True).clip(1e-6, 1 - 1e-6)
        return pd.Series(stats.norm.ppf(ranks), index=returns.index)

    def generate_dollar_bars(self, ticks_df, dollar_threshold):
        df = ticks_df.copy()
        df["dollar_value"] = df["price"] * df["volume"]
        df["cumulative_dollars"] = df["dollar_value"].cumsum()
        df["bar_id"] = (df["cumulative_dollars"] // dollar_threshold).astype(int)
        return df.groupby("bar_id").agg(
            open=("price", "first"),
            high=("price", "max"),
            low=("price", "min"),
            close=("price", "last"),
            volume=("volume", "sum"),
            dollars=("dollar_value", "sum"),
        )

    def _garch11_negloglik(self, params, r_scaled):
        omega, alpha, beta = params
        if omega <= 1e-10 or alpha < 0 or beta < 0 or alpha + beta >= 0.999:
            return 1e12
        h = np.empty(len(r_scaled))
        h[0] = np.var(r_scaled, ddof=1)
        for t in range(1, len(r_scaled)):
            h[t] = omega + alpha * (r_scaled[t - 1] ** 2) + beta * h[t - 1]
            if h[t] <= 1e-12 or not np.isfinite(h[t]):
                return 1e12
        return 0.5 * np.sum(np.log(2 * np.pi) + np.log(h) + (r_scaled ** 2) / h)

    def garch_standardize(self, returns):
        r = returns.dropna().to_numpy(dtype=float)
        scale = 100.0
        r_scaled = r * scale
        var0 = np.var(r_scaled, ddof=1)
        x0 = np.array([0.01 * var0, 0.05, 0.90])
        bounds = [(1e-10, None), (0.0, 0.999), (0.0, 0.999)]
        constraints = ({"type": "ineq", "fun": lambda x: 0.999 - (x[1] + x[2])},)
        res = minimize(
            self._garch11_negloglik,
            x0,
            args=(r_scaled,),
            method="SLSQP",
            bounds=bounds,
            constraints=constraints,
            options={"maxiter": 500},
        )
        if res.success:
            omega, alpha, beta = res.x
            h = np.empty(len(r_scaled))
            h[0] = var0
            for t in range(1, len(r_scaled)):
                h[t] = omega + alpha * (r_scaled[t - 1] ** 2) + beta * h[t - 1]
        else:
            lam = 0.94
            h = np.empty(len(r_scaled))
            h[0] = var0
            for t in range(1, len(r_scaled)):
                h[t] = (1 - lam) * (r_scaled[t - 1] ** 2) + lam * h[t - 1]
        cond_vol = np.sqrt(h) / scale
        return (
            pd.Series(r / cond_vol, index=returns.dropna().index),
            pd.Series(cond_vol, index=returns.dropna().index),
            {"success": bool(res.success), "params": res.x if res.success else np.array([np.nan, np.nan, np.nan])},
        )

    def shannon_entropy(self, series, bins=50):
        hist, _ = np.histogram(series, bins=bins, density=True)
        hist = hist[hist > 0]
        return -np.sum(hist * np.log2(hist))

    def entropy_winsorization(self, series, window=100, base_threshold=3.0, lambda_val=1.5):
        cleaned = series.copy()
        taus = pd.Series(index=series.index, dtype=float)
        for i in range(window, len(series)):
            chunk = series.iloc[i - window:i]
            entropy = self.shannon_entropy(chunk)
            tau = base_threshold * (1.0 / (entropy + 0.1))
            taus.iloc[i] = tau
            x = series.iloc[i]
            if abs(x) > tau:
                sign = np.sign(x)
                clipped = tau + lambda_val * np.tanh((abs(x) - tau) / lambda_val)
                cleaned.iloc[i] = clipped * sign
        return cleaned, taus

    def pca_orthogonalize(self, df_features, variance_threshold=0.95):
        z = (df_features - df_features.mean()) / df_features.std(ddof=0)
        pca_full = PCA()
        pca_full.fit(z)
        cum_var = np.cumsum(pca_full.explained_variance_ratio_)
        n_components = np.argmax(cum_var >= variance_threshold) + 1
        pca_reduced = PCA(n_components=n_components)
        comps = pca_reduced.fit_transform(z)
        cols = [f"PC_{i+1}" for i in range(n_components)]
        return pd.DataFrame(comps, index=df_features.index, columns=cols), pd.Series(
            pca_full.explained_variance_ratio_,
            index=[f"PC_{i+1}" for i in range(len(pca_full.explained_variance_ratio_))]
        )

    def takens_embedding(self, series, m, tau):
        N = len(series)
        if N < (m - 1) * tau + 1:
            raise ValueError("Time series is too short for these embedding parameters.")
        embedded = np.zeros((N - (m - 1) * tau, m))
        for i in range(m):
            embedded[:, i] = series.iloc[i * tau:N - (m - 1) * tau + i * tau]
        return embedded

Table of contents:

1. Introduction.

2. Escalation of war in the Gulf.

3. Hormuz as the strategic hinge.

4. Kharg island is the energy chokepoint.

5. Energy shock and the macro transmission channel.

6. Second-order economic effect of war.

7. Egypt external war pressure.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

The Gulf war now sits at the center of the global economic system. What once looked like a regional military confrontation now shapes the price of oil, the cost of maritime insurance, the movement of LNG, the availability of fertilizers, the outlook for inflation, and the fiscal stability of import-dependent states. The conflict has crossed the boundary between battlefield dynamics and macroeconomic reality. Every strike on infrastructure, every disruption in transit, and every shift in force posture now feeds directly into trade flows, industrial costs, sovereign risk, and household pressure across regions far from the Gulf.

At the center of that system stands the Strait of Hormuz. This narrow maritime corridor connects armed confrontation with global energy supply in a direct and immediate way. It is the hinge between missiles and inflation, between naval pressure and industrial costs, between regional violence and worldwide repricing across oil, gas, freight, food, and sovereign debt. Around it, Kharg Island emerges as a second critical node, a concentrated export point whose strategic value lies in its ability to convert Iranian production into global market pressure and fiscal survival.

What follows traces that chain with a clear line of sight. It begins with the expansion of war across the Gulf export system and then moves through Hormuz as the decisive chokepoint where military risk becomes macroeconomic shock. From there, the focus shifts to Kharg Island and the wider energy channel, where the conflict enters inflation, trade, growth, and monetary policy. The later sections turn to the deeper economic consequences of prolonged confrontation, including alliance strain, defence spending, fiscal burdens, and the exposure of vulnerable import-dependent economies such as Egypt.

You can go deeper by checking this report:

Gobal trade outlook

1.44MB ∙ PDF file

Download

Download

This moment also carries a strategic message for markets and states alike. The most important risks sit in the infrastructure that keeps economies supplied, in the narrow corridors that keep trade alive and domestic stability. Once those channels come under pressure, the effects spread with speed from ports to prices, from commodities to currencies, and from regional confrontation to global adjustment.

Subscribe now

Escalation of war in the Gulf

The war now sits inside the core of the global economy. The opening phase concentrated on Iranian leadership nodes, missile capacity, nuclear assets, and military infrastructure. The present phase covers the Gulf export system, maritime transit, shipping insurance, alliance bargaining, and market pricing across oil, gas, freight, food, and sovereign risk. Attacks hit major oil and gas sites across Iran, Saudi Arabia, Kuwait, Qatar, the United Arab Emirates, Bahrain, and Iraq. Bahrain declared force majeure after damage at the Sitra refinery, Iraq cut output as Hormuz disruption blocked crude flows, and Qatar reported the loss of around one-sixth of LNG capacity after damage to Ras Laffan. That sequence transformed the conflict into a regional energy and trade shock with global reach.

Persian Gulf oil shipments has lifted energy prices. Its adjusted high-energy scenario lowers 2026 world merchandise trade volume growth to 1.4% from 1.9% and trims global GDP growth to 2.5% from 2.8%. The IMF added that a persistent 10% increase in oil prices can raise global headline inflation by around 40 basis points and reduce global output by 0.1 to 0.2 percentage points. Together, those estimates show that the conflict now acts through the central channels of inflation, trade, and growth rather than through a narrow regional risk premium.

The Strait of Hormuz sits at the center of this transmission mechanism. UNCTAD estimated that the strait carries around one quarter of global seaborne oil trade along with major LNG and fertilizer flows. Its briefing shows ship transits through the strait falling by 97% from the February average, while oil and gas prices surged between 27 February and 9 March. In that setting, Hormuz becomes the hinge that connects military escalation with industrial input costs, European gas pricing, Asian energy security, fertilizer markets, freight rates, bunker fuel, and food pressure across import-dependent economies.

Washington is sending thousands of additional Marines and sailors to the region, adding to an existing U.S. military presence of about 50,000 personnel. At the same time, the White House has pressed allies to help secure passage through Hormuz, while European partners have adopted a posture centered on safe passage, inflation control, and de-escalation. This combination places deterrence, burden sharing, maritime security, and macro stabilization inside the same strategic field.

Outside powers have entered the same equation. China warned that the war threatens global energy supplies, trade routes, and economic stability. Russia declared political support for Tehran through a Kremlin message that described Russia as a loyal friend and reliable partner. These positions matter because they influence shipping expectations, sanctions politics, commodity diplomacy, and the pricing of geopolitical persistence across global markets.

Financial transmission has also accelerated. A U.S.-backed maritime reinsurance plan led by Chubb seeks to restore commercial shipping through Hormuz, which shows how finance and logistics now function as strategic infrastructure alongside naval and air power. UNCTAD’s assessment also tracks sharp rises in tanker costs, bunker fuel, and bond yields across exposed Gulf economies, confirming that the shock now moves through transport costs, refinancing conditions, and imported inflation as well as through crude and LNG benchmarks.

The confrontation therefore belongs to the category of wider-system conflict. Geography, energy, trade, shipping, insurance, alliance politics, inflation, and growth forecasts now move inside one connected field. Each strike on infrastructure or transit routes now feeds into industrial costs, consumer prices, sovereign funding conditions, and policy choices across regions far from the battlefield.

By the way, before we move to next section, a new TACO just happned today because of:

Hormuz as the strategic hinge

The closure of the strait has stopped the passage of about 20% of the world’s oil and liquefied natural gas since 28 February. The International Maritime Organization said around 20,000 seafarers on nearly 2,000 ships west of the strait are caught in the disruption, and Iraq has already declared force majeure because Hormuz disruption halted most crude exports and cut Basra output from 3.3 million barrels per day to 900,000. The narrow channel now links missile risk and naval pressure with global energy supply, shipping capacity, and state revenue.

The U.S. Energy Information Administration reported that Hormuz carried about 20 million barrels per day in 2024, equal to about 20% of global petroleum liquids consumption and more than one-quarter of seaborne oil trade. Around one-fifth of global LNG trade also moved through the strait. EIA estimated bypass capacity in Saudi and Emirati pipelines at about 2.6 million barrels per day, so rerouting covers only a small share of the volume at risk. UNCTAD’s adds that, one week before the conflict, crude oil represented 38% of the relevant seaborne trade categories moving through Hormuz, LPG 29%, LNG 19%, refined oil products 19%, and chemicals, including fertilizers, 13%. Asia receives 84% of Hormuz crude and 83% of Hormuz LNG, which gives the chokepoint immediate reach across the main import centers of the global economy.

Traffic data show how fast military pressure becomes market shock. UNCTAD recorded an average of 141 daily ship transits during 1–27 February, followed by a 97% collapse. Brent had risen 27% and Dutch TTF gas 74% from then. Chubb joined a $20 billion maritime reinsurance plan because shipowners, cargo interests, and financiers need war-risk cover before normal commercial flows can return. Passage through Hormuz therefore sets the marginal cost of energy movement, insurance, and cargo finance for economies far beyond the Gulf.

The macroeconomic spillover now reaches trade volumes, food costs, inflation expectations, and sovereign financing. Severe curtailment of Persian Gulf oil shipments could cut world merchandise trade volume growth in 2026 to 1.4% from 1.9% and lower world GDP growth to 2.5% from 2.8% under its high-energy-price scenario. This connects higher Gulf energy prices with fertilizer stress and broader food-system pressure. Besides, government bond yields in the United States and Europe jumped as investors priced a more persistent inflation shock, with the UK 10-year yield rising above 5% and markets shifting from rate-cut expectations toward rate-hike expectations. Hormuz therefore acts through central-bank reaction functions and sovereign borrowing costs as well as through the oil curve.

Kharg island is the energy chokepoint

Kharg Island now stands as one of the clearest points where battlefield logic and macroeconomic leverage meet. Kharg handles 90% of Iran’s oil exports, and some reports showed that Iran continued to ship between 1.1 million and 1.5 million barrels per day during the war. Iran exported about 1.7 million barrels per day so far in 2026, with about 1.55 million barrels per day moving through Kharg, after exports had reached about 2.17 million barrels per day in February. That concentration gives the island a dual role. It functions as Iran’s main export valve and as a target whose condition shapes global expectations for supply, escalation, and endurance.

The strategic value of Kharg comes from concentration of flow, storage, and loading capacity in a single hub. The island holds storage capacity of roughly 30 million barrels and held about 18 million barrels of crude. Its offshore position also matters. Kharg as sitting 16 miles from Iran’s coast in waters deep enough for very large tankers that cannot approach the shallow mainland coast. That gives Kharg a special role in converting upstream production into export revenue at scale. A strike, blockade, or sustained interdiction at that one node would reach revenue, shipping schedules, tanker availability, buyer confidence, and physical balances at the same time. Market analysis are pointing to a potential loss of around 2 million barrels per day from the market if Kharg infrastructure were taken out.

Kharg also carries the fiscal weight of Iran’s external energy system. The U.S. Energy Information Administration’s 2025 SHIP Act report estimated Iran’s crude oil and condensate export revenue at $43 billion in 2024, with exports at 1.483 million barrels per day. The same report estimated exports to China at 1.444 million barrels per day in 2024, which shows how strongly Iran’s export realization depends on a narrow buyer corridor linked to Asian demand. Iranian oil accounted for 11.6% of China’s seaborne imports so far in 2026. Kharg therefore sits at the intersection of Iranian fiscal resilience and Chinese energy procurement. The island matters because it converts production into cash flow, and it does so through a commercial route that remains central to a major importing power.

U.S. forces struck more than 90 Iranian military targets on Kharg while preserving the oil infrastructure. That decision revealed the hierarchy of escalation inside the war. Military assets on the island formed the first target set, while the export system itself remained the larger coercive lever. Trump administration was considering plans to occupy or blockade Kharg in order to pressure Iran to reopen the Strait of Hormuz. Kharg thus moved from vulnerable infrastructure into active strategic bargaining space. In this setting, the island serves as a pressure point whose control, impairment, or threatened seizure can alter both the battlefield and the oil market.

Energy shock and the macro transmission channel

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Sufficient statistics and minimal signals

This lecture defines a trading signal through sufficiency rather than through a rule, threshold, or pattern. A signal is a compression of the decision-time information set that preserves the conditional law of the declared outcome, so predictive meaning remains tied to the future law rather than to a single backtest metric. The lecture then develops minimality, bottleneck design, and representation stability as the conditions that make that compressed state usable under finite data, drift, and noise.

What’s inside:

1. Sufficiency as the definition of signal. A signal is a representation that preserves the conditional distribution of the outcome, which separates signal from strategy and makes feature relevance a property of conditional-law preservation rather than raw correlation.

2. Signal as compression with law preservation. Compression becomes signal construction when the reduced state carries the same predictive content as the full decision-time information set, which lets the system operate on a smaller and more stable representation without changing the law it conditions on.

3. Approximate sufficiency in practice. Exact sufficiency appears as the ideal case, while practical sufficiency is defined through a declared tolerance using divergence, proper-score regret, and the residual predictive content that remains outside the bottleneck.

4. Minimal sufficient sets. The lecture shows that a minimal signal is the smallest representation that still preserves the full conditional law, removes nuisance variation, lowers selection pressure, and keeps the signal portable across operators and model classes.

5. Non-uniqueness of sufficient representations. Many different representations can preserve the same predictive law, so the scientific question is whether preservation exists, while the engineering question is which representation remains most stable under noise, drift, and deployment constraints.

6. Bottleneck trade-offs. Bottleneck design becomes an information-budget problem that balances relevance against representational capacity, because too little compression increases variance and selection pressure, while too much compression removes predictive fragments the outcome still needs.

7. Overcompression and undercompression. The lecture explains that overcompression distorts the conditional law by blending distinct regimes, while undercompression creates a high-capacity overfitting surface where sharp in-sample structure often fails under recurrence and proper scoring.

8. Representation stability. A minimal signal must retain stable meaning under measurement noise, drift in state occupancy, and regime mixtures, so conditional beliefs, calibration, and divergence structure remain coherent when the signal moves from research into live deployment.

Sample

1.93MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. The API layer.

3. Protocol semantics, transport choice, and system state.

4. Risk and financial APIs limitations.

5. Historical-live non-equivalence and data reconstruction error.

6. Execution state machines and order routing.

7. Simulation and live trading.

8. Creating the architecture for a trading API.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Before moving on to the next series, there’s an important point that any algorithmic trader needs to know: the point about APIs.

A quant can state the issue in simple terms. Let s_t​ denote latent market state and let x_t​ denote observed state at decision time. The strategy trades on x_t​ rather than s_t​. The transformation from s_t​ to x_t​ includes vendor capture, transport, buffering, schema validation, clock alignment, and local state reconstruction. When that transformation changes, the strategy changes with it, even if the signal formula stays fixed.

The API layer belongs to the hypothesis itself because it determines the admissible information set on which the strategy acts.

This point matters because the gap between research and deployment tends to emerge at the boundary where information becomes software state. Bars in a notebook arrive sorted and complete. Live messages arrive through sessions, quotes, reconnect paths, and transport rules. Orders in a backtest pass from signal to fill in one step. Orders in production pass through acceptance, routing, queue interaction, etc. In a systematic process, those choices carry mathematical weight because they alter the state space on which the strategy operates.

The live strategy therefore admits a fuller description than the research signal alone. A useful expression is

Many teams place their energy on features, models, and optimization while the interface layer receives less formal treatment. That habit carries a cost. A signal derived from one information regime and deployed through another regime becomes a different object. A simulator that treats execution as an atomic event studies a market process distinct from the one live code encounters.

The API layer is where these fractures surface. It is also where discipline can restore coherence. If you want a stack to start with, here it is:

Algorithmic Trading in the Global FX Market

176KB ∙ PDF file

Download

Download

An API is an information contract, a timing contract, and an execution contract. That’s to say, data fields need causal meaning, timestamps need clear authority, order states need legal transitions, internal services need shared schemas across research, simulation, and live trading. Do you get the ida right? Under those conditions, performance claims acquire stronger footing because the path from observation to action remains explicit.

The subject here is the form through which market enters the strategy and the form through which strategy intent reaches the market. Code style, framework choice, and endpoint count matter through their effect on that form. A live trading stack earns coherence when the objects that move across it stay explicit, typed, and stable across the full path from research to execution. That coherence is a design choice and it begins at the API layer.

The API layer

In a trading system, the central question asks what is known, when it becomes known, and which representation carries it. The API layer lives at that boundary. It defines the admissible information set available to the strategy at decision time. That framing captures the practical source of failure with precision.

Let the latent market state at event index t be s_t. The strategy never accesses that state in raw form. It receives an encoded view x_t, produced through venue generation, vendor capture, normalization, transport, buffering, decoding, and local deserialization. If those operations are compressed into a single operator H, then the strategy acts on

In live systems, H belongs to a family of transformations shaped by endpoint choice, session state, codec, throttling behavior, aggregation policy, and recovery logic. The research question therefore asks two things at once: whether a mapping f(x_t) carries predictive value, and whether the x_t used in research belongs to the same family as the x_t available in deployment.

A useful decomposition is

The term H_t(s_0:t) shows that observed state may depend on a path rather than a single contemporaneous latent state. Corrections, delayed packets, local aggregation, and dropped messages create that path dependence. The residual term ε_t collects the user-facing uncertainty generated by packet loss, schema surprises, decoder failure, clock mismatch, and hidden vendor behavior. In a calm hourly-bar strategy, ε_t may occupy a small role. In a short-horizon or execution-sensitive strategy, it often defines the core challenge.

This perspective dissolves a common misconception. Many quants treat the API as a latency channel and connect latency risk with high-frequency trading. A richer view sees the API as a state-visibility mechanism. Consider a breakout rule triggered when price crosses a level and volume in the same interval exceeds a threshold. One endpoint may publish aggregated bars with finalized volume, while another may publish incremental trades that require local accumulation. The rule then sees different objects.

Speed alone does not resolve that difference. One representation is final. The other is provisional. One is bounded by bar closure. The other evolves through the interval. If research evaluates the final object while deployment trades the provisional one, then backtest and live trading inhabit different information worlds.

The official protocol and framework documentation supports this view from the software side. HTTP semantics are defined as stateless at the application-protocol level. That property makes HTTP well suited for resource retrieval, explicit commands, and idempotent inspection, while state continuity moves into higher layers. The OpenAPI specification makes HTTP service capabilities machine-readable through paths, parameters, schemas, and responses. FastAPI builds on that structure by generating OpenAPI descriptions and interactive documentation from Python declarations. Pydantic treats typed data models as executable contracts and emits JSON Schema from those models. Finance gives special weight to this alignment because state integrity sits at the center of the problem.

For a systematic trading system, the internal data model becomes a compact statement of admissible knowledge. A well-defined Bar object can enforce monotonic timestamps, nonnegative volume, high-low consistency, a declared timezone, and provenance metadata. A Signal object can enforce a causal timestamp, a horizon, a generation identifier, and a confidence field interpreted consistently by the risk layer. An OrderIntent object can separate intent from broker acknowledgment, so the system preserves a clean distinction between desired action, accepted action, and executed action. These structures carry operational value because they keep portfolio state, execution state, and research state aligned across time.

Subscribe now

Protocol semantics, transport choice, and system state

A familiar hierarchy pairs REST with slower interactions, WebSocket with streaming interactions, and FIX with institutional workflows. A stronger formulation sees each protocol as an organizer of interaction and, through that role, as a designer of local architecture.

Start with HTTP. Under RFC 9110, HTTP defines resource-oriented request-response semantics, methods, status codes, headers, and representations. It serves clear operations with strong structure: fetching historical bars, querying account state, placing an order, canceling an order, requesting configuration, and obtaining a health report. It also supports documentation with high clarity.

For internal trading services, that structure has high value. The stack gains clear contracts, reproducible input schemas, and rapid client generation.

HTTP also channels interaction through discrete exchanges. Applications that consume a continuous stream of trade events, order updates, heartbeats, or PnL deltas benefit from WebSocket. RFC 6455 defines a bidirectional protocol over TCP with an opening handshake and framed messages, built for two-way communication.

For example, Alpaca’s streaming documentation—I mean, the broker—presents trade, account, and order updates over WebSocket and highlights frame types across endpoints, plus authorization and stream subscription after connection. For a quant system, the implication is clear: WebSocket provides continuity, and continuity assigns responsibility. A persistent connection requires reconnect handling, duplicate control, interval recovery, subscription replay, sequence validation, and local buffering.

{
  "action": "auth",
  "key": "{YOUR_API_KEY_ID}",
  "secret": "{YOUR_API_SECRET_KEY}"
}

FIX addresses a different problem. The FIX Trading Community describes FIX as:

A standardized transaction language across the securities trade cycle, with message types aligned to steps such as quote request and new order, and with standardized fields as the building blocks.

That matters because institutional trading depends on precise business semantics across a workflow that spans indication, order, execution report, allocation, and post-trade reporting. FIX draws strength from a protocol that already speaks the language of the transaction. A modern retail or semi-professional quant stack may begin on other connectivity paths, yet it can still absorb the lesson: trading systems gain stability when message meanings stay explicit, durable, and tied to the lifecycle they govern.

The mathematics of transport stays simple. Let T_total denote end-to-end delay from the moment a signal becomes ready to the moment a fill event reaches local state. Then

A robust analysis evaluates the whole sum. In many implementations, T_broker and T_reconcile generate more variation than raw network travel time. Serialization, authentication, request signing, broker throttling, internal queuing, and local reconciliation can dominate the tails. Tail behavior matters more than averages because losses tend to concentrate there. A mean delay of 20 ms with a 95th percentile of 60 ms may fit one strategy and damage another whose edge concentrates in short-lived bursts.

The histogram decomposes end-to-end order latency into plausible components and shows the aggregate distribution. The plot reveals that total latency is a sum of heterogeneous terms and that the tail can widen even when most components remain well behaved.

Risk and financial APIs limitation

Once the API is treated as a formal component, the next task is classification. Risk in financial APIs represents a family of failure modes that interact. A useful taxonomy separates at least seven categories: data risk, temporal risk, schema risk, session risk, throughput risk, execution risk, and governance risk.

1. Data risk concerns what is transmitted. This includes missing fields, inconsistent bar construction, delayed corrections, undocumented transformations, and ambiguity about what a timestamp refers to. Is the bar timestamp the interval open, the interval close, or publication time? Are trades condition-filtered? Are odd lots included? Is volume consolidated at the same cadence in historical and live endpoints? Vendor documentation sometimes answers these questions, sometimes not. The point is that the risk exists even when the feed looks clean. A clean feed can still describe the wrong object for the strategy.

2. Temporal risk concerns when the data is available and in which order. Clock drift between client, broker, and data vendor is the obvious part. More subtle is the distinction between event time and arrival time. A message may encode one time but arrive at another. Streaming systems can also deliver temporary disorder. If the strategy assumes a total order while the transport offers only a best-effort order plus reconnect recovery, then the local state machine must restore order or at least detect that order was broken.

3. Schema risk concerns how the data is structured. An endpoint that changes a field name, adds a nullable attribute, changes a numeric type from integer to string, or alters nesting may be perfectly legal at the API level and still break live trading. This is why type validation libraries matter.

4. Session risk is operational but critical. Authentication expiry, refresh tokens, signed requests, environment separation, and broker session exclusivity determine whether the trading path remains live. Some APIs are quite clear that trading-enabled sessions have different restrictions than generic access, including brokerage-level single-session limits.

5. Throughput risk includes rate limits, quotas, pagination, backpressure, and resource exhaustion. This category is often underestimated by quants who work first in notebooks. Yet the feasible research surface is shaped by what the API allows. If a cross-sectional study requires more requests per minute than the provider permits, then the signal either needs a different collection design, a local cache, a licensed feed, or a narrower universe. Throughput is a constraint on the set of admissible strategies.

6. Execution risk concerns the gap between request acceptance and actual market interaction. Here a good example of why this matters: “the order object has identifiers, statuses, and a lifecycle that can be queried after placement. That is useful, but it also confirms the obvious point that the initial placement call is only one transition in a larger process. SEC routing and execution disclosures exist because the path from broker to venue materially affects outcomes”. In trading, the request path and the execution path should never be conflated.

7. Governance risk covers versioning, deprecation, environment drift, and organizational ambiguity. An internal API that is undocumented, lacks version tags, or mixes research-only fields with live-only fields can corrupt an otherwise sound strategy. Governance is what keeps the contract stable long enough for experiments to be comparable.

A convenient formalization is to model observed system risk as the union of these categories:

Each component can be associated with observables: gap counts, duplicate counts, validation failures, authentication renewals, request rejections, order rejects, and contract-version mismatches. The important point is that API risk becomes measurable only after categories are explicit.

The table below compresses this pitfalls:

Throughput deserves one extra comment because it changes research conclusions. Assume that trade opportunities survive for a characteristic time constant τ and that the system polls every Δ units of time rather than consuming a push stream. A rough first-order miss model is:

This formula is for illustrative purposes. It says that when opportunity duration is short relative to the observation interval, miss probability rises quickly. Coarser polling changes the distribution of observed opportunities. The picture below visualizes that relationship:

Quants often frame API problems as reliability problems. They are also selection problems. The feed, the protocol, and the quota choose which events you ever get the chance to model. That means a strategy discovered under one data-access regime may not even be defined under another. Once this is understood, the response becomes sharper.

Historical-live non-equivalence and data reconstruction error

Historical endpoints and live endpoints are seldom equivalent in the strict sense required by causal research. They may deliver related objects, but related is not enough. Historical data is often cleaned, deduplicated, repaired, aligned to bar boundaries, and packaged into stable records. Live data is incremental, provisional, and susceptible to temporary disorder. A strategy that learns on one and trades on the other inherits a reconstruction problem.

There are at least four sources of non-equivalence:

1. The first is finalization: Historical bars tend to be final objects. Live bars, unless the provider specifically publishes only completed intervals, may be in progress.

2. The second is correction policy: Historical vendors can revise prior records after exchange corrections or internal data-quality routines. Live systems receive the pre-correction path and may or may not later receive explicit correction messages.

3. The third is aggregation method: A historical endpoint may aggregate on server-side rules that differ from a client’s local reconstruction from trade ticks.

4. The last one is delivery semantics: Historical data is fetched as a completed array. Live data is delivered as a stream in which messages can be delayed, duplicated, or arrive after reconnect.

Any research process should ask a simple but uncomfortable question: is the historical object used in research reproducible from the live stream using only information available at the time? If the answer is no, then the strategy is at risk of hidden lookahead or deployment drift. Sometimes the mismatch is harmless. Often it is not. Opening-range logic, event-triggered strategies, and execution-sensitive systems are especially vulnerable because they depend on exact temporal boundaries.

One clean way to formalize the problem is to define a reconstruction operator G that maps a live stream path into a bar or state record:

The historical dataset supplies b_t^hist. The research assumption that often goes unstated is that b_t^hist = b_t^live_recon. This is rare. The true object of concern is the reconstruction error:

A strategy that is stable under small δ_t may survive deployment. A strategy that changes sign, entry timing, or position size under small δ_t may lose money even if the backtest looks good. This is one reason why mature groups insist on replay tests. They record live messages, reconstruct the internal state offline, and compare it with both the production decision log and the historical vendor view.

Out-of-order delivery is a particularly sharp example. Suppose messages carry event timestamps e_i but arrive at times a_i. A client that sorts by arrival time instead of event time assumes that sign(a_i - a_j) equals sign(e_i - e_j) for all pairs. That assumption is false in many real networks. Once it breaks, local bars, rolling indicators, or event-triggered conditions may shift. The correct response is to build reordering logic with bounded buffers and explicit policy for what to do when order cannot be restored with confidence.

Here is a minimal sketch of such a guard:

from dataclasses import dataclass
from heapq import heappush, heappop


@dataclass(order=True)
class MarketEvent:
    seq: int
    ts_event_ns: int
    payload: dict


class ReorderBuffer:
    def __init__(self, max_gap: int = 32):
        self.expected = 0
        self.max_gap = max_gap
        self.heap = []

    def push(self, event: MarketEvent):
        if event.seq < self.expected:
            return []  # duplicate or stale replay

        heappush(self.heap, event)
        out = []

        while self.heap and self.heap[0].seq == self.expected:
            out.append(heappop(self.heap))
            self.expected += 1

        if self.heap and self.heap[0].seq - self.expected > self.max_gap:
            raise RuntimeError("sequence gap too large")

        return out

The point of the snippet is to show the shape of the invariant. Local state should advance only when sequence integrity is acceptable. If the gap exceeds a declared tolerance, the client should raise an incident, resubscribe, or rebuild state from a safer source.

The next plot shows the operational consequence of delayed observations on a simple synthetic crossover-style entry. The delayed observation sees the same price path later and enters later. The lesson is that even modest observation delay can transform entry timing in a way that a clean historical backtest never exposed.

A common response is to smooth the signal until these issues disappear. That can help, but it does not solve the identification problem. If the strategy only works when final corrected bars are used, then the strategy depends on a state that may not exist in live trading.

First, define decision-time objects. Second, test historical-live parity by replaying captured live streams into the same internal schemas used in production. Third, store provenance metadata with every derived bar or indicator: source, reconstruction mode, timezone, sequence completeness, and whether the object is provisional or finalized. Fourth, make non-equivalence visible in research by introducing uncertainty envelopes. If a signal flips under plausible reconstruction perturbations, that signal should not be promoted.

Execution state machines and order routing

Read more

Table of contents:

1. Introduction.

2. Conflict and escalation dynamics.

3. Energy shock, inflation pass-through, and logistics friction.

4. Monetary policy, fiscal space, and financial stability channels.

5. Alliance, war, and strategic competition.

6. Defense-industrial economics and the cost curve of modern warfare.

7. Investment implications for multi-asset portfolios.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

Today we map the transmission system through which an Iran-centered war in the Gulf becomes a global pricing regime. The shock already extends far beyond the battlefield. It now moves through oil, equities, shipping, insurance, and industrial planning. Brent crude traded above $119 a barrel in early March 2026 as the conflict widened. Businesses and investors across Europe, Asia, and North America repriced energy exposure, freight risk, and supply-chain vulnerability. That response follows the structure of the system itself. The Strait of Hormuz remains one of the world’s decisive energy corridors. In 2025, flows through the strait averaged about 20 million barrels per day, equal to roughly one-fifth of global petroleum liquids consumption. Around one-fifth of global LNG trade also moved through that corridor.

The central frame is escalation through transmission channels. Missile exchanges raise shipping insurance. Higher shipping insurance raises fuel and freight costs. Higher fuel and freight costs strengthen inflation pressure. Stronger inflation pressure tightens monetary conditions, narrows fiscal room, and increases sensitivity in rates and credit markets. Geopolitical violence therefore enters asset prices through a cumulative macro-financial sequence. That sequence now operates in a constrained policy setting. On January 28, 2026, the Federal Reserve kept the federal funds target range at 3.5% to 3.75%. On February 5, 2026, the European Central Bank kept the deposit facility rate at 2.00%. The Congressional Budget Office projected a $1.9 trillion U.S. federal deficit for fiscal 2026. SIPRI reported that world military expenditure reached $2.718 trillion in 2024. Monetary authorities, fiscal states, and defense systems entered this phase with meaningful pre-existing constraints.

If you want to learn more, you’ll find more information here:

Short Term Energy Outlook 2026

7.03MB ∙ PDF file

Download

Download

Short Term Energy Outlook 2026

7.03MB ∙ PDF file

Download

Download

Read through that lens, the conflict functions as a regime test for portfolios, governments, and industrial systems. It tests whether logistics networks can reroute under pressure. It tests whether central banks can preserve credibility under renewed energy inflation. It tests whether fiscal states can absorb a larger strategic bill while debt burdens remain elevated. It also tests whether defense-industrial supply chains can sustain a modern cost curve shaped by expensive interceptors, long lead times, and repeated replenishment cycles. For that reason, the analysis that follows moves from escalation to energy pass-through. Then to monetary and fiscal constraints. Then to alliance management and defense-industrial economics. And finally to the asset-pricing implications that matter most for multi-asset investors in a world where geopolitical shocks increasingly behave like macro transmission systems rather than isolated events.

Subscribe now

Conflict and escalation dynamics

The market narrative around an Iran-centered conflict that expands across the broader Middle East, with spillovers into Gulf targets, regional infrastructure, and allied basing arrangements. That matches the pattern described by major wire services: a U.S.-Israeli campaign against Iran has coincided with missile and drone strikes across multiple Gulf states and with pressure on energy logistics and critical infrastructure.

A core structural feature of this conflict phase is its multi-node geometry. It links air and missile defense networks, energy export platforms, sea lanes, and diplomatic posture into a single operating system. Missile-defense radars and communications sites have emerged as strategic targets because they anchor interception chains for systems deployed by the United States and regional partners. This target set differs from a narrow, territory-bound war and aligns with a systems war logic: degrading sensors can reduce the marginal value of interceptors and raise the per-attack success probability for drones and missiles.

A gap between initial strike objectives and post-strike political end-states, framed as the absence of a defined phase two. A related public signal appeared in reporting from the Financial Times, which cited Senator Mark Warner’s view that administration briefings lacked an articulated next-step plan for the aftermath of the military campaign. In investor terms, that type of gap raises the probability distribution over duration, geographic spread, and infrastructure targeting, which then feeds into commodity risk premia and credit repricing.

Escalation pathways come from three channels that interact. The first is kinetic interaction across dense air-defense coverage, where a high volume of launches produces stress on interceptors, radars, and maintenance cycles. The second is economic coercion via chokepoints and storage constraints, which can force producers to curb output when exports face friction. The third is alliance-management stress, especially when operations touch bases, ports, and treaty obligations for states that seek distance from a campaign.

At the strategic layer, European governments have produced a legal framing that increases political distance from kinetic participation, which raises burden-sharing uncertainty and can bring additional volatility around NATO posture and basing in Southern Europe. This divergence matters for markets because alliance cohesion influences sanction coordination, shipping escorts, and the credibility of de-escalation off-ramps.

Conflict intensity has already shaped equity pricing. Intraday drawdowns in U.S. equities exceeded one thousand Dow points in at least one session during the first week of March, with oil-price acceleration as a primary driver in contemporaneous reporting. That linkage between war-duration uncertainty and inflation risk is central: markets can price tactical battlefield outcomes, yet they price duration and breadth through commodities, rates, and credit spreads.

Energy shock, inflation pass-through, and logistics friction

The dominant macro transmission mechanism in this phase is the energy channel, amplified by chokepoint concentration. The Strait of Hormuz serves as a major conduit for global petroleum liquids and for liquefied natural gas trade. The U.S. Energy Information Administration estimated average oil flows through the strait around twenty million barrels per day in 2024, which aligns with about one-fifth of global petroleum liquids consumption, and it assessed about one-fifth of global LNG trade transiting the strait in 2024, primarily from Qatar. The International Energy Agency provides a parallel framing, with LNG transiting volumes around one-fifth of global LNG trade and seaborne oil trade shares tied to Hormuz flows.

In this conflict phase, shipping disruption and storage constraints have emerged as binding constraints for producers and buyers. Recent reportes indicate producer firms have managed offshore output around storage restrictions and have used alternative export routes, while the broader war environment has pushed some Gulf producers toward output curtailment and force majeure declarations. In parallel, European policymakers convened contingency coordination groups after a sharp rise in regional gas prices.

Price action in crude and refined products has reflected a risk-premium regime. In early March, Brent traded in the mid-to-high eighties and moved into the nineties in subsequent sessions, while U.S. benchmarks also rose with large daily moves. A key market feature is convexity: small changes in expected duration of disruption can produce large changes in expected supply deficit, inventory draw rates, and implied volatility in oil options.

European natural gas provides a second amplification channel because LNG availability and shipping routes link Asia and Europe through spot competition. A surge exceeding fifty percent in European gas prices after the conflict began to disrupt key flows and to affect perceptions of LNG supply security, with storage levels and refill paths becoming a policy focus. This dynamic creates a macro wedge between the United States and the euro area: U.S. gas prices can move on domestic balances, whereas Europe clears at the margin through LNG cargo competition.

Freight and air logistics form a third channel that carries inflation into tradables beyond energy. The Washington Post described broad disruption to air and sea traffic, with airfreight rates between Asia and Europe rising by about forty-five percent in its reporting, alongside stalled shipping around Hormuz and suspended new shipments by major carriers. A shipping surcharge response reflects the same mechanism: higher fuel costs and routing risk raise the delivered price of intermediate and final goods, with the largest impulse for time-sensitive goods.

Inflation pass-through operates through headline energy prices, second-round wage and margin dynamics, and inflation expectations. Energy-driven inflation blocks rate cuts by the Federal Reserve and implies tighter conditions across Europe. A similar dilemma appears in mainstream reporting that highlights central bank hesitation under an oil-driven inflation impulse. Several analysis ahead of the strikes assessed that a prolonged disruption could push oil toward one hundred dollars and add roughly six to seven tenths of a percentage point to global inflation, illustrating how the energy channel can dominate the inflation path when the shock persists.

Monetary policy, fiscal space, and financial stability channels

The monetary side of the macro regime is shaped by a clash between disinflation progress and an energy shock that lifts headline inflation and shifts risk distributions. The Federal Open Market Committee held the target range for the federal funds rate at 3.5% to 3.75% in its January statement. A patient stance among Fed officials, with emphasis on holding policy steady while monitoring inflation dynamics and the war-driven energy impulse. That combination gives the policy path a more state-contingent structure: energy moves compress the room for easing, while labor-market weakening raises the weight on employment objectives.

In the euro area, the European Central Bank kept key rates unchanged in early February, with a deposit facility rate at 2.00% and related corridor rates as published by the ECB. The euro area energy shock risk differs from the U.S. risk because net import dependence remains meaningful and because the marginal LNG price clears through global competition. That asymmetry can widen growth and inflation dispersion within advanced economies and can shape currency and rate differentials.

Fiscal capacity and debt service costs form an additional constraint layer. The U.S. Congressional Budget Office projected a deficit of $1.9 trillion in fiscal year 2026 and a rising path for federal debt, reaching 120 percent of GDP by 2036 in its baseline outlook. Rising net interest costs represent a large share of that story, with outside analyses drawing from the same baseline highlighting a doubling path for net interest outlays across the next decade. These fiscal conditions matter for geopolitics because large, persistent deficits reduce political room for sustained high-cost expeditionary operations and raise the sensitivity of sovereign financing conditions to inflation shocks.

Private-sector balance sheet stress and credit repricing tighten the feedback loop between rates and growth. U.S. private credit defaults reached a record 9.2% in 2025 in a Fitch dataset, with floating-rate structures linking cash-flow stress to policy rates and to credit spreads. Household stress measures show a parallel dynamics: the Federal Reserve Bank of New York reported aggregate delinquency worsening in late 2025, with 4.8% of outstanding household debt in some stage of delinquency, and a rise in serious delinquency flows for several categories. Consumer balance sheets carry fatigue from prior inflation cycles, which raises the probability that an energy shock translates into demand compression rather than into a clean growth-through-inflation regime. The combination of household stress and private credit defaults suggests a financial system that prices duration risk in the real economy with greater sensitivity to energy-driven inflation.

Markets have begun to price this regime as a volatility state change rather than as a single event shock. The VIX moved above 20 amid the conflict, with attention to shock-absorber trades and hedging demand. Equity selloffs and partial rebounds have reflected a tug-of-war between the energy/inflation channel and the expectation that logistics protection or de-escalation steps might cap the duration of disruption.

Alliance, war, and strategic competition

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Calibration and scoring under uncertainty

This chapter turns an inefficiency claim into a measurable probabilistic object. A model earns credibility when its output remains a truthful conditional description of the future under the same decision-time information that defines the claim. Proper scoring rules provide that bridge. They test whether probabilities, quantiles, or densities align with realized outcomes, and they do so inside the exact state regions where the strategy intends to deploy capital.

What’s inside:

1. Scoring for conditional evidence. The chapter defines prediction as a conditional statement about the future and explains why scoring must reward truthful reporting of the conditional law rather than flattering in-sample behavior.

2. Proper scoring fundamentals. It establishes the role of strict propriety and separates probabilistic description from downstream action, so learning targets the conditional mechanism itself rather than the trade rule built on top of it.

3. Log score and likelihood geometry. Full conditional densities are evaluated through log score, which measures how much probability mass the model assigns to realized outcomes and turns distributional mismatch into an empirical cost.

4. Brier score and probability geometry. Binary event probabilities are scored through squared probability error, which makes calibration visible and tests whether reported event likelihoods match realized frequencies.

5. Pinball loss and quantile geometry. Conditional quantile boundaries are evaluated through asymmetric loss that checks whether reported boundaries retain the correct crossing behavior under the declared state and horizon.

6. Calibration under state selection. The chapter shows why a model can look acceptable on pooled data while failing in the narrow states where capital is actually deployed, so calibration must be checked on the same conditional sample that drives execution.

7. Conditional calibration. Local inefficiency claims require local validation, so probabilities and quantiles must retain their interpretation inside the same state regions where divergence suggests conditional structure.

8. Miscalibration as false inefficiency. The chapter explains how overconfident outputs can manufacture the illusion of edge when threshold-based decision rules amplify extreme signals that lack stable probabilistic meaning.

9. Decision-aware scoring. Evaluation is aligned with the architecture of risk, exposure, and constraints, so errors matter most where the portfolio carries the highest operational sensitivity.

Sample

1.94MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Global macro outlook.

3. Regional breakdown.

4. Major geopolitical flashpoints.

5. Commodity markets and supply chains.

6. Financial markets and capital flows.

7. Policy and regulatory trends.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

Growth can print while the distribution of outcomes widens, because the drivers that move expectations arrive as policy decisions sanction packages, court rulings, or any other events. Markets price that distribution every day through curves, basis, dispersion, and volatility surfaces, often faster than macro releases can confirm the regime shift.

Large refinancing calendars and ongoing portfolio runoff change who absorbs duration and at what price, which means the term premium becomes more active. In that setting, the same growth print can produce different outcomes depending on auction demand, collateral conditions, and the willingness of balance sheets to intermediate. This is why the discussion of central banks sits alongside issuance and liquidity: the reaction function only matters if the market can clear the supply without repricing risk.

Trade policy and regulation matter in the same way: they are rule changes that modify cash-flow boundaries and execution constraints. Tariff schedules alter landed costs, sourcing, and inventory timing. Legal pathways and exemption design shape how durable the tariff environment is, which in turn shapes how firms time capex and how markets price margins. Sanctions and enforcement innovations act through shipping, insurance, classification, and payments, changing the set of compliant routes and the economics of delivery. These are microstructure channels with macro consequences, and they move faster than quarterly accounts. If you trade cross-asset, you already know the pattern: the first clean signal often appears in basis, time spreads, freight, and vol before it reaches earnings or CPI.

But there are much more, indeed, if you want to go deeper don’t miss this:

Gas market report

3.83MB ∙ PDF file

Download

Download

Global Economic Prospects, January 2026

4.77MB ∙ PDF file

Download

Download

Economic bulletin

2.78MB ∙ PDF file

Download

Download

Financial markets and capital flows tie the regime together by showing who finances whom and under what conditions. Fund flows reveal risk appetite and breadth, while cross-border data anchors the foreign bid that matters for rate clearing and dollar dynamics. In 2026, these flow signals interact with policy stability, issuance pressure, and market depth in a way that can flip the sign of traditional relationships.

Subscribe now

Global macro outlook

The World Bank describes resilience amid persistent trade tensions and uncertainty, and it points to a mild step-down in growth as demand for traded goods cools after a tariff-related pull-forward in 2025. The UN’s 2026 outlook reinforces this texture by forecasting subdued global growth relative to the pre-pandemic average and by linking trade tensions and fiscal strains to the outlook. The IMF’s update frames the offsetting forces that keep growth steady, with technology investment and private-sector adaptability acting as counterweights to trade policy headwinds. In that IMF framing, technology-linked capex, including investment tied to artificial intelligence, plays a role in sustaining activity even when goods trade momentum softens.

A services-led expansion favors domestic demand signals, labor income dynamics, and pricing power inside non-tradable segments. A cooling goods cycle shifts the focus toward inventories, freight rates, and margin compression in trade-exposed firms. Policy uncertainty adds a layer of option-like convexity to macro outcomes: tariff announcements, legal rulings, and negotiated carve-outs create discrete jumps in expectations that propagate through FX basis, forward points, and equity dispersion. Reporting on tariff policy illustrates this channel by showing how legal constraints and policy substitutions sustain uncertainty even when headline measures change.

The front-loading mechanism matters because it alters the timing of realized activity. When firms advance imports ahead of higher tariff schedules, measured trade volumes and goods output pull demand from future quarters into the present. World Bank communication around Global Economic Prospects links this pattern to the 2025 trade impulse and expects that tailwind to fade, leaving a softer goods backdrop in 2026. The key step involves separating genuine final-demand strength from calendar-shifted flows. A model that treats the 2025 goods surge as durable can overstate cyclical beta and underweight defensive carry. A model that recognizes the inventory-and-tariff cycle can frame 2026 as a transition from goods momentum toward services endurance, with factor leadership that rotates across value chains and across regions in different phases.

Trade policy shifts influence external balances and currency pressures through re-routing of goods flows. When tariff schedules change, exporters redirect volume toward markets with lower barriers, and importers shift sourcing toward jurisdictions that preserve margin. Reporting on tariff developments highlights a setting where tariff policy remains a durable feature of the landscape, and it points to diversion of Chinese exports toward Europe and other regions as part of the adjustment. ECB analysis of China’s widening goods trade surplus since the pandemic provides a complementary lens on how global goods trade patterns evolve amid geopolitical tension and shifts in trade policy. These forces matter for quant strategies because external balances drive medium-horizon FX drift, while trade shocks drive short-horizon FX volatility, and the interaction between the two shapes carry-to-trend transitions in currency markets.

ECB analysis describes services activity that continues to expand while industry remains muted, using PMI measures as a signal of a recovery that runs through services first. This pattern appears in survey releases that show services readings holding above the expansion threshold while manufacturing moves toward stabilization. PMIs in Germany and France captures cross-country dispersion inside the euro area, with services carrying the composite in Germany and weaker demand weighing on French activity. These survey dynamics matter for trading research because PMI surprises map into short-horizon moves in rates, cyclicals, and currencies, while the level and composition of the PMI can guide medium-horizon positioning in curve steepeners, credit carry, and sector rotation.

PMI data provides a bridge between growth narrative and microstructure. A services-led composite often coincides with sticky wage dynamics, steadier cash-flow expectations in domestic-oriented sectors, and a tighter link between labor-market prints and rate pricing. A manufacturing stabilization phase tends to lift sensitivity to new orders, inventories, and global demand proxies. In systematic research, this implies that the same headline PMI level can support different trade decisions depending on whether services or manufacturing drives the move. It implies that signal design benefits from separating diffusion indexes by sector and by subcomponents such as new orders and employment, then aligning those readings with the asset classes that transmit the information with the least delay. The ECB’s characterization of a dual-speed recovery gives a qualitative anchor for this decomposition.

Central-bank decisions in the opening months of 2026 translate these macro signals into a policy-rate path that markets price each day. The Federal Reserve’s statement kept the target range unchanged and emphasized assessment of incoming data and the balance of risks. The ECB’s decision kept key rates unchanged and pointed to resilience supported by labor-market conditions and the rollout of public spending on defense and infrastructure. The Bank of England’s minutes show an internal split, with a narrow majority favoring an unchanged policy rate and a sizable minority favoring a cut. These decisions describe a global reaction function that prices a wide corridor for cuts, with emphasis on inflation evidence, services pricing, and the durability of disinflation.

Policy communication matters alongside the decision itself because it governs how markets map the same data to different paths for rates. A steady policy rate with language that emphasizes confidence in convergence to target supports carry regimes, where realized volatility remains low relative to the level of yields. A steady policy rate with language that emphasizes uncertainty around inflation drivers supports volatility regimes, where rates option markets price a wider distribution of paths. The ECB’s communication points to medium-term stabilization of inflation at target while describing a challenging global environment that includes policy uncertainty. This combination encourages a research posture that treats rate cuts as a function of inflation composition and of the persistence of growth in services, rather than as a function of headline growth alone.

Inflation dynamics form the hinge between macro and policy caution, and energy sits at the center of that hinge at the start of 2026. ECB analysis attributes the expected decline in headline inflation to base effects linked to energy prices, and separate ECB reporting shows a large negative reading for energy inflation alongside softer underlying inflation measures. Oil markets move in a direction that reduces the disinflation impulse that came from earlier price declines. As crude prices rise and year-over-year declines narrow, which pushes oil’s impact toward inflation pressure and raises the hurdle for policy easing. World Bank commodity data shows a rise in the energy price index driven by moves in natural gas and crude oil, reinforcing the picture of an energy complex that feeds into near-term inflation prints through transport and input costs. IEA market reporting adds a supply-side layer, with a large supply response projected that interacts with geopolitics and weather to drive volatility around the inflation path.

Energy-driven inflation pressure interacts with tariffs through pass-through channels that vary across economies and across consumption baskets. When tariff schedules rise, tradable-goods prices can reflect a combination of higher landed cost and of shifts in sourcing. When energy prices rise, freight and input costs amplify that pass-through. The World Bank’s emphasis on trade policy shifts and uncertainty outlook highlights the macro setting where these channels matter for inflation forecasts and for policy paths. The practical implication involves treating inflation as a composite of distinct drivers rather than as a single series. Energy, goods import prices, and services inflation each respond to different shocks and each maps into different parts of the curve and different equity sectors.

The euro-area inflation channel adds a second force that pulls in the opposite direction through goods prices. The ECB policymaker Fabio Panetta on the role of Chinese imports in the sharper-than-forecast drop in euro area inflation, linking disinflation in exposed goods categories to price and volume dynamics in import competition. An ECB Economic Bulletin focus box provides context by describing the rise in Chinese competitive pressure across export markets and its implications for European firms. This combination produces a cross-category split where goods disinflation can coexist with services stickiness, which keeps the inflation profile two-sided even when headline prints move below target for a time. This split carries direct implications for quant models that use inflation surprises as a single factor, since the market response can differ when the surprise comes from tradable goods versus domestic services.

Energy uncertainty and supply disruptions remain the second half of the euro-area inflation story. ECB communication frames geopolitical tensions as a source of two-sided inflation risks through energy markets, confidence, and business investment. Panetta’s comments point to energy instability and supply-chain disruptions as upside risks alongside the disinflationary pressure from import competition. This configuration supports an inflation process that behaves like a mixture of mean reversion in goods and jump risk in energy-sensitive components. That structure supports stress testing portfolios against energy spikes and supply-chain events, while mapping the influence of import prices on goods-sensitive sectors and on inflation-linked instruments.

Fiscal capacity shapes the medium-horizon risk premium landscape because it governs sovereign issuance, refinancing calendars, and the credibility of backstops. As refinancing needs rise and central bank balance sheets shrink through runoff, the marginal buyer of duration shifts toward price-sensitive private investors, which places emphasis on term premia as a driver of long yields. An ECB speech on safe asset abundance argues that large fiscal deficits and balance-sheet normalization reduce the safety and liquidity premia embedded in government bonds, compressing the convenience yield that investors pay for scarce safe assets and changing the forces that shaped real long-term rates during the prior decade.

This fiscal and balance-sheet backdrop matters for pricing in a way that reaches beyond government bonds. A rise in term premia raises discount rates that feed into equity valuations, increases the hurdle for private credit origination, and shifts the relative appeal of carry strategies versus volatility strategies. It places a premium on liquidity, since price-sensitive investors demand compensation for balance-sheet use during periods of heavy issuance. This argues for explicit monitoring of issuance calendars, auction tails, and measures of market depth in core sovereign futures, because those observables act as the proximate mechanism through which a fiscal narrative becomes a PnL driver. The IMF emphasis on public debt dynamics and the ECB focus on convenience yield provide the macro foundation for building those monitors.

Financial stability adds another layer because valuation and liquidity conditions shape the transmission mechanism from macro shocks into asset prices. The IMF’s October 2025 Global Financial Stability Report frames financial stability risks as elevated amid stretched asset valuations, sovereign bond market pressures, and the rising influence of nonbank financial institutions, and it highlights structural shifts in foreign exchange and emerging market bond markets that reshape both risk and resilience. This matters for systematic strategies that lean on historical diversification patterns, since structural market plumbing determines the speed and shape of drawdowns and the path of margin dynamics. The IMF’s discussion of stock-bond diversification complements this framing by linking bond-market dynamics to the absorption of supply by price-sensitive investors under balance-sheet runoff.

Major geopolitical flashpoints

Read more

Table of contents:

1. Introduction.

2. What is web scraping in a quantitative context?

3. Where does it come from?

4. How web scraping actually works.

5. Risks and limitations of the protocol.

6. Screener network topography.

7. Scraping fund positions.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

In fund research, the input that matters most is simple: what stocks are inside the funds right now, and in what weight. Without that look-through layer, fund momentum, category rotation, or risk exposure becomes a label-driven proxy.

We are going to construct and execute a systematic fund scraping operation. The explicit objective of this architecture is to expose what underlying stocks these otherwise opaque mutual funds and ETFs are holding.

If you intend to calculate true portfolio variance, isolate specific factor exposures, or detect hidden beta overlap across multiple assets, you must possess the exact equity constituents and their precise percentage weightings for every fund in your cross-section. Without ticker-level resolution, you might allocate capital to two distinct Diversified Value funds that both hold massive, highly correlated positions in the exact same mega-cap technology stock.

So, we will engineer a deterministic data pipelineto get a clean, structured JSON. The architecture we are deploying will execute in distinct, logical phases:

1. We will intercept the provider’s APIs to extract the core fund metadata and historical return vectors.

2. We will utilize concurrent, threaded I/O operations to query the global holdings API.

3. We will map this extracted data into memory-contiguous C-arrays.

4. We will apply algorithmic deduplication using tokenized lexical analysis and tuple signatures to eliminate share class multicollinearity.

The final output of this operation will be a sanitized, lightweight matrix of unique portfolios, ready for immediate ingestion into our engine.

What is web scraping in a quantitative context?

To understand the architecture we build for trading, we must first define what web scraping is at a fundamental, computational level. In the broader software engineering industry, web scraping is often defined as the automated gathering of information from the internet. In quantitative finance, we require a technical definition.

Web scraping is the automated, programmatic execution of HTTP network protocols to request, receive, and extract specific data structures from external servers, translating unstructured or semi-structured web payloads into strict, localized, memory-contiguous matrices for immediate algorithmic ingestion.

To know more about scraping check that pdf:

Web Scrapping: The basics explained

236KB ∙ PDF file

Download

Download

When a quantitative researcher builds a cross-sectional momentum strategy or a fundamental mean-reversion model, the model requires historical price vectors, dividend yields, expense ratios, constituent weights, etc. Historically, this data was locked behind expensive, proprietary data terminals operated by a few institutional vendors.

Web scraping bypasses this vendor lock-in. It allows a trading desk to aggregate fragmented public data from asset managers, regulatory filings, and financial portals directly into a local array or a structured SQL database. It is the direct acquisition of the raw material required for alpha generation.

Furthermore, standard vendor data is often commoditized. If every quant receives the exact same fundamental dataset at the exact same millisecond via an identical API feed, the alpha decays. Constructing custom extraction architecture allows a quantitative team to ingest alternative data—such as retail sentiment from localized forums, unstructured XBRL data from SEC 10-Q filings, or dynamically updating ETF composition lists directly from the issuer’s backend.

Where does it come from?

The methodology of data extraction has evolved in parallel with the evolution of web architecture itself. We can trace this lineage through distinct technological eras, driven by the need to parse complex data delivery mechanisms. Understanding this history is critical to understanding why modern quantitative infrastructure rejects certain parsing libraries.

Era 1: Static transfer and regular expressions

In the early days of the internet, web pages were static HTML documents stored directly on server hard drives. When a client sent an HTTP GET request, the server responded with the exact text file. Scraping in this era involved basic command-line tools executing standard network fetches.

The data was extracted using Regular Expressions (Regex). Regex operates as a finite state machine that matches specific string patterns. If a researcher wanted a stock price, they wrote a strictly defined string-matching algorithm to locate a sequence of numbers immediately following a specific HTML text block.

This was efficient computationally. A regex search over a 10-megabyte text file resolves in milliseconds. However, it was incredibly brittle from an operational standpoint. Consider the following HTML string: <td class="price">150.25</td>. A regex designed to capture the float inside the tags (r'class="price">(\d+\.\d+)<') works fine. But if the webmaster updates the website and injects a bold tag for visual styling—<td class="price"><b>150.25</b></td>—the regex logic fails. It returns a null value. In a trading pipeline, a null value propagates through the covariance matrix, zeroes out the optimization weights, and halts the entire trading execution.

Era 2: The Document Object Model and structural parsing

As web pages evolved into generated templates mapping to backend databases, relying on raw string matching became impossible. The industry shifted to structural parsing. Browsers render HTML into a hierarchical tree structure known as the Document Object Model (DOM).

Scrapers evolved to replicate this exact rendering process in memory. Python libraries, such as BeautifulSoup utilizing lxml parsers, were developed to download the HTML string, parse it into an internal tree data structure, and allow engineers to search the tree using XPath or CSS selectors. We stopped searching for text strings and started searching for specific structural nodes, such as the third cell of the second row within a specific table ID.

While more robust than Regex, structural parsing introduces severe computational overhead. Converting a 5-megabyte HTML string into a fully traversable DOM tree requires significant CPU cycles and RAM allocation. The search algorithm must execute O(N) traversal operations across thousands of nested nodes. When running high-frequency data ingestion across thousands of financial instruments, the CPU time spent building these DOM trees severely limits the concurrency capacity of the extraction server.

Era 3: Single Page Applications and JavaScript rendering

The critical inflection point occurred when web development shifted to reactive JavaScript frameworks (React, Angular, Vue). The server no longer returned HTML containing the financial data. Instead, the server returned an empty HTML shell and a massive JavaScript file. The client’s browser was expected to execute the JavaScript, which would then dynamically fetch the data and build the DOM locally on the user’s machine.

Standard HTML parsers broke; they would download the empty shell, parse it, and find zero data. To combat this, scrapers evolved into headless browsers. Infrastructure had to spin up entirely automated, invisible instances of Chromium in the background, load the page, wait for the V8 JavaScript engine to execute the rendering logic, wait for the network requests to resolve, and then extract the data from the rendered DOM.

This was disastrous. Running a single headless browser instance requires hundreds of megabytes of RAM. If a quantitative pipeline requires 100 concurrent workers to update the daily pricing matrix before the market opens, the server must allocate over 50 gigabytes of RAM strictly for browser overhead. This is inefficient and unscalable for certain types of trading systems.

Era 4: API interception

This brings us to the modern quantitative standard. Because Single Page Applications must retrieve their dynamic data from somewhere, they utilize Application Programming Interfaces (APIs)—typically REST or GraphQL architectures.

Instead of rendering a heavy headless browser, we utilize network monitoring tools to observe the traffic generated by the web application. We identify the exact API endpoint the JavaScript is querying, and we replicate that precise HTTP request from our Python architecture. We bypass the HTML entirely. We bypass the JavaScript rendering engine. We request the raw, serialized JSON data directly from the provider’s backend database. This is the optimal state of data extraction: maximum fidelity, minimum computational overhead.

How web scraping actually works

To extract data, you must understand the exact sequence of network events that occur during a programmatic request. Web scraping is the programmatic automation of the standard client-server architecture, executing at speeds unachievable by human interaction.

Step 1: The network handshake and request formulation

The process begins when our Python architecture instantiates a TCP connection. We formulate a strict HTTP request, this request consists of a specific method (GET for retrieving data, POST for submitting query parameters), the target URL, and a dictionary of Headers.

Headers are critical, they dictate the exact rules of engagement with the target server. We set the User-Agent header to declare what type of client we are. By default, Python networking libraries broadcast their standard identity (e.g., python-requests/2.31.0). Modern Web Application Firewalls flag this string and drop the TCP connection. We must inject standard browser headers to bypass basic heuristic checks.

Furthermore, we must configure advanced headers such as Accept-Encoding: gzip, deflate, br to inform the server that we can handle compressed byte streams, reducing network transit time.

Modern security also involves TLS (Transport Layer Security) fingerprinting. Servers analyze the specific cryptographic cipher suites our client proposes during the initial connection setup (the Client Hello packet). Standard Python networking libraries propose a different sequence of ciphers than a standard Google Chrome browser. If the server detects a mismatch between the User-Agent claiming to be Chrome and a TLS fingerprint matching Python, it blocks the request. Advanced quantitative architecture utilizes specialized network adapters to mimic exact browser TLS signatures, guaranteeing connection acceptance.

Before we even introduce anything, let’s start here with our practical example.

import requests

url = "https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener?page=1&pageSize=50&sortOrder=ReturnM120%20desc&outputType=json&version=1&languageId=es-ES&currencyId=EUR&universeIds=FOESP%24%24ALL&securityDataPoints=SecId%7CName%7CPriceCurrency%7CTenforeId%7CLegalName%7CClosePrice%7CYield_M12%7CCategoryName%7CAnalystRatingScale%7CStarRatingM255%7CQuantitativeRating%7CSustainabilityRank%7CReturnD1%7CReturnW1%7CReturnM1%7CReturnM3%7CReturnM6%7CReturnM0%7CReturnM12%7CReturnM36%7CReturnM60%7CReturnM120%7CFeeLevel%7CManagerTenure%7CMaxDeferredLoad%7CInitialPurchase%7CFundTNAV%7CEquityStyleBox%7CBondStyleBox%7CAverageMarketCapital%7CAverageCreditQualityCode%7CEffectiveDuration%7CMorningstarRiskM255%7CAlphaM36%7CBetaM36%7CR2M36%7CStandardDeviationM36%7CSharpeM36%7CTrackRecordExtension&filters=&term=&subUniverseId="

r = requests.get(url, timeout=20)
print("status:", r.status_code)
print("content-type:", r.headers.get("content-type"))
print("content-encoding:", r.headers.get("content-encoding"))
print("head:", r.text[:200])

Step 2: Server-side resolution and response

Upon receiving the formulated request, the target server evaluates it against its internal security protocols. It checks for IP reputation, rate limit thresholds (Token Bucket algorithms), and geographic blocking rules. If the request passes these validation checks, the server queries its internal SQL or NoSQL database, formats the result into the requested format, and transmits the payload back to our server over the open connection.

The server prefaces the payload with an HTTP Status Code. We evaluate this code before reading the data buffer. A 200 OK indicates absolute success. A 403 Forbidden indicates our IP or TLS signature was blocked by the WAF. A 429 Too Many Requests indicates we have exceeded the permissible concurrency limit. When a 429 is encountered, our infrastructure immediately halts execution on that specific thread and initiates a deterministic exponential backoff algorithm, calculating a precise wait time before re-attempting the connection.

status: 200
content-type: application/json
content-encoding: gzip
head: {"total": 49441,"page": 1,"pageSize": 50,"rows": [{"SecId": "F00000S8RT","Name": "Multipartner Konwave Gold Equity C USD","PriceCurrency": "USD","TenforeId": "52.8.LU1001014080","LegalName": "Multipar

Step 3: Payload deserialization and matrix mapping

Once the payload bytes are received locally via the network interface card, they must be transformed into operational data. In modern scraping (API interception), the response header Content-Type is typically application/json. The data arrives as a serialized, UTF-8 encoded byte string.

We execute a high-speed deserialization function. In environments prioritizing microseconds, standard libraries are replaced by C-compiled alternatives like orjson or ujson that map JSON strings to memory structures faster. The deserializer parses the string and maps the key-value pairs into native Python dictionaries and lists.

From this state, we execute our extraction logic. We iterate through the arrays, filtering out the specific financial metrics we require, and allocating them into contiguous memory blocks. We actively cast data types at this stage—ensuring strings representing numbers are cast directly to float64 objects in a structured array, preparing the data for immediate ingestion into linear algebra operations.

To demonstrate the evolutionary mechanics, observe the explicit difference in code execution between legacy DOM parsing and modern high-speed JSON deserialization:

import json
from bs4 import BeautifulSoup
import orjson
import numpy as np

# Legacy Mechanics: Structural DOM Parsing (Era 2) 
# The server returns a heavy string of markup logic mixed with data.
html_payload = """
<div class="financial-table">
    <div class="row">
        <span class="ticker">AAPL</span>
        <span class="price">150.25</span>
    </div>
</div>
"""
# We must allocate significant memory to build the entire tree structure just to find one float.
# This requires CPU cycles to parse the entire string, handle malformed tags, and build node relationships.
tree = BeautifulSoup(html_payload, "html.parser")
legacy_price = float(tree.find("span", class_="price").text)

# Modern Mechanics: API Deserialization (Era 4) 
# The server returns strictly serialized data with zero HTML rendering logic.
json_payload_bytes = b'{"data": [{"ticker": "AAPL", "price": 150.25}]}'

# We execute a direct memory mapping using a C-optimized parser (orjson).
# This bypasses standard Python object creation overhead where possible.
parsed_data = orjson.loads(json_payload_bytes)

# We extract the specific float and immediately allocate it into a NumPy array type.
modern_price = np.float64(parsed_data["data"][0]["price"])

The difference is structural and computational. Legacy scraping reads documents; modern scraping queries databases. By understanding the lineage and the strict network mechanics, we engineer scraping pipelines that don’t simulate human browsing. Instead, they operate as direct, high-speed, database-to-database communication protocols executing over the public internet.

Subscribe now

Risks and limitations of the protocol

The execution of thousands of sequential HTTP requests introduces specific, quantifiable, and systemic risks to the trading infrastructure. These risks are guaranteed events in any large-scale network operation. They manifest as missing data rows, unhandled exceptions, TCP timeouts, and truncated JSON payloads, which directly corrupt the backtesting matrix.

The primary operational risks are categorized by standard HTTP response codes. We anticipate HTTP 429 (Too Many Requests) errors when our extraction loops exceed the provider’s token bucket algorithm limits. We expect TCP connection timeouts when network routing degrades. We will encounter 502 Bad Gateway responses during dynamic load balancer reallocations, and 503 Service Unavailable responses when the backend database cluster is executing routine locking operations or snapshots.

To mitigate these risks, we construct a session adapter using the urllib3 library. We implement a deterministic exponential backoff algorithm. When a 429 or 5xx status code is returned, the system pauses before executing a retry, reducing the concurrency pressure on the provider’s server. We target the 429, 500, 502, 503, and 504 codes. We exclude 400 (Bad Request) and 404 (Not Found) codes, as those represent structural errors in our query parameters that a retry will never resolve.

Here is the implementation of the retry adapter used in our pipeline:

import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def _make_retry_adapter() -> HTTPAdapter:
    retry = Retry(
        total=6,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=("GET",),
        raise_on_status=False,
    )
    return HTTPAdapter(max_retries=retry, pool_connections=50, pool_maxsize=50)

def _lt_session() -> requests.Session:
    # We precisely replicate standard browser headers to bypass basic WAF heuristic checks.
    LT_HEADERS = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "es-ES,es;q=0.9,en;q=0.8",
        "Referer": "[https://lt.morningstar.com/](https://lt.morningstar.com/)",
    }
    s = requests.Session()
    s.headers.update(LT_HEADERS)
    adapter = _make_retry_adapter()
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    return s

Notice the specific performance parameters initialized in the adapter: pool_connections=50 and pool_maxsize=50. This keeps 50 distinct TCP connections open in the background, bound to the Session object. By utilizing the HTTP Keep-Alive header within the requests.Session architecture, we completely bypass the severe latency introduced by repeated Transport Layer Security handshakes.

Every new HTTPS request requires DNS resolution, a 3-way TCP handshake (SYN, SYN-ACK, ACK), and an expensive TLS cryptographic key exchange. This sequence requires 150 to 300 milliseconds of network overhead before a single byte of actual data is transmitted. This is the protocol tax. If we initiate a new session for 5,000 distinct requests, we incur 25 minutes of pure dead time negotiating connections. By pooling connections and persisting them, we reduce network overhead to the strict physical transfer time of the payload itself, dropping the marginal connection time to zero.

To visualize the progression of the exponential backoff risk mitigation, consider the following plot. It models the exact delay duration sequence based on the specified backoff_factor, demonstrating how the system slows down to prevent triggering a permanent IP ban.

This is the precise moment the entire architecture upgrades. We stop looking at the rendered web page, and we start interrogating the network layer.

By opening the developer tools within a controlled Chromium environment and monitoring the Fetch/XHR network traffic generated by the web application during standard user interaction, we discover and intercept the underlying API. We isolate the exact network request executing the database search. We identify the explicit endpoint serving the raw data to the frontend UI:

https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener.

This endpoint doesn’t return complex, nested HTML documents. It accepts defined query parameters that dictate the SQL-equivalent operations on the backend: pagination offsets, sorting directives, universe limiters, and projection columns (securityDataPoints). In return, it provides a raw JSON payload.

By interfacing with this REST API, the extraction process transitions from a heuristic, error-prone text-parsing operation into a deterministic database query. We send a formalized HTTP GET request, and we receive a serialized data structure that maps into our operational memory. A float value is transmitted as a float. A string is transmitted as a string. There is no string-manipulation required to strip percentage signs or currency symbols, which eliminates an entire class of data engineering bugs.

This single event renders all traditional web scraping techniques—BeautifulSoup HTML parsing, XPath selection, and resource-heavy headless browsers like Selenium or Playwright— obsolete within our infrastructure.

Screener network topography

Having established the foundational infrastructure and intercepted the correct REST API endpoint, we now face the challenge of extracting the financial data. The provider’s backend database contains tens of thousands of mutual funds, exchange-traded funds, and institutional share classes. This dataset is vastly larger than the maximum permitted payload size for a single HTTP response.

The endpoint enforces a strict pageSize limit. This is a direct mechanism to prevent memory exhaustion on their internal database cluster. When a client requests data, the provider’s server must execute a SQL or NoSQL query, allocate a continuous block of RAM to hold the result set, serialize that data into a JSON string, and transmit it over the network interface. Requesting 10,000 rows creates a massive serialization overhead. To bypass these hard limits, we define our extraction parameter as pageSize=50.

Extracting the complete cross-sectional universe requires mapping the network topography and iterating over a paginated endpoint recursively. We must calculate the required number of requests based on the total record count returned in the initial response. We can’t hardcode the total page count, as the universe size fluctuates daily due to fund liquidations, mergers, and new product launches.

The algorithmic execution proceeds sequentially for the initialization phase. We fetch page 1. We parse the initial JSON payload to read the total integer field, which dictates the absolute size of the dataset for that specific query. We calculate the ceiling of the division of total by pageSize (math.ceil(total / page_size)). This yields the exact integer value of total pages required to complete the extraction. If the total is 0, we default to 1 to avoid ZeroDivisionError exceptions in downstream processing logic.

Besides, we pass specific query parameters in the HTTP request: universeIds=FOESP$$ALL specifies the precise fund universe routing key on the provider’s backend. We inject the parameter securityDataPoints=SecId|LegalName|CategoryName|ReturnM120 to isolate only the necessary relational columns.

We request the SecId, the LegalName, the CategoryName, and the ReturnM120 (the 120-month historical return, which is the foundational variable for long-term quantitative momentum analysis).

If we omit the securityDataPoints parameter, the API defaults to returning every available column—over 100 data points including tracking error, manager tenure, and daily standard deviation. This inflates the JSON payload from 50 kilobytes to over 5 megabytes per page. Requesting unnecessary data points saturates network bandwidth, increases TLS decryption time, and degrades the JSON parsing speed. We limit the payload to our exact algorithmic requirements.

import math
import numpy as np
import requests

LT_BASE = "[https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener](https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener)"
SECURITY_POINTS = "SecId|LegalName|CategoryName|ReturnM120"

def fetch_lt_page(session: requests.Session, page: int, page_size: int) -> dict:
    params = {
        "page": int(page),
        "pageSize": int(page_size),
        "sortOrder": "ReturnM120 desc",
        "outputType": "json",
        "version": 1,
        "languageId": "es-ES",
        "currencyId": "EUR",
        "universeIds": "FOESP$$ALL",
        "securityDataPoints": SECURITY_POINTS,
        "filters": "",
        "term": "",
        "subUniverseId": "",
    }
    # Notice the exact tuple for the timeout: (4, 20)
    # 4 seconds allocated for the TCP SYN/ACK connection handshake.
    # 20 seconds allocated for the server to process the query and return the first byte.
    r = session.get(LT_BASE, params=params, timeout=(4, 20))
    ct = (r.headers.get("content-type") or "").lower()
    if r.status_code != 200 or "json" not in ct or not r.content:
        raise RuntimeError(f"LT bad response: {r.status_code} ct={ct} head={r.text[:200]!r}")
    return r.json()

def _normalize_pages(pages, total_pages: int):
    # This logic dictates the strict operational scope of the extraction matrix.
    if pages is None:
        # None implies a full universe extraction. WARNING: Huge data pull spanning gigabytes of memory.
        return range(1, total_pages + 1)
    if isinstance(pages, (int, np.integer)):
        # Integer implies pulling the top N pages strictly based on the sortOrder parameter.
        n = int(pages)
        return range(1, min(max(n, 0), total_pages) + 1)
    # List implies pulling specific, explicit page indices, useful for error-handling retry loops.
    # We apply a set comprehension to mathematically eliminate duplicate page requests.
    lst = sorted({int(p) for p in pages if 1 <= int(p) <= total_pages})
    return lst

The pagination handler _normalize_pages is engineered to accept three distinct input states, providing the system with deterministic flexibility. If a researcher is running a lightweight daily update to assess the top decile of momentum performers, they pass an integer pages=10 to scan only the top 500 funds. If the infrastructure team is bootstrapping the initial quantitative database from a blank state, they pass None to fetch the absolute entirety of the universe. If a previous extraction loop dropped specific network packets, the error handler passes a list pages=[14, 22, 105] to specifically target the missing vectors. The use of a set comprehension {int(p) ...} guarantees that duplicate page indices are stripped before execution, preventing redundant network I/O.

We append these scalar values into localized lists. Python lists operate with O(1) amortized append time complexity because they over-allocate memory buffers under the hood. We can’t initialize the final, rigid structured array at this specific stage because we don’t yet know the exact valid row count; we filter out invalid JSON objects—such as funds missing a critical SecId—during the loop.

The primary obstacle at the screener stage is managing the sequential nature of the pagination discovery. Because we don’t possess the total integer until the very first request resolves and the JSON is parsed, the initial discovery phase is linear. However, once the total_pages parameter is acquired, the subsequent page requests become entirely stateless and independent. This sets the theoretical stage for future parallelization. Yet, for this specific screener layer, executing sequential HTTP requests over an established, persistent TLS connection pool is efficient, completing the initial metadata extraction in a matter of a few seconds, preparing the matrix for the much heavier, threaded Global API processing phase.

Scraping fund positions

The screener endpoint provides the fund-level metadata, consisting of basic identifying parameters and historical return vectors. However, a fund’s surface-level metadata is insufficient for risk management and alpha generation. To construct a look-through portfolio, assess true factor exposure, or run Barra risk decomposition models, we require the exact equity holdings inside each fund. We need the underlying constituents, and we need them weighted.

This data requirement introduces a separate, more complex endpoint into our architecture: the Global API (api-global.morningstar.com). Unlike the public screener interface, this endpoint demands an explicit authentication mechanism. We must inject an apikey (it’s free) header into every request.

# 2) Direct holdings 
APIKEY = "HERE YOUR API KEY"

Furthermore, the endpoint requires the specific SecId extracted from the first phase to be formatted directly into the URL path.

The primary architectural obstacle is the sheer volume of network requests. The screener API supports bulk pagination; the Global API doesn’t. If the screener returns an array of 5,000 distinct funds, we must execute 5,000 separate, individual HTTP GET requests to the Global API to retrieve the constituent holdings for each individual fund.

A standard, sequential execution loop operates at approximately 0.5 seconds per request. This delay is a function of the physical network latency—the round-trip time between our trading servers and the provider’s data center, plus the provider’s internal database query execution time. Executing 5,000 requests sequentially requires 2,500 seconds, equating to over 40 minutes of pure execution time. This is unacceptable for a daily production pipeline that must process data, run optimizations, and generate execution orders before the market open.

We resolve this processing bottleneck through concurrent execution. We implement a ThreadPoolExecutor from the concurrent.futures module. Because HTTP network requests are I/O-bound tasks—meaning the CPU sits idle while waiting for the network interface card to receive the inbound packet from the remote server—the GIL doesn’t inhibit parallel execution in this specific scenario. The interpreter releases the GIL during standard I/O blocking operations. Therefore, we can launch 20 concurrent threads to process the requests without migrating to a complex multiprocessing architecture.

We chose thread pooling over an asynchronous event loop (like asyncio) because our existing network infrastructure relies on the requests library and custom urllib3 retry adapters. Threaded execution provides the exact same I/O concurrency benefits without requiring a complete rewrite of the underlying synchronous network stack.

import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
import numpy as np

APIKEY = "HERE YOUR API KEY"
MS_BASE = "[https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2](https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2)"
MS_HEADERS = {
    "apikey": APIKEY,
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
    "Accept": "application/json, text/plain, */*",
}

_thread_local = threading.local()

def _ms_session() -> requests.Session:
    # We must ensure strict thread safety across concurrent network operations.
    s = getattr(_thread_local, "session", None)
    if s is None:
        s = requests.Session()
        s.headers.update(MS_HEADERS)
        # Assuming _make_retry_adapter is defined as in Part 1 for exponential backoff
        # s.mount("https://", _make_retry_adapter())
        _thread_local.session = s
    return s

def fetch_holdings_us_topN(secid: str, country: str = "United States", top: int = 10):
    s = _ms_session()
    url = f"{MS_BASE}/{secid}/data"
    params = {
        "clientId": "MDC",
        "version": "4.71.0",
        "premiumNum": 10000,
        "freeNum": 10000,
    }
    r = s.get(url, params=params, timeout=(4, 25))
    if r.status_code != 200 or not r.content:
        return []
    
    data = r.json()
    page = (data.get("equityHoldingPage") or {})
    hlist = (page.get("holdingList") or [])
    
    out = []
    for h in hlist:
        c = h.get("country") or h.get("countryName")
        if c == country:
            t = h.get("ticker") or h.get("tradingSymbol")
            if t:
                out.append(t)
                if len(out) >= top:
                    break
    return out

def attach_holdings_all(secids, max_workers: int = 20, country: str = "United States", top: int = 10):
    holdings = [None] * len(secids)
    with ThreadPoolExecutor(max_workers=max_workers) as ex:
        futs = {ex.submit(fetch_holdings_us_topN, sid, country, top): i for i, sid in enumerate(secids)}
        for fut in as_completed(futs):
            i = futs[fut]
            try:
                holdings[i] = fut.result()
            except Exception:
                # We enforce strict failure handling to maintain array dimensions.
                holdings[i] = []
    return np.asarray(holdings, dtype=object)

Observe the use of threading.local(). This is an essential architectural detail that prevents systemic failure. We instantiate a unique requests.Session() object for every single active thread. When multiple concurrent connections interact with a single, shared urllib3 connection pool, race conditions occur at the socket level, leading to corrupted TCP streams and dropped connections. By isolating the session object to local thread memory storage, we guarantee that each thread maintains its own independent pool of Keep-Alive HTTP sockets. This ensures deterministic network execution.

We limit max_workers=20 as calculated threshold based on the provider’s implicit rate limits. Exceeding this thread count doesn’t yield linear performance gains; instead, it triggers the provider’s Web Application Firewall, which responds with HTTP 429 status codes, forcing our exponential backoff adapter into extended sleep cycles. Twenty workers provide the optimal equilibrium between maximum extraction speed and minimal server-side rejection.

The next part is to map the extracted data into a NumPy structured array. A structured array allocates a single, contiguous block of memory in C. We define exact byte boundaries for each field in the matrix. When the CPU executes operations on this array, it loads sequential memory blocks directly into the L1/L2 cache, executing instructions with zero pointer-chasing overhead.

We allocate a 16-character unicode string (U16) for the SecId, a 256-character unicode string (U256) for the Fund name, an 8-byte float (f8) for the return metric, and an object pointer (O) for the variable-length list of constituent stock holdings.

# Execution assembly demonstrating the structured array initialization
import numpy as np

# Assuming secids, names, rets, and stocks are already populated lists/arrays from the extraction phase
# We allocate exactly the required memory footprint before population.
arr = np.empty(len(secids), dtype=[("Id", "U16"), ("Fund", "U256"), ("ReturnM120", "f8"), ("Stocks", "O")])
arr["Id"] = np.asarray(secids, dtype="U16")
arr["Fund"] = np.asarray(names, dtype="U256")
arr["ReturnM120"] = rets
arr["Stocks"] = stocks

We acknowledge the use of the object pointer ("O") for the stocks column breaks strict C-contiguity for the holding string arrays themselves. However, this is computationally acceptable because our heavy numerical operations—the sorting and ranking logic—operate on the f8 float array, which remains contiguous and hardware-optimized.

Furthermore, we must handle missing data points at the numerical level. Web scraped data contains holes; some funds lack a 10-year track record. We temporarily replace NaN values with negative infinity (-np.inf) prior to sorting. Standard sorting functions in C don’t possess financial logic. Mathematical comparisons against NaN fail silently (NaN > 5 evaluates to False, NaN < 5 evaluates to False). If we don’t execute this specific substitution, NaN values sort unpredictably, injecting null pointers into the highest momentum deciles and breaking the downstream portfolio construction models. This substitution ensures that funds lacking complete return histories are forced to the absolute bottom of the generated matrix.

def sort_by_return_desc(arr: np.ndarray) -> np.ndarray:
    if arr.size == 0:
        return arr
    r = arr["ReturnM120"]
    
    # Isolate NaN values and push them to the lowest possible finite float boundary (-inf).
    # This prevents incomplete track records from ranking highly in momentum strategies.
    r2 = np.where(np.isfinite(r), r, -np.inf)
    
    # Execute the C-level sort on the substituted float array, and reverse it ([::-1]) for descending order.
    idx = np.argsort(r2)[::-1]
    
    # Return the newly aligned, completely contiguous memory view.
    return arr[idx]

The final, most complex important barrier in systematic fund scraping is the share class multiplicity problem. Asset managers construct a single underlying portfolio of equities, but they distribute that identical portfolio across dozens of distinct administrative wrappers. They do this to target different regulatory jurisdictions, implement varying management fee structures, and separate retail investors from institutional capital. Consequently, you will see Institutional classes, Retail classes, Accumulating (Acc) classes, Distributing (Inc) classes, GBP hedged variants, and EUR unhedged variants—all holding the exact same underlying assets in the exact same proportions.

A standard database unique constraint operating on the SecId is insufficient here because each distinct share class possesses a unique Morningstar SecId. If we include all share classes in a model without aggressive deduplication, we introduce multicollinearity into the output.

We must reduce the dataset to a single representative vector per underlying portfolio. We achieve this by cross-referencing two distinct topographical vectors: the exact list signature of the underlying top holdings, and a strict lexical analysis of the fund’s string name.

First, we define a tokenization function to strip punctuation and structural noise from the fund name strings. We use regular expressions to explicitly remove parentheses and brackets, as share classes are frequently denoted by suffixes like “(USD)” or “[Acc]”. Stripping these allows us to isolate the core lexical identity of the fund. We then define an algorithmic function to calculate the strict length of the common prefix between two tokenized arrays.

def _name_tokens(name: str) -> list[str]:
    s = (name or "").strip().lower()
    # Normalize common typographic variations in fund prospectuses
    s = s.replace("–", "-").replace("—", "-")
    # Strip isolating punctuation typically used for share class designations
    s = re.sub(r"[()\[\],]", " ", s)
    return [t for t in s.split() if t]

def _common_prefix_len(a: list[str], b: list[str]) -> int:
    n = min(len(a), len(b))
    k = 0
    # Calculate exact sequential token matches from index 0
    while k < n and a[k] == b[k]:
        k += 1
    return k

The clustering logic dictates a two-factor rule: two rows are identified as overlapping if and only if their top 10 holding tickers match exactly (creating a deterministic tuple signature) AND their fund names share a strong lexical prefix.

Why both? Relying solely on the holdings signature is dangerous. Two different passive index funds managed by different firms might hold the exact same top 10 mega-cap technology stocks. If we only checked holdings, we would collapse two distinct operational funds into one.

By enforcing the lexical prefix check, we ensure accuracy. If ‘Global Equity Fund Class A USD’ and ‘Global Equity Fund Class B EUR’ possess the exact same stocks, their holding tuple signature matches. Their tokenized prefix ['global', 'equity', 'fund'] also matches based on our defined threshold (e.g., 60% of the token length). Therefore, they are safely clustered together as a single entity.

Once clustered, we invoke a selection algorithm. We iterate over the array indices within the cluster, evaluate the ReturnM120 float vector, and retain the exact index containing the maximum finite return value.

We select for the maximum 120-month return because older, institutional share classes exhibit longer, uninterrupted historical track records and lower fee drags than newer retail classes. Retaining the asset with the longest valid track record is optimal for backtesting; it maximizes our sample size and statistical significance across different historical market regimes.

def _pick_best_index(arr: np.ndarray, idxs: list[int], keep: str = "max_return") -> int:
    if keep == "first":
        return idxs[0]
    
    best = idxs[0]
    best_r = arr["ReturnM120"][best]
    
    for i in idxs[1:]:
        r = arr["ReturnM120"][i]
        r_ok = np.isfinite(r)
        b_ok = np.isfinite(best_r)
        
        # We strictly prioritize finite, measurable returns over NaN or infinite values.
        # If the current best is NaN, any finite return replaces it immediately.
        if (r_ok and not b_ok) or (r_ok and b_ok and r > best_r):
            best = i
            best_r = r
            
    return best

def dedupe_unnecessary_share_classes(arr: np.ndarray, min_prefix_tokens=8, prefix_ratio=0.60, keep="max_return") -> np.ndarray:
    if arr.size == 0:
        return arr
        
    groups: dict[tuple, list[int]] = {}
    unique_empty_keys: list[int] = []
    tokens_cache = [None] * arr.size
    
    for i in range(arr.size):
        tokens_cache[i] = _name_tokens(str(arr["Fund"][i]))
        stocks = arr["Stocks"][i]
        try:
            # Convert the list of tickers into an immutable, hashable tuple for O(1) dictionary lookups
            sig = tuple(stocks) if stocks else tuple()
        except TypeError:
            sig = tuple()
            
        if len(sig) == 0:
            unique_empty_keys.append(i)
        else:
            groups.setdefault(sig, []).append(i)
            
    kept = set(unique_empty_keys)
    
    for sig, idxs in groups.items():
        # Sort by name to guarantee deterministic execution across multiple pipeline runs
        idxs_sorted = sorted(idxs, key=lambda i: arr["Fund"][i])
        clusters: list[list[int]] = []
        reps: list[int] = []
        
        for i in idxs_sorted:
            ti = tokens_cache[i]
            placed = False
            for c_idx, rep_i in enumerate(reps):
                tr = tokens_cache[rep_i]
                cpl = _common_prefix_len(ti, tr)
                
                # Dynamic threshold calculation ensures short names and long names are evaluated fairly
                threshold = max(min_prefix_tokens, int(math.ceil(prefix_ratio * min(len(ti), len(tr)))))
                if cpl >= threshold:
                    clusters[c_idx].append(i)
                    placed = True
                    break
            if not placed:
                reps.append(i)
                clusters.append([i])
                
        for cl in clusters:
            kept.add(_pick_best_index(arr, cl, keep=keep))
            
    kept_idx = np.array(sorted(kept), dtype=int)
    return arr[kept_idx]

So let’s see what we got from this. Time to test!

Oh! This looks good to me!

Screener Extract: 498 funds in 11.567s | Holdings Intercept: 7.711s | Total Architecture Execution: 19.278s | Matrix Dedupe Reduction: 498 -> 346 strict portfolios
[('F00000S8RT', 'Multipartner SICAV - Konwave Gold Equity Fund C USD', 25.92, list(['AU']))
 ('0P0000OMTB', 'Polar Capital Funds PLC - Polar Capital Global Technology Fund I Income EUR', 25.56, list(['NVDA', 'AVGO', 'AMD', 'META', 'LRCX', 'KLAC', 'LITE', 'COHR', 'SNDK', 'STX']))
 ('F0GBR04H0X', 'Multipartner SICAV - Konwave Gold Equity Fund B USD', 25.38, list(['AU']))
 ('F00000VI1X', 'Bakersteel Global Funds SICAV - Precious Metals Fund D EUR', 24.62, list(['NEM', 'AU', 'CDE']))
 ('F00001S8MC', 'CT (Lux) - Global Technology Class ZE (EUR Accumulation Shares)', 23.5 , list(['BE', 'LRCX', 'NVDA', 'AVGO', 'WDC', 'GOOGL', 'AMAT', 'MSFT', 'MRVL', 'AAPL']))]

Ticker    Count   PctFunds
--------------------------
NVDA        146      59.11
MSFT        132      53.44
AAPL        119      48.18
AVGO        113      45.75
GOOGL       110      44.53
AMZN        110      44.53
META        104      42.11
GOOG         68      27.53
NEM          44      17.81
LLY          40      16.19
MA           36      14.57
LRCX         33      13.36
MU           33      13.36
CRM          29      11.74
TSLA         28      11.34
AMAT         23       9.31
V            22       8.91
CDE          20       8.10
AMD          18       7.29
MRVL         18       7.29
FSLR         18       7.29
DC           16       6.48
PWR          16       6.48
RGLD         15       6.07
NFLX         15       6.07

Okay! No surprises here! This output matrix is formatted, computationally lightweight, and ready to be serialized to a Parquet file or whatelse. The architecture is resolved and ready for daily production scheduling.

Alright crew—good session today. Until the next article! And remember, may liquidity is with you! 💵​

PS: Help me to know what you think about the newsletter, give me feedback and rate it!

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full script:

"""
Morningstar fund screener (LT) + direct holdings (global API) + de-dup share classes
- No pandas in your code (NumPy structured array output)
- pages:
    * None  -> all pages (WARNING: huge)
    * int   -> 1..pages
    * list  -> explicit pages
- De-dup logic:
    * Only de-dups when TOP holdings signature matches AND fund names share a strong common prefix.
    * Keeps the class with max ReturnM120 by default.
"""

import math
import time
import re
import numpy as np
import requests
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# 1) LT Screener
LT_BASE = "https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener"

LT_HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "es-ES,es;q=0.9,en;q=0.8",
    "Referer": "https://lt.morningstar.com/",
}

SECURITY_POINTS = "SecId|LegalName|CategoryName|ReturnM120"


def _make_retry_adapter() -> HTTPAdapter:
    retry = Retry(
        total=6,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=("GET",),
        raise_on_status=False,
    )
    return HTTPAdapter(max_retries=retry, pool_connections=50, pool_maxsize=50)


def _lt_session() -> requests.Session:
    s = requests.Session()
    s.headers.update(LT_HEADERS)
    adapter = _make_retry_adapter()
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    return s


def fetch_lt_page(session: requests.Session, page: int, page_size: int) -> dict:
    params = {
        "page": int(page),
        "pageSize": int(page_size),
        "sortOrder": "ReturnM120 desc",
        "outputType": "json",
        "version": 1,
        "languageId": "es-ES",
        "currencyId": "EUR",
        "universeIds": "FOESP$$ALL",
        "securityDataPoints": SECURITY_POINTS,
        "filters": "",
        "term": "",
        "subUniverseId": "",
    }
    r = session.get(LT_BASE, params=params, timeout=(4, 20))
    ct = (r.headers.get("content-type") or "").lower()
    if r.status_code != 200 or "json" not in ct or not r.content:
        raise RuntimeError(f"LT bad response: {r.status_code} ct={ct} head={r.text[:200]!r}")
    return r.json()


def _normalize_pages(pages, total_pages: int):
    if pages is None:
        return range(1, total_pages + 1)
    if isinstance(pages, (int, np.integer)):
        n = int(pages)
        return range(1, min(max(n, 0), total_pages) + 1)
    lst = sorted({int(p) for p in pages if 1 <= int(p) <= total_pages})
    return lst


def scrape_lt_screener(pages=None, page_size: int = 50, category_contains: str = "RV", max_funds: int | None = None):
    secids, names, rets = [], [], []

    with _lt_session() as s:
        first = fetch_lt_page(s, 1, page_size)
        total = int(first.get("total", 0))
        total_pages = int(math.ceil(total / page_size)) if total > 0 else 1

        count = 0
        for pg in _normalize_pages(pages, total_pages):
            data = first if pg == 1 else fetch_lt_page(s, pg, page_size)
            for row in (data.get("rows") or []):
                cat = str(row.get("CategoryName") or "")
                if category_contains in cat:
                    sid = row.get("SecId")
                    if not sid:
                        continue
                    secids.append(sid)
                    names.append(row.get("LegalName") or "")
                    v = row.get("ReturnM120")
                    try:
                        rets.append(float(v) if v is not None else np.nan)
                    except Exception:
                        rets.append(np.nan)

                    count += 1
                    if max_funds is not None and count >= int(max_funds):
                        break
            if max_funds is not None and count >= int(max_funds):
                break

    return secids, names, np.asarray(rets, dtype="f8")

# 2) Direct holdings 
APIKEY = "HERE YOUR API KEY"

MS_BASE = "https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2"
MS_HEADERS = {
    "apikey": APIKEY,
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
}

_thread_local = threading.local()


def _ms_session() -> requests.Session:
    s = getattr(_thread_local, "session", None)
    if s is None:
        s = requests.Session()
        s.headers.update(MS_HEADERS)
        adapter = _make_retry_adapter()
        s.mount("https://", adapter)
        s.mount("http://", adapter)
        _thread_local.session = s
    return s


def fetch_holdings_us_topN(secid: str, country: str = "United States", top: int = 10):
    s = _ms_session()
    url = f"{MS_BASE}/{secid}/data"
    params = {
        "clientId": "MDC",
        "version": "4.71.0",
        "premiumNum": 10000,
        "freeNum": 10000,
    }
    r = s.get(url, params=params, timeout=(4, 25))
    if r.status_code != 200 or not r.content:
        return []

    data = r.json()
    page = (data.get("equityHoldingPage") or {})
    hlist = (page.get("holdingList") or [])

    out = []
    for h in hlist:
        c = h.get("country") or h.get("countryName")
        if c == country:
            t = h.get("ticker") or h.get("tradingSymbol")
            if t:
                out.append(t)
                if len(out) >= top:
                    break
    return out


def attach_holdings_all(secids, max_workers: int = 20, country: str = "United States", top: int = 10):
    holdings = [None] * len(secids)
    with ThreadPoolExecutor(max_workers=max_workers) as ex:
        futs = {ex.submit(fetch_holdings_us_topN, sid, country, top): i for i, sid in enumerate(secids)}
        for fut in as_completed(futs):
            i = futs[fut]
            try:
                holdings[i] = fut.result()
            except Exception:
                holdings[i] = []
    return np.asarray(holdings, dtype=object)

# 3) De-dup share classes (fixes your repeated funds)
#    Rule: same holdings signature AND strong common name prefix.
def _name_tokens(name: str) -> list[str]:
    s = (name or "").strip().lower()
    s = s.replace("–", "-").replace("—", "-")
    s = re.sub(r"[()\[\],]", " ", s)
    return [t for t in s.split() if t]


def _common_prefix_len(a: list[str], b: list[str]) -> int:
    n = min(len(a), len(b))
    k = 0
    while k < n and a[k] == b[k]:
        k += 1
    return k


def _pick_best_index(arr: np.ndarray, idxs: list[int], keep: str = "max_return") -> int:
    if keep == "first":
        return idxs[0]
    # keep == "max_return" (nan-safe)
    best = idxs[0]
    best_r = arr["ReturnM120"][best]
    for i in idxs[1:]:
        r = arr["ReturnM120"][i]
        r_ok = np.isfinite(r)
        b_ok = np.isfinite(best_r)
        if (r_ok and not b_ok) or (r_ok and b_ok and r > best_r):
            best = i
            best_r = r
    return best


def dedupe_unnecessary_share_classes(
    arr: np.ndarray,
    min_prefix_tokens: int = 8,
    prefix_ratio: float = 0.60,
    keep: str = "max_return",
) -> np.ndarray:
    """
    Dedup only when:
      - Stocks tuple matches exactly (same signature)
      - AND fund names share a long common prefix (token-based)

    This fixes cases like:
      '... Fund I Income GBP' vs '... Fund R Income'
      '... Fund B USD' vs '... Fund B EUR'
    """
    if arr.size == 0:
        return arr

    # Group by holdings signature; empty holdings => never dedup (keep all)
    groups: dict[tuple, list[int]] = {}
    unique_empty_keys: list[int] = []

    tokens_cache = [None] * arr.size

    for i in range(arr.size):
        toks = _name_tokens(str(arr["Fund"][i]))
        tokens_cache[i] = toks

        stocks = arr["Stocks"][i]
        try:
            sig = tuple(stocks) if stocks else tuple()
        except TypeError:
            sig = tuple()

        if len(sig) == 0:
            unique_empty_keys.append(i)
        else:
            groups.setdefault(sig, []).append(i)

    kept = set(unique_empty_keys)

    # For each signature group, cluster by name-prefix similarity
    for sig, idxs in groups.items():
        # sort by name (stable) to make clustering deterministic
        idxs_sorted = sorted(idxs, key=lambda i: arr["Fund"][i])

        clusters: list[list[int]] = []
        reps: list[int] = []  # representative index per cluster

        for i in idxs_sorted:
            ti = tokens_cache[i]
            placed = False

            for c_idx, rep_i in enumerate(reps):
                tr = tokens_cache[rep_i]
                cpl = _common_prefix_len(ti, tr)

                threshold = max(min_prefix_tokens, int(math.ceil(prefix_ratio * min(len(ti), len(tr)))))
                if cpl >= threshold:
                    clusters[c_idx].append(i)
                    placed = True
                    break

            if not placed:
                reps.append(i)
                clusters.append([i])

        # keep one per cluster
        for cl in clusters:
            best = _pick_best_index(arr, cl, keep=keep)
            kept.add(best)

    kept_idx = np.array(sorted(kept), dtype=int)
    return arr[kept_idx]


def sort_by_return_desc(arr: np.ndarray) -> np.ndarray:
    if arr.size == 0:
        return arr
    r = arr["ReturnM120"]
    r2 = np.where(np.isfinite(r), r, -np.inf)
    idx = np.argsort(r2)[::-1]
    return arr[idx]

# 4) Runner
def run(
    pages=10,
    page_size: int = 50,
    max_workers: int = 20,
    category_contains: str = "RV",
    holdings_country: str = "United States",
    holdings_top: int = 10,
    max_funds: int | None = None,  # recommended if pages=None
    dedupe: bool = True,
):
    t0 = time.perf_counter()

    secids, names, rets = scrape_lt_screener(
        pages=pages,
        page_size=page_size,
        category_contains=category_contains,
        max_funds=max_funds,
    )
    t1 = time.perf_counter()

    stocks = attach_holdings_all(secids, max_workers=max_workers, country=holdings_country, top=holdings_top)
    t2 = time.perf_counter()

    arr = np.empty(len(secids), dtype=[("Id", "U16"), ("Fund", "U256"), ("ReturnM120", "f8"), ("Stocks", "O")])
    arr["Id"] = np.asarray(secids, dtype="U16")
    arr["Fund"] = np.asarray(names, dtype="U256")
    arr["ReturnM120"] = rets
    arr["Stocks"] = stocks

    arr = sort_by_return_desc(arr)

    before = arr.size
    if dedupe:
        arr = dedupe_unnecessary_share_classes(
            arr,
            min_prefix_tokens=8,   # if you still see duplicates, increase to 10–12
            prefix_ratio=0.60,
            keep="max_return",
        )
        arr = sort_by_return_desc(arr)
    after = arr.size

    print(
        f"Screener: {len(secids)} funds in {t1 - t0:.3f}s | "
        f"Holdings in {t2 - t1:.3f}s | Total {t2 - t0:.3f}s | "
        f"Dedupe: {before}->{after}"
    )
    return arr


import numpy as np
import matplotlib.pyplot as plt
from collections import Counter

def topk_stock_table_and_plot(arr: np.ndarray, topk: int = 20, *, title: str = "Top holdings frequency"):
    """
    Counts how often each ticker appears in arr["Stocks"] (per-fund top holdings list),
    returns a NumPy table with (Ticker, Count, PctFunds) and shows a barplot in BLACK + CYAN.

    Percentage = Count / (# funds with non-empty Stocks) * 100
    """
    if arr.size == 0:
        out = np.empty(0, dtype=[("Ticker", "U16"), ("Count", "i4"), ("PctFunds", "f8")])
        return out

    counter = Counter()
    funds_with_holdings = 0

    for stocks in arr["Stocks"]:
        if not stocks:
            continue
        funds_with_holdings += 1
        # count presence within a fund once (avoid double-counting duplicates inside the list)
        seen = set()
        for t in stocks:
            if t is None:
                continue
            t = str(t).strip().upper()
            if not t:
                continue
            seen.add(t)
        counter.update(seen)

    if funds_with_holdings == 0 or not counter:
        out = np.empty(0, dtype=[("Ticker", "U16"), ("Count", "i4"), ("PctFunds", "f8")])
        return out

    items = counter.most_common(topk)
    tickers = [k for k, _ in items]
    counts = np.array([v for _, v in items], dtype=np.int32)
    pct = counts / float(funds_with_holdings) * 100.0

    table = np.empty(len(tickers), dtype=[("Ticker", "U16"), ("Count", "i4"), ("PctFunds", "f8")])
    table["Ticker"] = np.array(tickers, dtype="U16")
    table["Count"] = counts
    table["PctFunds"] = pct

    fig, ax = plt.subplots(figsize=(10, 6))
    fig.patch.set_facecolor("black")
    ax.set_facecolor("black")

    y = np.arange(len(tickers))
    ax.barh(y, pct, color="cyan", edgecolor="cyan")

    ax.set_yticks(y)
    ax.set_yticklabels(tickers)
    ax.invert_yaxis()

    ax.set_xlabel(f"% of funds (N={funds_with_holdings})")
    ax.set_title(title)

    ax.tick_params(colors="cyan")
    ax.xaxis.label.set_color("cyan")
    ax.yaxis.label.set_color("cyan")
    ax.title.set_color("cyan")
    for spine in ax.spines.values():
        spine.set_color("cyan")
    ax.grid(True, axis="x", linestyle="--", linewidth=0.6, color="cyan", alpha=0.35)

    plt.tight_layout()
    plt.show()

    return table


def print_topk_table(table: np.ndarray, decimals: int = 2):
    """Pretty-prints the output table (Ticker, Count, PctFunds)."""
    if table.size == 0:
        print("Empty table.")
        return
    w_t = max(6, int(max(len(t) for t in table["Ticker"])))
    header = f"{'Ticker':<{w_t}}  {'Count':>7}  {'PctFunds':>9}"
    print(header)
    print("-" * len(header))
    for row in table:
        print(f"{row['Ticker']:<{w_t}}  {int(row['Count']):>7}  {row['PctFunds']:>9.{decimals}f}")

if __name__ == "__main__":
    matrix = run(pages=10, max_workers=20)  # pages=None --> all pages (very large)
    print(matrix[:5])

    table = topk_stock_table_and_plot(matrix, topk=25, title="Most-held stocks (top holdings lists)")
    print_topk_table(table)

Table of contents:

1. Introduction.

2. The United States coercive diplomacy.

3. Iran as a cost imposer.

4. Nuclear thresholds and monitoring.

5. Israel, Gaza, and the legitimacy environment.

6. Saudi Arabia and Syria as pivots.

7. Ukraine load, Caucasus exposure, Middle East spillover.

8. Hormuz and Red Sea.

9. Taiwan pressure and U.S. allocation constraints

10. Sudan and multilateral bandwidth competition

11. Europe risk multiplier

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

Regional crises function as interconnected nodes within a broader architecture of geopolitical, economic, and military friction. From the contested maritime chokepoints of the Middle East to the industrial load bearing down on Europe and the persistent readiness burdens in the Indo-Pacific, the international system is experiencing simultaneous, compounded stress.

The expiration of the New START treaty in February 2026 serves as a structural inflection point for this new era. The loss of top-level arms control norms trickles down into regional theaters, replacing established escalation management with reciprocal fear dynamics. This vacuum elevates the importance of physical, measurable guardrails. Nuclear verification has transitioned from a technical non-proliferation tool into the central currency for de-escalation. The ability to monitor and verify intent—whether tracking Iranian breakout timelines or securing the fragile safety parameters at Ukraine’s Zaporizhzhya facility—now dictates the risk premiums applied by global financial markets and the preventive strike incentives of regional powers.

Within this volatile environment, statecraft has fundamentally shifted toward continuous force design and structural cost imposition. The dynamics analyzed in this report illustrate how major actors manipulate risk:

• The United States employs a visible escalation ladder, leveraging maritime dominance and advanced defense systems to enforce declarative realism, deter regional hegemons, and reassure exposed partners.

• Actors like Iran utilize dispersed proxy networks, drone swarms, and missile architectures to saturate defenses and impose systemic economic costs, altering global freight spreads and insurance premiums in critical corridors like the Red Sea and the Strait of Hormuz.

• The sheer volume of global flashpoints generates competition for diplomatic and military bandwidth. Legitimacy pressures emanating from the Gaza theater strain coalition durability, while severe humanitarian crises drain the diplomatic resources required for effective sanctions enforcement and crisis mediation.

• Geopolitical conflict now operates as financial architecture: Sanctions, export controls, and payment rails are weaponized, making the resilience of domestic industrial bases and the protection of strategic supply chains key.

As the geopolitical center of gravity shifts, regional pivots like Saudi Arabia and Syria determine the dynamics of these conflicts, influencing everything from global energy spare capacity to the friction of proxy routing across the Levant. Meanwhile, the persistent pressure around Taiwan forces a continuous recalibration of U.S. asset allocation, demonstrating that a surge in one theater degrades deterrence perceptions in another.

Subscribe now

The United States coercive diplomacy

The United States current application of coercive diplomacy reflects a transition toward force design as a primary tool for crisis management. Recent military movements in the Middle East illustrate this shift, where the deployment of the USS Abraham Lincoln carrier strike group and advanced missile defense systems like THAAD serves as a deliberate signal of readiness for high-intensity operations. This posture functions through a ladder logic where every deployment and public statement represents a specific rung on an escalatory scale. Officials monitor these thresholds to ensure that adversaries perceive a unified and unwavering message. The strategy relies on strict message to prevent the ambiguity that arises from distributed agency among local commanders and regional maritime actors.

Force design serves three critical functions within this philosophy. First, it establishes visible capability to extract concessions from Tehran regarding its nuclear enrichment and ballistic missile programs. Second, it provides physical assurance to regional partners who face the risk of direct exposure during heightened tensions. Third, it shapes domestic leadership perceptions by demonstrating capacity within an accelerating crisis loop. This internal-to-external cycle acts as a structural engine: domestic political pressure necessitates swift action, which leads to increased military visibility abroad. The outcomes of these crises then flow back into the domestic sphere, influencing legislative cycles and public media narratives.

A deeper structural layer guides these institutional choices beyond individual political identities. The maintenance of global hegemony and the stability of the economic regime create persistent incentives that survive changing administrations. This systemic position requires the United States to prioritize maritime dominance and the protection of strategic chokepoints like the Strait of Hormuz. The Maritime Action Plan reinforces this by seeking to revitalize domestic shipbuilding and imposing fees on foreign-built vessels to fund a Maritime Security Trust Fund. This focus on industrial capacity and robotic autonomous systems ensures that the American military remains the dominant force in contested waters.

Check the document here:

Restoring Americas Maritime Dominance

981KB ∙ PDF file

Download

Download

The 2026 National Defense Strategy formalizes this shift by replacing integrated deterrence with a clear hierarchy of priorities. This strategy explicitly identifies the defense of the American homeland and the containment of regional hegemons as the primary missions. By moving away from democratic idealism and open-ended nation-building, the current approach emphasizes declarative realism. This involves setting firm red lines and making the cost of aggression infeasible through denial. The use of force has become more prominent, as seen in recent operations in the Western Hemisphere and the intensified pressure on Iranian leadership. These actions suggest that diplomacy now serves as a prelude to or a partner of military coercion rather than a standalone alternative.

Iran as a cost imposer

The strategic posture of Iran emphasizes its role as a cost imposer through an integration of military dispersion and diplomatic signaling. This approach utilizes an expansive geography of exposure where missile and drone units maintain a mobile presence. These forces create a wide planning surface for any defensive entity because launch platforms distribute across vast terrains and reconstitute after operations. The sheer volume of these assets aims to saturate point defenses, which forces an opponent to coordinate responses across a multitude of nodes and decision authorities. This increase in coordination demand leads to inherent friction in execution and expands the window for adversary opportunity.

The leverage held by Tehran manifests through three synchronized layers of influence. Direct strike capabilities place regional bases and essential infrastructure under constant risk of engagement. Parallel to this, proxy depth extends the potential map of conflict across Iraq, Syria, Lebanon, and critical maritime zones. Narrative framing serves as the third layer, as Iranian officials craft messages centered on legitimacy, sovereignty, and proportionality to justify their defensive stance. These elements work together to manage escalation, where proxy incidents provide a means of signaling intent while maintaining a level of flexible attribution. In the maritime domain, these actions impose systemic economic costs by triggering a repricing of insurance and widening freight spreads.

Recent diplomatic activity reinforces this deterrent logic through formal communications to the United Nations. These messages frame potential retaliation as a defensive response tied to the protection of regional facilities and assets. This use of legal language seeks to establish international legitimacy while providing enough clarity to raise the overall effect of deterrence. By defining the scope of their claims, Iranian leaders constrain the space for miscalculation among partners and adversaries.

Information regarding indirect negotiations through the Omani Foreign Ministry highlights a parallel track of engagement. While military drills influence the pricing channel of global commodities, the cadence of these consultations suggests a deliberate attempt to balance pressure with dialogue. The uncertainty surrounding the safety of transit corridors carries convex effects on market demand and insurance terms. This dual-track strategy of maritime signaling and diplomatic mediation ensures that every tactical movement translates into a tangible economic or political cost for the international community.

Nuclear thresholds and monitoring

Nuclear verification functions as the essential medium for modern de-escalation because the transformation of political intent into measurable data provides the only reliable baseline for regional stability. This process replaces abstract estimates of breakout timelines with concrete measurements, thereby altering the incentive structure for preventive military action. The current bargaining environment centers on the specific modalities of access, including the physical sampling of materials, the continuous presence of international inspectors, and the robust protection of monitoring hardware against physical or cyber interference.

Developments this month illustrate the vulnerability of this verification infrastructure within active conflict zones. The International Atomic Energy Agency has drawn specific attention to the Zaporizhzhya Nuclear Power Plant, where operational safety relies on a single remaining external power line. This precarious situation demonstrates that monitoring capabilities remain tethered to local security conditions, which in turn depend on the shifting incentives of combatants. Any disruption to these essential links triggers a surge in uncertainty and fuels intense credibility contests between opposing parties. The resulting feedback loop shows that perceived threats to monitoring equipment directly influence the probability of kinetic strikes, as reduced visibility encourages more aggressive defensive postures.

Markets now interpret the continuity of nuclear monitoring as a leading indicator for escalation risks. Financial actors price the stability of these verification regimes rather than treating conflict as a binary outcome. The preservation of inspector access bounds the potential paths toward military confrontation, as it provides a cooling mechanism for political tensions. This economic dimension reinforces the status of verification as a central currency in international relations, where the transparency of a nuclear program serves as a prerequisite for broader trade and investment stability.

A secondary node of verification significance has emerged through the expansion of civil nuclear cooperation in the Gulf. Recent discussions regarding Saudi Arabian nuclear arrangements have introduced new variables concerning regional safeguards and enrichment constraints. This wave of civil nuclear development shifts the traditional balance of leverage and forces a recalibration of alliance commitments. The integration of advanced monitoring norms into these new programs serves as a stabilizing force, ensuring that the proliferation of technology aligns with established security equilibria. The pursuit of these verification standards remains the primary tool for managing the complex relationship between energy independence and regional strategic balance.

Israel, Gaza, and the legitimacy environment

Legitimacy pressure within the Gaza dictates the durability of international coalitions and the scope of regional policy. Data from February 2026 indicates that humanitarian reporting functions as a variable that shifts domestic opinion and legal forum activity in partner capitals. The United Nations human rights system confirms that displacement and demographic changes across Gaza and the West Bank create sustained pressure on diplomatic consent for basing and sanctions enforcement. The first fifteen months of the conflict, report mortality figures that exceed earlier counts. These findings increase litigation risk and impact coalition cohesion by altering the political cost of alignment.

The connection between the Gaza conflict and the broader regional landscape remains a central feature of current strategic planning. Narratives centered on Gaza mobilize support for maritime action and proxy activation, drawing energy from the humanitarian situation to shape the Iran file. This dynamic creates a chain where combat outcomes generate data on civilian harm, which then drives activity within international legal forums. Such legal developments shift domestic opinion, which in turn defines the space available for coalition partners to maintain sanctions and offer logistics support.

The International Committee of the Red Cross issued statements emphasizing obligations under international law regarding detainee and civilian protection. These communications affect the risk tolerance of partners for participation in kinetic operations. Regional actors monitor these signals to calibrate their level of cooperation with external powers. The relationship between humanitarian metrics and strategic leverage ensures that the legitimacy of military presence depends on the management of civilian protection norms. Coalition space remains tied to the perception of adherence to these standards, as seen in the current diplomatic discourse surrounding base access and crisis bargaining.

Saudi Arabia and Syria as pivots

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Local predictability via divergence

This chapter treats an inefficiency claim as a statement about distributional separation. Fix an outcome variable R and a horizon at which R is measured. The future becomes the probability law of R at decision time. The object of interest is the gap between the baseline future and the state-conditioned future, where X is a declared state variable. A divergence measures this gap by quantifying how much the predictive law of R changes when we move from “no state” to “state x,” and it keeps the research claim separate from the execution layer that turns structure into PnL.

What’s inside:

1. What divergence is for in trading research. Divergence serves as a dependence detector that lives in conditional laws, so the claim stays tied to how the future distribution changes across states.

2. Scope and limits of divergence. The chapter draws a boundary between divergence evidence and realized profit, and it frames the inefficiency claim inside a declared horizon, state region, and environment class.

3. Local divergence maps. The workflow treats R and X as design objects, then compares P(R|X≈x) to P(R) across state domains to build a field of separations over the state space.

4. Local inefficiency domains. A local inefficiency corresponds to a state domain that supports a stable statement across repeated visits, with an activation domain and a transition zone where the conditional law returns toward the baseline mixture.

5. Tail divergences. The chapter introduces tail-focused separation to target structure that lives inside extremes, where distributional shape changes matter most for constraint-relevant outcomes.

6. Divergence stability under perturbation. Evidence is checked under perturbations of the measurement lens (domain rules, estimation choices) and recomputed on held-out path segments to verify persistence.

7. Predictability has geometry. The chapter treats predictability as a geometric object with locality, boundaries, and transitions, so the research claim becomes measurable along the market’s movement through state regions.

Sample

2MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Asia-Pacific and strategic tensions.

3. U.S.-China relations.

4. U.S.-Russia dynamics.

5. Middle east energy risk.

6. Latin America’s alignments.

7. Debt and monetary shifts.

8. Portfolio strategy updates for Q1 2026.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

Markets don’t announce their turning points with volatility. More often, they drift—calm on the surface: alliances harden, supply routes get repriced, reserve assets get questioned, and policy tools lose degrees of freedom under debt. Early 2026 feels like one of those regimes. Risk isn’t higher in the headline sense; it’s more convex: fewer obvious warnings, but larger consequences when specific nodes break.

Think about that: a trade decision can move FX expectations; FX stress can alter reserve behavior; reserve behavior can reshape bond term premia; bond premia can tighten financial conditions; and tight conditions can force policy choices that echo back into geopolitics. The goal is to identify the handful of pressure points most likely to surprise portfolios in Q1 2026.

The structure follows a simple logic: start where momentum is strongest and mispricing is most tempting (Asia-Pacific), move through the two central strategic relationships that set the global risk budget (U.S.–China and U.S.–Russia), then into the corridor where energy volatility still has the cleanest transmission into inflation and rates (the Middle East). From there, it shifts to Latin America as the proving ground for spheres-of-influence dynamics, and finally to the macro plumbing—debt, reserves, and monetary credibility—where long-cycle risks are accumulating even in normal markets.

In regimes like this, the mistake is to look for a single catalyst. The real driver is coupling: more assets, policies, and institutions are linked through the same handful of constraints—financing, energy, security, and settlement. When coupling rises, diversification can fail quietly.

So the right question is less what happens next? and more where is the system least tolerant to surprise? Look for asymmetries: places where a small change forces a large adjustment; where hedges are expensive because everyone wants them; where policy is boxed in by debt; where supply is one incident away from being repriced; where credibility matters more than guidance. Early 2026 is full of those asymmetries.

Subscribe now

Asia-Pacific and strategic tensions

In Japan, Prime Minister Sanae Takaichi’s ruling party clinched a landslide snap-election victory in January, securing a supermajority in the lower house. This historic mandate has paved the way for expansive fiscal stimulus (including promised tax cuts and AI/semiconductor investments) while pledging responsible budgeting to avoid exacerbating Japan’s heavy debt load. Markets responded with enthusiasm: the Nikkei 225 surged 3.9% to an all-time record high above 56,000, reflecting investor confidence in a stable, pro-growth government. Japanese equities rallied across exporters, cyclicals, financials, and even defense stocks as part of a Japan is Back narrative.

Super-long Japanese bond yields spiked on fears of massive new issuance, but leveled off as Takaichi emphasized fiscal prudence – calming bond markets wary of Japan’s debt (the highest in the developed world). The Bank of Japan remains on intervention watch: the yen initially weakened to multi-decade lows, then rebounded after officials warned of potential yen-buying intervention to curb excess volatility. Overall, Japan’s political reflationary policy mix have turned it from a contrarian play into a reform-driven growth story, improving regional investor sentiment.

Asia-Pacific markets have benefited from both regional developments and a rotation out of overvalued U.S. tech stocks. The MSCI Asia Pacific Index jumped about 2.5% to reach an all-time high in early February. Alongside Japan’s rally, South Korea’s Kospi – seen as a proxy for AI-related investments – surged over 4%. Additionally, a surprise conservative election win in Thailand helped anchor optimism, strengthening the Thai baht. The region’s outperformers have been bolstered by comparatively lower equity valuations and stronger growth prospects than U.S. markets. Capital flows into Asia are accelerating as global investors fade uncertainty – shifting funds from pricey U.S. technology shares into Asian equities and other undervalued opportunities. Bitcoin and other risk assets have also rebounded alongside Asian stocks, reflecting improved risk appetite as immediate volatility subsides.

Despite a veneer of market calm, the Asia-Pacific region faces underlying strategic tensions. Taiwan remains a central flashpoint in U.S.-China relations. In a January phone call, President Trump and President Xi Jinping acknowledged the importance of a good relationship, but Xi pointedly warned that Taiwan is the most important issue and urged Washington to handle arms sales to Taiwan with prudence. Any misstep on Taiwan could reignite regional volatility. Additionally, the U.S.-China trade and tech rivalry continues to simmer. Beijing has been assertively pursuing self-reliance in semiconductors and AI, while Washington maintains tariffs and export controls. In fact, uncertainty around trade policy briefly rattled markets when President Trump announced new tariffs on South Korean imports in late January – a move seen as a protectionist signal that could indirectly pressure China. So far, however, Asia-Pacific stability has held, aided by diplomatic engagement (e.g. Trump planning to visit China in April, per official statements) and by the region’s internal economic momentum.

Asia-Pacific’s outlook appears favorable in the current climate. With Japan’s pro-growth policies and political stability, Japanese equities have become attractive – in sectors like industrials, finance, and consumer staples which stand to benefit from domestic stimulus and a stronger yen. Broader Asian emerging markets may see increased inflows as investors diversify away from U.S. tech; select Asian equity funds or ETFs could outperform given lower valuations. However, any escalation around Taiwan or in U.S.-China frictions could reverse Asia’s rally. Keeping an eye on upcoming policy decisions (e.g. Bank of Japan shifts or trade negotiations) is crucial.

U.S.-China relations

A noteworthy development in early 2026 is China’s move to shield itself from U.S. financial risks, which has significant geopolitical undertones. In February, Chinese regulators urged banks to curb US Treasuries exposure, cautioning lenders to limit purchases of U.S. government bonds and gradually reduce existing holdings. This directive, confirmed by people familiar with the matter, was framed as a risk-management measure against concentration and market volatility – not a geopolitical maneuver.

Even so, markets reacted and U.S. Treasury yields ticked higher (10-year yield up ~3 basis points to ~4.24%) as prices fell, the U.S. dollar slipped on the news, and U.S. equity futures pared gains. Strategists note this signal from one of the world’s largest U.S. creditors is a challenge to the dollar, especially as it comes on the heels of concerns over Washington’s fiscal policy. Indeed, global investors are questioning U.S. governance: Trump’s commitment to a strong dollar and the Federal Reserve’s independence have come under doubt.

China’s cautious diversification away from U.S. debt aligns with a broader de-dollarization trend, wherein central banks (including China’s) are raising gold reserves and settling more trade in yuan. This trend has helped drive gold to record highs above $5,000. For now, Beijing’s Treasury curbs have been modest and state entities (like China’s central bank) reportedly remain major holders of U.S. debt. However, the move telegraphs a long-term intent to reduce reliance on U.S. assets, potentially as a hedge against future U.S. sanctions or financial conflict. U.S. policymakers are taking note: the Achilles heel of U.S. fiscal health has always been its external financing needs, and China’s actions underscore that vulnerability.

The Trump Administration’s return has been marked by a revival of hardline trade tactics, not only toward China but even traditional allies. In addition to new tariffs on South Korean goods, the U.S. continues to enforce high tariffs on Chinese imports (a legacy of the 2018–19 trade war) and restrict China’s access to advanced semiconductor technology. China, for its part, has invested in tech and AI capabilities, but the global AI race has introduced new complexities. One of the biggest market vulnerabilities stemming from AI is hitting U.S. software companies – a development that also impacts Chinese tech indirectly.

In January, Wall Street saw a selloff dubbed software-mageddon: U.S. software and data service stocks plunged in value as investors feared that fast-advancing AI tools (like Anthropic’s Claude 4.6) could disrupt these firms’ business models. The S&P 500 Software & Services index dropped over 15% in January, erasing about $1 trillion in market cap since its peak, as hedge funds slashed exposure and short interest spiked. Even industry giants were not spared – e.g. Microsoft fell ~5% in one day. A Goldman Sachs strategist warned that while near-term earnings might hold up, they are insufficient to disprove the long-term downside risk from AI. This mass repricing reflects an existential worry: could generative AI automate away large swathes of software jobs and services? Data vendors saw sharp declines on fears that AI-driven platforms could replicate their offerings. Some stability returned by early February – analysts noted the sector had reached deep oversold conditions reminiscent of 2022, and value buyers emerged. But the episode highlights a key point: the AI arms race is disruptive, and not only to the U.S. tech sector. China’s tech industry faces similar AI-related disruption risks (and also U.S. export controls on chips), meaning this is a shared vulnerability even amid U.S.-China competition.

Major U.S. tech firms plan ~$630–650 billion in 2026 capital expenditures – to build AI data centers and infrastructure – a ~36% jump that is squeezing profits and alarming investors. This unprecedented AI arms race investment by Amazon, Alphabet, Meta, and others is shifting these companies from asset-light models to asset-intensive ones.

Despite economic tensions, Washington and Beijing have maintained cautious diplomatic engagement in early 2026. High-level communication lines are open – exemplified by the Trump-Xi phone call in January, where both leaders agreed on the importance of a stable relationship and discussed issues ranging from Iran to trade in U.S. soybeans and energy. This call resulted in plans for President Trump to visit China in April, signaling a willingness to manage differences through dialogue.

Nonetheless, core strategic disagreements persist. China’s latest Policy Paper on Latin America and the Caribbean (published December 2025) criticized U.S. hegemonism and pledged support for nations opposing unilateral bullying – rhetoric aimed at U.S. influence. Concurrently, the new U.S. National Security Strategy labels China a non-hemispheric competitor in the Americas. These documents underscore a widening ideological rift: China champions multipolarity and globalization in the right direction, whereas the U.S. is doubling down on its sphere-of-influence doctrine (as seen vividly in Latin America, discussed in a later section). In the Indo-Pacific, military posturing continues. The U.S. Navy maintains an increased presence in the South China Sea and Taiwan Strait, while China regularly conducts military drills around Taiwan. Both sides are wary of accidents or miscalculation – a Chatham House analysis warns that Trump’s initially ambiguous stance on Taiwan raised the risk of an accidental conflict, and calls for clearer U.S. policy to avoid escalation. So far, no direct crisis has erupted in 2026, but this is an ever-present tail risk.

If you want to know more about the US-National Defense Strategy check:

2026 US-National Defense Strategy

6.75MB ∙ PDF file

Download

Download

Heightened U.S.-China tensions argue for caution in sectors exposed to trade or regulatory crossfire. For instance, U.S. semiconductor and cloud computing firms face both export restrictions and rising competition from Chinese alternatives – any portfolio should limit over-concentration in these names. On the other hand, sectors like industrial machinery, infrastructure, and non-tech consumer goods in the U.S. and allied markets could be relative winners as governments channel investment into supply-chain resiliency (e.g. building chip plants domestically) and as consumers pivot spending patterns. On the Chinese side, equities continue to carry geopolitical risk discounts. Only risk-tolerant investors should consider partial exposure (for example, via broad emerging Asia funds) – and even then, allocations should be small and balanced against Western assets.

If de-dollarization efforts accelerate (e.g. more countries using yuan for trade), the dollar may face gradual depreciation pressure. Investors might diversify some currency exposure – holding a portion of assets in foreign currency funds or gold – as a hedge against any erosion in dollar dominance. Overall, portfolios should stay agile: U.S.-China developments can shift quickly with policy announcements or geopolitical events, so periodic rebalancing (quarterly or around key summits) is advisable.

U.S.-Russia dynamics

Read more

Table of contents:

1. Introduction.

2. The role of ETL in the design of trading systems.

3. ETL failure modes and risks.

4. Technical preliminaries.

5. Designing an ETL.

6. Layers, contracts and canonical schema.

7. ETL vs ELT for trading.

8. What breaks ETL in trading systems.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

We enter the part of the quant stack that drives everything: data. We treat data as the machinery that tells a trading system what the market is—minute by minute, venue by venue, revision by revision.

This new serie introduces a sequence of articles on infrastructure—ETLs, scraping, APIs—written from the quant-dev perspective: trading systems need repeatable market states. Research needs to recreate yesterday. Live trading needs to ingest today. Both need to refer to the same definitions.

A familiar pattern shows up in almost every desk, prop shop, or solo setup. Data starts as a handful of scripts, then grows into a maze of just this once fixes: a quick endpoint here, a patched column there, a notebook cell that becomes a dependency because it worked during a good backtest. The system still runs. Results still appear. But the pipeline becomes a moving object—shifting definitions, shifting assumptions, shifting outputs—until the only thing that remains stable is confusion.

That is why ETL deserves a dedicated article. The reason? A pipeline accumulates decisions: what gets stored, what gets normalized, what gets ignored, what gets re-requested, what gets overwritten, what gets trusted… Those decisions compound, and at some point they dominate the behavior of the entire stack. The strategy stops being the bottleneck. The data becomes the bottleneck.

Scraping and APIs deserve equal attention because they are how most modern trading pipelines actually exist in the wild. They impose constraints, quirks, and failure patterns: response truncation, pagination inconsistencies, silent field changes, intermittent outages, caching surprises, and vendor helpfulness.

The goal across this series is to make data infrastructure feel less like maintenance and more like a controlled instrument. A good pipeline behaves like a well-specified system: deterministic when it should be deterministic, conservative when it must be conservative, and transparent when something deviates.

Check this paper to know more about ETLs:

Data extraction, Transformation, and Loading process automation

299KB ∙ PDF file

Download

Download

So the promise of this article is concrete. By the end, it becomes natural to answer questions that otherwise stay vague: Where did this feature come from? Which definition did it use? When did it change? Can the dataset be recreated byte-for-byte? What happens when the vendor revises history? How does the system fail when the feed degrades? Those are operational questions, but they decide whether research can be trusted and whether live behavior can be diagnosed.

The role of ETL in the design of trading systems

Three components are typically treated as core in any trading system: strategy logic, execution, and risk. The ETL process—Extract, Transform, Load—is often categorized as a secondary support function. However, this categorization understates the structural role ETL plays in defining the parameters of the trading environment. It is the ETL process that establishes timestamp semantics and interval conventions, just as it defines the mechanics of volume aggregation across venues, determines the handling of missing prints or stale quotes, and establishes the boundaries of the trading day, including the definition of market close during partial sessions or under vendor-specific conventions.

A weak data pipeline permits a model to stay numerically consistent even when it lacks economic coherence. This failure manifests as an epistemic mismatch. The model trains on information unavailable at the time of execution or on data representations that drift between the training environment and the live feed. Backtests appear profitable but lack causal validity. Live trading degradation follows as a predictable consequence of this architecture.

Scale makes ETL automation a key pillar. But it is a challenge that requires automating the whole process. That means, new failures. From a quant dev perspective, this presents a tension between three competing criteria: velocity, correctness, and observability. Velocity is necessary because research iteration and feature publication are time-sensitive. Correctness is critical because a single semantic mismatch can introduce systematic lookahead bias without any explicit error in the code. Observability is essential because, without reproducibility and data lineage, it is impossible to distinguish between a failure of strategy logic and a failure of data construction. Once models are trained on warehouse-ready features, the ETL pipeline becomes an implicit component of the model itself.

Subscribe now

ETL failure modes and risks

ETL failure modes exist across domains, but in trading they are amplified because signals are marginal and backtests are sensitive to subtle causal violations. It is useful to name the principal failure modes, as a map from data defects to PnL leakage.

1. Temporal integrity risk is the most frequent source of silent contamination. Timezone normalization errors—exchange time, broker server time, and UTC being conflated—create misalignment that can masquerade as alpha. Bar timestamp conventions are often ambiguous: some providers label bars by start time, others by end time, and others by a vendor-defined label that is neither. DST discontinuities can duplicate or remove hourly timestamps. APIs frequently return date fields with implicit timezones. Resampling without explicit session calendars can manufacture bars during market closures. None of these defects need an explicit future reference to introduce lookahead, misalignment is sufficient.

2. Market microstructure risk emerges when sampling and aggregation rules are treated as interchangeable. For example, volume is venue-specific; cross-venue volume is not additive unless definitions match. VWAP computed from incomplete trade prints is not VWAP. OHLC bars from heterogeneous vendors can be numerically similar while being non-equivalent, which means return distributions can look stable while microstructure features drift. Mixing bid/ask and last-trade series destroys interpretability of returns, spread-sensitive features, and execution assumptions. Even quote filtering—removing stale quotes or enforcing spread constraints—changes the effective sampling distribution and can alter feature–target dependence.

3. Firm actions and instrument definition risk is where many useless backtests are born. Splits, dividends, symbol changes, roll schedules for futures, and specification changes in tick size or lot size all require explicit treatment. A toxic pattern is using adjusted close alongside raw open/high/low, which creates a synthetic object with inconsistent scale.

4. Data leakage risk is the highest-impact failure mode because it converts structural error into apparent predictive power. Vendor revisions—late prints, corrected OHLC—can enter historical features without an as-of policy. Using the close price for signals intended to be decided at open changes the filtration. Filling missing values using future data is leakage by construction. End-of-day fundamentals used for intraday signals without controlling for publication time inject future information. Even innocuous preprocessing, such as z-scoring using full-sample statistics, can leak information when deployed online.

5. Heterogeneity risk is both schema and semantics. NaN encodings differ across sources (null, zero, empty string). Identical field names can have different meaning, as when volume is shares, contracts, or notional. Data types differ and can change silently (float versus decimal, integer timestamps versus ISO strings). Precision and rounding differences can alter microstructure-sensitive computations and create spurious stability.

6. Operational risk appears when automation is treated as job finished equals success. Cron jobs can succeed while returning partial data. API rate limits can truncate responses. Network errors can create gaps, retries can create duplicates. Incremental loads can fail to be idempotent. Failures often cluster on holidays, early closes, vendor maintenance windows, and schedule drift that causes overlapping extraction windows.

Heavy transformations performed repeatedly inside modelling loops change the feasible hypothesis class. Recomputing features that should be materialized wastes iteration budget and induces inconsistent definitions when code evolves. Unbounded joins across large fact tables can make the research loop unstable under scaling.

Technical preliminaries

Trading doesn’t tolerate imprecise definitions because PnL is sensitive to small causal misalignments. Let P_t denote a price process under an explicit convention (bid, ask, mid, last). Let V_t denote an associated volume process with explicit units (shares, contracts, notional). Let I_t denote the sigma-algebra of information available up to time t. A trading signal S_t must be I_t-measurable.

An ETL pipeline can be modeled as a composition of maps,

where ε extracts from raw sources into an event stream, T transforms into conformed representations, and L loads into storage layers (OLTP and DW).

• Online transaction processing (OLTP) is a type of database system used in transaction-oriented applications, such as many operational systems. Online refers to the fact that such systems are expected to respond to user requests and process them in real-time (process transactions). The term is contrasted with online analytical processing (OLAP) which instead focuses on data analysis (for example planning and management systems).

• Data warehouse (DW), is a system used for reporting and data analysis and is a core component of goberning. Data warehouses are central repositories of data integrated from disparate sources.

The critical property is causal admissibility. For any feature F_t used at decision time t, F_t must be measurable with respect to I_t. ETL violates this property through mechanisms that look operationally reasonable: aggregations that use end-of-window values while decisions occur at window start; imputation via centered filters; backfills that overwrite previously published features without version control; and “helpful” resampling rules that use future ticks to fill an OHLC bar.

This is why causal admissibility must be enforced by automated tests. If the system can’t prove the property, the property doesn’t hold operationally.

To remove ambiguity, define three timelines. Observation time t^(e) (event-time) is when the market produced the observable. Processing time t^(p) is when the system ingested it. Decision time t^(d) is when the strategy commits to an action.

A bar at index k has an interval [t_k,start , t_k,end). If a strategy acts at bar open, then t_k^(d) . Features must not depend on events with t^(e)≥t_k,start.

Before diving into more complex staff, understanding where ETL pipelines typically struggle helps prioritize improvement efforts. Here’s a breakdown of common bottleneck areas:

Designing an ETL

Every strategy carries implicit assumptions about what constitutes a day, how the open is defined under multi-session trading, which clock governs decisions, what constitutes a trade, for example for a VWAP aggregation, and when information enters I_t (publication timestamps, vendor revisions, and the latency of ingestion itself). ETL either formalizes these assumptions or smuggles them.

The baseline automation flow is simple and sensible: collect data on a schedule, store it in OLTP, transform and load it into a DW, then use the DW for trading workflows. That pattern improves throughput. The difference from a traditional ETL lies in representing the pipeline as a reproducible computational object with explicit contracts so that performance does not amplify error.

Bounded complexity matters because a system that can’t be described will not remain stable. Verifiable correctness matters because a property that can’t be checked automatically is not a property of the system.

ETL design can be posed as constrained optimization. Let θ parametrize pipeline configuration (batch size, partitioning, retry policy, scheduling, feature versions). Let D(θ) denote the dataset emitted for modelling. We care about generalization performance G(D(θ)), latency L(θ), cost C(θ) (compute, storage, API calls), and data quality risk R(θ) (probability-weighted impact of faults). A compact objective can be written as

subject to causal admissibility, idempotence, and lineage. The key conceptual shift is that ETL is an optimization problem.

A useful operational shift is to treat the pipeline as a parameterized program with a semantic version v. The output dataset is then D_v(θ). If either v or θ changes, you must assume the output is a different dataset, even if schemas appear identical. This rule prevents a large fraction of unreproducible model results.

Scheduling ETL jobs reduces contention with operational systems. However, smart scheduling goes beyond simple timing:

Layers, contracts and canonical schema

A robust architecture is best described as layers governed by explicit contracts. At the perimeter, source connectors interact with APIs, brokers, file drops, and vendor feeds. Their contract is to emit raw events with source metadata, explicit time conventions, and enough information to reproduce the request.

Raw ingestion into OLTP should be append-heavy and minimally transformative. Its contract is to preserve raw information, enforce uniqueness, and preserve provenance. Canonicalization then converts heterogeneous source representations into a canonical schema by enforcing time normalization, type normalization, and symbol mapping. The DW layer provides partitioned fact tables and dimensions (assets, calendars, sources) with stable semantics optimized for analytical access.

Above the DW, a feature store publishes versioned feature vectors and tensors consumed by research and live engines. The feature store is the interface that enforces immutability by version and train/live parity. Finally, model inputs and backtest feeds are derived from the feature store as arrays, parquet datasets, or tight queries.

A canonical schema must minimize semantic ambiguity. At minimum, an asset dimension fixes identifiers, venues, tick sizes, and calendars. A calendar dimension fixes session definitions, holidays, and early closes. A source dimension fixes provider identity, endpoint sampling rules, time conventions, and revision policy. Facts separate raw events from derived bars and derived bars from derived features. This separation prevents circularity and leakage.

Establishing baseline metrics is essential before implementing optimization strategies. Track these KPIs to measure improvement:

A critical design choice is to represent bars with explicit start/end timestamps. Ambiguous labels are a recurring source of leakage.

CREATE TABLE IF NOT EXISTS fact_bar (
  asset_id           BIGINT NOT NULL,
  ts_start_utc       TIMESTAMPTZ NOT NULL,
  ts_end_utc         TIMESTAMPTZ NOT NULL,
  open               DOUBLE PRECISION,
  high               DOUBLE PRECISION,
  low                DOUBLE PRECISION,
  close              DOUBLE PRECISION,
  volume             DOUBLE PRECISION,
  source_id          TEXT NOT NULL,
  ingest_ts_utc      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  quality_flags      INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (asset_id, ts_start_utc, source_id)
);

CREATE INDEX IF NOT EXISTS idx_fact_bar_end
  ON fact_bar(asset_id, ts_end_utc);

This schema forces a commitment to interval semantics and prevents a broad class of timestamp-induced lookahead errors.

CREATE TABLE IF NOT EXISTS feature_vector (
  asset_id            BIGINT NOT NULL,
  ts_utc              TIMESTAMPTZ NOT NULL,
  feature_set_version TEXT NOT NULL,
  f1                  DOUBLE PRECISION,
  f2                  DOUBLE PRECISION,
  f3                  DOUBLE PRECISION,
  quality_flags       INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY(asset_id, ts_utc, feature_set_version)
);

ETL vs ELT for trading

The distinction between ETL and ELT is often discussed in any development team, as if one pattern is better than the other. In trading, that framing is unhelpful because the decision is about which transformations are definitional—therefore part of the market representation—and which transformations are computational—therefore a choice of execution venue. A trading dataset is not data until its semantics are pinned down, and semantics are what many transformations decide.

The first class of transformations is semantic. These transformations determine what your system claims to observe and when. Time conventions are semantic because they define the filtration. If the vendor labels a bar by end time but you interpret it as start time, you have built a causal violation into your feature history. Calendars are semantic because they define what a day means (and therefore what daily VWAP, overnight gap, and close-to-open return mean). Instrument mapping is semantic because it defines what the identifier refers to under symbol changes, roll schedules, corporate actions, and contract specification updates. Definitional conformance is semantic because it defines what a field means: whether volume is shares, contracts, or notional; whether a price is last, mid, bid, or ask; whether OHLC is constructed from trades or quotes; and whether adjusted prices are being mixed with unadjusted prices. These transformations belong before the warehouse-facing analytical layer in the sense that they must be controlled, testable, and versioned as part of the dataset definition. Once you accept that, the ETL/ELT question becomes secondary: the definitional layer must exist, regardless of where the subsequent compute occurs.

The second class of transformations is computational. These transformations take an already-defined representation and compute derived quantities that don’t change the meaning of the underlying observables. Rolling sums, rolling standard deviations, exponentially weighted averages, joins across conformed facts, feature aggregation, and windowed statistics are computational. They can be done in the warehouse (ELT), in an external feature job (ETL in the broad sense), or in a feature store materialization step. If a computational feature is used across many models and many experiments, materializing it can be economically rational because it stabilizes definitions and reduces iteration latency.

A practical way to decide is to ask one question: does this transformation change the information boundary of the dataset? If it does, it must be treated as part of the dataset specification and therefore requires explicit contracts, deterministic behavior, and versioning. If it doesn’t, then the transformation is a compute-placement decision driven by throughput and operational constraints. This avoids a common failure mode where teams ELT everything.

For example, if we continue with the VWAP example, the decision to reset it at the boundary of an exchange trading day is semantic because it depends on the session calendar and therefore on a definition of day. The computation of VWAP itself—cumulative sums of price times volume divided by cumulative volume—is computational once the day boundary has been fixed. Similarly, defining what constitutes an overnight return is semantic because it depends on which close and which open are used (and whether the asset trades continuously). Computing the return itself is computational.

This is also why trading pipelines benefit from an explicit semantic conformance layer even when the team prefers ELT. In practice, this layer may live as a set of controlled SQL models, a dbt project, a set of stored procedures, or a Python conformance job that writes canonical facts—like the one below. The key is that semantic logic is centralized, versioned, and tested.

# SQLite 
SQLITE_FACT_BAR_DDL = """
CREATE TABLE IF NOT EXISTS fact_bar (
  asset_id      INTEGER NOT NULL,
  ts_start_utc  TEXT NOT NULL,
  ts_end_utc    TEXT NOT NULL,
  open          REAL,
  high          REAL,
  low           REAL,
  close         REAL,
  volume        REAL,
  source_id     TEXT NOT NULL,
  ingest_ts_utc TEXT NOT NULL,
  quality_flags INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (asset_id, ts_start_utc, source_id)
);
CREATE INDEX IF NOT EXISTS idx_fact_bar_end ON fact_bar(asset_id, ts_end_utc);
"""

SQLITE_FEATURE_VECTOR_DDL = """
CREATE TABLE IF NOT EXISTS feature_vector (
  asset_id            INTEGER NOT NULL,
  ts_utc              TEXT NOT NULL,
  feature_set_version TEXT NOT NULL,
  f1                  REAL,
  f2                  REAL,
  f3                  REAL,
  quality_flags       INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY(asset_id, ts_utc, feature_set_version)
);
"""

SQLITE_UPSERT_BAR_SQL = """
INSERT INTO fact_bar(asset_id, ts_start_utc, ts_end_utc, open, high, low, close, volume, source_id, ingest_ts_utc, quality_flags)
VALUES (?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(asset_id, ts_start_utc, source_id)
DO UPDATE SET
  ts_end_utc    = excluded.ts_end_utc,
  open          = excluded.open,
  high          = excluded.high,
  low           = excluded.low,
  close         = excluded.close,
  volume        = excluded.volume,
  ingest_ts_utc = excluded.ingest_ts_utc,
  quality_flags = excluded.quality_flags;
"""

SQLITE_UPSERT_FEATURE_SQL = """
INSERT INTO feature_vector(asset_id, ts_utc, feature_set_version, f1, f2, f3, quality_flags)
VALUES (?,?,?,?,?,?,?)
ON CONFLICT(asset_id, ts_utc, feature_set_version)
DO UPDATE SET
  f1 = excluded.f1,
  f2 = excluded.f2,
  f3 = excluded.f3,
  quality_flags = excluded.quality_flags;
"""

A concise implementation pattern is to keep raw ingestion append-only and as close to source as possible, then apply a conformance step that pins down semantics, then perform computational features either in-warehouse or as a materialized feature job. In other words, a pragmatic hybrid is common: extract and load raw into OLTP with minimal mutation, transform into conformed DW tables, and materialize features into a versioned feature store. This preserves provenance while enabling analytical performance.

The ETL/ELT choice also interacts with latency and reliability. If you need intraday features with tight publication deadlines, pushing heavy joins and window functions into a shared warehouse can create contention and unpredictable runtimes. Conversely, if you compute everything outside the warehouse, you may replicate logic across scripts and drift into semantic inconsistency. A stable compromise is to push stable, high-fanout computations into materialized tables or feature store partitions, while keeping fragile semantics confined to a small, well-tested conformance layer.

To illustrate how semantic logic can remain explicit even in an ELT-heavy design, consider a canonical bar view that makes interval semantics and day identifiers visible to downstream consumers. The key idea is that the view should encode the definition, not only reshape columns.

CREATE OR REPLACE VIEW v_bar_canonical AS
SELECT
  asset_id,
  ts_start_utc,
  ts_end_utc,
  open, high, low, close,
  volume,
  source_id,
  date_trunc('day', ts_start_utc AT TIME ZONE 'UTC')::date AS day_id_utc,
  quality_flags
FROM fact_bar
WHERE ts_end_utc > ts_start_utc;

Once the semantics are fixed in canonical facts or views, computational features can be computed where it is most efficient. If you want the warehouse to own feature computation, you can materialize a feature table keyed by (asset_id, ts_utc, feature_set_version) and treat feature_set_version as a contract.

INSERT INTO feature_vector(asset_id, ts_utc, feature_set_version, f1, f2, f3, quality_flags)
SELECT
  asset_id,
  ts_end_utc AS ts_utc,
  '1.0.0' AS feature_set_version,
  ln(close) - ln(lag(close) OVER w) AS f1,
  close - (sum(close * volume) OVER wd) / NULLIF(sum(volume) OVER wd, 0) AS f2,
  stddev_samp(ln(close) - ln(lag(close) OVER w)) OVER w20 AS f3,
  max(quality_flags) OVER w AS quality_flags
FROM v_bar_canonical
WINDOW
  w  AS (PARTITION BY asset_id ORDER BY ts_end_utc),
  w20 AS (PARTITION BY asset_id ORDER BY ts_end_utc ROWS BETWEEN 19 PRECEDING AND CURRENT ROW),
  wd AS (PARTITION BY asset_id, day_id_utc ORDER BY ts_end_utc);

The goal of these snippets is to show that the definitional layer can remain explicit, inspectable, and testable even when feature computation is executed inside the warehouse. If, instead, you prefer external feature jobs, the same contract still applies.

What breaks ETL in trading systems

Heterogeneity and conformance are the first obstacles because they are structural. A conformance layer must normalize identifiers, time bases, session calendars, field semantics, and types. To make this enforceable, treat conformance as a mapping between typed domains. A source-level record belongs to a source domain X_s, and conformance maps it to a canonical domain X^* via Φ_s: X_s →X^*. If Φ_s changes, you have changed the dataset, even if downstream schemas remain identical.

Missing data is structural. For backtesting, NaN propagation is often preferable to imputation because it preserves ignorance. For live systems, “don’t crash” constraints exist, but fallbacks must be conservative: don’t invent OHLC; carry forward last observed values only for risk checks, not alpha computation; set quality flags; and disable signal generation for assets that violate completeness constraints. A useful principle is that a live fallback should minimize the probability of trading on corrupted information, even at the cost of missed trades.

API limits and latency budgets impose feasibility constraints. If the provider caps calls per minute and each call has a nontrivial wall time, there is a hard bound on how much data can be collected and published per day. If those bounds are violated, no amount of clever modelling will compensate; the design must change via batching, parallelization under caps, endpoint scope reduction, or provider selection.

Performance pathologies arise when feature computation is with conformance and aggregation. Recomputing features inside model selection loops expands iteration time and changes which hypotheses you can afford to test. Materialization reduces repeated computation, but it also increases the importance of versioning. A pipeline must separate semantic conformance, structural aggregation, computational feature construction, and statistical modelling.

Schema drift includes semantic drift: fields retain names while changing meaning. Without a schema/semantic registry, and without semantic versioning of transformations, a pipeline can mutate what the data means. Security and access control are baseline requirements because trading pipelines store proprietary features and operational metadata.

The following plan is minimalist and is designed as an example. In the appendix, I’ve included another, a little bit more detailed.

Extraction requires deterministic request generation, rate-limit aware scheduling, retries with backoff, and optional raw-response archiving for forensic analysis.

import time
import json
import random
from urllib.request import urlopen, Request

class RateLimiter:
    def __init__(self, max_per_minute: int):
        self.max_per_minute = int(max_per_minute)
        self.min_interval = 60.0 / float(self.max_per_minute)
        self._t_last = 0.0

    def wait(self):
        dt = time.time() - self._t_last
        if dt < self.min_interval:
            time.sleep(self.min_interval - dt)
        self._t_last = time.time()


def fetch_json(url: str, limiter: RateLimiter, max_retries: int = 5) -> dict:
    for attempt in range(max_retries):
        try:
            limiter.wait()
            req = Request(url, headers={"User-Agent": "quant-etl/1.0"})
            with urlopen(req, timeout=20) as r:
                return json.loads(r.read().decode("utf-8"))
        except Exception:
            sleep_s = (2 ** attempt) + random.random()
            time.sleep(sleep_s)
    raise RuntimeError(f"Failed to fetch after {max_retries} retries: {url}")

Transformation should be expressed as pure functions: normalize timestamps into UTC with explicit interval semantics, normalize numeric types, and attach quality metadata.

from datetime import datetime, timezone, timedelta

def to_utc(ts_str: str) -> datetime:
    dt = datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
    return dt.astimezone(timezone.utc)


def conform_bar(raw: dict, asset_id: int, source_id: str, bar_minutes: int) -> Bar:
    ts_end = to_utc(raw["date"])  # interpret explicitly as bar end
    ts_start = ts_end - timedelta(minutes=int(bar_minutes))

    open_  = float(raw.get("open", "nan"))
    high_  = float(raw.get("high", "nan"))
    low_   = float(raw.get("low",  "nan"))
    close_ = float(raw.get("close","nan"))
    vol_   = float(raw.get("volume","nan"))

    return Bar(
        asset_id=asset_id,
        ts_start_utc=ts_start,
        ts_end_utc=ts_end,
        open=open_, high=high_, low=low_, close=close_, volume=vol_,
        source_id=source_id,
    )

Loading should be batched and idempotent. Batching is an engineering constraint, idempotence is an epistemic constraint.

def chunked(iterable, size: int):
    buf = []
    for x in iterable:
        buf.append(x)
        if len(buf) >= size:
            yield buf
            buf = []
    if buf:
        yield buf

Finally, a quality gate must exist. If QC flags exceed thresholds, the system may still ingest and store raw data, but it shouldn’t publish features to the live engine. This is the smallest mechanism that prevents a strategy from trading on corrupted representations.

Let’s see what this example can offer us!

Okay! Basically what was expected.

Alright crew—solid session. Until the next bell: keep your process boring, your sizing honest, and your rules louder than your mood. May liquidity show up when you need it, slippage stay polite, and your cleanest edges stack quietly while the crowd chases noise 💵​

PS: What is more important for you?

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full script for an ETL

import time
import json
import random
import hashlib
import sqlite3
from dataclasses import dataclass
from datetime import datetime, timezone, timedelta
from urllib.request import urlopen, Request
import numpy as np
import matplotlib.pyplot as plt

# QL schemas
POSTGRES_FACT_BAR_DDL = r"""
-- PostgreSQL-style DDL (conceptually portable)
CREATE TABLE IF NOT EXISTS fact_bar (
  asset_id           BIGINT NOT NULL,
  ts_start_utc       TIMESTAMPTZ NOT NULL,
  ts_end_utc         TIMESTAMPTZ NOT NULL,
  open               DOUBLE PRECISION,
  high               DOUBLE PRECISION,
  low                DOUBLE PRECISION,
  close              DOUBLE PRECISION,
  volume             DOUBLE PRECISION,
  source_id          TEXT NOT NULL,
  ingest_ts_utc      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  quality_flags      INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (asset_id, ts_start_utc, source_id)
);

CREATE INDEX IF NOT EXISTS idx_fact_bar_end
  ON fact_bar(asset_id, ts_end_utc);
"""

POSTGRES_FEATURE_VECTOR_DDL = r"""
CREATE TABLE IF NOT EXISTS feature_vector (
  asset_id            BIGINT NOT NULL,
  ts_utc              TIMESTAMPTZ NOT NULL,
  feature_set_version TEXT NOT NULL,
  f1                  DOUBLE PRECISION,
  f2                  DOUBLE PRECISION,
  f3                  DOUBLE PRECISION,
  quality_flags       INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY(asset_id, ts_utc, feature_set_version)
);
"""

POSTGRES_UPSERT_BAR_SQL = r"""
INSERT INTO fact_bar(asset_id, ts_start_utc, ts_end_utc, open, high, low, close, volume, source_id, quality_flags)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
ON CONFLICT (asset_id, ts_start_utc, source_id)
DO UPDATE SET
  ts_end_utc    = EXCLUDED.ts_end_utc,
  open          = EXCLUDED.open,
  high          = EXCLUDED.high,
  low           = EXCLUDED.low,
  close         = EXCLUDED.close,
  volume        = EXCLUDED.volume,
  quality_flags = EXCLUDED.quality_flags;
"""

# SQLite 
SQLITE_FACT_BAR_DDL = """
CREATE TABLE IF NOT EXISTS fact_bar (
  asset_id      INTEGER NOT NULL,
  ts_start_utc  TEXT NOT NULL,
  ts_end_utc    TEXT NOT NULL,
  open          REAL,
  high          REAL,
  low           REAL,
  close         REAL,
  volume        REAL,
  source_id     TEXT NOT NULL,
  ingest_ts_utc TEXT NOT NULL,
  quality_flags INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (asset_id, ts_start_utc, source_id)
);
CREATE INDEX IF NOT EXISTS idx_fact_bar_end ON fact_bar(asset_id, ts_end_utc);
"""

SQLITE_FEATURE_VECTOR_DDL = """
CREATE TABLE IF NOT EXISTS feature_vector (
  asset_id            INTEGER NOT NULL,
  ts_utc              TEXT NOT NULL,
  feature_set_version TEXT NOT NULL,
  f1                  REAL,
  f2                  REAL,
  f3                  REAL,
  quality_flags       INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY(asset_id, ts_utc, feature_set_version)
);
"""

SQLITE_UPSERT_BAR_SQL = """
INSERT INTO fact_bar(asset_id, ts_start_utc, ts_end_utc, open, high, low, close, volume, source_id, ingest_ts_utc, quality_flags)
VALUES (?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(asset_id, ts_start_utc, source_id)
DO UPDATE SET
  ts_end_utc    = excluded.ts_end_utc,
  open          = excluded.open,
  high          = excluded.high,
  low           = excluded.low,
  close         = excluded.close,
  volume        = excluded.volume,
  ingest_ts_utc = excluded.ingest_ts_utc,
  quality_flags = excluded.quality_flags;
"""

SQLITE_UPSERT_FEATURE_SQL = """
INSERT INTO feature_vector(asset_id, ts_utc, feature_set_version, f1, f2, f3, quality_flags)
VALUES (?,?,?,?,?,?,?)
ON CONFLICT(asset_id, ts_utc, feature_set_version)
DO UPDATE SET
  f1 = excluded.f1,
  f2 = excluded.f2,
  f3 = excluded.f3,
  quality_flags = excluded.quality_flags;
"""

# YAML 
JOB_MANIFEST_YAML = r"""
job_name: ingest_intraday_bars
schedule: "0 6 * * 1-5"  # weekdays
source:
  provider: financial_api
  endpoint: historical-chart/15min
  symbols: ["SPY", "QQQ", "TLT"]
  tz_assumption: "UTC"   # explicit
transform:
  bar_minutes: 15
  interpret_vendor_time_as: "bar_end"
  canonical_time: "UTC"
load:
  target: fact_bar
  mode: upsert
  batch_size: 5000
qc:
  max_missing_bars_per_day: 0
  max_revision_fraction: 0.01
  return_outlier_z: 12
feature_publish:
  feature_set_version: "1.0.0"
  output: feature_vector
"""

# RateLimiter + fetch_json
class RateLimiter:
    def __init__(self, max_per_minute: int):
        self.max_per_minute = int(max_per_minute)
        self.min_interval = 60.0 / float(self.max_per_minute)
        self._t_last = 0.0

    def wait(self):
        dt = time.time() - self._t_last
        if dt < self.min_interval:
            time.sleep(self.min_interval - dt)
        self._t_last = time.time()


def fetch_json(url: str, limiter: RateLimiter, max_retries: int = 5) -> dict:
    for attempt in range(max_retries):
        try:
            limiter.wait()
            req = Request(url, headers={"User-Agent": "quant-etl/1.0"})
            with urlopen(req, timeout=20) as r:
                return json.loads(r.read().decode("utf-8"))
        except Exception:
            sleep_s = (2 ** attempt) + random.random()
            time.sleep(sleep_s)
    raise RuntimeError(f"Failed to fetch after {max_retries} retries: {url}")

# Bar dataclass + hash
@dataclass(frozen=True)
class Bar:
    asset_id: int
    ts_start_utc: datetime
    ts_end_utc: datetime
    open: float
    high: float
    low: float
    close: float
    volume: float
    source_id: str

    def natural_key(self) -> tuple:
        return (self.asset_id, self.ts_start_utc, self.source_id)

    def payload_hash(self) -> str:
        s = (
            f"{self.asset_id}|{self.ts_start_utc.isoformat()}|{self.ts_end_utc.isoformat()}|"
            f"{self.open}|{self.high}|{self.low}|{self.close}|{self.volume}|{self.source_id}"
        )
        return hashlib.sha256(s.encode("utf-8")).hexdigest()

#Transform helpers
def to_utc(ts_str: str) -> datetime:
    dt = datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
    return dt.astimezone(timezone.utc)

def conform_bar(raw: dict, asset_id: int, source_id: str, bar_minutes: int) -> Bar:
    ts_end = to_utc(raw["date"])  # interpret explicitly as bar end
    ts_start = ts_end - timedelta(minutes=int(bar_minutes))

    open_  = float(raw.get("open", "nan"))
    high_  = float(raw.get("high", "nan"))
    low_   = float(raw.get("low",  "nan"))
    close_ = float(raw.get("close","nan"))
    vol_   = float(raw.get("volume","nan"))

    return Bar(
        asset_id=asset_id,
        ts_start_utc=ts_start,
        ts_end_utc=ts_end,
        open=open_, high=high_, low=low_, close=close_, volume=vol_,
        source_id=source_id,
    )

def chunked(iterable, size: int):
    buf = []
    for x in iterable:
        buf.append(x)
        if len(buf) >= size:
            yield buf
            buf = []
    if buf:
        yield buf

# Causality assertion
def assert_causal_features(feature_ts_end_utc: np.ndarray, decision_ts_utc: np.ndarray):
    if np.any(decision_ts_utc < feature_ts_end_utc):
        bad = np.where(decision_ts_utc < feature_ts_end_utc)[0][:10]
        raise AssertionError(f"Causality violation at rows: {bad}")

# QC returns (robust z using MAD)
def qc_returns(close: np.ndarray, z_thresh: float = 12.0) -> int:
    close = np.asarray(close, dtype=np.float64)
    r = np.diff(np.log(close))
    med = np.nanmedian(r)
    mad = np.nanmedian(np.abs(r - med))
    if (not np.isfinite(mad)) or mad == 0.0:
        return 1
    z = (r - med) / (1.4826 * mad)
    return int(np.nanmax(np.abs(z)) > float(z_thresh))

# Feature pipeline (daily VWAP reset + rolling std)
def daily_reset_vwap(price: np.ndarray, vol: np.ndarray, day_id: np.ndarray) -> np.ndarray:
    price = np.asarray(price, dtype=np.float64)
    vol = np.asarray(vol, dtype=np.float64)
    day_id = np.asarray(day_id)

    out = np.empty_like(price, dtype=np.float64)
    pv = price * vol

    boundaries = np.flatnonzero(np.r_[True, day_id[1:] != day_id[:-1]])
    boundaries = np.r_[boundaries, len(day_id)]

    for s, e in zip(boundaries[:-1], boundaries[1:]):
        cum_pv = np.cumsum(pv[s:e])
        cum_v  = np.cumsum(vol[s:e])
        out[s:e] = np.where(cum_v > 0.0, cum_pv / cum_v, np.nan)
    return out

def rolling_std(x: np.ndarray, w: int) -> np.ndarray:
    x = np.asarray(x, dtype=np.float64)
    out = np.full_like(x, np.nan)
    w = int(w)
    if w < 2:
        return out
    for i in range(w - 1, len(x)):
        seg = x[i - w + 1:i + 1]
        out[i] = np.nanstd(seg, ddof=1)
    return out

def compute_features(close: np.ndarray, vol: np.ndarray, day_id: np.ndarray, w_vol: int = 20):
    close = np.asarray(close, dtype=np.float64)
    vol = np.asarray(vol, dtype=np.float64)

    r = np.empty_like(close)
    r[0] = np.nan
    r[1:] = np.diff(np.log(close))

    vwap = daily_reset_vwap(close, vol, day_id)
    sig = rolling_std(r, w=w_vol)

    X = np.column_stack([r, close - vwap, sig])
    return X

# DB layer (SQLite) - idempotent upsert + reads
def init_db(conn: sqlite3.Connection):
    cur = conn.cursor()
    cur.executescript(SQLITE_FACT_BAR_DDL)
    cur.executescript(SQLITE_FEATURE_VECTOR_DDL)
    conn.commit()

def upsert_bars(conn: sqlite3.Connection, bars: list[Bar], quality_flags: int = 0, batch_size: int = 5000):
    cur = conn.cursor()
    ingest_now = datetime.now(timezone.utc).isoformat()

    rows = []
    for b in bars:
        rows.append((
            int(b.asset_id),
            b.ts_start_utc.isoformat(),
            b.ts_end_utc.isoformat(),
            float(b.open), float(b.high), float(b.low), float(b.close), float(b.volume),
            str(b.source_id),
            ingest_now,
            int(quality_flags),
        ))

    for chunk in chunked(rows, batch_size):
        cur.executemany(SQLITE_UPSERT_BAR_SQL, chunk)

    conn.commit()

def count_fact_bars(conn: sqlite3.Connection) -> int:
    cur = conn.cursor()
    cur.execute("SELECT COUNT(*) FROM fact_bar;")
    return int(cur.fetchone()[0])

def load_bars_arrays(conn: sqlite3.Connection, asset_id: int, source_id: str):
    cur = conn.cursor()
    cur.execute(
        """
        SELECT ts_start_utc, ts_end_utc, open, high, low, close, volume, quality_flags
        FROM fact_bar
        WHERE asset_id = ? AND source_id = ?
        ORDER BY ts_start_utc ASC;
        """,
        (int(asset_id), str(source_id)),
    )
    rows = cur.fetchall()
    if not rows:
        raise RuntimeError("No bars found.")

    ts_start = np.array([np.datetime64(r[0]) for r in rows])
    ts_end   = np.array([np.datetime64(r[1]) for r in rows])
    open_  = np.array([r[2] for r in rows], dtype=np.float64)
    high_  = np.array([r[3] for r in rows], dtype=np.float64)
    low_   = np.array([r[4] for r in rows], dtype=np.float64)
    close_ = np.array([r[5] for r in rows], dtype=np.float64)
    vol_   = np.array([r[6] for r in rows], dtype=np.float64)
    qf     = np.array([r[7] for r in rows], dtype=np.int64)
    return ts_start, ts_end, open_, high_, low_, close_, vol_, qf

def upsert_feature_vectors(conn: sqlite3.Connection,
                           asset_id: int,
                           ts_utc: np.ndarray,
                           X: np.ndarray,
                           feature_set_version: str,
                           quality_flags: int = 0,
                           batch_size: int = 5000):
    ts_utc = np.asarray(ts_utc)
    X = np.asarray(X, dtype=np.float64)
    if X.ndim != 2 or X.shape[1] < 3:
        raise ValueError("X must be (n,>=3) for f1,f2,f3.")

    cur = conn.cursor()
    rows = []
    for i in range(len(ts_utc)):
        # store ISO string
        ts_s = str(ts_utc[i]).replace(" ", "T")
        rows.append((int(asset_id), ts_s, str(feature_set_version),
                     float(X[i, 0]), float(X[i, 1]), float(X[i, 2]),
                     int(quality_flags)))

    for chunk in chunked(rows, batch_size):
        cur.executemany(SQLITE_UPSERT_FEATURE_SQL, chunk)

    conn.commit()

# Make synthetic "raw API" bars
def make_synthetic_raw_bars(start_end_utc: datetime,
                            n_bars: int,
                            bar_minutes: int = 15,
                            seed: int = 7):
    """
    Produces raw bars where "date" is the *bar end* timestamp in ISO Z (like many vendors).
    """
    rng = np.random.default_rng(seed)
    out = []

    # price random walk
    price = 100.0
    dt = timedelta(minutes=int(bar_minutes))
    ts_end = start_end_utc

    for _ in range(n_bars):
        # simulate end-to-end bar
        ret = rng.normal(0.0, 0.001)          # small drift/noise
        close = price * np.exp(ret)
        high = max(price, close) * (1.0 + abs(rng.normal(0.0, 0.0007)))
        low  = min(price, close) * (1.0 - abs(rng.normal(0.0, 0.0007)))
        open_ = price
        vol = float(max(1.0, rng.lognormal(mean=4.5, sigma=0.4)))  # positive

        out.append({
            "date": ts_end.isoformat().replace("+00:00", "Z"),
            "open": open_,
            "high": high,
            "low": low,
            "close": close,
            "volume": vol,
        })

        price = close
        ts_end = ts_end + dt

    return out


def day_id_from_ts_end(ts_end: np.ndarray) -> np.ndarray:
    # day_id = YYYYMMDD integer using ts_end UTC date
    days = np.array([str(t)[:10].replace("-", "") for t in ts_end], dtype=np.int64)
    return days


# ETL + QC + Features + Causality test + plot
def main():
    # demo config
    asset_id = 1
    source_id = "demo_vendor"
    bar_minutes = 15
    feature_set_version = "1.0.0"
    w_vol = 20

    # (1) EXTRACT (synthetic)
    start_end_utc = datetime(2026, 2, 2, 0, 0, tzinfo=timezone.utc)
    n_bars = 3 * 24 * (60 // bar_minutes)  # 3 days of 15-min bars
    raw = make_synthetic_raw_bars(start_end_utc=start_end_utc, n_bars=n_bars, bar_minutes=bar_minutes)

    # (2) TRANSFORM (conform)
    bars = [conform_bar(r, asset_id=asset_id, source_id=source_id, bar_minutes=bar_minutes) for r in raw]

    # (3) LOAD (idempotent upsert)
    conn = sqlite3.connect(":memory:")
    init_db(conn)

    upsert_bars(conn, bars, quality_flags=0, batch_size=5000)
    n1 = count_fact_bars(conn)

    # rerun same load to prove idempotence
    upsert_bars(conn, bars, quality_flags=0, batch_size=5000)
    n2 = count_fact_bars(conn)

    print(f"fact_bar row count after first load:  {n1}")
    print(f"fact_bar row count after second load: {n2}  (should match)")

    # read back as arrays
    ts_start, ts_end, open_, high_, low_, close_, vol_, qf = load_bars_arrays(conn, asset_id, source_id)
    day_id = day_id_from_ts_end(ts_end)

    # QC (dataset-level flag, minimal demo)
    qc_flag = qc_returns(close_, z_thresh=12.0)
    print(f"qc_returns flag: {qc_flag}  (1 means outlier risk detected)")

    # FEATURES (train/live parity rule for bar-open execution)
    # compute features using series as-given (bars labeled by end-time)
    X_raw = compute_features(close_, vol_, day_id, w_vol=w_vol)

    # For bar-open decisions at ts_start[k], enforce "only info strictly before that open":
    # shift features by 1 bar so X_used[k] depends on bars ending <= ts_start[k].
    X_used = np.empty_like(X_raw)
    X_used[:] = np.nan
    X_used[1:] = X_raw[:-1]

    # causality timestamps: feature window "ends" at previous bar end (ts_end[k-1])
    feature_ts_end = np.empty_like(ts_end)
    feature_ts_end[:] = np.datetime64("NaT")
    feature_ts_end[1:] = ts_end[:-1]
    decision_ts = ts_start.copy()

    # causality test (should pass)
    assert_causal_features(feature_ts_end, decision_ts)
    print("Causality check (shifted features): PASS")

    # demonstrate failure if you (incorrectly) use current bar end as if available at open
    try:
        assert_causal_features(ts_end, decision_ts)  # BAD: decision at start < end
        print("Causality check (unshifted): PASS (unexpected)")
    except AssertionError as e:
        print(f"Causality check (unshifted): FAIL as expected -> {e}")

    # (5) PUBLISH features into feature store table
    upsert_feature_vectors(
        conn,
        asset_id=asset_id,
        ts_utc=decision_ts,          # label features at decision time
        X=X_used,
        feature_set_version=feature_set_version,
        quality_flags=qc_flag,
    )
    print("feature_vector upsert: OK")

    # (6) Tiny example signal + PnL (just to validate arrays end-to-end)
    # mean-reversion toy: if close - vwap > 0 => short, else long; 0 when NaN
    f2 = X_used[:, 1]
    sig = np.where(np.isfinite(f2), np.where(f2 > 0.0, -1.0, 1.0), 0.0)

    # bar PnL if entering at open and exiting at close of same bar
    pnl = sig * (close_ - open_)
    pnl_cum = np.nancumsum(np.where(np.isfinite(pnl), pnl, 0.0))

    # Plot 1: close vs VWAP (from X_raw, because VWAP exists at bar end)
    vwap_end = close_ - X_raw[:, 1]

    plt.figure()
    plt.plot(close_, label="close")
    plt.plot(vwap_end, label="vwap (daily reset)")
    plt.title("Close vs Daily-Reset VWAP (computed at bar end)")
    plt.legend()
    plt.show()

    # Plot 2: decision-time signal markers on close (using shifted features)
    plt.figure()
    plt.plot(close_, label="close")
    idx_long = np.where(sig == 1.0)[0]
    idx_short = np.where(sig == -1.0)[0]
    plt.scatter(idx_long, close_[idx_long], marker="^", label="long (decision @ open)")
    plt.scatter(idx_short, close_[idx_short], marker="v", label="short (decision @ open)")
    plt.title("Toy Signal (causally shifted features)")
    plt.legend()
    plt.show()

    # Plot 3: cumulative PnL (toy)
    plt.figure()
    plt.plot(pnl_cum, label="cumulative PnL (toy)")
    plt.title("Cumulative PnL (enter @ open, exit @ close)")
    plt.legend()
    plt.show()

    print("\n JOB MANIFEST")
    print(JOB_MANIFEST_YAML)

main()

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Mutual information as evidence of structure

This chapter treats quantitative trading research as a sequence of testable dependence claims, not as an automatic story about why markets move. The goal is to separate what the data actually supports—a conditional advantage linking X to R—from what remains underdetermined—any single structural narrative we might want to attach to it. The workflow moves from defining what it means to have information, to checking that claim against what exists in raw data, to identifying when structure appears (state conditions S), and finally to compressing the feature set without introducing leakage or manufacturing evidence through selection.

What’s inside:

1. What mutual information is for in trading research. Mutual information is used to detect and quantify dependence without committing to a particular functional form.

2. Separating dependence from functional form. Different decision rules can exploit the same underlying dependence. Dependence is the object; the specific rule is only a vehicle.

3. Quantifying the strategy’s implied information. Realized performance can be read as an information claim per decision.

4. Orthogonality to utility. Dependence is not monetization. A pattern can be real and still fail to generate PnL under a given execution scheme.

5. Dependence is the object, not the model. This reverses the usual pipeline where a backtest creates belief and justification is constructed afterward.

6. Information is not edge. The section distinguishes layers: (i) existence of dependence, (ii) robustness under refuters, (iii) translation into executable edge.

7. Incremental information. The practical question is whether X adds anything beyond a baseline information set Z that is already available, already priced, or already captured by your stack.

8. Conditional dependence, sample size, and non-circular boundaries. Dependence is not uniform; it often concentrates when the market enters a configuration S.

9. Why feature sets behave differently than single features. Weak variables can become informative in combination (gating/synergy), and apparently strong variables can become redundant once others are known.

10. Dependence can move across time scales. A pattern can exist at one horizon and disappear at another, or shift with regime.

11. The smallest information set that preserves structure. Compression is an active process: reduce while verifying that the evidence you care about remains intact and that the compression introduces no leakage.

12. Mutual information is a target, not a number you trust by default. I(X;R) depends on estimation choices, regularization, and temporal dependence.

Check a sample of what you will find inside:

Sample

1.94MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. The architecture of escalation.

   1. Motif 1: Ultimatum + army + readiness.

   2. Motif 2: Airspace denial as the real constraint.

   3. Motif 3: Europe is choosing a sanctions path.

   4. Motif 4: U.S. governance becomes a market input again.

   5. Motif 5: Energy scarcity as leverage in the Americas.

   6. Motif 6: War remains industrial: drones, grids, heating.

   7. Motif 7: Asia is about patrol tempo + treaty signaling.

3. What is confirmed in late January 2026.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Introduction

Here’s the uncomfortable truth: you can’t read it headline by headline anymore. The signal is in the way events fit together. In geopolitics, the market-relevant details are often boring on purpose: a routing constraint, a sanctions trajectory, an enforcement memo, a posture shift, a logistics denial. They read like bureaucracy, but each one changes the friction in the system.

The real danger is the accumulation of constraints that make miscalculation more likely. Constraints reshape risk. They can make deliberate escalation harder while making accidents more likely, simply because actors operate with less slack. Detours add steps. Steps add failure points. Failure points increase the chance of misreads, probing, brinkmanship, and asymmetric responses that sit below the threshold of war but above the threshold of irrelevant. That’s how you get the defining pattern of a constraint regime: spot markets can look orderly while hedges stay expensive.

Sanctions work the same way. The market impact isn’t limited to the sanctioned party’s direct exposure. It spreads through the plumbing—documentation, payment rails, due diligence, shipping paperwork, correspondent banking behavior, insurer risk appetite. Even before every rule is enforced, the direction of travel changes private behavior. Firms and intermediaries start pricing the risk of being on the wrong side of tomorrow’s interpretation.

When several constraints tighten in the same window, the effects compound. Multiple theaters begin to hit the same transmission channels—shipping, insurance, enforcement, fiscal sequencing—so correlation rises for reasons that have nothing to do with fundamentals.

Subscribe now

The architecture of escalation

Escalation is usually narrated as a sequence of events: a statement, a strike, a retaliation. But it is a engineered shape of decision-making: what can move, what cannot, what must route around a constraint, and what becomes legible enough to model. That legibility is the key transmission mechanism where each motif is a repeatable building block. Taken alone, each looks local; together, they describe a scenario rotating toward constraint-driven pricing.

Motif 1: Ultimatum + army + readiness

Iran’s posture of immediate readiness to respond by land, air, and sea presents the American posture as a naval force prepared for speed and violence, with the aircraft carrier USS Abraham Lincoln elevated to a symbol rather than a mere asset. This detail is important because markets don’t value symbols as they value inventories. Symbols compress time. They turn maybe later into could be soon, which is what increases the cost of being wrong.

Multiple reports described President Trump using the language of war and explicitly linking the posture to the USS Abraham Lincoln and accompanying destroyers moving into the region. The central move is to fuse that war language with Iran’s readiness claim into a single decision-loop: action, response, and the promise of a larger follow-through if the first strike is limited.

For investors, the immediate trade is often misunderstood. The largest early repricing does not require missiles landing. Iters and shippers to believe the probability of disruption has changed. Indeed, insurance is a probability business. When political signaling becomes operationally legible the market channel that moves first tends to be risk transfer: war-risk premia, marine insurance riders, freight spreads, and the volatility surface of energy and transportation equities. A carrier is a mechanism that makes tail risk financeable, because it gives counterparties something concrete to model.

Notice that all of this does not stay on Iran. It pivots into other theatres. That pivot is the tell. The story is not Iran as a one-off. The story is the world is back to pricing corridor disruption as a generalized condition.

We’re going to show you a sequence of plots that, after seeing a couple of them, will help you understand the whole picture.

The deployment is centered on the USS Abraham Lincoln Carrier Strike Group:

The presence of the USS Iwo Jima is symbolic and operationally significant. Having served as the holding facility for Nicolás Maduro following his capture earlier in the month, its deployment to the Gulf signals that the Venezuela Model—decapitation strikes against leadership—is on the table for Iran. Trump explicitly compared the two, noting the armada facing Iran is Bigger than Venezuela.

The credibility of this force is underpinned by Operation Midnight Hammer, a recent military action described by the President as a warning shot. While specific operational details remain classified, the invocation of this operation suggests that the threshold for kinetic engagement has already been crossed. The administration is signaling that it has moved beyond red lines to active measures.

Despite the martial posturing, a diplomatic track remains open, albeit narrow. Iranian Foreign Minister Abbas Araghchi is scheduled to hold talks in Turkiye. The U.S. administration has defined a strict two-week decision window to observe the Iranian response to the ultimatum. This timeline creates a period of volatility; any miscalculation by Iranian naval forces or proxy groups during this window could trigger the US army to fulfill its missions with speed and violence.

The Iranian regime faces a dilemma: capitulation on nuclear enrichment would be a humiliating retreat for Supreme Leader Ali Khamenei, endangering the regime’s domestic legitimacy amidst ongoing protests. Conversely, rebuffing the demand risks a strike that could destabilize the government entirely. The U.S. bet is that the regime’s survival instinct will force it to the table, but the rigid timeline increases the risk of accidental war.

Motif 2: Airspace denial as the real constraint

Saudi Crown Prince Mohammed bin Salman told Iran’s president that Saudi Arabia will not allow its airspace or territory to be used for military action against Iran. This is the kind of detail that sounds diplomatic, but trades like logistics.

Airspace is the hidden scaffolding of escalatial state signals denial, it changes routing, refueling geometry, the set of feasible strike paths, and the number of steps required to execute a plan. Operational complexity is not a moral category; it is a probability category. More steps means more chances to abort, miscalculate, leak, or be deterred.

This denial is driven by a rational calculation of vulnerability. Saudi Arabia and the UAE are the glass houses of the region—their economies rely on vulnerable oil infrastructure and futuristic cities (Neom, Dubai) that are within minutes of Iranian missile flight times. The detente initiated between Riyadh and Tehran (brokered by China) has prioritized regional stability over U.S. security objectives. MBS’s affirmation of support for dialogue indicates that the Gulf states are choosing to manage Iran rather than defeat it.

The denial of airspace fundamentally alters the operational perspective of any U.S. strike.

1. Maritime dependence: The U.S. must rely almost exclusively on carrier-based aviation launching from international waters in the Gulf of Oman or the Arabian Sea, rather than land-based tactical air.

2. Routing complexity: To reach targets in northern or central Iran without crossing Saudi or Emirati airspace, U.S. assets must fly through Iraq (politically sensitive) or rely on long-range standoff munitions (Tomahawks) that do not require overflight permissions for pilots.

3. Diplomatic isolation: The U.S. is isolated operationally. By refusing to participate, the Gulf states strip any potential conflict of regional coalition legitimacy, framing it strictly as a U.S.-Iran bilateral war.

The sovereign constraints on U.S. power projection can be summarized as:

From an investment point of view, airspace denial is one of the clearest examples of how politics translates into convexity. Constraints can lower the probability of deliberate escalation while simultaneously increasing the probability of incidents (misreads, brinkmanship, probing flights, asymmetric responses) because actors operate in tighter corridors with less slack. That is why this signal can support a regime where spot markets look calm, while option markets remain bid: uncertainty is not eliminated but reshaped.

In practical portfolio terms, this motif matters most for exposures that depend on routing: air freight, certain energy-linked shipping lanes, and any supply chain whose cost base is sensitive to detours and insurance terms. Even if you are not trading these directly, they feed into headline inflation volatility and therefore into rate volatility.

This motif underscores that while the U.S. retains overwhelming global power, its local power is subject to the veto of its partners. The real constraint on the architecture of escalation is Arab neutrality.

Motif 3: Europe is choosing a sanctions path.

France and European leaders are moving toward sanctions related to Iranian repression and intensifying pressure on the IRGC. The EU moved toward designating Iran’s Revolutionary Guard as a terrorist organizn France’s position, alongside approval of new sanctions targeting individuals and entities linked to protest suppression and support to Russia.

For positioning, the critical insight is that sanctions regimes do not reprice markets only through the sanctioned entity’s direct exposure. They reprice through: documentation, payment rails, due diligence, shipping paperwork, correspondent banking behavior, and the risk appetite of insurers and freight forwarders. Even if the sanction list is not yet operationalized in every legal sense, the trajectory itself shifts behavior, because private actors price the risk of being caught on the wrong side of tomorrow’s rule.

That is why you can see a sanctions tightening as a kind of tax on frictionless globalization. It just needs to make trade slower, more expensive, and more legally conditional. For investors, that tends to express through wider dispersion within Europe—firms with clean supply chains and robust compliance cultures gain relative advantage—and through higher premia for EM assets that depend on cross-border capital confidence.

This matters because it places Europe’s sanctions momentum alongside Middle East escalation risk. That pairing is not accidental. Sanctions are a second channel of escalation. When military geometry is constrained, policy geometry often tightens instead. Markets price that as a regime shift toward persistent friction, which is a different macro environment than 2010s smooth trade assumptions.

The sanctions path complements the U.S. strategy by closing the economic escape valves for Tehran.

1. Economic strangulation: The IRGC controls vast swathes of the Iranian economy (construction, energy, telecommunications). Designating it as a terrorist organization makes it legally perilous for any European bank or corporation to engage in any significant commerce with Iran, creating a de facto total embargo.

2. Transatlantic unity: This moves Europe into alignment with the U.S. maximum pressure campaign. While Europe is not sending ships, it is weaponizing its financial system.

3. Diplomatic rupture: The designation likely ends any hope of reviving the nuclear deal. Iran has consistently stated that the IRGC designation is a red line. By crossing it, Europe is accepting that the diplomatic era is over and the containment era has begun.

The EU’s sanctions architecture:

Motif 4: U.S. governance becomes a market input again.

A potential partial U.S. government shutdown tied to DHS funding, politically amplified by the Minnesota shooting case and subsequent operational guidance to ICE about avoiding agitators. Now there are new orders instructing ICE officers in Minnesota not to interact with agitators, reflecting operational shifts amid public tension. At the same time, the renewed shutdown risk around a January 30 funding deadline and the political dynamics that could push the U.S. toward a partial shutdown:

1. The killing of Alex Pretti, a 37-year-old VA nurse, by Border Patrol agents on January 24 became a national symbol of excessive force.

2. Minneapolis saw protests of 50,000 people, creating a domestic security crisis that mirrored the international tensions.

3. The funding lapse affects key agencies including DHS, Transportation, and Labor. This creates operational friction; the very agencies needed to secure the homeland during a potential war with Iran are operating without funding.

Besides, the governance crisis was compounded by a contentious Federal Reserve meeting on January 28, 2026. The Fed chose to hold interest rates steady, marking a strategic pause in its easing cycle.

The dissent by Governors Miran and Waller highlights a central bank unsure of the terrain. The sticky inflation combined with social unrest creates a stagflationary risk profile. Markets reacted with a focus on governance risk—the idea that the U.S. political system is becoming too dysfunctional to manage its economic policy effectively.

The shutdown and the Fed pause create a feedback loop that weakens the U.S. global posture.

1. It is difficult to project competence abroad (e.g., in the South China Sea) when the federal government is closed and citizens are being shot by agents at home.

2. The shutdown complicates the logistics of military funding and aid distribution.

3. Rivals like Iran and Russia may calculate that a U.S. president facing domestic turmoil is either distracted (opportunity) or desperate (danger), adding volatility to their decision-making.

The investment meaning is that the United States remains both the world’s anchor for risk-free discounting and one of the world’s largest sources of policy impulses. When governance volatility rises, global term premia tend to rise—not necessarily because investors fear default tomorrow, but because investors price the possibility of policy discontinuity across immigration enforcement, trade, defense posture, sanctions enforcement, and fiscal sequencing.

This is especially effective in how it frames this: it treats U.S. politics as local, a variable that interacts with every other motif. If you are pricing Middle East escalation, sanctions enforcement, energy corridor coercion, and China-adjacent trade policy, then U.S. governance volatility becomes correlation fuel. In that regime, diversification can fail: assets that are normally uncorrelated begin reacting to the same policy headlines because the marginal buyer’s constraint becomes political uncertainty rather than fundamentals.

Motif 5: Energy scarcity as leverage in the Americas.

Read more

Table of contents:

1. Introduction.

2. When fast becomes slow.

3. Risks and framework limitations.

4. Lifecycle engineering for low latency data structures.

   1. Point 1 — Capacity-constrained processing.

   2. Point 2 — Tail telemetry.

   3. Point 3 — Consistency domains without global locks.

   4. Point 4 — Schema and layout hashing.

   5. Point 5 — Deterministic replay.

   6. Point 6 — Fault containment, controlled degradation, and restart integrity.

   7. Point 7 — Latency distribution.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Most performance failures in trading systems don’t look like failures. You know, impressive microbenchmarks, clean profiles, and average latency that inspire confidence. Then the market compresses time. A volatility microburst lands, the arrival process stops behaving like expected and the engine doesn’t slow down uniformly. One thread stays current, another drifts into backlog, and a third starts making decisions with a reality that no longer exists. This sounds familiar, doesn’t it?

The strategy still runs. Orders still go out. PnL still prints. But the system has quietly crossed a boundary. That boundary is the actual subject of low-latency engineering. Not how fast can you execute a loop, but under what load does your system lose coherence—and what happens to decision quality when it does.

Whether the state you believed you had matches the book you actually traded against. A system that is quick most of the day yet slow precisely during clustered order-flow is not a fast system. The result? The trading system becomes hardware-dependent, shaped by queue depth, cache residency, coherence traffic, parser variability, and scheduling noise. At that point, edge becomes a random variable.

To know more about that, check this:

Insights into trading system dynamics

2.36MB ∙ PDF file

Download

Download

Today we review more than simple micro-optimizations. Once allocator jitter, false sharing, and accidental copies are tamed, the remaining problem is lifecycle. How is state created, mutated, published, observed, persisted, replayed, and recovered—without letting burst dynamics rewrite the signal?

If your data structures only describe access patterns, they are incomplete. In a low-latency trading engine, the data structure is a protocol: it defines visibility, atomicity, backpressure, and the admissible ways in which time is allowed to slip between components. When those rules are implicit, the market discovers them for you, usually at the worst possible moment.

We therefore shift the unit of design from fast-optimized code to bounded behavior. Bounded queues that force overload policies instead of hiding them. Telemetry that measures tails without creating tails. Consistency domains that deliver coherent snapshots without global locks. Schemas treated as binary contracts, so layout drift can’t silently tax the cache. Replay that reconstructs state transitions bit-for-bit, turning production into a debuggable artifact.

When fast becomes slow

I would say that by the time you reach this stage of architectural maturity, you have likely solved the easy problems. You have eliminated allocator-driven jitter by pooling objects, you have enforced cache-line alignment to prevent false sharing between cores, and you have stopped letting hidden copies in your language runtime rewrite your latency distribution. You have moved beyond the naive optimization of average-case behavior and have started treating the CPU’s branch predictor, the Translation Lookaside Buffer, and the L3 cache as constraints.

However, a new problem appears once the micro-optimizations are complete. You can build a pipeline that is optimized for single-threaded throughput—clocking nano or microsecond-scale benchmarks in isolation—and yet remains strategically slow in production.

This paradox exists because the market doesn’t compensate participants based on mean latency or high-throughput benchmarks under ideal conditions. The market compensates based on the distributional arrival of information. Specifically, the market pays for decision accuracy conditioned on the state of the order book at the moment of execution. A system that is fast 99% of the time but slow during the critical 1% of volatility is not a fast system in the eyes of the PnL.

The dilemma is that speed (wire-to-wire latency) and truth (state consistency) are often orthogonal goals in naive architectures. To go faster, engineers often relax consistency guarantees, introduce unbounded buffers to absorb bursts, or rely on optimistic concurrency models that assume low contention. In a clustered arrival regime—which describes 100% of relevant market microstructure events due to the self-similar nature of order flow—these optimizations create a divergence between the internal state of the trading engine and the external reality of the exchange.

The data structure problem therefore shifts from a question of memory layout (Part I and II of the implied series) to a lifecycle problem: How is state created, mutated, observed, snapshotted, and recovered under bursty load without violating the semantic integrity of the strategy?

You can check them here:

Risks and framework limitations

When a trading engine moves from a controlled research environment to live production, the mathematical definition of its operation changes fundamentally.

In research, the strategy is modeled as a deterministic function F:

Where A_t is the action at time t, S_t is the market state, and θ are parameters. In this environment, S_t is perfect, and A_t is instantaneous.

In production, the strategy degrades into a stochastic operator F̂:

Here, 𝜉_t represents hidden system variables that act as noise terms:

• Queue depth (Q_t): The backlog of unprocessed messages, which acts as a proxy for the information gap between the exchange and the CPU.

• Cache residency: The probability that the relevant state S_t is in L1/L2 cache versus main memory (RAM), introducing a variance of ~200 cycles vs ~300 nanoseconds per access.

• Coherence traffic: The latency penalty induced by other cores invalidating shared cache lines, often caused by the OS scheduler or noisy neighbor threads.

• Parsing jitter: The variance in time required to decode the wire format, often depending on the specific template ID or message schema variability.

The dominant risk is not that the system is slow in a linear sense. The risk is that ε (staleness) and 𝛿 (execution delay) are correlated with market volatility.

If Cov(ε, | ΔP_t} > 0, your system is statistically most likely to be acting on stale data exactly when the price is moving the most. For example, during a sudden liquidity collapse, the incoming message rate spikes, increasing ε. Simultaneously, |ΔP_t| is large. Your strategy might try to hit a bid that vanished 50 microseconds ago because it is still processing the queue from 100 microseconds ago. This converts what looks like technical edge into noise, effectively randomizing the PnL of the strategy. The decision function is no longer deterministic, it is a random variable dependent on the micro-state of the hardware.

The event that exposes this fragility is the microburst.

Standard Poisson arrival models fail to capture the reality of market data, which is characterized by self-similarity, heavy tails, and clustering. A microburst is is a phase transition in the queuing system (a very short, intense spike in activity).

As visualized in our queue topology simulation (plot 1 below), when arrival intensity λ(t) momentarily exceeds service rate μ(t), the queue length Q_t does not just increase linearly; it undergoes a regime shift. The system transitions from a real-time state (Q≈0), where the CPU is waiting for network packets, to a batch state (Q≫0), where the network card is buffering packets faster than the PCI bus or the CPU can ingest them.

Crucially, in this batch state, the system is partially ordered. Because different components of a trading system (feed handlers, strategy logic, execution gateways, risk checks) often run on decoupled threads or separate cores, a burst desynchronizes their views of the world:

1. The feed handler receives update U₁, U₂, U₃.

2. The strategy calculates a signal based on U₁.

3. The risk/order manager checks limits based on U₃ (because it processes a different stream or shares a flag updated asynchronously).

At this moment, the system has no single time. The strategy logic and the execution logic occupy different temporal frames. This is the main problem: The burst breaks the semantic integrity of the engine. The system is a collection of race conditions executing in parallel, where the winner is determined by cache arbitration rather than financial logic.

Subscribe now

We must shift our perspective on what a low-latency data structure actually is.

In standard quant dev, a data structure is defined by its asymptotic complexity (O(1) lookup, O(log N) insertion). In low latency architectures, this view is insufficient because constants matter, and the cost of the worst-case operation dominates the PnL profile.

A data structure must be seen as a set of rules. It is a boundary definition that governs:

• Visibility: When exactly does a write by the feed handler become visible to the strategy thread? Is it after a memory fence? Is it relaxed?

• Atomicity: Can the strategy see a torn price (e.g., a bid from time t and an ask from time t+1)? If so, the strategy might compute a phantom arbitrage opportunity that exists only in the corrupted memory state.

• Backpressure: How does the structure communicate to the upstream socket that it is full? Does it block? Does it drop? Does it signal the sender to slow down (which is impossible with UDP multicast)?

• Lifecycle: How is the data persisted for replay without blocking the read/write path? Can we reconstruct the exact sequence of memory states from the log files?

This is different from optimizing code for instruction count. In this point we are optimizing the conditional law of decision latency. We are engineering the system such that the latency distribution P(L_t > x) decays super-exponentially, ensuring that even in the 99.99th percentile of events (the burst), the system remains within the coherence window of the market.

The implementation of this approach faces severe structural obstacles that cannot be solved by simply writing faster code:

1. Boundedness vs. blindness:

   • Infinite or dynamically resizing buffers (like std::vector or Python lists) hide overload. They allow the system to ingest a burst it cannot process, creating a standing wave of latency that persists long after the burst ends. The system continues to process old data while the market has moved on.

   • We need buffers that are strictly bounded. However, bounding a buffer requires a policy for rejection (drop vs. overwrite). This forces the strategy to handle gaps in data. A gap in a price series creates a discontinuity in derivative calculations, complicating the quant logic and requiring robust exception handling in the math itself.

2. Observation vs. perturbation:

   • Measuring latency typically involves getting timestamps (rdtsc or clock_gettime) and logging them. Writing to a log file involves blocking I/O; writing to memory involves cache pollution. The act of measuring the tail often causes the tail by evicting critical strategy data from the L1 cache to make room for telemetry structs.

   • We need telemetry that is zero-cost in the cache-hot path, storing only summary statistics (quantiles) rather than raw logs, without allocating memory or contending for lock resources.

3. Consistency vs. contention:

   • To ensure the strategy sees a consistent snapshot (e.g., best bid and best ask from the same packet), the naive solution is a mutex or rwlock. Locks induce context switches, force the OS to schedule threads, and stall the CPU pipeline. In a microsecond environment, a context switch is a massive penalty.

   • We need a wait-free snapshot mechanism that guarantees consistency without blocking the writer, even if it requires the reader to retry. We trade CPU cycles (retries) for latency stability (no blocking).

4. Replay fidelity:

   • Production systems often differ from backtests because the production system has state (sequence numbers, partial fills, TCP retransmits) that the backtest ignores. A strategy that worked in the simulator fails in production because the simulator did not model the sequence of partial updates caused by network fragmentation.

   • We need a data structure layout that supports deterministic replay. We must be able to feed the raw wire capture into the engine and produce the exact same state transitions, bit-for-bit, as occurred in live trading. This requires treating the entire engine as a deterministic finite state machine derived solely from the input log.

This sets the stage for building a deterministic state machine capable of surviving the hostile environment of live market microstructure.

Lifecycle engineering for low latency data structures

This type of engine rests on seven pillars. These pillars are are the axioms that prevent a system from becoming non-deterministic. To continue with the previous articles, we will proceed with the examples implemented in Python.

Point 1 — Capacity-constrained processing

The first misconception to dismantle is the idea that larger buffers equal greater safety. In latency-sensitive systems, buffers are debt. They convert instantaneous overload (a bandwidth problem) into delayed decision error (a latency problem).

Let arrivals be a clustered process with intensity λ(t) and service rate μ(t). If we use a dynamically resizing queue (like a Python list or C++ std::vector), we implicitly allow queue length Q_t ↦ ∞.

During a burst, Q_t grows. If the burst lasts 100ms and the system lags by 50ms, the strategy calculates signals based on prices that are 50ms old. This is an eternity, the alpha has decayed to zero or reversed. Furthermore, the very act of resizing a dynamic container during a burst exacerbates the problem. When a std::vector or dynamic list hits capacity, the runtime must allocate a larger block of memory, copy all existing elements to the new block, and free the old one. This operation causes a massive latency spike exactly when the system can least afford it, effectively pausing processing while the queue is flooding.

The solution involves bounded and pre-assigned queues with explicit overflow. So we must use fixed-capacity, circular buffers. The overflow behavior must be defined. This forces the engineering team to confront the reality of finite resources before the market does it for them.

• Drop newest: Ignore the new packet (good for keeping the strategy processing contiguous history, bad for recency). This effectively sheds load at the gateway, ensuring the internal engine never processes data faster than it can consume, but it risks blindness to the current market state.

• Drop oldest: Overwrite the tail (maximizes recency, but introduces data gaps). This is equivalent to saying, “If I can’t process everything, I will at least process the most current thing.” It breaks the assumption of a continuous time series, forcing the strategy to be robust against missing ticks.

For algorithmic trading, drop oldest (circular overwrite) is often preferred for market data (we care about the now price), while drop newest might be preferred for order entry (we don’t want to skip a sequence number).

We use a numpy based Structure of Arrays. This ensures that reading a specific field (e.g., all timestamps) utilizes full cache lines, unlike an Array of Structures or a list of objects. By keeping data contiguous in memory, we minimize the CPU’s need to fetch data from main RAM, keeping the hot path executing almost entirely within L1 or L2 cache.

import numpy as np

class BoundedEventQueue:
    """
    Preallocated circular queue for fixed-schema events.
    Overflow is explicit. No GC. No resizing.
    Uses Structure of Arrays (SoA) to maximize cache locality for column-based analysis.
    """
    __slots__ = ("cap", "mask", "write", "read",
                 "ts", "typ", "sym", "px", "qty",
                 "overflow_count")

    DROP_NEWEST = 0
    DROP_OLDEST = 1

    def __init__(self, cap_pow2: int):
        # Enforcing power of 2 allows bitwise AND masking (x & mask) 
        # which is significantly faster than modulo (x % cap) and avoids division instructions.
        if cap_pow2 & (cap_pow2 - 1) != 0:
            raise ValueError("Capacity must be a power of 2.")
        self.cap = int(cap_pow2)
        self.mask = self.cap - 1
        self.write = 0
        self.read = 0

        # Fixed schema (SoA) - Preallocation prevents runtime allocation
        # We assume 64-bit timestamps and 32-bit prices/quantities.
        self.ts  = np.empty(self.cap, dtype=np.uint64)
        self.typ = np.empty(self.cap, dtype=np.uint8)
        self.sym = np.empty(self.cap, dtype=np.uint16)
        self.px  = np.empty(self.cap, dtype=np.int32)
        self.qty = np.empty(self.cap, dtype=np.int32)

        self.overflow_count = 0

    def push(self, ts, typ, sym, px, qty, overflow_policy=DROP_OLDEST) -> bool:
        """
        Push operation that is strictly O(1) and allocation-free.
        """
        # Check occupancy
        size = self.write - self.read
        if size >= self.cap:
            self.overflow_count += 1
            if overflow_policy == self.DROP_NEWEST:
                return False
            # DROP_OLDEST: We forcibly advance the read pointer, effectively
            # "forgetting" the oldest unread message to make space for the new one.
            # This maintains the "freshness" of the buffer at the cost of history.
            self.read += 1

        # Bitwise mask for circular wrapping
        i = self.write & self.mask
        
        # Direct memory access without object overhead
        self.ts[i]  = ts
        self.typ[i] = typ
        self.sym[i] = sym
        self.px[i]  = px
        self.qty[i] = qty
        
        self.write += 1
        return True

Note: Full low latency data structures is in the provided code of the appendix.

This structure forces the architect to answer the question: What happens when overflow_count > 0? The answer usually involves entering a risk reduction mode. If the queue overflows, the strategy is legally blind; the only safe action is to widen quotes or pull orders until the overflow_count stabilizes and the system catches up.

Point 2 — Tail telemetry

The second pillar is observability. You can’t manage what you can’t measure, but measuring the 99.9th percentile (Q_0.999) of latency is statistically difficult.

A naive average (mean) is useless in HFT because latency distributions are heavy-tailed (Power Law or Pareto-like). A slow event might be 1000x slower than the median due to an OS context switch, a minor page fault, or a Garbage Collection pause. In these distributions, the mean is a misleading artifact of the tail.

The standard approach of storing all latency samples to compute an exact quantile is memory-prohibitive and causes cache pollution. Writing every single latency sample to a log file acts as an observer effect—the I/O overhead of logging the latency creates the latency you are trying to measure. We need a streaming quantile estimator—an algorithm that updates the estimate of the tail using constant memory.

The P² (piecewise-parabolic) algorithm is ideal here. It maintains 5 marker points that track the shape of the CDF around the target quantile. As data streams in, these markers adjust their height and position to approximate the curve, allowing for accurate tail estimation without ever storing the raw data points.

This matters because if your telemetry system allocates memory (e.g., creating a string for a log message logger.info(f"Latency: {lat}")), that allocation itself can trigger a Garbage Collection pause or a malloc lock, causing the very latency spike you are trying to record. Zero-allocation telemetry is a requirement, not an optimization.

import numpy as np

class P2Quantile:
    """
    Streaming quantile estimator using P^2 algorithm.
    Memory usage: Fixed (small arrays).
    Allocations per update: Zero.
    """
    def __init__(self, p: float):
        self.p = p
        # Markers for the P^2 algorithm
        # We only need 5 doubles to track the curve shape.
        self.q = np.zeros(5, dtype=np.float64) 
        # ... (initialization logic omitted for brevity) ...

    def update(self, x: float):
        # Complex logic to adjust markers based on parabolic interpolation.
        # This allows us to estimate the 99.9th percentile with high accuracy
        # after observing N samples, without storing N samples.
        # This function must be allocation-free to run in the hot path.
        pass

The output of this estimator should be fed directly into the autotrader’s logic. If Q_0.999 > Threshold, the strategy should automatically widen spreads or reduce size, acknowledging that its reaction time has degraded. This turns telemetry from a passive debugging tool into an active risk control.

Point 3 — Consistency domains without global locks

The third pillar addresses the torn read problem.

Imagine a book update arrives:

1. Bid Price: 100.05

2. Ask Price: 100.07

The strategy thread reads the Bid (100.05). Before it reads the Ask, a new update arrives:

1. Bid Price: 100.06

2. Ask Price: 100.08

If the strategy reads the new Ask (100.08) combined with the old Bid (100.05), it sees a spread of 0.03 (100.08 - 100.05). But the real spread was always 0.02. If the strategy’s logic is “Buy if spread > 0.025”, this phantom spread triggers a trade that immediately loses money. This is a Heisenbug that disappears when you add logging (because logging slows the writer down enough to avoid the race).

To prevent this without locking the writer (which stalls the feed handler), we use a Sequence Lock (Seqlock). A standard Mutex would force the writer to wait for readers, which is unacceptable for a feed handler that must process millions of messages per second.

from multiprocessing import Value
import ctypes

class SharedSnapshot:
    def write(self, values):
        s = self.seq.value + 1
        self.seq.value = s       # Odd = Locked / Dirty
        # Implicit memory barrier needed here in C++ (std::memory_order_release)
        self.arr[:] = values     # Bulk copy
        # Implicit memory barrier needed here in C++ (std::memory_order_release)
        self.seq.value = s + 1   # Even = Unlocked / Clean

    def read(self, out):
        s1 = self.seq.value
        if s1 & 1: return False  # Collision: Writer is currently writing
        
        # Optimistic read: assume no one is writing
        out[:] = self.arr        
        
        s2 = self.seq.value
        # If s1 != s2, it means the writer started writing while we were reading.
        # The snapshot in 'out' is corrupted.
        return s1 == s2          # Verification

This defines a consistency domain. All fields inside arr are guaranteed to be from the same logical timestamp if read() returns True. This effectively eliminates a class of synchronization errors where the strategy acts on state combinations that never existed in reality. It shifts the cost of concurrency from the writer (who must never block) to the reader (who must occasionally retry), which aligns perfectly with the priorities of a trading system.

Point 4 — Schema and layout hashing

Read more