Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

What changed this month

I have decided to introduce a variation in the MarketOps section. From now on, I will share two of my portfolios with you.

One based on monthly rotation using momentum stocks, and another based on mean reversion, which I will share whenever stocks meet the governing criteria for entry.

Each update combines long-term performance context with the portfolios’ current positioning, recent activity and next scheduled actions.

The mean-reversion model currently has four next-session signals in WDC, AMD, GOOGL and GOOG. Its report tracks active and expired signals, completed trades, entry levels, targets, exit deadlines and recent net results after transaction costs.

At the latest market close, the momentum portfolio remains in a risk-on regime and holds five positions: AMAT, LITE, MU, TER and WDC. The report also shows the portfolio’s current volatility allocation, position weights, recent transactions and the schedule for the next monthly review.

Together, the two reports provide a complete view of how the portfolios are behaving across different market conditions. Momentum seeks to remain exposed to persistent relative strength, while mean reversion looks for shorter-term dislocations within a filtered Nasdaq universe.

Reports

Read more

Table of contents:

1. Introduction.

2. US–Iran escalation.

3. Shipping routes, and the global supply.

4. Regional alliances and proxy networks.

5. United States, China, Russia, and the Dollar system.

6. Macro-financial transmission across asset classes.

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

From research question to testable hypothesis

Quantitative trading research begins with market intuition, but it becomes empirically useful only when the researcher defines the instruments, market state, activation trigger, response window, expected price path, and causal mechanism. Strong hypothesis fixes these elements before testing, separates the source of the edge from its execution, and establishes the conditions under which the claim remains valid or expires.

What’s inside:

1. Narrowing the phenomenon: The researcher reduces a broad market observation to a specific inefficiency by defining the relevant instruments, environmental state, and natural time horizon.

2. Selecting instruments, states, and horizons: Assets should be close to the source of the market force, conditions should support the proposed mechanism, and the holding period should match the speed at which the effect unfolds.

3. Defining the expected effect: The hypothesis specifies the direction, shape, magnitude, duration, and decay of the anticipated response, including mean reversion, continuation, spread compression, or ranking changes.

4. Connecting price action to economic meaning: Expected returns require a causal explanation based on information diffusion, liquidity recovery, forced order flow, institutional execution, funding constraints, or another identifiable market process.

5. Defining the trade lifecycle: Entry conditions activate the claim, maintenance conditions verify that its mechanism remains present, and breakdown conditions identify when the original hypothesis has expired.

6. Separating observation from explanation: Similar chart patterns can emerge from different forces. Research must look beyond visual geometry to identify participants, incentives, liquidity constraints, venues, and transmission channels.

7. Separating idea, hypothesis, and execution: A trading idea provides intuition, the research hypothesis identifies the source of predictive value, and the execution method determines how to capture it. Each portfolio component must possess its own causal foundation.

Sample

1.92MB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

From candidate features to validated trading models

Wrapper-based feature selection evaluates predictive inputs through the exact model and validation framework intended for live deployment. Unlike statistical filters that judge variables in isolation, wrappers score candidate subsets by repeatedly training the target algorithm under purged cross-validation. This structure ties feature value directly to net economic performance, accounting for turnover, execution fees, and collinearity risks.

What’s inside:

1. Defining the wrapper paradigm: The framework wraps a search algorithm around a target model, scoring subsets on out-of-sample performance to capture joint feature interactions that univariate statistical filters miss.

2. Accounting for market frictions: Objective functions must penalize execution friction, turnover, and subset dimension, ensuring statistical predictive power translates into net strategy returns.

3. Sequential Forward Selection: Building compact feature sets from zero limits latency and operational pipeline fragility, though greedy forward steps risk missing variables that only add value in specific pairs.

4. Sequential Backward Selection: Starting with the complete feature matrix preserves joint interaction structures, but early pruning steps require regularization to manage matrix instability from collinear market inputs.

5. Exhaustive Feature Selection: Evaluating all possible subsets guarantees locating the global in-sample optimum, but exponential search spaces create massive computational costs and high selection bias under noise.

6. Sequential Floating Selection: Integrating conditional backward passes into forward search paths dynamically purges early variable additions that become redundant as complementary features enter the model.

7. Recursive Feature Elimination: Pruning variables based on standardized coefficients or block-permutation scores directly leverages internal model weights, requiring grouped evaluations when features share underlying market drivers.

Sample

575KB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

From market observation to research question

The process begins with visible market behavior, but it becomes useful only when the researcher defines the phenomenon, classifies the market setting, identifies the mechanism, specifies the scope, and formulates a testable claim. Strong research moves from intuition toward measurable, comparable, and falsifiable market processes.

What’s inside:

1. Market observation. Trading ideas begin with repeated contact between price, volume, liquidity, time, events, and relative asset behavior.

2. Phenomenon classification. The researcher separates observations into single-instrument responses, delayed adjustments, event-driven reactions, and cross-sectional or relative-value behavior.

3. Reversal and continuation patterns. Price responses are studied through their lifecycle: initial pressure, dislocation, liquidity return, continuation path, or movement back toward balance.

4. Delayed adjustment. Some opportunities arise when one instrument reacts first and another follows later because of market clocks, venues, liquidity pools, or transmission delays.

5. Event-driven research. Earnings, macro releases, order-flow shocks, scheduled events, and surprise events create clean windows for studying price discovery, liquidity withdrawal, drift, dispersion, and repricing.

6. Cross-sectional and relative-value behavior. We study how dispersion, peer-group misalignment, and linked-instrument spreads reveal hidden capital migration and market friction beneath index-level returns.

7. Research question formation. A raw idea becomes research-ready when it defines the trigger, asset universe, market state, response window, expected path, and comparison benchmark.

8. Existence, conditional, and mechanism questions. Research asks whether the effect appears, when it strengthens or weakens, and why it exists through identifiable forces, transmission channels, and sustaining agents.

Sample

1.9MB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

From raw market data to scalable statistical filters

Filter-based feature selection evaluates the intrinsic statistical properties of candidate variables independent of downstream model training. By screening out noisy inputs before fitting complex algorithms, statistical filters reduce dimensional overreach, limit execution friction, and protect systematic strategies from temporal leakage.

What’s inside:

1. Defining the filter paradigm: Filters evaluate features through direct statistical tests against target labels outside the model training loop, providing a fast screening layer against high-dimensional noise.

2. Controlling selection risks: Ranking variables on statistical scores can mislead researchers if a feature increases turnover, inflates execution costs, or relies on unavailable forward bar data.

3. Measuring Information Gain: Discretizing continuous metrics into quantile bins isolates entropy reduction, revealing non-linear regime thresholds that standard linear filters overlook.

4. Class separation with Fisher Score and ANOVA: Comparing between-class variance against within-class dispersion highlights features with distinct conditional distributions, while Welch’s ANOVA corrects for fat tails and non-constant variance.

5. Capturing non-linear signals through Mutual Information: Applying continuous nearest-neighbor estimators detects non-linear dependencies, preserving valuable predictive signals such as U-shaped volume reversals.

6. Uncovering joint dependencies with ReliefF: Evaluating instances against their nearest hits and misses in multi-dimensional space isolates localized feature interactions without training a full model.

7. Preventing collinearity with mRMR: Optimizing target relevance against pairwise mutual information keeps feature sets diverse and stops models from clustering around identical risk drivers.

Sample

386KB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

INDEX

Quality standards of a strong trading hypothesis

A strong trading hypothesis turns a market idea into an research object. It defines what is claimed, where the claim applies, how the effect appears in data, and how evidence improves the hypothesis over time. Precision, measurability, and falsifiability give the hypothesis structure and make the research process coherent, testable, and repeatable.

What’s inside:

1. Precision. A strong hypothesis defines the exact claim, the market setting, and the expected response.

2. Precise wording. Clear wording transforms broad ideas such as pressure, stress, momentum, or dislocation into specific market processes.

3. Precise scope. The hypothesis identifies the asset universe, market state, event type, time horizon, and environment where the effect belongs.

4. Precise outcome definition. The research defines the empirical object used to judge the effect, such as return, spread compression, rank shift, abnormal movement, path behavior, or persistence.

5. Measurability. The hypothesis enters the data through observable variables, classifiable states, and outcomes that can be scored consistently.

6. Observable variables and classifiable states. Market concepts such as crowding, hidden demand, liquidity pressure, or absorption become defensible proxies and coherent state definitions.

7. Falsifiability. The hypothesis exposes the claim to evidence by defining the conditions that weaken, contradict, or challenge the proposed mechanism.

8. Revision after contradiction. Evidence improves the hypothesis by narrowing the trigger, refining the state filter, adjusting the horizon, improving the outcome, or replacing the mechanism when needed.

Sample

1.92MB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

From market endpoints to research-ready data

Quantitative trading strategies depend on the historical records that feed them. Acquiring free market data through custom web scrapers lowers upfront capital costs and expands research flexibility, but it introduces operational risks like protocol shifts, rate limits, and missing fields. Building a transparent pipeline around raw WebSocket connections ensures data transformations remain visible and verifiable before strategy testing begins.

What’s inside:

1. Data trade-offs: Free feeds provide rapid experimentation and broad market access, but lack the service-level guarantees, corporate-action histories, and delisting data typical of institutional vendors.

2. Managing protocol fragility: Relying on unofficial endpoints exposes pipelines to breakage when vendors adjust message headers, change packet envelopes, or enforce stricter rate limits.

3. Maintaining a tight dependency envelope: Constructing sockets and HTTP requests using standard library modules keeps data transformations visible and prevents third-party wrappers from hiding parsing errors.

4. Handling raw WebSocket transport: Direct management of TCP connections, TLS handshakes, frame masking, and ping-pong liveness maintains link stability across large multi-symbol requests.

5. Parsing TradingView message envelopes: Reading length-prefixed packet markers prevents buffer corruption, while explicit session and adjustment flags document whether prices reflect raw or split-adjusted trades.

6. Enforcing structured schema validation: Formatting raw bars into typed NumPy arrays preserves missing volume as unallocated values rather than zero, protecting liquidity filters from distortion.

7. Harvesting and de-duplicating universes: Paginating scanner endpoints allows researchers to build multi-asset ticker lists, though static current-day universes carry survivorship bias that requires explicit research controls.

Sample

429KB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

INDEX

Internal anatomy of a good hypothesis

A genuine hypothesis has an internal structure. It defines when the claim begins, what market response should appear, and why that response should emerge. Its value comes from connecting activation, response, mechanism, and evidence into one coherent research object that can be tested, challenged, refined, and extended.

What’s inside:

1. Internal anatomy of a good hypothesis. The lecture presents a hypothesis as an organized structure whose parts must fit together.

2. The condition that activates the claim. A hypothesis begins by identifying the market state in which the effect should appear.

3. Trigger variable. The trigger translates pressure, dislocation, surprise, imbalance, or regime transition into a variable that can select relevant episodes.

4. State filter. The state filter places the trigger inside the right market environment.

5. Eligibility rule. The eligibility rule protects the empirical sample and decides which observations belong to the same research object by controlling for trading session, liquidity, data quality, structural breaks, and special distortions.

6. The response that defines the effect. A hypothesis must specify what should happen after activation.

7. Sign, magnitude, and timing. Sign defines direction, such as continuation, reversal, convergence, or divergence. Magnitude defines economic substance after costs and frictions. Timing defines the internal path of adjustment after the trigger appears.

8. The mechanism that explains the effect. The final part explains why the trigger should lead to the response.

Sample

1.96MB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking below.

INDEX

From raw text streams to market trading signals

Text preprocessing in quantitative trading acts as an active signal-shaping operator rather than a passive data-cleaning routine. Splitting tokens, stripping punctuation, or deleting words directly alters the geometry of the feature space, changing the conditional expectation of forward returns and impacting strategy execution.

What’s inside:

1. Evaluating preprocessing as a signal operator: Preprocessing logic directly modifies input vectors, meaning every string transformation parameterizes the feature space and dictates what the classifier can observe.

2. Preventing semantic and structural failure modes: Unaligned text rules introduce structural risks, including semantic inversion from deleted negations, unstable feature representations across vendor feeds, and collapsed entity definitions.

3. Preserving entity meaning through canonicalization: Universal lowercasing acts as a destructive projection operator, merging distinct ticker symbols, corporate entities, and ordinary nouns into identical feature coordinates.

4. Defining typed token boundaries: Protected regular expressions preserve specialized financial spans, keeping currency pairs, percentages, and execution log values intact before standard tokenizers shatter them.

5. Protecting logical sign with targeted stopword filters: Retaining direction-governing terms like “not,” “under,” and “without” prevents semantic inversion in sentiment classification and stops false trade signals.

6. Validating morphological reduction under economic loss: Naive stemming merges distinct temporal and probabilistic risk states, making it necessary to distill contextual language model insights into deterministic lookup tables.

7. Combining multi-channel sparse and dense representations: Ensembling deterministic sparse vectors with dense contextual embeddings balances explicit, auditable feature attribution against abstract semantic interpretation.

Sample

759KB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

INDEX

What a trading hypothesis is not

The lecture explains that charts, indicators, and backtests can all be useful, but none of them are complete hypotheses on their own. Their value appears only when they are connected to mechanism, scope, timing, variable design, and evidence, so the researcher can move from surface regularity to a claim that can be tested, challenged, and refined.

What’s inside:

1. What a trading hypothesis is not. The lecture begins by separating a genuine hypothesis from three incomplete objects (a chart pattern, an indicator, and a backtest result).

2. A chart pattern without explanatory content. A visual formation may reveal tension, compression, rupture, or recovery, but it remains only an empirical clue until the researcher explains what market process could have produced it.

3. Visual regularity without mechanism. Repeated shapes on a chart can create the impression of structure, but research begins only when those shapes are translated into participants, frictions, information, liquidity, or structural relations.

4. Repetition without causal interpretation. A pattern that appears many times does not automatically become a hypothesis. Repetition gains research value only when it is linked to a stable cause and a clear condition under which it should persist or disappear.

5. Description without research value. Statements such as “price tends to bounce after strong declines” are useful starting observations, but they need scope, variables, comparison design, and evidence criteria before they become researchable.

6. An indicator without market logic. Indicators compress data into signals, thresholds, scores, or ratios, but they matter scientifically only when they represent a real market condition such as crowding, strain, dislocation, liquidity withdrawal, or delayed adjustment.

7. A backtest result without a prior claim. A good historical result is not enough. The lecture emphasizes that a backtest should examine a claim stated in advance, rather than becoming a machine for discovering attractive performance and inventing explanations afterward.

8. The importance of forward logic. Strong research moves from observation to mechanism, from mechanism to formal claim, and from claim to evidence. This order gives meaning to both success and failure, allowing the researcher to learn from the market rather than simply collect historical artifacts.

Sample

1.9MB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

INDEX

From raw price paths to structurally aligned features

Data-shape transformation represents the first structural choice in model construction rather than a routine cleanup step. Every transformation operator selects specific variations in a market path while suppressing others, determining whether a model observes genuine economic signals or sample-specific noise. Aligning these operators with the strategy’s target holding period prevents false statistical stability and preserves the underlying alpha.

What’s inside:

1. Reframing data transformations: Treating transformations as active structural choices ensures the mathematical shape of the input feature matches the economic mechanism behind the trade.

2. Recognizing transformation risks: Indiscriminate differencing or scaling can amputate trend signals, introduce causality leakage through full-sample parameters, or distort high-dimensional distance geometry.

3. Extracting state-space surprises: Applying local linear trend models yields one-step-ahead forecast errors, isolating orthogonal price surprises from predictable local drift.

4. Neutralizing dynamic benchmarks: Rolling regression residuals project asset returns into the null space of a benchmark, stripping out market beta to expose pure relative-value dislocations.

5. Eliminating stochastic drift via cointegration: Engle-Granger equilibrium spreads remove shared unit-root trends between paired assets, producing stationary inputs designed for mean-reversion models.

6. Isolating multiscale wavelet details: Discrete Haar wavelet filter banks decouple overlapping frequency bands, targeting scale-specific shocks while filtering out macro drift and microstructural noise.

7. Encoding bounded state ranges: Rolling min-max positions compress nominal prices into a [0, 1] interval, removing cross-sectional scale disparities and allowing tree algorithms to process boundary rejections.

Sample

450KB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Why a trading hypothesis?

A trading hypothesis is presented as the first scientific form of a market idea, one that transforms intuition into a structured claim with an activating condition, an expected response, a time horizon, and a proposed mechanism. The lecture develops this framework as the bridge between market logic and empirical evidence, and explains how restriction, precision, and coherence turn a vague observation into a research object that can be tested, challenged, and refined.

What’s inside:

1. Why a trading hypothesis matters. A trading hypothesis is the first formal research object in quantitative trading because it gives direction, causal meaning, and empirical consequences to an otherwise informal market observation.

2. Hypothesis as a conditional claim about market behavior. The lecture defines a genuine hypothesis as a structured conditional statement that specifies when a market condition appears, what response should follow, over what horizon, and because of what underlying process.

3. The activating condition. A hypothesis begins with a trigger that identifies when the market enters a meaningful state, such as a volatility expansion, spread dislocation, order flow imbalance, or liquidity shock, so the sample becomes selective in a principled way.

4. The expected market response. The claim must state what should happen after activation by defining the object of movement, the direction of adjustment, and the scale or relevance of that movement, so the idea acquires a clear empirical identity.

5. The horizon over which the response should emerge. Time is treated as part of the hypothesis itself because different mechanisms operate on different clocks, and the validity of the research depends on matching the response window to the real rhythm of the market process.

6. A bridge between market logic and empirical evidence. The lecture shows that a good hypothesis connects two domains at once, the logic of participants, frictions, and structural forces on one side, and the language of variables, samples, outcomes, and comparisons on the other.

7. Empirical signature and supporting evidence. A strong hypothesis does not predict profitability alone, but a specific patterned trace in the data, including direction, structure, timing, conditional strength, and comparative contrast, so support comes from alignment rather than from one backtest statistic.

8. The hypothesis as a restricted object. The lecture explains that every valid hypothesis must be restricted to a coherent asset universe, a defined market state, and a proper time horizon, because restriction is what gives the claim scientific shape and protects it from vague breadth.

Sample

1.87MB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

INDEX

From raw market series to stationary model inputs

Raw price series carry unbounded drift, non-constant variance, and heavy tails that break numerical optimizers through singular matrices and vanishing gradients. Effective time series preparation replaces naive preprocessing with targeted mathematical operators that make market structure legible to learning algorithms without destroying predictive memory or leaking future data.

What’s inside:

1. Controlling transformation risks: Standard Z-scores distort skewed downside risks, hard Winsorization erases capitulation signals, and global scaling leaks future variance into historical training sets.

2. Preserving long-range memory: Integer differencing (d=1) discards over 60% of predictive memory, whereas fractional differencing uses exponentially decaying weights to achieve stationarity while retaining historical path memory.

3. Mapping fat-tailed distributions: Converting skewed returns through Cornish-Fisher expansions or rank-to-Gaussian transforms prevents optimizers from treating real market tail shocks as mathematically impossible anomalies.

4. Sampling on information time: Chronological time bars generate zero-inflated distributions and intraday volatility spikes, while Dollar Bars sample data on dynamic fiat thresholds to produce more stable return properties.

5. Eliminating volatility ghost effects: Standardizing returns with GARCH(1,1) conditional volatility adjusts for autoregressive variance clustering, avoiding the abrupt artificial drops caused by flat rolling lookback windows.

6. Mitigating noise and multicollinearity: Combining Shannon entropy soft-clipping with Marchenko-Pastur PCA thresholding compresses extreme shocks without losing rank order and strips out correlated noise vectors.

7. Reconstructing phase-space geometry: Applying Takens’ Delay Embedding Theorem lifts 1D price series into multi-dimensional phase space, exposing hidden market regime structures to spatial classifiers.

Sample

650KB ∙ PDF file

Download

Download

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

INDEX

Sufficient statistics and minimal signals

This lecture defines a trading signal through sufficiency rather than through a rule, threshold, or pattern. A signal is a compression of the decision-time information set that preserves the conditional law of the declared outcome, so predictive meaning remains tied to the future law rather than to a single backtest metric. The lecture then develops minimality, bottleneck design, and representation stability as the conditions that make that compressed state usable under finite data, drift, and noise.

What’s inside:

1. Sufficiency as the definition of signal. A signal is a representation that preserves the conditional distribution of the outcome, which separates signal from strategy and makes feature relevance a property of conditional-law preservation rather than raw correlation.

2. Signal as compression with law preservation. Compression becomes signal construction when the reduced state carries the same predictive content as the full decision-time information set, which lets the system operate on a smaller and more stable representation without changing the law it conditions on.

3. Approximate sufficiency in practice. Exact sufficiency appears as the ideal case, while practical sufficiency is defined through a declared tolerance using divergence, proper-score regret, and the residual predictive content that remains outside the bottleneck.

4. Minimal sufficient sets. The lecture shows that a minimal signal is the smallest representation that still preserves the full conditional law, removes nuisance variation, lowers selection pressure, and keeps the signal portable across operators and model classes.

5. Non-uniqueness of sufficient representations. Many different representations can preserve the same predictive law, so the scientific question is whether preservation exists, while the engineering question is which representation remains most stable under noise, drift, and deployment constraints.

6. Bottleneck trade-offs. Bottleneck design becomes an information-budget problem that balances relevance against representational capacity, because too little compression increases variance and selection pressure, while too much compression removes predictive fragments the outcome still needs.

7. Overcompression and undercompression. The lecture explains that overcompression distorts the conditional law by blending distinct regimes, while undercompression creates a high-capacity overfitting surface where sharp in-sample structure often fails under recurrence and proper scoring.

8. Representation stability. A minimal signal must retain stable meaning under measurement noise, drift in state occupancy, and regime mixtures, so conditional beliefs, calibration, and divergence structure remain coherent when the signal moves from research into live deployment.

Sample

1.93MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. The API layer.

3. Protocol semantics, transport choice, and system state.

4. Risk and financial APIs limitations.

5. Historical-live non-equivalence and data reconstruction error.

6. Execution state machines and order routing.

7. Simulation and live trading.

8. Creating the architecture for a trading API.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Before moving on to the next series, there’s an important point that any algorithmic trader needs to know: the point about APIs.

A quant can state the issue in simple terms. Let s_t​ denote latent market state and let x_t​ denote observed state at decision time. The strategy trades on x_t​ rather than s_t​. The transformation from s_t​ to x_t​ includes vendor capture, transport, buffering, schema validation, clock alignment, and local state reconstruction. When that transformation changes, the strategy changes with it, even if the signal formula stays fixed.

The API layer belongs to the hypothesis itself because it determines the admissible information set on which the strategy acts.

This point matters because the gap between research and deployment tends to emerge at the boundary where information becomes software state. Bars in a notebook arrive sorted and complete. Live messages arrive through sessions, quotes, reconnect paths, and transport rules. Orders in a backtest pass from signal to fill in one step. Orders in production pass through acceptance, routing, queue interaction, etc. In a systematic process, those choices carry mathematical weight because they alter the state space on which the strategy operates.

The live strategy therefore admits a fuller description than the research signal alone. A useful expression is

Many teams place their energy on features, models, and optimization while the interface layer receives less formal treatment. That habit carries a cost. A signal derived from one information regime and deployed through another regime becomes a different object. A simulator that treats execution as an atomic event studies a market process distinct from the one live code encounters.

The API layer is where these fractures surface. It is also where discipline can restore coherence. If you want a stack to start with, here it is:

Algorithmic Trading in the Global FX Market

176KB ∙ PDF file

Download

Download

An API is an information contract, a timing contract, and an execution contract. That’s to say, data fields need causal meaning, timestamps need clear authority, order states need legal transitions, internal services need shared schemas across research, simulation, and live trading. Do you get the ida right? Under those conditions, performance claims acquire stronger footing because the path from observation to action remains explicit.

The subject here is the form through which market enters the strategy and the form through which strategy intent reaches the market. Code style, framework choice, and endpoint count matter through their effect on that form. A live trading stack earns coherence when the objects that move across it stay explicit, typed, and stable across the full path from research to execution. That coherence is a design choice and it begins at the API layer.

The API layer

In a trading system, the central question asks what is known, when it becomes known, and which representation carries it. The API layer lives at that boundary. It defines the admissible information set available to the strategy at decision time. That framing captures the practical source of failure with precision.

Let the latent market state at event index t be s_t. The strategy never accesses that state in raw form. It receives an encoded view x_t, produced through venue generation, vendor capture, normalization, transport, buffering, decoding, and local deserialization. If those operations are compressed into a single operator H, then the strategy acts on

In live systems, H belongs to a family of transformations shaped by endpoint choice, session state, codec, throttling behavior, aggregation policy, and recovery logic. The research question therefore asks two things at once: whether a mapping f(x_t) carries predictive value, and whether the x_t used in research belongs to the same family as the x_t available in deployment.

A useful decomposition is

The term H_t(s_0:t) shows that observed state may depend on a path rather than a single contemporaneous latent state. Corrections, delayed packets, local aggregation, and dropped messages create that path dependence. The residual term ε_t collects the user-facing uncertainty generated by packet loss, schema surprises, decoder failure, clock mismatch, and hidden vendor behavior. In a calm hourly-bar strategy, ε_t may occupy a small role. In a short-horizon or execution-sensitive strategy, it often defines the core challenge.

This perspective dissolves a common misconception. Many quants treat the API as a latency channel and connect latency risk with high-frequency trading. A richer view sees the API as a state-visibility mechanism. Consider a breakout rule triggered when price crosses a level and volume in the same interval exceeds a threshold. One endpoint may publish aggregated bars with finalized volume, while another may publish incremental trades that require local accumulation. The rule then sees different objects.

Speed alone does not resolve that difference. One representation is final. The other is provisional. One is bounded by bar closure. The other evolves through the interval. If research evaluates the final object while deployment trades the provisional one, then backtest and live trading inhabit different information worlds.

The official protocol and framework documentation supports this view from the software side. HTTP semantics are defined as stateless at the application-protocol level. That property makes HTTP well suited for resource retrieval, explicit commands, and idempotent inspection, while state continuity moves into higher layers. The OpenAPI specification makes HTTP service capabilities machine-readable through paths, parameters, schemas, and responses. FastAPI builds on that structure by generating OpenAPI descriptions and interactive documentation from Python declarations. Pydantic treats typed data models as executable contracts and emits JSON Schema from those models. Finance gives special weight to this alignment because state integrity sits at the center of the problem.

For a systematic trading system, the internal data model becomes a compact statement of admissible knowledge. A well-defined Bar object can enforce monotonic timestamps, nonnegative volume, high-low consistency, a declared timezone, and provenance metadata. A Signal object can enforce a causal timestamp, a horizon, a generation identifier, and a confidence field interpreted consistently by the risk layer. An OrderIntent object can separate intent from broker acknowledgment, so the system preserves a clean distinction between desired action, accepted action, and executed action. These structures carry operational value because they keep portfolio state, execution state, and research state aligned across time.

Subscribe now

Protocol semantics, transport choice, and system state

A familiar hierarchy pairs REST with slower interactions, WebSocket with streaming interactions, and FIX with institutional workflows. A stronger formulation sees each protocol as an organizer of interaction and, through that role, as a designer of local architecture.

Start with HTTP. Under RFC 9110, HTTP defines resource-oriented request-response semantics, methods, status codes, headers, and representations. It serves clear operations with strong structure: fetching historical bars, querying account state, placing an order, canceling an order, requesting configuration, and obtaining a health report. It also supports documentation with high clarity.

For internal trading services, that structure has high value. The stack gains clear contracts, reproducible input schemas, and rapid client generation.

HTTP also channels interaction through discrete exchanges. Applications that consume a continuous stream of trade events, order updates, heartbeats, or PnL deltas benefit from WebSocket. RFC 6455 defines a bidirectional protocol over TCP with an opening handshake and framed messages, built for two-way communication.

For example, Alpaca’s streaming documentation—I mean, the broker—presents trade, account, and order updates over WebSocket and highlights frame types across endpoints, plus authorization and stream subscription after connection. For a quant system, the implication is clear: WebSocket provides continuity, and continuity assigns responsibility. A persistent connection requires reconnect handling, duplicate control, interval recovery, subscription replay, sequence validation, and local buffering.

{
  "action": "auth",
  "key": "{YOUR_API_KEY_ID}",
  "secret": "{YOUR_API_SECRET_KEY}"
}

FIX addresses a different problem. The FIX Trading Community describes FIX as:

A standardized transaction language across the securities trade cycle, with message types aligned to steps such as quote request and new order, and with standardized fields as the building blocks.

That matters because institutional trading depends on precise business semantics across a workflow that spans indication, order, execution report, allocation, and post-trade reporting. FIX draws strength from a protocol that already speaks the language of the transaction. A modern retail or semi-professional quant stack may begin on other connectivity paths, yet it can still absorb the lesson: trading systems gain stability when message meanings stay explicit, durable, and tied to the lifecycle they govern.

The mathematics of transport stays simple. Let T_total denote end-to-end delay from the moment a signal becomes ready to the moment a fill event reaches local state. Then

A robust analysis evaluates the whole sum. In many implementations, T_broker and T_reconcile generate more variation than raw network travel time. Serialization, authentication, request signing, broker throttling, internal queuing, and local reconciliation can dominate the tails. Tail behavior matters more than averages because losses tend to concentrate there. A mean delay of 20 ms with a 95th percentile of 60 ms may fit one strategy and damage another whose edge concentrates in short-lived bursts.

The histogram decomposes end-to-end order latency into plausible components and shows the aggregate distribution. The plot reveals that total latency is a sum of heterogeneous terms and that the tail can widen even when most components remain well behaved.

Risk and financial APIs limitation

Once the API is treated as a formal component, the next task is classification. Risk in financial APIs represents a family of failure modes that interact. A useful taxonomy separates at least seven categories: data risk, temporal risk, schema risk, session risk, throughput risk, execution risk, and governance risk.

1. Data risk concerns what is transmitted. This includes missing fields, inconsistent bar construction, delayed corrections, undocumented transformations, and ambiguity about what a timestamp refers to. Is the bar timestamp the interval open, the interval close, or publication time? Are trades condition-filtered? Are odd lots included? Is volume consolidated at the same cadence in historical and live endpoints? Vendor documentation sometimes answers these questions, sometimes not. The point is that the risk exists even when the feed looks clean. A clean feed can still describe the wrong object for the strategy.

2. Temporal risk concerns when the data is available and in which order. Clock drift between client, broker, and data vendor is the obvious part. More subtle is the distinction between event time and arrival time. A message may encode one time but arrive at another. Streaming systems can also deliver temporary disorder. If the strategy assumes a total order while the transport offers only a best-effort order plus reconnect recovery, then the local state machine must restore order or at least detect that order was broken.

3. Schema risk concerns how the data is structured. An endpoint that changes a field name, adds a nullable attribute, changes a numeric type from integer to string, or alters nesting may be perfectly legal at the API level and still break live trading. This is why type validation libraries matter.

4. Session risk is operational but critical. Authentication expiry, refresh tokens, signed requests, environment separation, and broker session exclusivity determine whether the trading path remains live. Some APIs are quite clear that trading-enabled sessions have different restrictions than generic access, including brokerage-level single-session limits.

5. Throughput risk includes rate limits, quotas, pagination, backpressure, and resource exhaustion. This category is often underestimated by quants who work first in notebooks. Yet the feasible research surface is shaped by what the API allows. If a cross-sectional study requires more requests per minute than the provider permits, then the signal either needs a different collection design, a local cache, a licensed feed, or a narrower universe. Throughput is a constraint on the set of admissible strategies.

6. Execution risk concerns the gap between request acceptance and actual market interaction. Here a good example of why this matters: “the order object has identifiers, statuses, and a lifecycle that can be queried after placement. That is useful, but it also confirms the obvious point that the initial placement call is only one transition in a larger process. SEC routing and execution disclosures exist because the path from broker to venue materially affects outcomes”. In trading, the request path and the execution path should never be conflated.

7. Governance risk covers versioning, deprecation, environment drift, and organizational ambiguity. An internal API that is undocumented, lacks version tags, or mixes research-only fields with live-only fields can corrupt an otherwise sound strategy. Governance is what keeps the contract stable long enough for experiments to be comparable.

A convenient formalization is to model observed system risk as the union of these categories:

Each component can be associated with observables: gap counts, duplicate counts, validation failures, authentication renewals, request rejections, order rejects, and contract-version mismatches. The important point is that API risk becomes measurable only after categories are explicit.

The table below compresses this pitfalls:

Throughput deserves one extra comment because it changes research conclusions. Assume that trade opportunities survive for a characteristic time constant τ and that the system polls every Δ units of time rather than consuming a push stream. A rough first-order miss model is:

This formula is for illustrative purposes. It says that when opportunity duration is short relative to the observation interval, miss probability rises quickly. Coarser polling changes the distribution of observed opportunities. The picture below visualizes that relationship:

Quants often frame API problems as reliability problems. They are also selection problems. The feed, the protocol, and the quota choose which events you ever get the chance to model. That means a strategy discovered under one data-access regime may not even be defined under another. Once this is understood, the response becomes sharper.

Historical-live non-equivalence and data reconstruction error

Historical endpoints and live endpoints are seldom equivalent in the strict sense required by causal research. They may deliver related objects, but related is not enough. Historical data is often cleaned, deduplicated, repaired, aligned to bar boundaries, and packaged into stable records. Live data is incremental, provisional, and susceptible to temporary disorder. A strategy that learns on one and trades on the other inherits a reconstruction problem.

There are at least four sources of non-equivalence:

1. The first is finalization: Historical bars tend to be final objects. Live bars, unless the provider specifically publishes only completed intervals, may be in progress.

2. The second is correction policy: Historical vendors can revise prior records after exchange corrections or internal data-quality routines. Live systems receive the pre-correction path and may or may not later receive explicit correction messages.

3. The third is aggregation method: A historical endpoint may aggregate on server-side rules that differ from a client’s local reconstruction from trade ticks.

4. The last one is delivery semantics: Historical data is fetched as a completed array. Live data is delivered as a stream in which messages can be delayed, duplicated, or arrive after reconnect.

Any research process should ask a simple but uncomfortable question: is the historical object used in research reproducible from the live stream using only information available at the time? If the answer is no, then the strategy is at risk of hidden lookahead or deployment drift. Sometimes the mismatch is harmless. Often it is not. Opening-range logic, event-triggered strategies, and execution-sensitive systems are especially vulnerable because they depend on exact temporal boundaries.

One clean way to formalize the problem is to define a reconstruction operator G that maps a live stream path into a bar or state record:

The historical dataset supplies b_t^hist. The research assumption that often goes unstated is that b_t^hist = b_t^live_recon. This is rare. The true object of concern is the reconstruction error:

A strategy that is stable under small δ_t may survive deployment. A strategy that changes sign, entry timing, or position size under small δ_t may lose money even if the backtest looks good. This is one reason why mature groups insist on replay tests. They record live messages, reconstruct the internal state offline, and compare it with both the production decision log and the historical vendor view.

Out-of-order delivery is a particularly sharp example. Suppose messages carry event timestamps e_i but arrive at times a_i. A client that sorts by arrival time instead of event time assumes that sign(a_i - a_j) equals sign(e_i - e_j) for all pairs. That assumption is false in many real networks. Once it breaks, local bars, rolling indicators, or event-triggered conditions may shift. The correct response is to build reordering logic with bounded buffers and explicit policy for what to do when order cannot be restored with confidence.

Here is a minimal sketch of such a guard:

from dataclasses import dataclass
from heapq import heappush, heappop


@dataclass(order=True)
class MarketEvent:
    seq: int
    ts_event_ns: int
    payload: dict


class ReorderBuffer:
    def __init__(self, max_gap: int = 32):
        self.expected = 0
        self.max_gap = max_gap
        self.heap = []

    def push(self, event: MarketEvent):
        if event.seq < self.expected:
            return []  # duplicate or stale replay

        heappush(self.heap, event)
        out = []

        while self.heap and self.heap[0].seq == self.expected:
            out.append(heappop(self.heap))
            self.expected += 1

        if self.heap and self.heap[0].seq - self.expected > self.max_gap:
            raise RuntimeError("sequence gap too large")

        return out

The point of the snippet is to show the shape of the invariant. Local state should advance only when sequence integrity is acceptable. If the gap exceeds a declared tolerance, the client should raise an incident, resubscribe, or rebuild state from a safer source.

The next plot shows the operational consequence of delayed observations on a simple synthetic crossover-style entry. The delayed observation sees the same price path later and enters later. The lesson is that even modest observation delay can transform entry timing in a way that a clean historical backtest never exposed.

A common response is to smooth the signal until these issues disappear. That can help, but it does not solve the identification problem. If the strategy only works when final corrected bars are used, then the strategy depends on a state that may not exist in live trading.

First, define decision-time objects. Second, test historical-live parity by replaying captured live streams into the same internal schemas used in production. Third, store provenance metadata with every derived bar or indicator: source, reconstruction mode, timezone, sequence completeness, and whether the object is provisional or finalized. Fourth, make non-equivalence visible in research by introducing uncertainty envelopes. If a signal flips under plausible reconstruction perturbations, that signal should not be promoted.

Execution state machines and order routing

The industry still tolerates expressions like: “send the order”, “the order is in”, “the order got filled”. For software, these phrases hide critical distinctions. For PnL, they hide money. However, execution should be modeled as a state machine.

A request to place an order is only a statement of intent until the broker acknowledges it. The acknowledgment is not yet a fill or the fill may be partial. The remaining quantity may rest, cancel, expire, or be modified by subsequent logic. Some APIs reflect these transitions in a REST resource that can be queried after placement; others also emit asynchronous trade or order updates over a stream; sometimes users place, monitor, and cancel orders through the Trading API, and that order identifiers and status objects support later inspection. The best approach is to maintain two separate layers: account and order routing. That separation is healthy because it prevents one channel from pretending to be the whole lifecycle.

Okay, let’s talk about the second one: routing. SEC rules on order execution and order routing disclosure exist because the path an order takes after leaving the customer-facing system affects economic results. A retail API client sees only part of the venue-level journey, and the system designer should treat that journey as relevant. Even if the internal API abstracts away venue detail, it should preserve enough state to distinguish user intent, broker acceptance, downstream routing outcome, and final position reconciliation.

A minimal lifecycle can be written as:

with additional branches for rejected, canceled, expired, and replaced. Not every broker surfaces each branch in the same way. The internal system should anyway. Doing so has three advantages. First, it prevents false certainty. Second, it enables reconciliation. Third, it supports simulation. A sound backtest represents the full state machine between signal and filled position, because those intermediate transitions reveal failure modes that materially shape PnL.

Consider expected execution price. A crude but useful decomposition is:

Here P_ref is a chosen reference price, S_queue captures queue-position effects and spread interaction, S_latency captures adverse movement while the order is in flight, S_impact captures self-induced price movement for non-negligible size, and S_fees captures commissions, exchange fees, rebates, and financing where relevant. Many prototype systems focus on S_latency and ignore the rest. That is tolerable for a toy simulator, but not for a research workflow that wants to claim tradability. Even a simple simulator should at least model partial fills, expiry, and side-dependent slippage under volatility.

An internal order model should therefore separate three objects:

1. OrderIntent: what the strategy wants.

2. BrokerOrderState: what the broker has acknowledged and how it has evolved.

3. PositionLedgerEntry: what the portfolio believes happened after reconciliation.

Conflating them creates the classic bug where the strategy believes it is flat because its own intent was to exit, while the broker still shows a partially open quantity.

A minimal type sketch might look like this:

from enum import Enum
from pydantic import BaseModel, Field, field_validator
from typing import Optional


class OrderSide(str, Enum):
    BUY = "BUY"
    SELL = "SELL"


class OrderState(str, Enum):
    INTENT = "INTENT"
    ACCEPTED = "ACCEPTED"
    WORKING = "WORKING"
    PARTIALLY_FILLED = "PARTIALLY_FILLED"
    FILLED = "FILLED"
    REJECTED = "REJECTED"
    CANCELED = "CANCELED"
    EXPIRED = "EXPIRED"


class OrderIntent(BaseModel):
    client_order_id: str
    symbol: str
    side: OrderSide
    qty: float = Field(gt=0)
    limit_price: Optional[float] = Field(default=None, gt=0)


class BrokerOrderState(BaseModel):
    client_order_id: str
    broker_order_id: Optional[str] = None
    state: OrderState
    filled_qty: float = Field(ge=0)
    avg_fill_price: Optional[float] = None

    @field_validator("avg_fill_price")
    @classmethod
    def positive_price_if_present(cls, v):
        if v is not None and v <= 0:
            raise ValueError("avg_fill_price must be positive")
        return v

The exact fields can be adjusted. Once these objects are defined, live logic becomes cleaner. Reconciliation compares the latest broker state with the internal ledger. Simulation advances through legal states instead of jumping from signal to fill. Risk logic defines kill-switch conditions in terms of state divergence, not only PnL.

Quants often focus on price improvement as the hard part of execution. In many systems, the hard part is state certainty. A robust system determines with high confidence whether an order is live, whether a cancel request succeeded, and whether a partial fill changed net exposure. Alpha only matters when the process itself is reliable. The real objective is to build an execution engine whose states are clear enough that later improvements rest on solid assumptions.

This leads to prototype API design. Even at the prototyping stage, order endpoints benefit from idempotency, accept client-generated identifiers, distinguish acceptance from fill, and expose a machine-readable state model. Internal consumers rely on structured state as the source of truth. Human-readable logs still help, but they support the contract rather than define it.

The external side of the problem therefore becomes clear: financial APIs present data and orders as evolving states carried through channels with meaningful semantics. The constructive response is to assign those states formal names and transitions. Once that structure is in place, the internal API becomes the stable surface across which research, simulation, and live code remain aligned.

Send me a copy

Simulation and live trading

One script uses the same field names as the next, timezones remain consistent across services, the execution engine works with normalized order states, the backtester uses the same fill object as live trading, and the monitoring service reads position exposure from a shared contract rather than reconstructing it from logs. That is manageable at small scale and essential at medium scale. The solution is to design an internal API.

The main purpose of an internal API is contract preservation. Research, paper trading, backtesting, and live services should consume and emit the same conceptual objects even when their data sources differ. If one system produces Signal(direction, generated_at, horizon, confidence) and another expects alpha_score, entry_ts, and weight_hint, the boundary needs stronger alignment. The internal API creates that alignment by enforcing shared names, units, valid ranges, and temporal interpretation.

A sensible contract design starts with five invariants:

1. Causality: Every object that can influence a decision carries a timestamp or event key that identifies when it became admissible. This includes bars, features, signals, risk checks, and order updates. With that structure in place, replay testing becomes meaningful.

2. Provenance: A bar is more than OHLCV. It carries information about whether it came from vendor history, live reconstruction, a synthetic resampler, or a replay archive. That small field turns future debugging into straightforward engineering.

3. State separability: Data objects, signal objects, order-intent objects, broker-state objects, and portfolio-ledger objects each remain distinct. This separation keeps assumptions from leaking across modules and preserves clarity at the interfaces.

4. Versioning: Schemas evolve, and they should evolve explicitly. A schema_version field together with artifacted OpenAPI snapshots provides inexpensive and reliable protection.

5. Idempotency: Commands that change state, especially order-related commands, should support replay without accidental duplication. Client order IDs provide the standard mechanism.

These invariants can be supported with concise Pydantic models. Here is a small example:

from datetime import datetime, timezone
from pydantic import BaseModel, Field, field_validator
from typing import Literal


class Bar(BaseModel):
    schema_version: str = "1.0"
    source: Literal["vendor_history", "live_recon", "replay"]
    symbol: str
    ts_open: datetime
    ts_close: datetime
    open: float = Field(gt=0)
    high: float = Field(gt=0)
    low: float = Field(gt=0)
    close: float = Field(gt=0)
    volume: float = Field(ge=0)

    @field_validator("ts_open", "ts_close")
    @classmethod
    def enforce_timezone(cls, v: datetime):
        if v.tzinfo is None:
            raise ValueError("timestamps must be timezone-aware")
        return v.astimezone(timezone.utc)


class Signal(BaseModel):
    schema_version: str = "1.0"
    generated_at: datetime
    symbol: str
    direction: int
    horizon_s: int = Field(gt=0)
    confidence: float = Field(ge=0.0, le=1.0)
    model_tag: str

The snippet already delivers three concrete benefits. Invalid states fail early, schemas become inspectable by both humans and machines and, client libraries can be generated or written against a stable description rather than inherited through oral tradition.

The internal API also makes research and live mode comparable by design. A strong pattern places all market-data acquisition behind adapter interfaces and exposes only canonical internal models to the rest of the stack. The rest of the code then works with the same objects whether bars came from Alpaca, IBKR, a CSV archive, a replay file, or MetaTrader. Sources are not interchangeable, but their specific oddities belong at the boundary, where they can be normalized, instead of spreading through the strategy codebase.

Do you know what is premature? To build a large platform before the contracts are understood. A lean internal API can live inside a single FastAPI application with a few strict models and a small set of endpoints. What matters is that the contracts are defined before the complexity arrives.

A practical minimal endpoint set for a quant prototyping API includes /health, /bars, /signal/run, /orders/simulate, /orders/submit, /positions, /portfolio/state, and /backtest/run. Each one benefits from typed inputs and outputs, deterministic validation, and a precise statement of time semantics.

The internal API also supports replay, allowing a team to verify that the object evaluated in research matches the object the live system would have seen. An endpoint that consumes archived messages and emits canonical internal states often adds more value than a second optimization endpoint. Little by little we will be building things like this:

Creating the architecture for a trading API

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

INDEX

Calibration and scoring under uncertainty

This chapter turns an inefficiency claim into a measurable probabilistic object. A model earns credibility when its output remains a truthful conditional description of the future under the same decision-time information that defines the claim. Proper scoring rules provide that bridge. They test whether probabilities, quantiles, or densities align with realized outcomes, and they do so inside the exact state regions where the strategy intends to deploy capital.

What’s inside:

1. Scoring for conditional evidence. The chapter defines prediction as a conditional statement about the future and explains why scoring must reward truthful reporting of the conditional law rather than flattering in-sample behavior.

2. Proper scoring fundamentals. It establishes the role of strict propriety and separates probabilistic description from downstream action, so learning targets the conditional mechanism itself rather than the trade rule built on top of it.

3. Log score and likelihood geometry. Full conditional densities are evaluated through log score, which measures how much probability mass the model assigns to realized outcomes and turns distributional mismatch into an empirical cost.

4. Brier score and probability geometry. Binary event probabilities are scored through squared probability error, which makes calibration visible and tests whether reported event likelihoods match realized frequencies.

5. Pinball loss and quantile geometry. Conditional quantile boundaries are evaluated through asymmetric loss that checks whether reported boundaries retain the correct crossing behavior under the declared state and horizon.

6. Calibration under state selection. The chapter shows why a model can look acceptable on pooled data while failing in the narrow states where capital is actually deployed, so calibration must be checked on the same conditional sample that drives execution.

7. Conditional calibration. Local inefficiency claims require local validation, so probabilities and quantiles must retain their interpretation inside the same state regions where divergence suggests conditional structure.

8. Miscalibration as false inefficiency. The chapter explains how overconfident outputs can manufacture the illusion of edge when threshold-based decision rules amplify extreme signals that lack stable probabilistic meaning.

9. Decision-aware scoring. Evaluation is aligned with the architecture of risk, exposure, and constraints, so errors matter most where the portfolio carries the highest operational sensitivity.

Sample

1.94MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. What is web scraping in a quantitative context?

3. Where does it come from?

4. How web scraping actually works.

5. Risks and limitations of the protocol.

6. Screener network topography.

7. Scraping fund positions.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

In fund research, the input that matters most is simple: what stocks are inside the funds right now, and in what weight. Without that look-through layer, fund momentum, category rotation, or risk exposure becomes a label-driven proxy.

We are going to construct and execute a systematic fund scraping operation. The explicit objective of this architecture is to expose what underlying stocks these otherwise opaque mutual funds and ETFs are holding.

If you intend to calculate true portfolio variance, isolate specific factor exposures, or detect hidden beta overlap across multiple assets, you must possess the exact equity constituents and their precise percentage weightings for every fund in your cross-section. Without ticker-level resolution, you might allocate capital to two distinct Diversified Value funds that both hold massive, highly correlated positions in the exact same mega-cap technology stock.

So, we will engineer a deterministic data pipelineto get a clean, structured JSON. The architecture we are deploying will execute in distinct, logical phases:

1. We will intercept the provider’s APIs to extract the core fund metadata and historical return vectors.

2. We will utilize concurrent, threaded I/O operations to query the global holdings API.

3. We will map this extracted data into memory-contiguous C-arrays.

4. We will apply algorithmic deduplication using tokenized lexical analysis and tuple signatures to eliminate share class multicollinearity.

The final output of this operation will be a sanitized, lightweight matrix of unique portfolios, ready for immediate ingestion into our engine.

What is web scraping in a quantitative context?

To understand the architecture we build for trading, we must first define what web scraping is at a fundamental, computational level. In the broader software engineering industry, web scraping is often defined as the automated gathering of information from the internet. In quantitative finance, we require a technical definition.

Web scraping is the automated, programmatic execution of HTTP network protocols to request, receive, and extract specific data structures from external servers, translating unstructured or semi-structured web payloads into strict, localized, memory-contiguous matrices for immediate algorithmic ingestion.

To know more about scraping check that pdf:

Web Scrapping: The basics explained

236KB ∙ PDF file

Download

Download

When a quantitative researcher builds a cross-sectional momentum strategy or a fundamental mean-reversion model, the model requires historical price vectors, dividend yields, expense ratios, constituent weights, etc. Historically, this data was locked behind expensive, proprietary data terminals operated by a few institutional vendors.

Web scraping bypasses this vendor lock-in. It allows a trading desk to aggregate fragmented public data from asset managers, regulatory filings, and financial portals directly into a local array or a structured SQL database. It is the direct acquisition of the raw material required for alpha generation.

Furthermore, standard vendor data is often commoditized. If every quant receives the exact same fundamental dataset at the exact same millisecond via an identical API feed, the alpha decays. Constructing custom extraction architecture allows a quantitative team to ingest alternative data—such as retail sentiment from localized forums, unstructured XBRL data from SEC 10-Q filings, or dynamically updating ETF composition lists directly from the issuer’s backend.

Subscribe now

Where does it come from?

The methodology of data extraction has evolved in parallel with the evolution of web architecture itself. We can trace this lineage through distinct technological eras, driven by the need to parse complex data delivery mechanisms. Understanding this history is critical to understanding why modern quantitative infrastructure rejects certain parsing libraries.

Era 1: Static transfer and regular expressions

In the early days of the internet, web pages were static HTML documents stored directly on server hard drives. When a client sent an HTTP GET request, the server responded with the exact text file. Scraping in this era involved basic command-line tools executing standard network fetches.

The data was extracted using Regular Expressions (Regex). Regex operates as a finite state machine that matches specific string patterns. If a researcher wanted a stock price, they wrote a strictly defined string-matching algorithm to locate a sequence of numbers immediately following a specific HTML text block.

This was efficient computationally. A regex search over a 10-megabyte text file resolves in milliseconds. However, it was incredibly brittle from an operational standpoint. Consider the following HTML string: <td class="price">150.25</td>. A regex designed to capture the float inside the tags (r'class="price">(\d+\.\d+)<') works fine. But if the webmaster updates the website and injects a bold tag for visual styling—<td class="price"><b>150.25</b></td>—the regex logic fails. It returns a null value. In a trading pipeline, a null value propagates through the covariance matrix, zeroes out the optimization weights, and halts the entire trading execution.

Era 2: The Document Object Model and structural parsing

As web pages evolved into generated templates mapping to backend databases, relying on raw string matching became impossible. The industry shifted to structural parsing. Browsers render HTML into a hierarchical tree structure known as the Document Object Model (DOM).

Scrapers evolved to replicate this exact rendering process in memory. Python libraries, such as BeautifulSoup utilizing lxml parsers, were developed to download the HTML string, parse it into an internal tree data structure, and allow engineers to search the tree using XPath or CSS selectors. We stopped searching for text strings and started searching for specific structural nodes, such as the third cell of the second row within a specific table ID.

While more robust than Regex, structural parsing introduces severe computational overhead. Converting a 5-megabyte HTML string into a fully traversable DOM tree requires significant CPU cycles and RAM allocation. The search algorithm must execute O(N) traversal operations across thousands of nested nodes. When running high-frequency data ingestion across thousands of financial instruments, the CPU time spent building these DOM trees severely limits the concurrency capacity of the extraction server.

Era 3: Single Page Applications and JavaScript rendering

The critical inflection point occurred when web development shifted to reactive JavaScript frameworks (React, Angular, Vue). The server no longer returned HTML containing the financial data. Instead, the server returned an empty HTML shell and a massive JavaScript file. The client’s browser was expected to execute the JavaScript, which would then dynamically fetch the data and build the DOM locally on the user’s machine.

Standard HTML parsers broke; they would download the empty shell, parse it, and find zero data. To combat this, scrapers evolved into headless browsers. Infrastructure had to spin up entirely automated, invisible instances of Chromium in the background, load the page, wait for the V8 JavaScript engine to execute the rendering logic, wait for the network requests to resolve, and then extract the data from the rendered DOM.

This was disastrous. Running a single headless browser instance requires hundreds of megabytes of RAM. If a quantitative pipeline requires 100 concurrent workers to update the daily pricing matrix before the market opens, the server must allocate over 50 gigabytes of RAM strictly for browser overhead. This is inefficient and unscalable for certain types of trading systems.

Era 4: API interception

This brings us to the modern quantitative standard. Because Single Page Applications must retrieve their dynamic data from somewhere, they utilize Application Programming Interfaces (APIs)—typically REST or GraphQL architectures.

Instead of rendering a heavy headless browser, we utilize network monitoring tools to observe the traffic generated by the web application. We identify the exact API endpoint the JavaScript is querying, and we replicate that precise HTTP request from our Python architecture. We bypass the HTML entirely. We bypass the JavaScript rendering engine. We request the raw, serialized JSON data directly from the provider’s backend database. This is the optimal state of data extraction: maximum fidelity, minimum computational overhead.

How web scraping actually works

To extract data, you must understand the exact sequence of network events that occur during a programmatic request. Web scraping is the programmatic automation of the standard client-server architecture, executing at speeds unachievable by human interaction.

Step 1: The network handshake and request formulation

The process begins when our Python architecture instantiates a TCP connection. We formulate a strict HTTP request, this request consists of a specific method (GET for retrieving data, POST for submitting query parameters), the target URL, and a dictionary of Headers.

Headers are critical, they dictate the exact rules of engagement with the target server. We set the User-Agent header to declare what type of client we are. By default, Python networking libraries broadcast their standard identity (e.g., python-requests/2.31.0). Modern Web Application Firewalls flag this string and drop the TCP connection. We must inject standard browser headers to bypass basic heuristic checks.

Furthermore, we must configure advanced headers such as Accept-Encoding: gzip, deflate, br to inform the server that we can handle compressed byte streams, reducing network transit time.

Modern security also involves TLS (Transport Layer Security) fingerprinting. Servers analyze the specific cryptographic cipher suites our client proposes during the initial connection setup (the Client Hello packet). Standard Python networking libraries propose a different sequence of ciphers than a standard Google Chrome browser. If the server detects a mismatch between the User-Agent claiming to be Chrome and a TLS fingerprint matching Python, it blocks the request. Advanced quantitative architecture utilizes specialized network adapters to mimic exact browser TLS signatures, guaranteeing connection acceptance.

Before we even introduce anything, let’s start here with our practical example.

import requests

url = "https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener?page=1&pageSize=50&sortOrder=ReturnM120%20desc&outputType=json&version=1&languageId=es-ES&currencyId=EUR&universeIds=FOESP%24%24ALL&securityDataPoints=SecId%7CName%7CPriceCurrency%7CTenforeId%7CLegalName%7CClosePrice%7CYield_M12%7CCategoryName%7CAnalystRatingScale%7CStarRatingM255%7CQuantitativeRating%7CSustainabilityRank%7CReturnD1%7CReturnW1%7CReturnM1%7CReturnM3%7CReturnM6%7CReturnM0%7CReturnM12%7CReturnM36%7CReturnM60%7CReturnM120%7CFeeLevel%7CManagerTenure%7CMaxDeferredLoad%7CInitialPurchase%7CFundTNAV%7CEquityStyleBox%7CBondStyleBox%7CAverageMarketCapital%7CAverageCreditQualityCode%7CEffectiveDuration%7CMorningstarRiskM255%7CAlphaM36%7CBetaM36%7CR2M36%7CStandardDeviationM36%7CSharpeM36%7CTrackRecordExtension&filters=&term=&subUniverseId="

r = requests.get(url, timeout=20)
print("status:", r.status_code)
print("content-type:", r.headers.get("content-type"))
print("content-encoding:", r.headers.get("content-encoding"))
print("head:", r.text[:200])

Step 2: Server-side resolution and response

Upon receiving the formulated request, the target server evaluates it against its internal security protocols. It checks for IP reputation, rate limit thresholds (Token Bucket algorithms), and geographic blocking rules. If the request passes these validation checks, the server queries its internal SQL or NoSQL database, formats the result into the requested format, and transmits the payload back to our server over the open connection.

The server prefaces the payload with an HTTP Status Code. We evaluate this code before reading the data buffer. A 200 OK indicates absolute success. A 403 Forbidden indicates our IP or TLS signature was blocked by the WAF. A 429 Too Many Requests indicates we have exceeded the permissible concurrency limit. When a 429 is encountered, our infrastructure immediately halts execution on that specific thread and initiates a deterministic exponential backoff algorithm, calculating a precise wait time before re-attempting the connection.

status: 200
content-type: application/json
content-encoding: gzip
head: {"total": 49441,"page": 1,"pageSize": 50,"rows": [{"SecId": "F00000S8RT","Name": "Multipartner Konwave Gold Equity C USD","PriceCurrency": "USD","TenforeId": "52.8.LU1001014080","LegalName": "Multipar

Step 3: Payload deserialization and matrix mapping

Once the payload bytes are received locally via the network interface card, they must be transformed into operational data. In modern scraping (API interception), the response header Content-Type is typically application/json. The data arrives as a serialized, UTF-8 encoded byte string.

We execute a high-speed deserialization function. In environments prioritizing microseconds, standard libraries are replaced by C-compiled alternatives like orjson or ujson that map JSON strings to memory structures faster. The deserializer parses the string and maps the key-value pairs into native Python dictionaries and lists.

From this state, we execute our extraction logic. We iterate through the arrays, filtering out the specific financial metrics we require, and allocating them into contiguous memory blocks. We actively cast data types at this stage—ensuring strings representing numbers are cast directly to float64 objects in a structured array, preparing the data for immediate ingestion into linear algebra operations.

To demonstrate the evolutionary mechanics, observe the explicit difference in code execution between legacy DOM parsing and modern high-speed JSON deserialization:

import json
from bs4 import BeautifulSoup
import orjson
import numpy as np

# Legacy Mechanics: Structural DOM Parsing (Era 2) 
# The server returns a heavy string of markup logic mixed with data.
html_payload = """
<div class="financial-table">
    <div class="row">
        <span class="ticker">AAPL</span>
        <span class="price">150.25</span>
    </div>
</div>
"""
# We must allocate significant memory to build the entire tree structure just to find one float.
# This requires CPU cycles to parse the entire string, handle malformed tags, and build node relationships.
tree = BeautifulSoup(html_payload, "html.parser")
legacy_price = float(tree.find("span", class_="price").text)

# Modern Mechanics: API Deserialization (Era 4) 
# The server returns strictly serialized data with zero HTML rendering logic.
json_payload_bytes = b'{"data": [{"ticker": "AAPL", "price": 150.25}]}'

# We execute a direct memory mapping using a C-optimized parser (orjson).
# This bypasses standard Python object creation overhead where possible.
parsed_data = orjson.loads(json_payload_bytes)

# We extract the specific float and immediately allocate it into a NumPy array type.
modern_price = np.float64(parsed_data["data"][0]["price"])

The difference is structural and computational. Legacy scraping reads documents; modern scraping queries databases. By understanding the lineage and the strict network mechanics, we engineer scraping pipelines that don’t simulate human browsing. Instead, they operate as direct, high-speed, database-to-database communication protocols executing over the public internet.

Risks and limitations of the protocol

The execution of thousands of sequential HTTP requests introduces specific, quantifiable, and systemic risks to the trading infrastructure. These risks are guaranteed events in any large-scale network operation. They manifest as missing data rows, unhandled exceptions, TCP timeouts, and truncated JSON payloads, which directly corrupt the backtesting matrix.

The primary operational risks are categorized by standard HTTP response codes. We anticipate HTTP 429 (Too Many Requests) errors when our extraction loops exceed the provider’s token bucket algorithm limits. We expect TCP connection timeouts when network routing degrades. We will encounter 502 Bad Gateway responses during dynamic load balancer reallocations, and 503 Service Unavailable responses when the backend database cluster is executing routine locking operations or snapshots.

To mitigate these risks, we construct a session adapter using the urllib3 library. We implement a deterministic exponential backoff algorithm. When a 429 or 5xx status code is returned, the system pauses before executing a retry, reducing the concurrency pressure on the provider’s server. We target the 429, 500, 502, 503, and 504 codes. We exclude 400 (Bad Request) and 404 (Not Found) codes, as those represent structural errors in our query parameters that a retry will never resolve.

Here is the implementation of the retry adapter used in our pipeline:

import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def _make_retry_adapter() -> HTTPAdapter:
    retry = Retry(
        total=6,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=("GET",),
        raise_on_status=False,
    )
    return HTTPAdapter(max_retries=retry, pool_connections=50, pool_maxsize=50)

def _lt_session() -> requests.Session:
    # We precisely replicate standard browser headers to bypass basic WAF heuristic checks.
    LT_HEADERS = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "es-ES,es;q=0.9,en;q=0.8",
        "Referer": "[https://lt.morningstar.com/](https://lt.morningstar.com/)",
    }
    s = requests.Session()
    s.headers.update(LT_HEADERS)
    adapter = _make_retry_adapter()
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    return s

Notice the specific performance parameters initialized in the adapter: pool_connections=50 and pool_maxsize=50. This keeps 50 distinct TCP connections open in the background, bound to the Session object. By utilizing the HTTP Keep-Alive header within the requests.Session architecture, we completely bypass the severe latency introduced by repeated Transport Layer Security handshakes.

Every new HTTPS request requires DNS resolution, a 3-way TCP handshake (SYN, SYN-ACK, ACK), and an expensive TLS cryptographic key exchange. This sequence requires 150 to 300 milliseconds of network overhead before a single byte of actual data is transmitted. This is the protocol tax. If we initiate a new session for 5,000 distinct requests, we incur 25 minutes of pure dead time negotiating connections. By pooling connections and persisting them, we reduce network overhead to the strict physical transfer time of the payload itself, dropping the marginal connection time to zero.

To visualize the progression of the exponential backoff risk mitigation, consider the following plot. It models the exact delay duration sequence based on the specified backoff_factor, demonstrating how the system slows down to prevent triggering a permanent IP ban.

This is the precise moment the entire architecture upgrades. We stop looking at the rendered web page, and we start interrogating the network layer.

By opening the developer tools within a controlled Chromium environment and monitoring the Fetch/XHR network traffic generated by the web application during standard user interaction, we discover and intercept the underlying API. We isolate the exact network request executing the database search. We identify the explicit endpoint serving the raw data to the frontend UI:

https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener.

This endpoint doesn’t return complex, nested HTML documents. It accepts defined query parameters that dictate the SQL-equivalent operations on the backend: pagination offsets, sorting directives, universe limiters, and projection columns (securityDataPoints). In return, it provides a raw JSON payload.

By interfacing with this REST API, the extraction process transitions from a heuristic, error-prone text-parsing operation into a deterministic database query. We send a formalized HTTP GET request, and we receive a serialized data structure that maps into our operational memory. A float value is transmitted as a float. A string is transmitted as a string. There is no string-manipulation required to strip percentage signs or currency symbols, which eliminates an entire class of data engineering bugs.

This single event renders all traditional web scraping techniques—BeautifulSoup HTML parsing, XPath selection, and resource-heavy headless browsers like Selenium or Playwright— obsolete within our infrastructure.

Screener network topography

Having established the foundational infrastructure and intercepted the correct REST API endpoint, we now face the challenge of extracting the financial data. The provider’s backend database contains tens of thousands of mutual funds, exchange-traded funds, and institutional share classes. This dataset is vastly larger than the maximum permitted payload size for a single HTTP response.

The endpoint enforces a strict pageSize limit. This is a direct mechanism to prevent memory exhaustion on their internal database cluster. When a client requests data, the provider’s server must execute a SQL or NoSQL query, allocate a continuous block of RAM to hold the result set, serialize that data into a JSON string, and transmit it over the network interface. Requesting 10,000 rows creates a massive serialization overhead. To bypass these hard limits, we define our extraction parameter as pageSize=50.

Extracting the complete cross-sectional universe requires mapping the network topography and iterating over a paginated endpoint recursively. We must calculate the required number of requests based on the total record count returned in the initial response. We can’t hardcode the total page count, as the universe size fluctuates daily due to fund liquidations, mergers, and new product launches.

The algorithmic execution proceeds sequentially for the initialization phase. We fetch page 1. We parse the initial JSON payload to read the total integer field, which dictates the absolute size of the dataset for that specific query. We calculate the ceiling of the division of total by pageSize (math.ceil(total / page_size)). This yields the exact integer value of total pages required to complete the extraction. If the total is 0, we default to 1 to avoid ZeroDivisionError exceptions in downstream processing logic.

Besides, we pass specific query parameters in the HTTP request: universeIds=FOESP$$ALL specifies the precise fund universe routing key on the provider’s backend. We inject the parameter securityDataPoints=SecId|LegalName|CategoryName|ReturnM120 to isolate only the necessary relational columns.

We request the SecId, the LegalName, the CategoryName, and the ReturnM120 (the 120-month historical return, which is the foundational variable for long-term quantitative momentum analysis).

If we omit the securityDataPoints parameter, the API defaults to returning every available column—over 100 data points including tracking error, manager tenure, and daily standard deviation. This inflates the JSON payload from 50 kilobytes to over 5 megabytes per page. Requesting unnecessary data points saturates network bandwidth, increases TLS decryption time, and degrades the JSON parsing speed. We limit the payload to our exact algorithmic requirements.

import math
import numpy as np
import requests

LT_BASE = "[https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener](https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener)"
SECURITY_POINTS = "SecId|LegalName|CategoryName|ReturnM120"

def fetch_lt_page(session: requests.Session, page: int, page_size: int) -> dict:
    params = {
        "page": int(page),
        "pageSize": int(page_size),
        "sortOrder": "ReturnM120 desc",
        "outputType": "json",
        "version": 1,
        "languageId": "es-ES",
        "currencyId": "EUR",
        "universeIds": "FOESP$$ALL",
        "securityDataPoints": SECURITY_POINTS,
        "filters": "",
        "term": "",
        "subUniverseId": "",
    }
    # Notice the exact tuple for the timeout: (4, 20)
    # 4 seconds allocated for the TCP SYN/ACK connection handshake.
    # 20 seconds allocated for the server to process the query and return the first byte.
    r = session.get(LT_BASE, params=params, timeout=(4, 20))
    ct = (r.headers.get("content-type") or "").lower()
    if r.status_code != 200 or "json" not in ct or not r.content:
        raise RuntimeError(f"LT bad response: {r.status_code} ct={ct} head={r.text[:200]!r}")
    return r.json()

def _normalize_pages(pages, total_pages: int):
    # This logic dictates the strict operational scope of the extraction matrix.
    if pages is None:
        # None implies a full universe extraction. WARNING: Huge data pull spanning gigabytes of memory.
        return range(1, total_pages + 1)
    if isinstance(pages, (int, np.integer)):
        # Integer implies pulling the top N pages strictly based on the sortOrder parameter.
        n = int(pages)
        return range(1, min(max(n, 0), total_pages) + 1)
    # List implies pulling specific, explicit page indices, useful for error-handling retry loops.
    # We apply a set comprehension to mathematically eliminate duplicate page requests.
    lst = sorted({int(p) for p in pages if 1 <= int(p) <= total_pages})
    return lst

The pagination handler _normalize_pages is engineered to accept three distinct input states, providing the system with deterministic flexibility. If a researcher is running a lightweight daily update to assess the top decile of momentum performers, they pass an integer pages=10 to scan only the top 500 funds. If the infrastructure team is bootstrapping the initial quantitative database from a blank state, they pass None to fetch the absolute entirety of the universe. If a previous extraction loop dropped specific network packets, the error handler passes a list pages=[14, 22, 105] to specifically target the missing vectors. The use of a set comprehension {int(p) ...} guarantees that duplicate page indices are stripped before execution, preventing redundant network I/O.

We append these scalar values into localized lists. Python lists operate with O(1) amortized append time complexity because they over-allocate memory buffers under the hood. We can’t initialize the final, rigid structured array at this specific stage because we don’t yet know the exact valid row count; we filter out invalid JSON objects—such as funds missing a critical SecId—during the loop.

The primary obstacle at the screener stage is managing the sequential nature of the pagination discovery. Because we don’t possess the total integer until the very first request resolves and the JSON is parsed, the initial discovery phase is linear. However, once the total_pages parameter is acquired, the subsequent page requests become entirely stateless and independent. This sets the theoretical stage for future parallelization. Yet, for this specific screener layer, executing sequential HTTP requests over an established, persistent TLS connection pool is efficient, completing the initial metadata extraction in a matter of a few seconds, preparing the matrix for the much heavier, threaded Global API processing phase.

Send me a copy

Scraping fund positions

The screener endpoint provides the fund-level metadata, consisting of basic identifying parameters and historical return vectors. However, a fund’s surface-level metadata is insufficient for risk management and alpha generation. To construct a look-through portfolio, assess true factor exposure, or run Barra risk decomposition models, we require the exact equity holdings inside each fund. We need the underlying constituents, and we need them weighted.

This data requirement introduces a separate, more complex endpoint into our architecture: the Global API (api-global.morningstar.com). Unlike the public screener interface, this endpoint demands an explicit authentication mechanism. We must inject an apikey (it’s free) header into every request.

# 2) Direct holdings 
APIKEY = "HERE YOUR API KEY"

Furthermore, the endpoint requires the specific SecId extracted from the first phase to be formatted directly into the URL path.

The primary architectural obstacle is the sheer volume of network requests. The screener API supports bulk pagination; the Global API doesn’t. If the screener returns an array of 5,000 distinct funds, we must execute 5,000 separate, individual HTTP GET requests to the Global API to retrieve the constituent holdings for each individual fund.

A standard, sequential execution loop operates at approximately 0.5 seconds per request. This delay is a function of the physical network latency—the round-trip time between our trading servers and the provider’s data center, plus the provider’s internal database query execution time. Executing 5,000 requests sequentially requires 2,500 seconds, equating to over 40 minutes of pure execution time. This is unacceptable for a daily production pipeline that must process data, run optimizations, and generate execution orders before the market open.

We resolve this processing bottleneck through concurrent execution. We implement a ThreadPoolExecutor from the concurrent.futures module. Because HTTP network requests are I/O-bound tasks—meaning the CPU sits idle while waiting for the network interface card to receive the inbound packet from the remote server—the GIL doesn’t inhibit parallel execution in this specific scenario. The interpreter releases the GIL during standard I/O blocking operations. Therefore, we can launch 20 concurrent threads to process the requests without migrating to a complex multiprocessing architecture.

We chose thread pooling over an asynchronous event loop (like asyncio) because our existing network infrastructure relies on the requests library and custom urllib3 retry adapters. Threaded execution provides the exact same I/O concurrency benefits without requiring a complete rewrite of the underlying synchronous network stack.

import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
import numpy as np

APIKEY = "HERE YOUR API KEY"
MS_BASE = "[https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2](https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2)"
MS_HEADERS = {
    "apikey": APIKEY,
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
    "Accept": "application/json, text/plain, */*",
}

_thread_local = threading.local()

def _ms_session() -> requests.Session:
    # We must ensure strict thread safety across concurrent network operations.
    s = getattr(_thread_local, "session", None)
    if s is None:
        s = requests.Session()
        s.headers.update(MS_HEADERS)
        # Assuming _make_retry_adapter is defined as in Part 1 for exponential backoff
        # s.mount("https://", _make_retry_adapter())
        _thread_local.session = s
    return s

def fetch_holdings_us_topN(secid: str, country: str = "United States", top: int = 10):
    s = _ms_session()
    url = f"{MS_BASE}/{secid}/data"
    params = {
        "clientId": "MDC",
        "version": "4.71.0",
        "premiumNum": 10000,
        "freeNum": 10000,
    }
    r = s.get(url, params=params, timeout=(4, 25))
    if r.status_code != 200 or not r.content:
        return []
    
    data = r.json()
    page = (data.get("equityHoldingPage") or {})
    hlist = (page.get("holdingList") or [])
    
    out = []
    for h in hlist:
        c = h.get("country") or h.get("countryName")
        if c == country:
            t = h.get("ticker") or h.get("tradingSymbol")
            if t:
                out.append(t)
                if len(out) >= top:
                    break
    return out

def attach_holdings_all(secids, max_workers: int = 20, country: str = "United States", top: int = 10):
    holdings = [None] * len(secids)
    with ThreadPoolExecutor(max_workers=max_workers) as ex:
        futs = {ex.submit(fetch_holdings_us_topN, sid, country, top): i for i, sid in enumerate(secids)}
        for fut in as_completed(futs):
            i = futs[fut]
            try:
                holdings[i] = fut.result()
            except Exception:
                # We enforce strict failure handling to maintain array dimensions.
                holdings[i] = []
    return np.asarray(holdings, dtype=object)

Observe the use of threading.local(). This is an essential architectural detail that prevents systemic failure. We instantiate a unique requests.Session() object for every single active thread. When multiple concurrent connections interact with a single, shared urllib3 connection pool, race conditions occur at the socket level, leading to corrupted TCP streams and dropped connections. By isolating the session object to local thread memory storage, we guarantee that each thread maintains its own independent pool of Keep-Alive HTTP sockets. This ensures deterministic network execution.

We limit max_workers=20 as calculated threshold based on the provider’s implicit rate limits. Exceeding this thread count doesn’t yield linear performance gains; instead, it triggers the provider’s Web Application Firewall, which responds with HTTP 429 status codes, forcing our exponential backoff adapter into extended sleep cycles. Twenty workers provide the optimal equilibrium between maximum extraction speed and minimal server-side rejection.

The next part is to map the extracted data into a NumPy structured array. A structured array allocates a single, contiguous block of memory in C. We define exact byte boundaries for each field in the matrix. When the CPU executes operations on this array, it loads sequential memory blocks directly into the L1/L2 cache, executing instructions with zero pointer-chasing overhead.

We allocate a 16-character unicode string (U16) for the SecId, a 256-character unicode string (U256) for the Fund name, an 8-byte float (f8) for the return metric, and an object pointer (O) for the variable-length list of constituent stock holdings.

# Execution assembly demonstrating the structured array initialization
import numpy as np

# Assuming secids, names, rets, and stocks are already populated lists/arrays from the extraction phase
# We allocate exactly the required memory footprint before population.
arr = np.empty(len(secids), dtype=[("Id", "U16"), ("Fund", "U256"), ("ReturnM120", "f8"), ("Stocks", "O")])
arr["Id"] = np.asarray(secids, dtype="U16")
arr["Fund"] = np.asarray(names, dtype="U256")
arr["ReturnM120"] = rets
arr["Stocks"] = stocks

We acknowledge the use of the object pointer ("O") for the stocks column breaks strict C-contiguity for the holding string arrays themselves. However, this is computationally acceptable because our heavy numerical operations—the sorting and ranking logic—operate on the f8 float array, which remains contiguous and hardware-optimized.

Furthermore, we must handle missing data points at the numerical level. Web scraped data contains holes; some funds lack a 10-year track record. We temporarily replace NaN values with negative infinity (-np.inf) prior to sorting. Standard sorting functions in C don’t possess financial logic. Mathematical comparisons against NaN fail silently (NaN > 5 evaluates to False, NaN < 5 evaluates to False). If we don’t execute this specific substitution, NaN values sort unpredictably, injecting null pointers into the highest momentum deciles and breaking the downstream portfolio construction models. This substitution ensures that funds lacking complete return histories are forced to the absolute bottom of the generated matrix.

def sort_by_return_desc(arr: np.ndarray) -> np.ndarray:
    if arr.size == 0:
        return arr
    r = arr["ReturnM120"]
    
    # Isolate NaN values and push them to the lowest possible finite float boundary (-inf).
    # This prevents incomplete track records from ranking highly in momentum strategies.
    r2 = np.where(np.isfinite(r), r, -np.inf)
    
    # Execute the C-level sort on the substituted float array, and reverse it ([::-1]) for descending order.
    idx = np.argsort(r2)[::-1]
    
    # Return the newly aligned, completely contiguous memory view.
    return arr[idx]

The final, most complex important barrier in systematic fund scraping is the share class multiplicity problem. Asset managers construct a single underlying portfolio of equities, but they distribute that identical portfolio across dozens of distinct administrative wrappers. They do this to target different regulatory jurisdictions, implement varying management fee structures, and separate retail investors from institutional capital. Consequently, you will see Institutional classes, Retail classes, Accumulating (Acc) classes, Distributing (Inc) classes, GBP hedged variants, and EUR unhedged variants—all holding the exact same underlying assets in the exact same proportions.

A standard database unique constraint operating on the SecId is insufficient here because each distinct share class possesses a unique Morningstar SecId. If we include all share classes in a model without aggressive deduplication, we introduce multicollinearity into the output.

We must reduce the dataset to a single representative vector per underlying portfolio. We achieve this by cross-referencing two distinct topographical vectors: the exact list signature of the underlying top holdings, and a strict lexical analysis of the fund’s string name.

First, we define a tokenization function to strip punctuation and structural noise from the fund name strings. We use regular expressions to explicitly remove parentheses and brackets, as share classes are frequently denoted by suffixes like “(USD)” or “[Acc]”. Stripping these allows us to isolate the core lexical identity of the fund. We then define an algorithmic function to calculate the strict length of the common prefix between two tokenized arrays.

def _name_tokens(name: str) -> list[str]:
    s = (name or "").strip().lower()
    # Normalize common typographic variations in fund prospectuses
    s = s.replace("–", "-").replace("—", "-")
    # Strip isolating punctuation typically used for share class designations
    s = re.sub(r"[()\[\],]", " ", s)
    return [t for t in s.split() if t]

def _common_prefix_len(a: list[str], b: list[str]) -> int:
    n = min(len(a), len(b))
    k = 0
    # Calculate exact sequential token matches from index 0
    while k < n and a[k] == b[k]:
        k += 1
    return k

The clustering logic dictates a two-factor rule: two rows are identified as overlapping if and only if their top 10 holding tickers match exactly (creating a deterministic tuple signature) AND their fund names share a strong lexical prefix.

Why both? Relying solely on the holdings signature is dangerous. Two different passive index funds managed by different firms might hold the exact same top 10 mega-cap technology stocks. If we only checked holdings, we would collapse two distinct operational funds into one.

By enforcing the lexical prefix check, we ensure accuracy. If ‘Global Equity Fund Class A USD’ and ‘Global Equity Fund Class B EUR’ possess the exact same stocks, their holding tuple signature matches. Their tokenized prefix ['global', 'equity', 'fund'] also matches based on our defined threshold (e.g., 60% of the token length). Therefore, they are safely clustered together as a single entity.

Once clustered, we invoke a selection algorithm. We iterate over the array indices within the cluster, evaluate the ReturnM120 float vector, and retain the exact index containing the maximum finite return value.

We select for the maximum 120-month return because older, institutional share classes exhibit longer, uninterrupted historical track records and lower fee drags than newer retail classes. Retaining the asset with the longest valid track record is optimal for backtesting; it maximizes our sample size and statistical significance across different historical market regimes.

def _pick_best_index(arr: np.ndarray, idxs: list[int], keep: str = "max_return") -> int:
    if keep == "first":
        return idxs[0]
    
    best = idxs[0]
    best_r = arr["ReturnM120"][best]
    
    for i in idxs[1:]:
        r = arr["ReturnM120"][i]
        r_ok = np.isfinite(r)
        b_ok = np.isfinite(best_r)
        
        # We strictly prioritize finite, measurable returns over NaN or infinite values.
        # If the current best is NaN, any finite return replaces it immediately.
        if (r_ok and not b_ok) or (r_ok and b_ok and r > best_r):
            best = i
            best_r = r
            
    return best

def dedupe_unnecessary_share_classes(arr: np.ndarray, min_prefix_tokens=8, prefix_ratio=0.60, keep="max_return") -> np.ndarray:
    if arr.size == 0:
        return arr
        
    groups: dict[tuple, list[int]] = {}
    unique_empty_keys: list[int] = []
    tokens_cache = [None] * arr.size
    
    for i in range(arr.size):
        tokens_cache[i] = _name_tokens(str(arr["Fund"][i]))
        stocks = arr["Stocks"][i]
        try:
            # Convert the list of tickers into an immutable, hashable tuple for O(1) dictionary lookups
            sig = tuple(stocks) if stocks else tuple()
        except TypeError:
            sig = tuple()
            
        if len(sig) == 0:
            unique_empty_keys.append(i)
        else:
            groups.setdefault(sig, []).append(i)
            
    kept = set(unique_empty_keys)
    
    for sig, idxs in groups.items():
        # Sort by name to guarantee deterministic execution across multiple pipeline runs
        idxs_sorted = sorted(idxs, key=lambda i: arr["Fund"][i])
        clusters: list[list[int]] = []
        reps: list[int] = []
        
        for i in idxs_sorted:
            ti = tokens_cache[i]
            placed = False
            for c_idx, rep_i in enumerate(reps):
                tr = tokens_cache[rep_i]
                cpl = _common_prefix_len(ti, tr)
                
                # Dynamic threshold calculation ensures short names and long names are evaluated fairly
                threshold = max(min_prefix_tokens, int(math.ceil(prefix_ratio * min(len(ti), len(tr)))))
                if cpl >= threshold:
                    clusters[c_idx].append(i)
                    placed = True
                    break
            if not placed:
                reps.append(i)
                clusters.append([i])
                
        for cl in clusters:
            kept.add(_pick_best_index(arr, cl, keep=keep))
            
    kept_idx = np.array(sorted(kept), dtype=int)
    return arr[kept_idx]

So let’s see what we got from this. Time to test!

Oh! This looks good to me!

Screener Extract: 498 funds in 11.567s | Holdings Intercept: 7.711s | Total Architecture Execution: 19.278s | Matrix Dedupe Reduction: 498 -> 346 strict portfolios
[('F00000S8RT', 'Multipartner SICAV - Konwave Gold Equity Fund C USD', 25.92, list(['AU']))
 ('0P0000OMTB', 'Polar Capital Funds PLC - Polar Capital Global Technology Fund I Income EUR', 25.56, list(['NVDA', 'AVGO', 'AMD', 'META', 'LRCX', 'KLAC', 'LITE', 'COHR', 'SNDK', 'STX']))
 ('F0GBR04H0X', 'Multipartner SICAV - Konwave Gold Equity Fund B USD', 25.38, list(['AU']))
 ('F00000VI1X', 'Bakersteel Global Funds SICAV - Precious Metals Fund D EUR', 24.62, list(['NEM', 'AU', 'CDE']))
 ('F00001S8MC', 'CT (Lux) - Global Technology Class ZE (EUR Accumulation Shares)', 23.5 , list(['BE', 'LRCX', 'NVDA', 'AVGO', 'WDC', 'GOOGL', 'AMAT', 'MSFT', 'MRVL', 'AAPL']))]

Ticker    Count   PctFunds
--------------------------
NVDA        146      59.11
MSFT        132      53.44
AAPL        119      48.18
AVGO        113      45.75
GOOGL       110      44.53
AMZN        110      44.53
META        104      42.11
GOOG         68      27.53
NEM          44      17.81
LLY          40      16.19
MA           36      14.57
LRCX         33      13.36
MU           33      13.36
CRM          29      11.74
TSLA         28      11.34
AMAT         23       9.31
V            22       8.91
CDE          20       8.10
AMD          18       7.29
MRVL         18       7.29
FSLR         18       7.29
DC           16       6.48
PWR          16       6.48
RGLD         15       6.07
NFLX         15       6.07

Okay! No surprises here! This output matrix is formatted, computationally lightweight, and ready to be serialized to a Parquet file or whatelse. The architecture is resolved and ready for daily production scheduling.

Alright crew—good session today. Until the next article! And remember, may liquidity is with you! 💵​

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full script:

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

INDEX

Local predictability via divergence

This chapter treats an inefficiency claim as a statement about distributional separation. Fix an outcome variable R and a horizon at which R is measured. The future becomes the probability law of R at decision time. The object of interest is the gap between the baseline future and the state-conditioned future, where X is a declared state variable. A divergence measures this gap by quantifying how much the predictive law of R changes when we move from “no state” to “state x,” and it keeps the research claim separate from the execution layer that turns structure into PnL.

What’s inside:

1. What divergence is for in trading research. Divergence serves as a dependence detector that lives in conditional laws, so the claim stays tied to how the future distribution changes across states.

2. Scope and limits of divergence. The chapter draws a boundary between divergence evidence and realized profit, and it frames the inefficiency claim inside a declared horizon, state region, and environment class.

3. Local divergence maps. The workflow treats R and X as design objects, then compares P(R|X≈x) to P(R) across state domains to build a field of separations over the state space.

4. Local inefficiency domains. A local inefficiency corresponds to a state domain that supports a stable statement across repeated visits, with an activation domain and a transition zone where the conditional law returns toward the baseline mixture.

5. Tail divergences. The chapter introduces tail-focused separation to target structure that lives inside extremes, where distributional shape changes matter most for constraint-relevant outcomes.

6. Divergence stability under perturbation. Evidence is checked under perturbations of the measurement lens (domain rules, estimation choices) and recomputed on held-out path segments to verify persistence.

7. Predictability has geometry. The chapter treats predictability as a geometric object with locality, boundaries, and transitions, so the research claim becomes measurable along the market’s movement through state regions.

Sample

2MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. The role of ETL in the design of trading systems.

3. ETL failure modes and risks.

4. Technical preliminaries.

5. Designing an ETL.

6. Layers, contracts and canonical schema.

7. ETL vs ELT for trading.

8. What breaks ETL in trading systems.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

We enter the part of the quant stack that drives everything: data. We treat data as the machinery that tells a trading system what the market is—minute by minute, venue by venue, revision by revision.

This new serie introduces a sequence of articles on infrastructure—ETLs, scraping, APIs—written from the quant-dev perspective: trading systems need repeatable market states. Research needs to recreate yesterday. Live trading needs to ingest today. Both need to refer to the same definitions.

A familiar pattern shows up in almost every desk, prop shop, or solo setup. Data starts as a handful of scripts, then grows into a maze of just this once fixes: a quick endpoint here, a patched column there, a notebook cell that becomes a dependency because it worked during a good backtest. The system still runs. Results still appear. But the pipeline becomes a moving object—shifting definitions, shifting assumptions, shifting outputs—until the only thing that remains stable is confusion.

That is why ETL deserves a dedicated article. The reason? A pipeline accumulates decisions: what gets stored, what gets normalized, what gets ignored, what gets re-requested, what gets overwritten, what gets trusted… Those decisions compound, and at some point they dominate the behavior of the entire stack. The strategy stops being the bottleneck. The data becomes the bottleneck.

Scraping and APIs deserve equal attention because they are how most modern trading pipelines actually exist in the wild. They impose constraints, quirks, and failure patterns: response truncation, pagination inconsistencies, silent field changes, intermittent outages, caching surprises, and vendor helpfulness.

The goal across this series is to make data infrastructure feel less like maintenance and more like a controlled instrument. A good pipeline behaves like a well-specified system: deterministic when it should be deterministic, conservative when it must be conservative, and transparent when something deviates.

Check this paper to know more about ETLs:

Data extraction, Transformation, and Loading process automation

299KB ∙ PDF file

Download

Download

So the promise of this article is concrete. By the end, it becomes natural to answer questions that otherwise stay vague: Where did this feature come from? Which definition did it use? When did it change? Can the dataset be recreated byte-for-byte? What happens when the vendor revises history? How does the system fail when the feed degrades? Those are operational questions, but they decide whether research can be trusted and whether live behavior can be diagnosed.

The role of ETL in the design of trading systems

Three components are typically treated as core in any trading system: strategy logic, execution, and risk. The ETL process—Extract, Transform, Load—is often categorized as a secondary support function. However, this categorization understates the structural role ETL plays in defining the parameters of the trading environment. It is the ETL process that establishes timestamp semantics and interval conventions, just as it defines the mechanics of volume aggregation across venues, determines the handling of missing prints or stale quotes, and establishes the boundaries of the trading day, including the definition of market close during partial sessions or under vendor-specific conventions.

A weak data pipeline permits a model to stay numerically consistent even when it lacks economic coherence. This failure manifests as an epistemic mismatch. The model trains on information unavailable at the time of execution or on data representations that drift between the training environment and the live feed. Backtests appear profitable but lack causal validity. Live trading degradation follows as a predictable consequence of this architecture.

Scale makes ETL automation a key pillar. But it is a challenge that requires automating the whole process. That means, new failures. From a quant dev perspective, this presents a tension between three competing criteria: velocity, correctness, and observability. Velocity is necessary because research iteration and feature publication are time-sensitive. Correctness is critical because a single semantic mismatch can introduce systematic lookahead bias without any explicit error in the code. Observability is essential because, without reproducibility and data lineage, it is impossible to distinguish between a failure of strategy logic and a failure of data construction. Once models are trained on warehouse-ready features, the ETL pipeline becomes an implicit component of the model itself.

Subscribe now

ETL failure modes and risks

ETL failure modes exist across domains, but in trading they are amplified because signals are marginal and backtests are sensitive to subtle causal violations. It is useful to name the principal failure modes, as a map from data defects to PnL leakage.

1. Temporal integrity risk is the most frequent source of silent contamination. Timezone normalization errors—exchange time, broker server time, and UTC being conflated—create misalignment that can masquerade as alpha. Bar timestamp conventions are often ambiguous: some providers label bars by start time, others by end time, and others by a vendor-defined label that is neither. DST discontinuities can duplicate or remove hourly timestamps. APIs frequently return date fields with implicit timezones. Resampling without explicit session calendars can manufacture bars during market closures. None of these defects need an explicit future reference to introduce lookahead, misalignment is sufficient.

2. Market microstructure risk emerges when sampling and aggregation rules are treated as interchangeable. For example, volume is venue-specific; cross-venue volume is not additive unless definitions match. VWAP computed from incomplete trade prints is not VWAP. OHLC bars from heterogeneous vendors can be numerically similar while being non-equivalent, which means return distributions can look stable while microstructure features drift. Mixing bid/ask and last-trade series destroys interpretability of returns, spread-sensitive features, and execution assumptions. Even quote filtering—removing stale quotes or enforcing spread constraints—changes the effective sampling distribution and can alter feature–target dependence.

3. Firm actions and instrument definition risk is where many useless backtests are born. Splits, dividends, symbol changes, roll schedules for futures, and specification changes in tick size or lot size all require explicit treatment. A toxic pattern is using adjusted close alongside raw open/high/low, which creates a synthetic object with inconsistent scale.

4. Data leakage risk is the highest-impact failure mode because it converts structural error into apparent predictive power. Vendor revisions—late prints, corrected OHLC—can enter historical features without an as-of policy. Using the close price for signals intended to be decided at open changes the filtration. Filling missing values using future data is leakage by construction. End-of-day fundamentals used for intraday signals without controlling for publication time inject future information. Even innocuous preprocessing, such as z-scoring using full-sample statistics, can leak information when deployed online.

5. Heterogeneity risk is both schema and semantics. NaN encodings differ across sources (null, zero, empty string). Identical field names can have different meaning, as when volume is shares, contracts, or notional. Data types differ and can change silently (float versus decimal, integer timestamps versus ISO strings). Precision and rounding differences can alter microstructure-sensitive computations and create spurious stability.

6. Operational risk appears when automation is treated as job finished equals success. Cron jobs can succeed while returning partial data. API rate limits can truncate responses. Network errors can create gaps, retries can create duplicates. Incremental loads can fail to be idempotent. Failures often cluster on holidays, early closes, vendor maintenance windows, and schedule drift that causes overlapping extraction windows.

Heavy transformations performed repeatedly inside modelling loops change the feasible hypothesis class. Recomputing features that should be materialized wastes iteration budget and induces inconsistent definitions when code evolves. Unbounded joins across large fact tables can make the research loop unstable under scaling.

Technical preliminaries

Trading doesn’t tolerate imprecise definitions because PnL is sensitive to small causal misalignments. Let P_t denote a price process under an explicit convention (bid, ask, mid, last). Let V_t denote an associated volume process with explicit units (shares, contracts, notional). Let I_t denote the sigma-algebra of information available up to time t. A trading signal S_t must be I_t-measurable.

An ETL pipeline can be modeled as a composition of maps,

where ε extracts from raw sources into an event stream, T transforms into conformed representations, and L loads into storage layers (OLTP and DW).

• Online transaction processing (OLTP) is a type of database system used in transaction-oriented applications, such as many operational systems. Online refers to the fact that such systems are expected to respond to user requests and process them in real-time (process transactions). The term is contrasted with online analytical processing (OLAP) which instead focuses on data analysis (for example planning and management systems).

• Data warehouse (DW), is a system used for reporting and data analysis and is a core component of goberning. Data warehouses are central repositories of data integrated from disparate sources.

The critical property is causal admissibility. For any feature F_t used at decision time t, F_t must be measurable with respect to I_t. ETL violates this property through mechanisms that look operationally reasonable: aggregations that use end-of-window values while decisions occur at window start; imputation via centered filters; backfills that overwrite previously published features without version control; and “helpful” resampling rules that use future ticks to fill an OHLC bar.

This is why causal admissibility must be enforced by automated tests. If the system can’t prove the property, the property doesn’t hold operationally.

To remove ambiguity, define three timelines. Observation time t^(e) (event-time) is when the market produced the observable. Processing time t^(p) is when the system ingested it. Decision time t^(d) is when the strategy commits to an action.

A bar at index k has an interval [t_k,start , t_k,end). If a strategy acts at bar open, then t_k^(d) . Features must not depend on events with t^(e)≥t_k,start.

Before diving into more complex staff, understanding where ETL pipelines typically struggle helps prioritize improvement efforts. Here’s a breakdown of common bottleneck areas:

Designing an ETL

Every strategy carries implicit assumptions about what constitutes a day, how the open is defined under multi-session trading, which clock governs decisions, what constitutes a trade, for example for a VWAP aggregation, and when information enters I_t (publication timestamps, vendor revisions, and the latency of ingestion itself). ETL either formalizes these assumptions or smuggles them.

The baseline automation flow is simple and sensible: collect data on a schedule, store it in OLTP, transform and load it into a DW, then use the DW for trading workflows. That pattern improves throughput. The difference from a traditional ETL lies in representing the pipeline as a reproducible computational object with explicit contracts so that performance does not amplify error.

Bounded complexity matters because a system that can’t be described will not remain stable. Verifiable correctness matters because a property that can’t be checked automatically is not a property of the system.

ETL design can be posed as constrained optimization. Let θ parametrize pipeline configuration (batch size, partitioning, retry policy, scheduling, feature versions). Let D(θ) denote the dataset emitted for modelling. We care about generalization performance G(D(θ)), latency L(θ), cost C(θ) (compute, storage, API calls), and data quality risk R(θ) (probability-weighted impact of faults). A compact objective can be written as

subject to causal admissibility, idempotence, and lineage. The key conceptual shift is that ETL is an optimization problem.

A useful operational shift is to treat the pipeline as a parameterized program with a semantic version v. The output dataset is then D_v(θ). If either v or θ changes, you must assume the output is a different dataset, even if schemas appear identical. This rule prevents a large fraction of unreproducible model results.

Scheduling ETL jobs reduces contention with operational systems. However, smart scheduling goes beyond simple timing:

Layers, contracts and canonical schema

A robust architecture is best described as layers governed by explicit contracts. At the perimeter, source connectors interact with APIs, brokers, file drops, and vendor feeds. Their contract is to emit raw events with source metadata, explicit time conventions, and enough information to reproduce the request.

Raw ingestion into OLTP should be append-heavy and minimally transformative. Its contract is to preserve raw information, enforce uniqueness, and preserve provenance. Canonicalization then converts heterogeneous source representations into a canonical schema by enforcing time normalization, type normalization, and symbol mapping. The DW layer provides partitioned fact tables and dimensions (assets, calendars, sources) with stable semantics optimized for analytical access.

Above the DW, a feature store publishes versioned feature vectors and tensors consumed by research and live engines. The feature store is the interface that enforces immutability by version and train/live parity. Finally, model inputs and backtest feeds are derived from the feature store as arrays, parquet datasets, or tight queries.

A canonical schema must minimize semantic ambiguity. At minimum, an asset dimension fixes identifiers, venues, tick sizes, and calendars. A calendar dimension fixes session definitions, holidays, and early closes. A source dimension fixes provider identity, endpoint sampling rules, time conventions, and revision policy. Facts separate raw events from derived bars and derived bars from derived features. This separation prevents circularity and leakage.

Establishing baseline metrics is essential before implementing optimization strategies. Track these KPIs to measure improvement:

A critical design choice is to represent bars with explicit start/end timestamps. Ambiguous labels are a recurring source of leakage.

CREATE TABLE IF NOT EXISTS fact_bar (
  asset_id           BIGINT NOT NULL,
  ts_start_utc       TIMESTAMPTZ NOT NULL,
  ts_end_utc         TIMESTAMPTZ NOT NULL,
  open               DOUBLE PRECISION,
  high               DOUBLE PRECISION,
  low                DOUBLE PRECISION,
  close              DOUBLE PRECISION,
  volume             DOUBLE PRECISION,
  source_id          TEXT NOT NULL,
  ingest_ts_utc      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  quality_flags      INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (asset_id, ts_start_utc, source_id)
);

CREATE INDEX IF NOT EXISTS idx_fact_bar_end
  ON fact_bar(asset_id, ts_end_utc);

This schema forces a commitment to interval semantics and prevents a broad class of timestamp-induced lookahead errors.

CREATE TABLE IF NOT EXISTS feature_vector (
  asset_id            BIGINT NOT NULL,
  ts_utc              TIMESTAMPTZ NOT NULL,
  feature_set_version TEXT NOT NULL,
  f1                  DOUBLE PRECISION,
  f2                  DOUBLE PRECISION,
  f3                  DOUBLE PRECISION,
  quality_flags       INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY(asset_id, ts_utc, feature_set_version)
);

ETL vs ELT for trading

Read more