Table of contents:

1. Introduction.

2. Dilution analysis.

3. Income.

4. Balance.

5. Cash flow.

6. Ratios.

7. Segment.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

We are opening a new subsection inside MarketOps dedicated to small-cap opportunities. The goal of this series is to analyze companies that may be overlooked by the broader market but still present interesting financial, operational, or strategic characteristics for investors. This first report marks the beginning of that new series!!!

Rush Street Interactive, Inc. (ticker: RSI) is an online gambling and sports betting company operating within the Consumer Cyclical sector, specifically in the Gambling industry. RSI should be analyzed as a digital gaming operator whose value depends on its ability to grow revenue, retain users, improve margins, and manage dilution in a highly competitive and regulated market.

The company’s business is centered on online casino gaming, sports betting, and related digital wagering products. Unlike traditional casino operators, RSI does not depend primarily on physical casino properties, hotel capacity, or destination traffic. Its business model is more closely tied to digital platform performance, market access, customer acquisition, regulatory approvals, payment infrastructure, product experience, and player retention. This gives RSI a different investment profile from land-based gambling businesses.

The main attraction of RSI is its exposure to the continued legalization and expansion of online gambling. As more jurisdictions allow regulated online betting and iGaming, companies with existing platforms, licenses, and operating experience can expand their addressable market. Growth in this industry requires significant spending on marketing, promotions, compliance, technology, and customer support.

RSI’s investment case depends on four main areas:

1. Revenue quality.

2. Margin expansion.

3. Balance sheet strength

4. Dilution control.

Strong revenue growth is valuable only if it leads to better operating leverage and cash generation. Improving EBITDA is useful, but investors should also examine net income, operating cash flow, free cash flow, and the difference between adjusted and reported profitability.

Overall, RSI can be introduced as a small-cap digital gambling company with exposure to a structurally growing industry, but also with meaningful execution and capital structure risks. The company operates in an attractive market, but its investment value depends on whether management can convert market growth into sustainable profitability and improved per-share value. For that reason, RSI deserves a detailed analysis across dilution, income statement quality, balance sheet position, cash flow generation, operating ratios, and segment performance.

All information and data presented in this article have been sourced from the QuantX platform.

Dilution analysis

Rush Street Interactive’s dilution profile is a relevant part of the equity analysis because the company has a meaningful public share base and a visible increase in shares outstanding over time. The dilution panel shows 104.15 million shares outstanding and a float of 98.29 million shares, which means that most of the company’s share base is already available for public trading.

The first important observation is that RSI is classified with medium overall dilution risk. The share-count chart shows a clear upward trend from roughly 58 million shares in 2020 to 104.15 million shares currently. That increase is material from a shareholder perspective because the ownership base has expanded over the period shown.

The chart shows that RSI’s share count increases in a gradual way. A company can increase revenue, improve its operating footprint, and expand into new markets, but if the share count also rises materially, the economic benefit must be measured on a per-share basis.

At the same time, the dilution risk is partly offset by the company’s cash position. RSI’s data shows net cash per share of $5.68, estimated cash of $576.94 million, and quarterly burn of $20.11 million. These figures confirm that RSI is not under immediate financing pressure. The cash need indicator is marked as low, which is consistent with the idea that the company has enough liquidity to reduce the near-term probability of raising capital through additional equity issuance.

The cash position chart shows a sharp increase around 2020, followed by a multi-year decline and then a recovery into the most recent period. This pattern shows that RSI’s balance sheet has experienced both cash drawdown and cash rebuilding. The decline from the earlier peak indicates that cash has been consumed or redeployed over time, while the recent recovery suggests improved liquidity.

The quarterly burn of $20.11 million is also important. Even with a large estimated cash balance, burn rate must be monitored because persistent cash consumption can gradually change the dilution profile. At the current level shown, the company appears to have enough liquidity to absorb ongoing burn. If burn decreases as revenue scales and margins improve, dilution risk declines further. If burn remains persistent or increases, the company’s future need for capital becomes more relevant.

Offering ability is marked low, which indicates that the screener does not currently identify strong near-term offering pressure. Overhead supply is also marked low, suggesting that visible selling pressure from prior issuance or overhang is not considered severe by the screener. Historical dilution is marked low, although this should be interpreted carefully because the chart still shows a meaningful increase in shares outstanding over time.

The institutional ownership figure is also notable. Data shows 98.2% institutional ownership, which implies that RSI’s shareholder base is concentrated among institutions. This can be interpreted in two ways. On one hand, high institutional ownership can indicate that professional investors are involved in the stock. On the other hand, it can also make the stock more sensitive to institutional portfolio adjustments, block sales, or changes in sentiment. When institutional ownership is this high, investors should monitor ownership changes, insider activity, secondary offerings, and large holder transactions.

Short interest is shown at 9.2%. A short interest level of this size can indicate skepticism or hedging activity, but dilution risk depends more directly on the company’s share issuance behavior, cash burn, capital needs, and financing structure. For RSI, the short interest figure should be viewed as a secondary market signal rather than the main determinant of shareholder risk.

Overall, RSI’s dilution profile is mixed but positive. The company has a rising share count, which creates a real per-share issue. However, the current risk indicators are mostly low, and the cash position reduces immediate financing pressure.

RSI must show that growth translates into better per-share economics. The most relevant confirmation would come from stable or declining share count, improving net income, stronger operating cash flow, lower cash burn, and consistent growth in cash flow per share. If the company continues to grow while limiting new issuance, the dilution risk becomes manageable. If the share count continues to rise faster than profitability and cash generation, shareholder

Subscribe now

Income

The income statement shows a company with strong top-line growth and improving operating leverage. Total revenue increased from $262.4M in Q1 2025 to $370.4M in Q1 2026, representing approximately 41% year-over-year growth. Sequentially, revenue also increased from $324.9M in Q4 2025 to $370.4M in Q1 2026, showing that the company continued to expand beyond the previous quarter.

Operating income also improved significantly. RSI generated $42.8M in Q1 2026, compared with $14.6M in Q1 2025. That represents an increase of almost 193% year over year. This is a positive signal because operating income grew much faster than revenue, which indicates that the company is gaining operating leverage. Basically, RSI is converting a larger portion of its revenue into operating profit.

Cost of revenue increased from $170.9M in Q1 2025 to $238.2M in Q1 2026. This increase is expected given the higher revenue base, but it grew slightly slower than revenue. As a result, gross profit improved from $91.5M in Q1 2025 to approximately $132.2M in Q1 2026. Gross margin also improved slightly, from around 34.9% to 35.7%. This margin stability is important because it shows that RSI’s revenue growth is not being achieved through a major deterioration in direct profitability.

Marketing and advertising expense is especially important for an online gambling company because customer acquisition is one of the main drivers of both growth and risk. RSI spent $47.4M on marketing and advertising in Q1 2026, compared with $42.1M in Q1 2025. In absolute terms, marketing expense increased. However, as a percentage of revenue, it declined from 16.0% in Q1 2025 to 12.8% in Q1 2026. It indicates that RSI generated more revenue without increasing marketing spend at the same pace. From an investor’s perspective, this suggests improved marketing efficiency and better customer monetization.

General and administrative expense increased from $25.3M in Q1 2025 to $31.3M in Q1 2026. The increase is material, but it remained controlled relative to revenue growth. As a percentage of revenue, G&A declined from approximately 9.6% to 8.5%. This supports the same operating leverage argument. The company is scaling revenue faster than some of its operating cost categories.

Total costs and expenses increased from $247.8M in Q1 2025 to $327.6M in Q1 2026, an increase of 32%. Since revenue increased by 41%, the income statement shows a favorable spread between revenue growth and expense growth. This spread is what drives the improvement in operating income. This shows that the company’s growth is translating into better profitability.

Net income also improved. RSI reported $26.2M of net income in Q1 2026, compared with $11.2M in Q1 2025. That represents growth of 134% year over year. Sequentially, net income increased from $19.1M in Q4 2025 to $26.2M in Q1 2026. This confirms that the improvement is visible also at the bottom line.

However, the data also shows that not all net income belongs directly to common shareholders. Net income attributable to the parent was $9.1M in Q1 2026, while net income attributable to noncontrolling interest was $17.1M. RSI’s structure means that investors should continue separating consolidated net income from income available to common shareholders.

The Q2 2025 column contains unusual items that distort comparability. Other nonoperating income was negative $110.5M, while income tax expense showed a benefit of $115.0M. These two large items heavily affected pre-tax income and tax expense, making Q2 2025 less useful as a clean operating comparison. For that reason, Q1 2026 should be compared more directly with Q1 2025, Q3 2025, and Q4 2025 when analyzing recurring income statement performance.

Earnings per share is shown as $0 across the quarters. This means EPS is not very informative, likely because of rounding, share structure, or the relationship between consolidated income and income attributable to common stockholders.

Basic shares outstanding increased from 93.9M in Q1 2025 to 102.2M in Q1 2026, an increase of approximately 8.8%. This is relevant because the company’s profitability improved, but shareholders also experienced dilution. The key investor question is whether earnings and cash flow are growing faster than the share count. In Q1 2026, profit growth exceeded share count growth, but the share count still needs to be monitored.

Overall, RSI’s income statement shows a stronger operating profile than the prior year. Revenue growth is strong, operating income is expanding faster than revenue, marketing expense is becoming more efficient as a percentage of sales, and net income is improving. The main investor watchpoints are the split between consolidated net income and income attributable to common shareholders, the unusual nonoperating and tax effects in Q2 2025, and the continued increase in basic shares outstanding.

Balance

RSI’s balance sheet shows a company whose asset base has expanded over the period shown. Total assets increased from $314.8M in Q3 2023 to $677.3M in Q1 2026. This is a material increase and indicates that the company has built a larger financial base over the last several quarters. The increase is visible from Q1 2025 onward, when total assets moved from $387.0M to $543.0M in Q2 2025, $593.2M in Q3 2025, and $677.3M in Q1 2026.

The growth in total assets is supported by the increase in current assets and cash. Current assets rose from $223.4M in Q3 2023 to $415.2M in Q1 2026. Cash and cash equivalents increased from $171.2M to $330.6M over the same period. This is one of the strongest elements in the balance sheet because cash represents a liquid resource that can support operations, absorb volatility, and reduce the need for external financing.

The short-term balance sheet position also looks stronger over time. Current assets consistently exceed current liabilities. In Q1 2026, current assets were $415.2M, while current liabilities were $212.0M. This produces a current ratio of approximately 1.96x, which indicates that RSI has almost two dollars of current assets for every dollar of current liabilities. This is a liquidity signal because it shows that the company has enough short-term assets to cover its short-term obligations.

Total liabilities also increased, rising from $152.5M in Q3 2023 to $358.7M in Q1 2026. This increase should be monitored because liabilities expanded alongside the asset base. However, the balance sheet does not show a simple deterioration, because equity also increased. Total stockholders’ equity including noncontrolling interest rose from $162.3M in Q3 2023 to $318.6M in Q1 2026.

The composition of equity is important. Stockholders’ equity attributable to the parent increased from $51.5M in Q3 2023 to $159.1M in Q1 2026. Equity attributable to noncontrolling interest also increased, moving from $110.8M to $159.4M. For common shareholders, the parent-level equity figure is more relevant than total equity including noncontrolling interest. RSI’s parent stockholders’ equity has improved, but investors should continue separating parent equity from noncontrolling interest when assessing book value and shareholder value.

The retained earnings line shows an improving accumulated deficit. Retained earnings were -$136.6M in Q3 2023 and improved to -$93.6M in Q1 2026. This remains negative, which means the company still carries an accumulated deficit, but the trend is improving.

Additional paid-in capital increased from $188.8M in Q3 2023 to $250.4M in Q1 2026. This indicates that the capital base has expanded over time. This connects directly with the dilution analysis, an increase in paid-in capital can reflect equity issuance, stock-based compensation, or other capital transactions.

Accounts payable and accrued liabilities also increased. Accounts payable rose from $23.1M in Q3 2023 to $42.7M in Q1 2026, while accrued liabilities increased from $56.3M to $81.0M. This is consistent with a larger operating base. The key question is whether these liabilities grow in line with revenue and operating activity, or whether they indicate increasing pressure on the company’s working capital.

Overall, RSI’s balance sheet shows a stronger liquidity position, a larger asset base, and improving equity. The most positive elements are the growth in cash, the increase in current assets, the improvement in parent stockholders’ equity, and the reduction in accumulated deficit. The main watchpoints are the increase in total liabilities, the large role of noncontrolling interest within total equity, and the continued expansion in additional paid-in capital.

Cash flow

RSI’s cash flow statement shows a company that has generated positive operating cash flow across every visible quarter. Net cash provided by operating activities was $20.1M in Q1 2026, $69.1M in Q4 2025, $95.8M in Q3 2025, $54.8M in Q2 2025, and $28.7M in Q1 2025. The company also generated strong operating cash flow in the earlier visible quarters, with $81.1M in Q2 2024 and $80.6M in Q3 2024. This indicates that RSI’s operating model has produced recurring cash inflows, even though the quarterly pattern is volatile.

The comparison between net income and operating cash flow is especially relevant. Net income improved from -$2.2M in Q2 2024 to $26.2M in Q1 2026, while operating cash flow remained positive throughout the full period. In several quarters, operating cash flow was much higher than net income. This reflects the impact of noncash expenses, working-capital movements, and other cash flow adjustments. This is a positive signal because the business is producing operating cash.

The cash flow by activity shows a clear structure, operating cash flow is positive, while investing and financing cash flows are generally negative. Investing cash flow was negative in every visible quarter, including -$9.2M in Q1 2026, -$8.1M in Q4 2025, and -$28.8M in Q3 2025. The negative investing cash flow is linked to software development, intangible assets, and other investment activity.

Financing cash flow was also negative in every visible quarter. The most notable outflows were -$34.4M in Q3 2025, -$33.9M in Q2 2025, -$27.2M in Q1 2025, and -$22.7M in Q1 2026. A major component of these financing outflows is tax withholding related to share-based payment arrangements. Share-based compensation can affect both the income statement and the share count, while tax withholding related to stock awards creates real cash outflows.

The period cash change including foreign exchange effects was positive in most visible quarters, but it turned negative in Q1 2026, with a decrease of -$3.8M. It shows that operating cash generation was not enough to fully offset investing and financing outflows in that quarter. By contrast, Q4 2025 showed a strong positive cash increase of $63.0M, supported by $69.1M of operating cash flow and relatively limited investing and financing outflows.

The noncash adjustment lines are important because they explain part of the gap between net income and operating cash flow. Depreciation, depletion, and amortization increased from $7.6M in Q2 2024 to $12.7M in Q1 2026. Share-based payment noncash expense was more volatile, with high values such as $26.6M in Q3 2024, $23.8M in Q3 2025, and $8.7M in Q1 2026.

The investing section shows that RSI continues to spend on software and intangible assets. Payments to develop software were $8.6M in Q1 2026, $7.0M in Q4 2025, and $21.8M in Q3 2025. For a digital gambling company, technology investment can be productive if it improves platform reliability, customer retention, product depth, and operating scalability.

The company is funding investing activity and financing-related outflows from operations in most quarters. However, the quality of cash flow needs to be assessed because operating cash flow is influenced by noncash adjustments, share-based compensation, working-capital movements, and tax items.

Ratios

RSI’s ratios show a company with improving profitability, strong liquidity, no visible balance-sheet debt pressure, and positive cash-flow generation. The ratio table is useful because it connects the income statement, balance sheet, and cash flow statement into a single analytical view. Instead of looking only at revenue growth or net income, the ratios show whether RSI is becoming more profitable, more liquid, less levered, and more cash-generative on a relative basis.

Operating margin increased from 1.90% in Q2 2024 to 11.55% in Q1 2026. It shows that RSI is converting more revenue into operating income over time. The path is not perfectly linear, but the overall direction is positive. The increase from 5.56% in Q1 2025 to 11.55% in Q1 2026 is important because it shows year-over-year margin expansion.

Net margin also improved compared with the earlier periods, although it remains much lower than operating margin. Net margin moved from -0.05% in Q2 2024 to 2.45% in Q1 2026. The data also shows a temporary peak of 6.20% in Q2 2025. The more relevant point is that RSI has moved from near-breakeven or slightly negative net margin into positive net margin. This indicates that profitability is improving, but the gap between operating margin and net margin still matters for common shareholders.

Return metrics also show progress. ROE was 5.70% in Q1 2026, compared with -0.16% in Q2 2024 and 1.71% in Q3 2024. ROA was 1.34% in Q1 2026, compared with -0.03% in Q2 2024 and 0.33% in Q3 2024. RSI is generating positive returns on both equity and assets, which supports the idea that the business is becoming more productive. The Q2 2025 ROE of 13.57% and ROA of 3.07% stand out, but that quarter should be treated cautiously because it appears influenced by unusual items.

The current ratio increased from 1.62 in Q3 2023 to 1.96 in Q1 2026. The quick ratio is the same as the current ratio across the data, which indicates that liquid or near-liquid current assets represent most of the current asset base. The cash ratio also improved from 1.24 in Q3 2023 to 1.56 in Q1 2026. This is a positive liquidity signal because cash alone covers a large portion of current liabilities. RSI’s liquidity profile reduces near-term financing pressure and supports the earlier conclusion that immediate dilution risk is not the main concern.

The leverage section is also favorable. Debt / equity and debt / assets are both shown as 0.00 across the visible periods. This indicates that RSI does not appear to be carrying meaningful balance-sheet debt. The absence of debt pressure is important because it reduces financial risk and gives the company more flexibility.

The equity / assets ratio improved from 16.36% in Q3 2023 to 23.49% in Q1 2026. This means equity represents a larger share of the asset base over time. The improvement is positive because it shows that the company’s capital structure has strengthened. However, this ratio should still be analyzed together with dilution. A rising equity base is good, but investors must also check whether equity growth comes from retained profitability, capital issuance, or stock-based compensation. The strongest version of the investment case occurs when equity / assets improves because profits and cash flow are building the company’s capital base without excessive share expansion.

RSI generated positive free cash flow in every visible quarter where the metric is reported. Free cash flow was $19.9M in Q1 2026, $68.9M in Q4 2025, $95.3M in Q3 2025, $54.1M in Q2 2025, and $28.6M in Q1 2025. This shows that RSI is generating cash after capital expenditures and investment requirements. The company’s free cash flow profile supports the balance sheet and reduces dependence on external financing.

FCF margin is positive but volatile. It was 5.36% in Q1 2026, down from 21.21% in Q4 2025 and 34.30% in Q3 2025. The decline in Q1 2026 does not eliminate the positive cash-flow profile, but it does show that cash conversion can fluctuate significantly from quarter to quarter. For investors, the key question is whether RSI can sustain positive free cash flow margins over a full year rather than only in isolated quarters.

RSI reported OCF / net income of 2.22x in Q1 2026, 13.17x in Q4 2025, and 15.84x in Q3 2025. These values show that operating cash flow has generally exceeded net income. That is positive because it indicates strong cash conversion.

The main watchpoints are the volatility of net margin, the unusual return metrics in Q2 2025, the decline in FCF margin in Q1 2026, and the need to separate true per-share value creation from accounting improvements. The ratios show a company moving in the right direction, but the final investor judgment depends on whether these improvements remain durable across future quarters.

The most important focus going forward is sustainability. RSI becomes more attractive if operating margins stay above prior levels, free cash flow remains positive, and equity growth translates into better per-share value.

Segment

RSI’s segment data shows a company whose revenue base is highly concentrated in online casino and online sports betting. Total revenue increased from $691.2M in FY2023 to $924.1M in FY2024 and $1.13B in FY2025. This represents a clear multi-year expansion in the company’s revenue base. The growth profile is important because it confirms that RSI is improving quarter by quarter, and also scaling on a full-year basis.

Online casino and online sports betting generated $674.1M in FY2023, $917.1M in FY2024, and $1.13B in FY2025. This segment accounts for almost all company revenue in every visible year. This confirms that RSI’s economic value is tied primarily to the performance of its digital wagering platform, not to retail betting or social gaming.

Retail sports betting is a very small part of the revenue base and has declined. Retail sports betting revenue decreased from $12.8M in FY2023 to $2.4M in FY2024 and $2.0M in FY2025. This decline confirms that RSI’s business mix has become even more digital. This is relevant because the company’s future performance should be analyzed through online customer acquisition, online casino engagement, sportsbook activity, retention, and platform economics rather than physical retail betting exposure.

Social gaming is also small but stable. Revenue increased from $4.3M in FY2023 to $4.6M in FY2024 and $4.9M in FY2025. The important point is that neither retail sports betting nor social gaming drives the investment thesis. RSI’s business is structurally dependent on online casino and online sports betting.

The geographic breakdown shows that the United States and Canada remain the core revenue region. Revenue from the United States and Canada increased from $611.9M in FY2023 to $785.3M in FY2024 and $979.6M in FY2025. This region represents the majority of total revenue and remains the primary engine of the business. For investors, this means RSI’s performance depends on the North American online gambling market, including regulation, market access, customer acquisition costs, promotional intensity, and competitive pressure.

Latin America and Mexico also increased, from $79.3M in FY2023 to $138.8M in FY2024 and $154.9M in FY2025. This region is smaller than the United States and Canada, but it provides an additional growth channel. Its growth rate from FY2023 to FY2024 was strong, while FY2025 growth was more moderate. From an investor’s perspective, this region adds diversification, but it does not yet change the company’s dependence on the United States and Canada.

The geographic revenue share confirms this concentration. The United States and Canada represented approximately 88.5% of revenue in FY2023, 85.0% in FY2024, and 86.7% in FY2025. Latin America and Mexico represented approximately 11.5%, 15.0%, and 13.7% over the same years. This means RSI remains a North America-led business with a smaller but meaningful Latin America and Mexico contribution. The geographic mix is not overly diversified, but it is not single-region either.

RSI is a digital-first gambling company, and its revenue growth is being driven almost entirely by online casino and online sports betting. The company’s smaller product segments are not material enough to change the investment thesis.

We conclude the review by noting that the next visible catalyst is the upcoming Earnings & Revenue event, with an estimated date around August 5, 2026. The market estimate is EPS of 0.148 and revenue of $362.44M. This event can become an important confirmation point for the investment case. If RSI reports revenue above expectations, maintains operating leverage, and shows continued cash generation, the market can receive another positive signal. If the company disappoints on revenue, margins, or guidance, the stock can become vulnerable because the price already reflects strong expectations.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Quality standards of a strong trading hypothesis

A strong trading hypothesis turns a market idea into an research object. It defines what is claimed, where the claim applies, how the effect appears in data, and how evidence improves the hypothesis over time. Precision, measurability, and falsifiability give the hypothesis structure and make the research process coherent, testable, and repeatable.

What’s inside:

1. Precision. A strong hypothesis defines the exact claim, the market setting, and the expected response.

2. Precise wording. Clear wording transforms broad ideas such as pressure, stress, momentum, or dislocation into specific market processes.

3. Precise scope. The hypothesis identifies the asset universe, market state, event type, time horizon, and environment where the effect belongs.

4. Precise outcome definition. The research defines the empirical object used to judge the effect, such as return, spread compression, rank shift, abnormal movement, path behavior, or persistence.

5. Measurability. The hypothesis enters the data through observable variables, classifiable states, and outcomes that can be scored consistently.

6. Observable variables and classifiable states. Market concepts such as crowding, hidden demand, liquidity pressure, or absorption become defensible proxies and coherent state definitions.

7. Falsifiability. The hypothesis exposes the claim to evidence by defining the conditions that weaken, contradict, or challenge the proposed mechanism.

8. Revision after contradiction. Evidence improves the hypothesis by narrowing the trigger, refining the state filter, adjusting the horizon, improving the outcome, or replacing the mechanism when needed.

Sample

1.9MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Where to find data?

3. Risks and web-scraping failure modes.

4. Dependency envelope and system boundary definition.

5. WebSocket transport.

6. TradingView message, session state, and symbols.

7. OHLCV and structured NumPy arrays.

8. Symbol acquisition loop.

9. Metadata normalization, and universe construction.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Algorithmic trading begins with a data question before it becomes a model question. Every signal, backtest, risk estimate, execution assumption, and portfolio rule depends on the historical record that feeds the system. A strategy is a function of its data. When the data source changes, the object under study changes with it.

Independent researchers face this problem under capital constraints. Market data subscriptions, platform limits, and research time compete for the same budget. Everybody knows how hard is to find a profitable strategy so traders may need data before premium infrastructure becomes affordable. However, the researcher needs enough historical evidence to test an idea before expanding spending.

Free data has value. It gives the researcher access, speed, breadth, and experimentation capacity. Its risk profile comes from the conditions attached to that access. Endpoint changes, rate limits, missing fields, symbol inconsistencies, session conventions, adjustment modes, and partial responses can enter the research process as hidden assumptions.

The practical answer is to stop treating acquisition as a casual download. This builds a small research stack around that idea. One part retrieves OHLCV bars from TradingView. The other builds the symbol universe that feeds those requests. The code stays close to the source, so the researcher can see how raw access becomes a structured object for testing.

From there, the workflow becomes research-ready. The researcher can trace the source path, preserve the request logic, validate the output, and understand which data conventions shape the sample before the first model decision occurs.

Where to find data?

The first question in algorithmic trading concerns data origin. The question comes before model choice. A trading strategy is a function over a historical record. When that record contain unstable fields, adjustment errors, or hidden provider changes, the strategy test becomes unstable.

Independent traders enter the problem from a cash constraint. They seek cheap data because the platform account, broker margin, hosting, and research time consume capital. Paying hundreds per month for a data subscription before a strategy proves itself creates an uncomfortable sequence. The market still imposes the same standard. A weak data feed can make a bad strategy look acceptable, make a decent strategy look broken, and hide a good idea under avoidable noise. Free data can be true. The core difficulty is that free data comes with weak evidence about error conditions.

Premium data reduces many operational risks and keeps responsibility on the researcher. A paid feed may provide service-level agreements, corporate-action files, delisting history, explicit licensing, historical corrections, and support channels. Those advantages have economic value because they reduce unknowns. The premium label still has to match the strategy. A daily equity strategy needs corporate-action logic. A futures strategy needs roll methodology. A foreign-exchange strategy needs session boundaries. An intraday strategy needs timezone, holiday, daylight-saving, and microstructure details. A paid vendor can deliver observations that are correct in format and wrong in use.

The trader needs research data before institutional data becomes affordable. The correct response is to treat free data as a constrained research source with documented guarantees and documented gaps. The trader gets data from an available source, then records what that source can guarantee and what it leaves uncertain.

Algorithmic trading contains many arguments about alpha built on casual data pipelines. Alpha is a small signal estimated inside a noisy environment. A two-basis-point edge can vanish through timestamp leakage, holiday mismatch, stale universe selection, or a split adjustment imported after the backtest export. The data layer defines the sample and which trades could have existed. When the sample is wrong, the alpha estimate describes a different object.

Risks and web-scraping failure modes

The uncomfortable truth is that a data pipeline is one of the sources of model risk. In algorithmic trading, the market data loader is part of the strategy, even when it looks like infrastructure. A bad entry signal is easy to blame. A silent data defect is harder.

Free data carries a different risk profile. The visible risks are rate limits, endpoint changes, missing values, duplicated bars, inconsistent symbol formats, unavailable historical depth, and provider-side throttling. Indeed, a free source can return adjusted close with raw open, split-adjusted prices with incomplete dividend treatment, or bars whose volume semantics differ across markets. It can omit delisted names, creating a survivorship-biased universe that looks stronger than the universe available at the time. It can make calendars look aligned because the stored sample contains successful requests and much more.

Let’s look at some of these risks in more detail:

1. The first limitation is protocol fragility. The script handles the WebSocket layer: socket creation, TLS wrapping, HTTP upgrade, WebSocket masking, ping/pong responses, frame parsing, TradingView packet packing, and ~m~<length>~m~<payload> message extraction. This is technically strong because it avoids third-party dependencies, but it also means the script depends on message conventions that may change without warning.

   A premium data vendor normally gives an official API contract: documented fields, deprecation policy, status pages, versioned endpoints, and support. A scraped or reverse-engineered feed gives none of that. If TradingView changes a header requirement, modifies the session protocol, renames a method, changes the packet envelope, throttles unauthenticated requests, or alters how heartbeats behave, the script may stop working.

2. The second limitation is rate limits, throttling and sampling bias. The ticker scanner already includes a pause argument between paginated requests. That is a design decision that reduces aggressive request behavior and lowers the probability of rate-limit problems. In fact, it can create sampling bias.

3. The final limitation is the most important, web-scraped or unofficially acquired data is usually not execution-grade by default. Execution-grade data needs stronger guarantees: timestamp precision, exchange-specific calendars, correction handling, corporate-action methodology, symbol mapping, delisting coverage, latency expectations, outage reporting, and licensing clarity. The current scripts are useful for acquisition and research, but they do not yet implement a full institutional data stack.

Subscribe now

Dependency envelope and system boundary definition

In a research stack, every dependency is implicit. A wrapper library may save time, but it can also hide protocol assumptions, retry behavior, schema changes, rate-limit responses, and timestamp conversions. In this case, we chose a narrow dependency envelope so the researcher can see what happens between the socket and the structured array.

The purpose of a minimal acquisition layer is to reduce the number of unknown transformations between the remote data source. In market data, unknown transformations are expensive:

• Did the wrapper adjust for splits?

• Did it include dividends?

• Did it localize timestamps?

• Did it align sessions?

• Did it forward-fill missing volume?

• Did it retry after a failed request and mix partial data into a final table?

The less visible the path, the more forensic work is required when a strategy behaves too well.

The header of the OHLCV script names the supported output. A dictionary mapping TradingView symbols such as NASDAQ:AAPL or TVC:VIX to structured NumPy arrays with datetime, open, high, low, close, and volume fields. That output is the first useful data model in the project. This makes the next stage of research straightforward because every downstream function can rely on named fields and stable numeric dtypes.

# SCRIPT 1 - OHLCV
TV_HOST = "data.tradingview.com"
TV_PATH = "/socket.io/websocket"
TV_ORIGIN = "https://www.tradingview.com"

Portability across vendors should be built at the interface, not faked inside the source adapter. This script is a source adapter. It should speak TradingView’s language and then emit a local object with a stable schema. The right boundary is therefore a small function that returns arrays in a controlled dtype. A separate layer can later convert those arrays into pandas, Polars, Arrow, Parquet, a feature store, or a database. Mixing the vendor protocol and the research table abstraction too early would make the system harder to test.

When a library wrapper fails, you often see only the wrapper exception. Here, the possible failures are closer to the actual layers—DNS or socket creation, TLS wrapping, WebSocket upgrade, frame serialization or a TradingView message. That is more work, but it gives a better map where debugging becomes easier.

The second script uses the same philosophy but applies it to ticker extraction. It imports json, time, urllib.request, and urllib.error. The scanner client therefore depends on the standard library rather than requests. The script wants to POST JSON to a known scanner endpoint, receive JSON back, and build dictionaries. Nothing about that requires a large HTTP stack. If later the user needs sessions, proxies, rotating headers, backoff middleware, caching, or async dispatch, those can be added.

# SCRIPT 2 - Tickers
import json
import time
import urllib.request
import urllib.error

TV_SCANNER_BASE_URL = "https://scanner.tradingview.com"

The data script starts when the universe is defined.

WebSocket transport

The WebSocket portion is the most technically interesting part of the OHLCV script. It opens a TCP connection, wraps it with TLS, creates a Sec-WebSocket-Key, writes the HTTP upgrade request, reads the upgrade response, and checks for the 101 status. Many researchers never see this layer because websocket-client or a vendor SDK does it for them. Here it is visible, and that visibility is useful because the data feed is a stream with a handshake, framing rules, liveness behavior, and failure states.

The function _ws_connect is the gatekeeper. If it fails, there is no chart session, no symbol resolution, and no bars. In production, this function is part of the availability budget. If a research script runs once per day, failure is annoying. If a scanner-plus-fetcher runs across thousands of symbols, even a small connection failure probability becomes a measurable drag on data completeness. A proper acquisition engine would count connection attempts, upgrade failures, elapsed time, remote closes, and timeout frequency. The current function raises clear errors, which is a good starting point.

def _ws_connect(timeout: int = 20):
    raw_sock = socket.create_connection((TV_HOST, 443), timeout=timeout)
    ctx = ssl.create_default_context()
    sock = ctx.wrap_socket(raw_sock, server_hostname=TV_HOST)

    key = base64.b64encode(os.urandom(16)).decode()
    request = (
        f"GET {TV_PATH} HTTP/1.1\r\n"
        f"Host: {TV_HOST}\r\n"
        "Upgrade: websocket\r\n"
        "Connection: Upgrade\r\n"
        f"Sec-WebSocket-Key: {key}\r\n"
        "Sec-WebSocket-Version: 13\r\n"
        f"Origin: {TV_ORIGIN}\r\n"
        "User-Agent: Mozilla/5.0\r\n"
        "\r\n"
    )
    sock.sendall(request.encode())

That makes the TradingView dependency obvious and exposes the browser-like headers. If TradingView changes accepted headers, this is where the break will happen.

The next important design point is frame masking. WebSocket clients must mask frames sent to the server. The _ws_send_text function constructs the frame header, chooses the correct payload-length encoding, generates a random four-byte mask, applies it byte by byte, and sends the final frame. This is the kind of code that looks low-level for a research workflow, but it is exactly the level at which many unofficial data clients break when protocol assumptions drift.

This implementation is compact and correct for basic text frames. It implements the WebSocket behavior needed for this research task. For a batch OHLCV downloader, that boundary is appropriate. The researcher understands that the code targets the data-download workflow rather than the full surface area of a general-purpose WebSocket library, including fragmentation, compression extensions, advanced close-code handling, and reconnect state.

def _ws_send_text(sock, text: str):
    payload = text.encode("utf-8")
    length = len(payload)

    header = bytearray()
    header.append(0x81)  # FIN + text frame

    if length <= 125:
        header.append(0x80 | length)
    elif length < 65536:
        header.append(0x80 | 126)
        header += struct.pack(">H", length)
    else:
        header.append(0x80 | 127)
        header += struct.pack(">Q", length)

    mask = os.urandom(4)
    masked_payload = bytes(payload[i] ^ mask[i % 4] for i in range(length))
    sock.sendall(header + mask + masked_payload)

The mask is required for client-to-server frames. If this function is wrong, the failure can look like a remote-side refusal even though the symbol, interval, and TradingView method are all valid.

The receive side is equally important. The function _ws_recv_text reads two bytes, extracts the opcode, checks the payload length, optionally unmasks the payload, handles close frames, responds to pings with pongs, ignores pongs, and returns text frames.

The script’s ping-pong handling is a strength because many quick data clients ignore it. In a cross-sectional downloader, slow symbols, large n_bars values, and network jitter make liveness important. The code treats the stream as a conversation rather than a one-way file download.

def _ws_recv_text(sock) -> str:
    while True:
        b1, b2 = _recv_exact(sock, 2)
        opcode = b1 & 0x0F
        masked = b2 & 0x80
        length = b2 & 0x7F

        if length == 126:
            length = struct.unpack(">H", _recv_exact(sock, 2))[0]
        elif length == 127:
            length = struct.unpack(">Q", _recv_exact(sock, 8))[0]

        mask = _recv_exact(sock, 4) if masked else None
        payload = _recv_exact(sock, length) if length else b""

        if opcode == 0x8:
            raise ConnectionError("TradingView closed the WebSocket.")
        if opcode == 0x9:
            _ws_send_pong(sock, payload)
            continue
        if opcode == 0x1:
            return payload.decode("utf-8", errors="ignore")

It suppresses control-frame noise and returns only text payloads to the TradingView parser. This separation keeps the higher-level OHLCV function focused on messages such as timescale update and series completed.

The current script raises or prints errors in sensible places, but a production research engine should turn those errors into events. A CSV or JSON log with symbol, interval, n_bars, error class, elapsed seconds, attempt number, and returned bar count would make data acquisition auditable.

TradingView message, session state, and symbols

Once the WebSocket is alive, the helper _tv_pack wraps a JSON payload inside the TradingView envelope format (a marker, the payload length, another marker, and the JSON body). The length prefix is key because TradingView can send multiple logical messages inside a single WebSocket text frame, and the client needs a deterministic way to separate them.

The _extract_tv_packets function is therefore a core parser. It scans the accumulated buffer for the envelope marker, reads the declared payload length, waits if the current buffer is incomplete, and returns complete packets with a residual buffer. This is the kind of logic that prevents subtle parse corruption. If the client assumes one socket frame equals one TradingView message, it will work until it does not. When it fails, the error will look random. A truncated JSON object, a missing series update, or a swallowed heartbeat. The script avoids that by treating the TradingView envelope as the true unit of protocol state.

def _tv_pack(method: str, params: list) -> str:
    payload = json.dumps({"m": method, "p": params}, separators=(",", ":"))
    return f"~m~{len(payload)}~m~{payload}"


def _extract_tv_packets(buffer: str):
    packets = []
    pos = 0
    while True:
        start = buffer.find("~m~", pos)
        if start == -1:
            return packets, ""
        len_start = start + 3
        len_end = buffer.find("~m~", len_start)
        if len_end == -1:
            return packets, buffer[start:]
        length_str = buffer[len_start:len_end]
        if not length_str.isdigit():
            pos = len_end + 3
            continue
        payload_len = int(length_str)
        payload_start = len_end + 3
        payload_end = payload_start + payload_len
        if len(buffer) < payload_end:
            return packets, buffer[start:]
        packets.append(buffer[payload_start:payload_end])
        pos = payload_end

The session layer begins with _random_session, which generates identifiers such as cs_xxxxxxxxxxxx. This is more than a convenience. TradingView chart sessions multiplex state. The downloader creates a chart session, resolves a symbol within that session, and creates a series attached to that symbol. The random suffix makes a session identifier collision unlikely, but the key point is conceptual. The requested bars are a stateful resource. The client builds a stateful context and then asks the server to stream a series update.

The main get_tradingview_ohlcv function sends messages in a sequence that makes sense, like set_auth_token, chart_create_session, resolve_symbol, create_series.

_tv_send(sock, "set_auth_token", ["unauthorized_user_token"])
_tv_send(sock, "chart_create_session", [chart_session, ""])

resolved_symbol = _symbol_payload(
    symbol=symbol,
    session=session,
    adjustment=adjustment,
)

_tv_send(sock, "resolve_symbol", [chart_session, "symbol_1", resolved_symbol])
_tv_send(sock, "create_series", [
    chart_session,
    series_id,
    series_id,
    "symbol_1",
    interval,
    int(n_bars),
])

The complete request goes beyond a ticker string. It includes a session mode, an adjustment mode, an interval, and a bar count. In our research, those fields define the dataset and belong with the downloaded array.

The symbol payload function deserves special attention because it exposes two parameters that directly affect conclusions—session and adjustment. Session defines whether the data uses regular or extended trading hours. Adjustment defines whether prices are adjusted for splits, dividends, or left raw. These choices are research-defining inputs. A moving-average strategy, volatility estimate, overnight return study, breakout rule, or volume filter can change materially depending on regular-session versus extended-session bars and split-adjusted versus raw prices.

The script defaults to regular session and splits adjustment. That is a defensible default for many equity research tasks because it avoids some discontinuities caused by stock splits while keeping pre-market and after-hours activity separate from regular bars. The convention still needs to be documented. When a strategy shows edge, the first reviewer should ask “edge under which session and adjustment convention?” A reproducible backtest records that answer clearly.

def _symbol_payload(symbol: str, session: str = "regular", adjustment: str = "splits") -> str:
    payload = {
        "symbol": symbol,
        "adjustment": adjustment,
        "session": session,
    }
    return "=" + json.dumps(payload, separators=(",", ":"))

Adjustment mode defines the price process being modeled. Let C_t​ be the close at time t. A split-adjusted close C_t^adj​ is a different stochastic process from a raw close C_t^raw​. A return model usually requires continuity through corporate actions, so it often uses adjusted data. An execution simulator uses raw tradable prices and handles corporate actions. The script can request different adjustment modes, so the research layer records the adjustment convention as metadata attached to each array.

The equation is simple, but it is enough to invalidate a backtest when ignored. A split in raw data creates a large negative return that does not represent an economic loss. A split-adjusted series removes that discontinuity for return research. Dividend adjustment introduces another explicit convention. The script’s flexibility is valuable because it supports these different conventions, and the metadata makes that flexibility reproducible. A useful extension returns both the structured array and a small metadata object containing source, symbol, interval, n_bars, session, adjustment, request_time_utc, and downloader_version.

Send me a copy

OHLCV and structured NumPy arrays

The parser _parse_timescale_update turns TradingView's timescale_update payload into a list of numeric tuples. The function expects bars with a vector v containing timestamp, open, high, low, close, and possibly volume. It skips bars with fewer than five values and sets volume to np.nan if volume is absent. This is one of the best practical decisions in the script. Missing volume remains missing rather than being converted to zero. Zero volume is a statement, missing volume is uncertainty. Preserving that distinction avoids a common retail-data mistake.

A zero-volume bar can mean no volume, depending on the instrument and venue. A missing volume field means the source did not provide the value in that message. Those are different observations. In algorithmic trading, volume feeds liquidity filters, execution-cost models, volatility scaling, breakout confirmation, and regime classification. Treating missing volume as zero can falsely classify a liquid symbol as untradable or distort turnover constraints. Using np.nan preserves the distinction and forces downstream code to decide how to handle the missing value.

def _parse_timescale_update(msg: dict, series_id: str):
    try:
        payload = msg["p"][1]
        series = payload.get(series_id, {})
        bars = series.get("s", [])
    except Exception:
        return []

    out = []
    for bar in bars:
        values = bar.get("v", [])
        if len(values) < 5:
            continue

        timestamp = float(values[0])
        open_ = float(values[1])
        high = float(values[2])
        low = float(values[3])
        close = float(values[4])
        if len(values) >= 6 and values[5] is not None:
            volume = float(values[5])
        else:
            volume = np.nan

The next function, _bars_to_array, defines the local schema. It creates a structured NumPy dtype with datetime64 seconds and float64 fields for open, high, low, close, and volume. The timestamp conversion uses integer seconds and casts to datetime64[s]. This is compact and efficient. It also avoids the overhead of a DataFrame at the source boundary.

A structured array is serializable, typed, compact, and easy to validate. DataFrames are excellent for research manipulation, but they can hide dtype drift. A structured array forces this schema. In a larger system, one could convert to a DataFrame after validation.

dtype = [
    ("datetime", "datetime64[s]"),
    ("open", "f8"),
    ("high", "f8"),
    ("low", "f8"),
    ("close", "f8"),
    ("volume", "f8"),
]

arr = np.zeros(len(bars), dtype=dtype)
arr["datetime"] = timestamps.astype("datetime64[s]")
arr["open"] = [b[1] for b in bars]
arr["high"] = [b[2] for b in bars]
arr["low"] = [b[3] for b in bars]
arr["close"] = [b[4] for b in bars]
arr["volume"] = [b[5] for b in bars]

The minimum validation layer around this array should enforce OHLC inequalities. For each bar, low should not exceed open, high, or close. High should not be below open, low, or close. Prices should be finite and positive for standard equities, indices, and spot FX rates. Timestamps should increase after sorting, unless the source returns reverse chronological data and the script normalizes it. The current code does not perform these checks, but it is okay for the moment.

A violated OHLC bar can break volatility estimates, range features, candlestick rules, stop-loss simulations, and intrabar execution assumptions. One corrupted high can create a false breakout. One negative close can explode logarithmic returns. One non-monotonic timestamp can produce lookahead when arrays are joined across assets. The array function should remain simple, but the caller should immediately pass the output through a validator.

The validation snippet looks like this:

Read more

Table of contents:

1. Introduction.

2. Inflation reopens.

3. Energy and chokepoints.

4. Global trade.

5. U.S.–China.

6. Europe’s energy decline continues.

7. East mediterranean gas and the new energy geography.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Growth remains alive, yet price formation has moved from demand cycles into shipping lanes, fuel markets, mineral chains, defence budgets, and regulatory systems. The global economy still expands, but each point of growth now carries a higher security cost.

Today we review the pressure points that define the current environment. Inflation has reopened through energy, fertilizers, freight, and expectations. Hormuz has become a macro variable because oil, LNG, refined products, and insurance prices move through the same corridor. Global trade has entered a selective phase, where AI infrastructure expands while tariffs, standards, and compliance rules reshape market access.

To go deeper you can check this:

EIA

4.39MB ∙ PDF file

Download

Download

Global Trade Update

11.1MB ∙ PDF file

Download

Download

Gas Market Report, Q2 2026

2.19MB ∙ PDF file

Download

Download

The U.S.–China relationship sits at the centre of this structure. Both powers use the same channels for commerce and leverage: rare earths, chips, agriculture, Treasury holdings, industrial capacity, and maritime routes. Europe faces a harsher version of the same problem. Its energy model has shifted away from Russia, but the replacement system depends on LNG, external suppliers, cloud platforms, AI hardware, and fiscal room that bond markets continue to test.

The East Mediterranean has gained strategic weight within this context. Gas fields, pipelines, LNG terminals, electricity cables, hydrogen plans, and maritime claims now form part of a larger energy map shaped by Gulf disruption and European supply risk. The region offers optionality, but also concentrates infrastructure, diplomacy, and security inside one contested space.

Subscribe now

Inflation reopens

The central feature of this macro regime is the coexistence of positive growth with a renewed inflation impulse. The IMF places global growth at 3.1% in 2026 and 3.2% in 2027, below the recent pace of about 3.4% seen in 2024 and 2025 and below the 3.7% average recorded during 2000–2019. Global headline inflation is expected to rise to 4.4% in 2026 before easing to 3.7% in 2027, which turns the 2026 cycle into a test of disinflation credibility rather than a clean continuation of the previous normalization path.

The Middle East war has changed the macro baseline because the shock enters through commodity markets, inflation expectations, transport costs, and financial conditions at the same time. The IMF states that, absent the war, its 2026 growth forecast would have received a slight upward revision to 3.4%. The downward move to 3.1% therefore comes from the conflict shock, partly offset by recent strong data and lower tariff rates. The current slowdown reflects an external constraint on supply and confidence rather than a broad collapse in private demand.

The baseline also hides a wide dispersion across economies. Advanced economies show a more contained revision, while commodity-importing emerging and developing economies carry the largest pressure. These countries face a triple squeeze from higher energy bills, weaker external balances, and tighter financing conditions. The IMF’s downside scenarios show the scale of that vulnerability. Under an adverse path with larger and more persistent energy-price increases, global growth falls to 2.5% in 2026 and inflation reaches 5.4%. Under a more severe path with greater damage to regional energy infrastructure, global growth falls near 2%, while headline inflation moves above 6% by 2027.

The OECD reinforces this picture with a slightly weaker growth profile. It projects global GDP growth at 2.9% in 2026 and 3.0% in 2027. The same outlook shows that the energy-price surge and the Middle East conflict raise costs and lower demand, while strong technology-related investment, lower effective tariff rates, and the momentum inherited from 2025 provide support. This creates a two-engine global economy. One engine is constrained by fuel, freight, insurance, and import costs.

Early 2026 activity data show that the world entered the shock with strength across both services and manufacturing. The OECD notes that global growth remained solid in 2025, expanding at an annualized pace of about 3.25% in the second half of the year. Private consumption and investment were major drivers across many economies, while technology-related business investment carried the U.S. expansion through tariff headwinds. In Asia and the United States, technology-related industrial production continued to rise in the opening months of 2026, supported by stronger capital-expenditure intentions among large listed technology firms.

This detail changes the interpretation of the global cycle. The world is dealing with a supply shock layered on top of an investment upswing. The AI capital cycle supports manufacturing, electricity demand, high-end components, and corporate spending. The energy shock pushes in the opposite direction by compressing margins, lowering household purchasing power, increasing transport costs, and raising the hurdle rate for investment in energy-intensive sectors. The result is a macro regime where the same economy can show strong capex in strategic technology and weak real-income momentum in consumer-facing sectors.

Inflation has reopened through the energy channel. The OECD expects G20 inflation to rise from 3.4% in 2025 to 4.0% in 2026, then ease to 2.7% in 2027. In advanced G20 economies, headline inflation rises from 2.5% to 3.5% in 2026, while core inflation stays around 2.6% before easing in 2027. In the United States, higher energy prices more than offset the reduction in effective tariff rates, lifting projected headline inflation from 2.6% in 2025 to 4.2% in 2026.

The World Bank projects overall commodity prices to rise 16% in 2026, driven by energy, fertilizer, and several key metals. Its energy price index is forecast to rise 24% in 2026, reaching its highest level since the 2022 energy shock. The same forecast describes the 2026 energy move as a near 40% shock relative to January expectations. This means the macro forecast now depends on physical routing, spare production capacity, strategic reserves, and conflict duration.

The WTO projects a slowdown in global trade growth during 2026, while AI-related goods create an offsetting source of demand. Continued strength in AI-enabling goods trade could add 0.5 percentage points to world merchandise trade growth in 2026. This turns trade into a split system. Strategic technology flows remain strong, while energy-linked freight costs, security risks, insurance premia, and compliance barriers weigh on broader merchandise movement.

UNCTAD reports that non-tariff measures now impose higher export costs than tariffs for 88% of countries. In 2025, tariffs on exports rose 10% for developed economies, 16% for developing economies, and 18% for least-developed economies. For smaller exporters, technical regulations, certification requirements, sanitary rules, and product standards have become a larger obstacle than headline tariff rates.

Fiscal policy sits under pressure from the same shock structure. Finances entered 2026 with weak debt dynamics, while the Middle East war added a new source of pressure through energy disruption, tighter financial conditions, and demands for household support. Governments now face a narrow balance between cushioning the energy shock and preserving fiscal credibility. This is one reason the macro regime has become more state-driven. Defence spending, energy subsidies, industrial policy, reconstruction finance, and debt service all compete for the same fiscal space.

The result is a global economy that still expands, but expands through a narrower and more political channel. Growth depends on AI investment, defence production, energy substitution, and public-sector intervention. Inflation depends on chokepoints, commodity flows, and expectations. Trade depends on standards and strategic access. Financial stability depends on the capacity of bond markets to absorb larger public-sector needs. May 2026 therefore marks a shift from a classic business-cycle framework toward a security-constrained macro framework, where physical infrastructure, military risk, energy routing, and fiscal credibility determine the path of output and prices.

Energy and chokepoints

The Strait of Hormuz handled around 20 million barrels per day of crude oil and oil products in 2025, close to 25% of world seaborne oil trade. The route also carried around 110 bcm of LNG, equal to almost one-fifth of global LNG trade. Qatar and the UAE sit at the centre of this exposure because around 93% of Qatar’s LNG exports and 96% of UAE LNG exports passed through Hormuz in 2025. This turns a local maritime disruption into a world pricing event because Asian utilities, European importers, petrochemical firms, airlines, refiners, and shipping operators compete for the same displaced molecules and cargoes.

Saudi Arabia and the UAE hold the main operational bypass capacity through pipelines that can redirect crude flows toward the Red Sea and Fujairah. The available rerouting capacity is estimated between 3.5 million and 5.5 million barrels per day, which covers only a fraction of normal Gulf export flows. Iran, Iraq, Kuwait, Qatar, and Bahrain remain tied to the Strait for most oil exports. The shock therefore prices the difference between gross energy supply and deliverable energy supply. A barrel inside the Gulf has a market value only when it can reach a buyer, pass through insurance, secure shipping, clear port risk, and arrive inside a refinery schedule.

The gas shock has a separate macro channel because LNG markets have thinner buffers than oil markets. The IEA estimates that transit disruption through Hormuz has reduced LNG supplies from Qatar and the UAE by more than 300 million cubic metres per day since 1 March, equal to more than 2 bcm per week. Ras Laffan, the world’s largest liquefaction facility, has been offline since the 2 March attack, while associated gas output has also been affected by oil-field shut-ins. This creates pressure on Asian spot LNG prices, European gas benchmarks, power prices, fertilizer costs, and industrial input prices.

Asia carries the largest direct import exposure. About 80% of oil and oil products moving through Hormuz in 2025 went to Asia, and almost 90% of LNG volumes exiting the Persian Gulf through the Strait also went to Asia. China, India, and Japan are among the main crude importers exposed to the route, with China and India together receiving 44% of crude exports passing through Hormuz. This gives the conflict an Asian growth channel. Energy-importing economies face higher import bills, weaker terms of trade, margin compression, and pressure on administered fuel systems.

Europe faces a different exposure. Its direct share of Gulf crude flows is smaller, with around 4% of the region’s crude routed into Europe, yet the price effect reaches Europe through global Brent pricing, LNG competition, electricity markets, aviation fuel, fertilizer, and industrial chemicals. The ECB describes the shock as a negative supply shock for the euro area because Hormuz affects LNG, refined products, aluminium, helium, sulphur, and fertilizers. These commodities enter the cost base of electricity, aircraft fuel, food production, manufacturing, construction, and strategic industries.

The inflation channel is visible in Europe’s April data. Euro area headline inflation rose to 3% in April, driven by a 10.9% increase in energy prices, while inflation excluding energy fell to 2.2%. This composition matters because the shock comes from imported energy and critical inputs rather than from broad domestic overheating. The ECB also reported longer delivery times, rising input costs, and supply shortages, with the latest observations for April 2026. That combination creates a policy dilemma because monetary policy can anchor expectations, while the physical shortage originates outside the domestic credit cycle.

The oil-price path remains the central market gauge. Brent averaged was $103 per barrel in March and expects a peak near $115 per barrel in the second quarter of 2026. The same outlook keeps a risk premium through the forecast period because supply disruption uncertainty keeps prices above pre-conflict levels. The forecast sees Brent moving below $90 in the fourth quarter of 2026 and averaging $76 in 2027, based on a gradual return of flows and lower shut-ins. This makes the forecast path contingent on shipping access, infrastructure damage, spare capacity, inventories, and diplomatic sequencing.

The Brent-WTI spread shows the geographic nature of the shock. The EIA states that the spread averaged $12 per barrel in March and peaks near $15 per barrel in April because Brent carries greater exposure to Middle East flows, shipping costs, and Asian consuming markets. This spread turns the conflict into a regional balance-sheet issue. U.S. refiners, European buyers, Asian utilities, Gulf exporters, and tanker owners experience the same conflict through different price curves.

The World Bank projects energy prices to rise 24% to their highest level since 2022. Overall commodity prices are forecast to rise 16%. Fertilizer prices are projected to increase 31%, driven by a 60% jump in urea prices. This links the Hormuz shock to food affordability because natural gas is a key feedstock for nitrogen fertilizers, while transport fuel affects every stage of agricultural logistics. The World Bank warns that prolonged conflict pressure could push up to 45 million more people into acute food insecurity.

The latest monthly commodity data reinforces the same direction. The World Bank’s May commodity update shows the energy price index rising 12.1% in April, driven by crude oil gains of 8.7%. Food prices edged up 1.5%, fertilizer prices jumped 14%, raw materials rose 2.5%, and metals gained 1.4%. These moves show that the energy shock has started to move across adjacent commodity baskets rather than staying isolated inside crude oil.

Strategic reserves can smooth the shock, but they cannot recreate Gulf export capacity. The market now prices the speed at which governments release reserves, the capacity of refineries to adapt crude slates, the availability of tankers, the cost of war-risk insurance, the integrity of pipelines, and the ability of Gulf producers to sustain production under threat. Saudi Aramco is warning that the oil market could lose around 100 million barrels per week if the disruptions continue at the current rate and Hormuz remains closed. This figure gives the shock a weekly depletion rhythm that matters for inventories, refinery planning, and political pressure.

Global trade

The global trade regime has moved into a split structure. One side carries the weight of tariffs, energy shocks, insurance costs, route risk, sanctions screening, and regulatory barriers. The other side carries the force of the AI capital cycle, where chips, servers, networking equipment, electrical systems, cooling units, data-centre components, and advanced manufacturing inputs keep crossing borders because artificial intelligence requires a large physical supply chain.

Merchandise trade volume is projected to fall from 4.6% growth in 2025 to 1.9% in 2026, then recover to 2.6% in 2027. Services trade volume is projected to ease from 5.3% in 2025 to 4.8% in 2026, then rise to 5.1% in 2027. Goods and services trade grew around 4.7% in 2025, above world GDP growth of 2.9%, while 2026 places trade growth near GDP growth. This means trade has shifted from an expansion phase to a filtering phase, where strategic sectors expand while broad merchandise flows face macro and policy friction.

The trade rebound came from two sources with different political meanings. First, companies brought imports forward ahead of expected tariff changes. Second, AI-related demand created a hardware boom across semiconductor supply chains, server systems, and electronic components. That combination lifted trade before the full impact of new policy barriers entered the system. The frontloading effect fades, while AI hardware demand remains a support factor. The result is a trade cycle where last year’s strength creates this year’s comparison problem, and where the quality of trade growth matters more than the headline number.

AI has become the strongest structural trade engine because it transforms digital demand into physical shipments. AI-enabling goods trade reached $4.18 trillion after 21.9% annual growth. These goods accounted for 42% of global trade growth, despite representing a smaller share of total merchandise trade. Asia remained the global hub with 62% of AI-related goods trade, while North America became the fastest-growing market and accounted for 16% of the total. This structure gives AI a geopolitical footprint because the model layer depends on the chip layer, the chip layer depends on capital equipment, and the equipment layer depends on minerals, power systems, logistics, and standards.

The earlier trade model rewarded low-cost production and efficient container movement. The current model rewards secure access to compute inputs, energy, advanced components, trusted suppliers, and export licenses. A server shipment now reflects a chain of permissions. It requires chips, memory, networking systems, cooling equipment, energy infrastructure, cybersecurity rules, and destination controls. Trade remains open in aggregate, while the strategic layer becomes more selective.

The tariff layer adds uncertainty to that selective structure. The OECD estimates that changes in U.S. bilateral tariff rates lowered the effective tariff rate on U.S. merchandise imports to 9.9%, compared with the 14% assumption used in its outlook. The same assessment describes a new 10 percentage point U.S. tariff applied across countries, with exclusions for sector-specific goods and exempted products, including many pharmaceuticals and semiconductor-related goods. This gives companies a changing tariff map where country exposure, product classification, exemption status, and origin rules shape sourcing choices.

The tariff map has a second effect, it extends planning risk. Firms can reroute supply chains around a tariff, but repeated rule changes alter inventory strategy, contract length, supplier selection, customs classification, and hedging costs. Trade policy now affects the working-capital cycle of companies. Importers carry more inventory when they expect border costs to rise. Exporters diversify buyers when access terms become unstable. Multinationals split production when rules of origin become strategic.

Regulatory barriers have become the deeper trade constraint. Tariffs rose by 10% for developed economies, 16% for developing economies, and 18% for least-developed economies. Yet regulatory trade measures now impose higher export costs than tariffs for 88% of countries. These measures include technical regulations, health and safety requirements, certification procedures, administrative rules, and product standards. They determine which firms can access a market, which goods qualify, and which producers can absorb the compliance cost.

This shifts the centre of trade policy from border taxes to rule architecture. A tariff is visible at the border. A product standard sits inside the production process. A certification rule affects testing capacity. A cybersecurity requirement affects software design. A carbon rule affects energy sourcing. A data rule affects cloud architecture. A rule-of-origin test affects supplier geography. For large firms, these rules become manageable costs. For smaller exporters, they become market-entry barriers.

Data shows the development impact of this rule-based trade system. Least-developed economies lose around 10% of their exports to G20 markets because firms lack the capacity to meet regulatory requirements. Smaller exporters face higher costs when local testing and certification facilities are scarce, since products require external verification before market entry. This creates a trade hierarchy based on compliance capacity. The firm with legal teams, testing access, documentation systems, and certification budgets enters the market. The firm with production capacity alone faces exclusion from high-value markets.

Regulatory convergence offers one of the largest trade gains inside the current system. Rules or recognizing standards can reduce regulatory-measure costs by 15% to 30%. In Africa, limited cooperation could reduce costs by 30% to 40% in sectors such as agriculture and manufacturing. This shows that trade expansion depends less on classical tariff cuts and more on mutual recognition, shared testing systems, transparent documentation, and regional standards architecture.

A country or bloc that defines safety rules, digital rules, carbon rules, product rules, payment rules, and certification rules can shape the direction of trade. The largest markets gain leverage because exporters adapt to their requirements. This creates a form of regulatory gravity. Producers build around the rulebook of the market they want to enter. Trade flows follow demand, but production systems follow standards.

The AI trade boom intensifies this regulatory gravity. AI hardware crosses borders, but each shipment sits inside controls on chips, cloud access, cybersecurity, data governance, model safety, export licensing, and critical minerals. The value of trade has moved toward goods that carry strategic meaning. A chip is a commercial product, a defence input, a data-centre component, a productivity asset, and a geopolitical lever. A server rack is infrastructure for commerce, surveillance, science, finance, and military logistics.

China’s role in this structure is central because manufacturing depth, mineral processing, component scale, and export capacity give it leverage across the trade stack. China uses rare earths and bilateral commercial agreements as tools of bargaining, while stressing that China’s financial strength still depends on exports and dollar inflows. That combination creates a system of conflict inside dependence. China gains leverage from supply-chain concentration. The United States gains leverage from demand, finance, technology standards, and security alliances. The trade regime becomes a negotiation over who controls the bottleneck.

For Europe, the same regime creates a strategic dilemma. European firms need AI hardware, cloud infrastructure, energy inputs, and export access. European regulators want sovereignty through rules, but the physical supply chain for AI and the energy shock sit outside Europe’s full control. This gives Europe strong influence over market access and weaker influence over upstream technology capacity. The result is a continent that can write standards, enforce compliance, and shape consumer-market rules, while depending on external actors for several inputs that define the next productivity cycle.

The investment implication is that trade exposure must be read through bottlenecks rather than through country labels alone. The relevant question is where a company sits inside the chain of controlled inputs, certification rules, tariff exposure, energy intensity, and AI infrastructure demand. A firm tied to AI hardware can benefit from the capex boom while facing export-license risk. A firm tied to consumer imports can face margin pressure from tariff shifts and freight costs. A firm selling into regulated markets can face growth limits from certification costs. Trade beta has become rule beta, energy beta, and bottleneck beta.

U.S.–China

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Internal anatomy of a good hypothesis

A genuine hypothesis has an internal structure. It defines when the claim begins, what market response should appear, and why that response should emerge. Its value comes from connecting activation, response, mechanism, and evidence into one coherent research object that can be tested, challenged, refined, and extended.

What’s inside:

1. Internal anatomy of a good hypothesis. The lecture presents a hypothesis as an organized structure whose parts must fit together.

2. The condition that activates the claim. A hypothesis begins by identifying the market state in which the effect should appear.

3. Trigger variable. The trigger translates pressure, dislocation, surprise, imbalance, or regime transition into a variable that can select relevant episodes.

4. State filter. The state filter places the trigger inside the right market environment.

5. Eligibility rule. The eligibility rule protects the empirical sample and decides which observations belong to the same research object by controlling for trading session, liquidity, data quality, structural breaks, and special distortions.

6. The response that defines the effect. A hypothesis must specify what should happen after activation.

7. Sign, magnitude, and timing. Sign defines direction, such as continuation, reversal, convergence, or divergence. Magnitude defines economic substance after costs and frictions. Timing defines the internal path of adjustment after the trigger appears.

8. The mechanism that explains the effect. The final part explains why the trigger should lead to the response.

Sample

1.94MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Risks and transformation limitations.

3. Processing text and features related to text.

4. Lexical canonicalization as a trading signal operator.

5. Token boundary specification and market microstructure semantics.

6. Stopword removal, negation retention, and signal integrity.

7. Lemmatization, stemming, and contextual LLM preprocessing.

8. Sparse and dense feature construction for event-driven alpha.

9. Alignment, and log-template parsing.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Between a market event and a model-generated trading signal exists a sequence of transformations that determines what the model is permitted to observe. Standard literature describes this sequence as text preprocessing. Tokenization partitions a character stream into discrete units. Normalization maps heterogeneous strings into canonical forms. Stopword removal deletes tokens according to frequency-based exclusion lists. Stemming and lemmatization compress morphological variants into common representations. Multiword grouping binds adjacent tokens into higher-order semantic objects. In conventional natural language processing, these operations are often treated as preliminary cleaning procedures. In algorithmic trading, they are active signal-shaping mechanisms.

Trading model receives the version of language produced by the preprocessing layer. A headline, filing paragraph, policy statement, earnings transcript, or execution log become usable information only after a pipeline decides which words survive, which strings are merged, which entities are protected, which symbols are erased, and which boundaries define a tradable event. Every rule in that pipeline determines what the classifier can see and what it is forced to ignore.

The dilemma materializes when quantitative researchers treat these transformations as benign standardization rather than active parameterizations of the feature space. A text feature is realized when a preprocessing policy determines the semantic boundary under which that word becomes measurable. Splitting, merging, deleting, lowercasing, masking, or embedding a token changes the geometry of the model input. It alters the sparsity of the design matrix, changes the relationships among features, shifts the timing of event triggers, and reweights the probability of a future market move conditional on the processed text.

Consider the ingestion of a Federal Open Market Committee press release. The raw document is a non-stationary sequence of characters containing policy language, forward guidance, inflation references, balance-sheet terminology, and deliberately controlled ambiguity. If the tokenizer splits on hyphens, the phrase “mortgage-backed” becomes two separate objects rather than one economically coherent instrument descriptor. If a stopword filter removes “not” or “without,” a restrictive sentence may be pushed toward the same representation as an accommodative sentence. If a normalization rule lowercases all symbols, a ticker, acronym, institution, and ordinary noun may collapse into a single token.

The same problem appears in higher-frequency environments. A headline, order-routing message, or execution log may be transformed within milliseconds of arrival. A small parsing decision can determine whether “USD/JPY” remains a currency pair or becomes two unrelated tokens, whether “5.2%” remains a magnitude or becomes an unanchored digit, whether “Order rejected: Code 404” remains distinct from “Order routed: Latency 404ms.” When the preprocessing layer destroys these boundaries, the downstream model may still behave correctly according to its training objective, but it is acting on a distorted version of the event.

Do you want to know more about this? Check that:

How does a text preprocessing pipeline affect ontology matching?

3.08MB ∙ PDF file

Download

Download

Preprocessing is all you need

324KB ∙ PDF file

Download

Download

Today we argues that text preprocessing in trading systems must be evaluated as part of the signal-generation process rather than as a detachable data-cleaning stage. The relevant question is whether it preserves economically meaningful distinctions under the latency, auditability, and chronological constraints of live trading. A transformation that improves linguistic compactness can still generate negative alpha if it deletes directional modifiers, collapses distinct issuers, introduces future information, or destabilizes the feature basis across time.

Risks and transformation limitations

When preprocessing is decoupled from the objective function of the trading strategy, four structural failure modes emerge. The first risk is semantic inversion. This occurs when a transformation rule reverses the directional polarity of the information. The deletion of negation markers or contrast conjunctions maps risk-reducing statements to risk-escalating feature vectors. In credit trading, the semantic distance between “will default” and “will not default” is the difference between a short and a long position. A generic stopword filter that removes “not” projects both text inputs onto the exact same coordinate in the feature space, forcing the model to calculate an expected return based on a corrupted prior.

The second risk is unstable representation. A fixed concept receives disparate numerical encodings across time intervals or data sources because the preprocessing logic lacks invariance to input formatting. Data vendors frequently update their schema. A news feed might transition from capitalizing standard tickers to formatting them in lowercase within brackets. If the canonicalization logic relies on strict case-matching without entity recognition fallback, the historical feature vector for that asset decays to zero, while a spurious new feature dimension begins accumulating frequency. The model observes this as a sudden regime shift in the underlying asset when it is a mechanical artifact of the data handler.

The third risk is false alignment. Two distinct economic entities or states are collapsed into identical token representations. Stemming protocols force independent market concepts to share an etymological root, inflating feature counts and degrading the precision of the classifier. Truncating “organization” and “organic” to the root “organ” collapses corporate structural news into agricultural commodities data. The classifier inherits a dense, overlapping feature representation that dilutes the predictive power of both original terms.

The fourth risk is operational opacity. Algorithmic systems generate execution and state logs, often communicating via standard protocols such as FIX. Preprocessing frameworks convert these continuous text streams into discrete templates for anomaly detection and latency monitoring. Lossy transformations mask critical dynamic variables, rendering the reconstructed event templates insufficient for post-trade attribution or incident resolution. If a regular expression intended to mask order quantities accidentally masks routing destination tags, the quantitative team loses the ability to diagnose venue-specific slippage.

The key problem for redefining text transformations is a live trading drawdown linked to an information extraction failure. A sentiment classifier parameterized to trade sovereign policy headlines initiated long positions following restrictive policy announcements and short positions following neutral macroeconomic updates. Post-trade analysis isolated the failure to the data transformation layer. A static text processing rule removed contrast terms, reduced specific sovereign entities to generic geographic tokens, and applied morphological stemming that equated distinct central bank actions.

During a critical trading session, a headline reading “Central Bank pauses rate hikes, despite inflation pressures” was ingested. The preprocessing layer stripped “despite” and stemmed “hikes” and “pauses”. The resulting token array fed to the support vector machine lacked the logical dependency structure of the original sentence. The model output a high-confidence positive sentiment score, triggering a large, unhedged long position in the sovereign bond market seconds before a massive sell-off.

The classification algorithm executed correctly given the input vector. The input vector misrepresented the market event because the text transformation policy was optimized for corpus reduction rather than economic fidelity. The post-diagnostics proved that the strategy’s negative alpha was generated within the first twenty milliseconds of the text handling pipeline. This event forces the next question. Must text processing remain rigid, deterministic, and auditable, or should it become contextual, adaptive, and reliant on large language models?

Subscribe now

Processing text and features related to text

The prevailing workflow in quantitative text analysis isolates natural language processing from financial modeling. Text is ingested, cleaned, and vectorized using generic linguistic conventions before the quantitative researcher trains a predictive model. This separation is flawed. Let the raw text at time t be x_t, and let the preprocessing policy be P. The prediction model f does not observe x_t but z_t = P(x_t). The trading signal is s_t = f(P(x_t)). Therefore, P is a functional operator embedded within the trading strategy. The parameters of P modulate the conditional expectation of the forward return,

We can formalize the optimization problem. The strategy seeks to minimize a risk-adjusted loss function J(θ, P), where θ represents the continuous weights of the classification model and P represents the discrete parameters of the preprocessing policy. Because P consists of non-differentiable string operations, gradient-based optimization fails. Researchers bypass this computational bottleneck by freezing P at generic heuristic defaults and optimizing over θ. This guarantees a suboptimal solution because the feature space itself remains unoptimized for the specific financial task.

Recent advancements in large language models present an alternative to static rules. Neural architectures can resolve linguistic ambiguity conditional on the surrounding text—the regular LLMs everybody knows. They can preserve negation, identify domain-specific entities, and differentiate identical surface strings based on usage. However, replacing deterministic algorithms with generative models introduces latency, variable execution costs, and non-deterministic outputs. High-frequency and mid-frequency statistical arbitrage strategies operate under temporal bounds. Executing a transformer network forward pass introduces latency measured in milliseconds, violating the execution constraints of a strategy engineered for microsecond-level reactions. The quantitative challenge is to engineer a text processing pipeline that extracts the semantic precision of contextual models while maintaining the execution speed and exact reproducibility of static rules.

The first obstacle is the absence of economic loss functions for text operations. Standard linguistic tasks measure success using metrics such as classification accuracy or F₁ scores on static text corpora, optimizing for average case performance. The cost of a false positive classification is asymmetric to the cost of a false negative. A transformation rule that improves overall text categorization accuracy by two percent but simultaneously degrades precision on severe, fat-tailed drawdown events produces negative alpha. The preprocessing layer must be calibrated against actual capital deployment metrics, such as maximum drawdown or turnover-adjusted return.

The second obstacle is source heterogeneity. Market text originates from diverse distributions with distinct generative processes. Regulatory filings, such as SEC 10-K and 10-Q documents, are dense, structured, and rely on formal accounting lexicons. Social media feeds are sparse, adversarial, non-standard, and populated with cashtags. Execution logs are machine-generated deterministic strings with dynamic alphanumeric variables. Applying a uniform, global preprocessing operator across these distinct domains guarantees information loss. The pipeline requires domain-conditioned transformation paths.

The third obstacle is look-ahead bias induced by contextual processing. If a large language model relies on weights trained on data generated after time t to process text observed at time t, the resulting feature vector z_t contains future information. Pre-trained language representations, whether produced by Word2Vec-style embeddings, BERT-like encoders, GPT-style transformers, or modern embedding models, inherit the temporal context of their training corpus. If a model was pre-trained on a corpus containing 2020 macroeconomic data, validating a 2018 trading strategy using those embeddings constitutes a forward-looking leak. Validating complex text transformations requires chronological segregation of vocabulary sets, embedding spaces, and rule dictionaries. Every artifact must be timestamped and generated only from data available prior to the simulation step.

Lexical canonicalization as a trading signal operator

Lexical canonicalization standardizes heterogeneous text strings into a controlled vocabulary. Market data feeds transmit text with arbitrary capitalization, markup artifacts, and non-standard character encodings. The standard approach applies universal lowercasing and punctuation stripping to reduce the vocabulary dimension and force text convergence.

This reduction changes the basis of the feature space. Consider a text sequence mapped to a sparse vector c(x) ∈ N^|V|, where |V| is the dimension of the vocabulary and c_j(x) represents the occurrence count of token j. A generic canonicalization policy P_a produces vocabulary V_a. A domain policy P_b produces vocabulary V_b. The linear combination of features

The transition from a high-dimensional raw text space to a lower-dimensional canonical space is an explicit projection operator.

Canonicalization must be evaluated as an information-theoretic operator. If a token carries distinct economic meaning based on its capitalization—such as the ticker symbol for a listed equity versus a common noun—lowercasing acts as a destructive operator. The string “APPLE” extracted from a financial data vendor carries a probability mass concentrated entirely on a specific technology equity. The string “Apple” might refer to the equity, or it might initiate a sentence. The string “apple” refers to an agricultural commodity. A universal case-folding mapping function collapses these three distinct nodes into a single coordinate. The mutual information between the raw text feature and the target variable is truncated, reducing the upper bound of the classifier’s predictive capability.

To quantify this instability, we define feature turnover when a canonicalization policy is updated or when the underlying data feed shifts formats from P_old to P_new at time t. Let V_old and V_new be the active vocabularies observed over a trailing window. Feature turnover is the Jaccard distance between the active feature sets:

High feature turnover indicates that the canonicalization operator is shifting the representation of the market. If τ_t spikes without a corresponding macroeconomic regime shift or market microstructure event, the canonicalization operator is malfunctioning and injecting mechanical noise into the feature vectors.

import re
from dataclasses import dataclass
from typing import List, Dict, Set

NEGATION = {"not", "no", "never", "without", "neither", "nor"}
UNCERTAINTY = {"may", "might", "could", "expects", "guides", "sees"}

@dataclass(frozen=True)
class ProcessedText:
    tokens: List[str]
    entities: List[str]
    flags: Dict[str, bool]

def normalize_market_text(text: str, known_entities: Set[str]) -> ProcessedText:
    """
    Standardizes text while preserving case-sensitive market entities and 
    extracting logical flags before irreversible lowercasing.
    """
    # Remove basic HTML markup
    text = re.sub(r"<[^>]+>", " ", text)
    # Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    
    # Extract entities preserving case. This prevents "APPLE" from becoming "apple"
    entities = [e for e in known_entities if e in text]
    
    # Tokenize broadly, capturing percentages and punctuation
    raw_tokens = re.findall(r"[A-Za-z][A-Za-z.'\-]*|\$?\d+(?:\.\d+)?%?|[!?]", text)
    tokens = []
    
    for tok in raw_tokens:
        if tok in entities:
            # Preserve exact case for known entities
            tokens.append(tok)
        elif tok.lower() in NEGATION or tok.lower() in UNCERTAINTY:
            # Standardize logical operators
            tokens.append(tok.lower())
        else:
            # Fallback canonicalization
            tokens.append(tok.lower())
            
    # Extract boolean flags representing the logical geometry of the sequence
    flags = {
        "has_negation": any(t in NEGATION for t in tokens),
        "has_uncertainty": any(t in UNCERTAINTY for t in tokens),
        "has_percent": any(t.endswith("%") for t in tokens),
    }
    return ProcessedText(tokens=tokens, entities=entities, flags=flags)

The next plot visualizes the baseline accuracy variances across disparate datasets under different canonicalization regimes, demonstrating that baseline controls are necessary. A robust canonicalization policy avoids global destructiveness. It extracts protected entities using deterministic gazetteers and named entity recognition modules before applying generic, dimension-reducing transformations to the residual text.

Token boundary specification and market microstructure semantics

Token boundary specification determines the minimum atomic unit of information available to the prediction model. Generic tokenizers partition text using whitespace and standard punctuation delimiters. Financial text consistently violates the core assumptions of generic tokenization, relying on special characters to denote specialized meaning.

Consider market identifiers and quantitative formats. Ticker symbols contain periods to denote share classes (BRK.B). Currency pairs rely on forward slashes (USD/JPY). ISINs and CUSIPs contain structured alphanumeric sequences without spacing. Magnitudes are similarly complex. Interest rates combine digits, decimal points, and percentage signs without whitespace, while financial statements combine currency symbols, numerical digits, and alphabetical magnitude modifiers (e.g., $1.5B). A generic tokenizer fractures these sequences blindly based on static delimiter rules. The string “USD/JPY” becomes

The explicit, hard-coded relationship between the base currency and the quote currency is severed.

When token boundaries are misspecified, the classifier must relearn the fractured relationship through computationally expensive sequence modeling or n-gram concatenation. This increases the data requirement to reach statistical significance and reduces the overall statistical power of the model. Token boundary specification in algorithmic trading must construct typed semantic spans.

Let x be the raw text string. A financial tokenizer first applies an ordered set of regular expressions R to identify protected spans E(x). Conflicts and overlaps are resolved using explicit priority queues, ensuring that a longer, more specific match supersedes a generic match. The tokenizer then partitions the complement sequence x\E(x) using standard whitespace rules. The output is an ordered sequence of discrete tokens interleaved with immutable typed entities.

The probability distribution of market events is conditional on these typed spans. An earnings surprise is a function of a specific reporting entity combined with a numerical magnitude measured relative to a predefined consensus estimate. If the tokenizer fractures the magnitude, the model receives unanchored digits. The feature vector contains noise rather than signal. Let’s implement it.

def protected_tokenize(text: str) -> List[str]:
    """
    Applies regex to extract specific market boundaries (money, percentages, ratings)
    before standard tokenization shatters them into unanchored digits.
    """
    patterns = {
        "MONEY": r"\$\d+(?:\.\d+)?(?:\s?-\s?\$?\d+(?:\.\d+)?)?",
        "PCT": r"[-+]?\d+(?:\.\d+)?%",
        "RATING": r"\b(?:AAA|AA|A|BBB|BB|B|CCC|CC|C|D)[+-]?\b",
        "FX": r"\b[A-Z]{3}/[A-Z]{3}\b",
        "FILING": r"\b(?:10-K|10-Q|8-K|S-1)\b"}
    
    spans = []
    for label, pat in patterns.items():
        for m in re.finditer(pat, text):
            spans.append((m.start(), m.end(), label, m.group()))
            
    # Sort by start index, resolve overlaps by taking the longest match
    spans = sorted(spans, key=lambda s: (s[0], -(s[1]-s[0])))
    
    tokens = []
    i = 0
    for start, end, label, raw in spans:
        if start < i:
            continue # Skip overlapping spans
        # Tokenize text BEFORE the protected span
        tokens.extend(re.findall(r"[A-Za-z][A-Za-z'\-]*|\d+|[!?]", text[i:start]))
        # Append the protected span as a single, typed token
        tokens.append(f"<{label}:{raw}>")
        i = end
        
    # Tokenize any remaining text after the last span
    tokens.extend(re.findall(r"[A-Za-z][A-Za-z'\-]*|\d+|[!?]", text[i:]))
    return tokens

The impact of tokenization extends into the temporal precision of market microstructure events. Execution logs and limit order book updates record exact latencies, side, and sequence numbers. A message reading “ADD order 12345 100@150.50” must not be parsed into arbitrary digits. Misspecifying the boundary of a timestamp, an order identifier, or a price-quantity tuple corrupts the sequence alignment. If the text pipeline fails to generate <QTY>@<PRICE>, the parser cannot reconstruct the state of the limit order book accurately, destroying the integrity of the downstream order flow imbalance calculations.

Send me a copy

Stopword removal, negation retention, and signal integrity

Stopword removal algorithms eliminate tokens to reduce computational overhead and feature space dimensionality. In generic text retrieval, articles, prepositions, and auxiliary verbs offer minimal discriminative power. The frequency of a token is orthogonal to its economic value. Function words frequently encode the logical geometry of a statement.

Consider the conditional probabilities of market reactions. Terms such as “not”, “without”, “against”, “under”, and “over” establish the directionality of the adjacent verbs and nouns. A corporate filing stating a firm is “not in breach of debt covenants” maps to a specific default probability distribution. A generic stopword filter removes “not”, “in”, and “of”, passing the sequence “breach debt covenants” to the classifier. The text processing rule has executed a semantic inversion.

Directional signal integrity requires that the preprocessing operator P preserves the sign of the expected return conditional on the text. We formalize this constraint. Let y be the future asset return, and let g(·) be a scoring function mapping text to a real number. Let N(x) = 1 denote the presence of a negating token in the raw text x. A stopword policy P fails the integrity constraint if:

To prevent sign errors, the preprocessing layer must implement a protected vocabulary. Words that govern logical contrast, temporal ordering, and directional magnitude bypass the deletion filter. Furthermore, the calculation of term frequency-inverse document frequency (TF-IDF) is sensitive to stopword deletion. The document length denominator decreases, artificially inflating the weight of the remaining tokens.

# A standard generic list, typically destructive in finance
BASE_STOPWORDS = {
    "the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
    "this", "that", "these", "those", "it", "its", "he", "she", "they", "in", "of"}

# The whitelist: terms that must survive the filter to preserve logical sign
TRADING_KEEP = {
    "not", "no", "never", "without", "despite", "against", "under", "over",
    "before", "after", "from", "to", "above", "below", "between", "near", "will", "has"}

def remove_stopwords_for_trading(tokens: List[str]) -> List[str]:
    """
    Filters stopwords while strictly retaining logical and directional modifiers.
    Prevents semantic inversion.
    """
    cleaned = []
    for tok in tokens:
        low = tok.lower()
        if low in TRADING_KEEP:
            cleaned.append(low)
        elif low in BASE_STOPWORDS:
            continue
        else:
            cleaned.append(low)
    return cleaned

If negation markers are removed, the surrounding nouns receive higher statistical weights for the wrong directional classification. The plot below illustrates the comparative agreement rates of contextual models versus deterministic baselines in handling such logical operators.

Lemmatization, stemming, and contextual LLM preprocessing

Morphological reduction maps morphological variants of a word to a common representation. Stemming applies deterministic, rule-based truncation to remove suffixes. Lemmatization utilizes vocabulary databases and morphological analysis to return the dictionary form of a word. The motivation for both techniques is to increase the observation count for sparse features, thereby reducing the variance of the parameter estimates in the classification model.

The assumption underlying morphological reduction is that inflected forms share an invariant economic meaning. Markets violate this assumption. Modality and tense encode the probability and timeline of an event. In distressed debt trading, the sequence defaulted represents a realized absorbing state requiring a specific recovery pricing model. The sequence defaulting represents a continuous process, implying ongoing negotiation and uncertain bondholder recovery. The sequence defaults may represent a generic noun in a macroeconomic report. Standard stemming algorithms, such as the Porter or Snowball stemmers, operate via cascading regular expressions without semantic awareness. Stemming these variants into the single root default collapses three distinct temporal and probabilistic states into one dimension.

Stemming operates maximizing recall at the expense of precision. In low-resource text classification environments where the goal is topic modeling, feature merging reduces overfitting. In trade generation, false positives exact direct capital costs. Consider the tokens “securities” and “securing”. A generic stemmer truncates both to secur. A machine learning model trained to identify regulatory risk might flag a document discussing the securing of physical assets as a regulatory event because the feature space merged it with unregistered securities. If a classifier triggers a short position based on this stemmed root, spuriously merging a regulatory risk term with a standard operational term, the variance of the strategy returns increases and the Sharpe ratio degrades.

Lemmatization offers a more conservative mapping but requires part-of-speech (POS) tagging for accuracy. The lemma of a word depends on its grammatical function. The word “cut” functions as a noun in “dividend cut” and as a verb in “will cut rates”. Static lemmatizers struggle with financial vernacular where nouns and verbs are overloaded and sentence structures are frequently abbreviated. Financial headlines drop articles and auxiliary verbs, confusing standard POS taggers trained on formal literature. When the POS tagger fails, the lemmatizer defaults to an incorrect base form, generating misaligned features.

Large language models process morphology conditioned on the global sequence. The model computes attention weights across the entire input, resolving the lemma based on the surrounding context. The query, key, and value vectors in a transformer architecture map the dependency between “cut”, “rates”, and “Fed”, allowing the network to identify the precise economic action.

We evaluate morphological operators using an economic validation loss function. Let IC(P) be the information coefficient of the signal generated using policy P. Let τ(P) be the feature turnover across rolling windows, D(P) be the maximum drawdown contribution attributable to false positive classifications, and λ(P) be the computational latency in milliseconds. The loss function is:

The penalty parameters λ₁, λ₂ and λ₃ constrain the preprocessing policy to trading limits. A morphological reduction policy is deployed only if it minimizes this objective function relative to vanilla tokenization. In high-frequency environments, the latency penalty λ₃ is severe, eliminating LLMs from the live data path.

The optimal integration of LLMs restricts them to offline candidate generation and research validation. Quantitative teams use LLMs over historical data to identify complex morphological mappings that standard lemmatizers miss.

These insights are then distilled into deterministic, N-gram hash maps. The live execution path utilizes an O(1) dictionary lookup, transforming dynamic morphological insights into static rules. This hybrid architecture captures the semantic precision of the transformer network while guaranteeing the microsecond execution speeds required to capture the alpha.

def economic_validation_loss(ic: float, feature_turnover: float, 
                             drawdown_contrib: float, latency_ms: float,
                             lam_turn: float=0.20, lam_dd: float=0.50, lam_lat: float=0.01) -> float:
    """
    Calculates the loss function penalized by latency, turnover, and drawdown 
    to empirically validate a preprocessing policy. Lower values indicate a better policy.
    
    Args:
        ic: Information Coefficient (predictive power) of the signal.
        feature_turnover: Jaccard distance measuring pipeline instability.
        drawdown_contrib: The strategy drawdown attributable to false positive classifications.
        latency_ms: Computational overhead per document.
    """
    # Negative IC because we want to minimize the loss
    return -ic + (lam_turn * feature_turnover) + (lam_dd * drawdown_contrib) + (lam_lat * latency_ms)

This models the trajectory of feature set turnover under competing morphological policies, demonstrating the instability introduced by naive stemming.

Sparse and dense feature construction for event-driven alpha

The transformation of discrete tokens into numerical vectors defines the geometry of the feature space. The choice between sparse lexical representations and dense embedding vectors dictates the capacity, interpretability, and vulnerability of the trading model.

Sparse representations, such as TF-IDF matrices, map text into a high-dimensional, orthogonal space where each dimension corresponds exactly to a discrete token or n-gram. Let N be the total number of documents in the historical corpus and df_j be the document frequency of token j (the number of documents containing token j). The TF-IDF weight for token j in a specific document x is given by:

where c_j(x) is the raw count of token j in document x. Sparse models possess a critical property for trading: absolute, deterministic feature attribution. The inner product β^T w(P(x)), where β is the learned coefficient vector, allows a risk management system to isolate the exact text tokens driving a position. If an automated strategy initiates a sudden, massive short position, the risk manager can trace the decision back to the specific non-zero weights in w(P(x)). When the preprocessing policy P modifies the token stream—perhaps by introducing a new multiword grouping like “credit_default_swap”—the specific impact on w_j(x) and the corresponding coefficient β_j is mathematically traceable. You know what the model saw and how much it cared.

However, sparse representations suffer from semantic blindness. The vectors for “acquire” and “buyout” are orthogonal, their dot product is zero, even though their economic implications are nearly identical. This forces the model to learn the economic equivalence from scratch through historical labels, requiring large amounts of training data.

Dense representations leverage neural architectures to project text into a lower-dimensional, continuous vector space. Embeddings capture semantic proximity, words with similar contextual distributions occupy adjacent coordinates in the vector space. Here, “acquire” and “buyout” will have a high cosine similarity. Modern Transformer models utilize self-attention mechanisms to generate dynamic, contextual embeddings. The representation of the word “bank” in “river bank” will be distinct from its representation in “central bank”.

The interaction between static preprocessing rules and dense embeddings is non-linear and often destructive. Preprocessing destroys the syntactic structure required by transformer models to compute accurate attention weights. Removing stopwords, conjunctions, and punctuation degrades the contextual resolution of the embedding. A transformer relies on the relative positioning of words to understand the dependencies, stripping “the”, “and”, and “to” compresses the sequence and confuses the attention heads.

Conversely, failing to preprocess domain-specific entities contaminates the embedding space. If alphanumeric identifiers, timestamps, and order magnitudes are not masked or typed during preprocessing, the neural model allocates embedding dimensions to transient noise. The model tries to learn a semantic meaning for “10:04:23.004” or the specific order ID “O-99382”, which will never appear again.

An optimal architecture deploys multiple, parallel representation channels.

• Channel A (anchor): Computes sparse TF-IDF vectors from text with conservative normalization and explicit tokenization. This channel preserves interpretable, hard event triggers (e.g., the presence of the exact token “bankruptcy_chapter_11”).

• Channel B (context): Utilizes dense embeddings extracted from a domain-adapted language model. This channel processes minimally altered text sequences (retaining punctuation and stopwords) to capture abstract semantics and tone.

• Channel C (structure): Encodes explicit numerical magnitudes and structured entity relationships extracted via the protected tokenization rules (e.g., <MAGNITUDE_PCT: 5.2>).

The prediction algorithm ensembles these distinct representations. This multi-channel approach allows the system to monitor the divergence between explicit lexical signals (Channel A) and abstract semantic vectors (Channel B). If the dense model says “sell” but the sparse model sees no explicit negative triggers, the execution logic can require human confirmation or reduce the trade size, treating the divergence as a measure of model uncertainty.

Alignment, and log-template parsing

Algorithmic trading infrastructure requires text processing far beyond predictive sentiment analysis. Risk aggregation across portfolios depends on exact entity resolution. System monitoring, latency arbitrage, and error detection depend on log parsing. Both of these critical infrastructural tasks require preprocessing policies that map volatile, unstructured text strings to static identifiers without generating false equivalencies.

Entity resolution aligns heterogeneous string references across disparate data ontologies. A single corporate issuer might be referenced by its legal name (International Business Machines Corporation), its ticker symbol (IBM), exchange codes, subsidiary brands, or common vernacular (Big Blue). The preprocessing policy must normalize these diverse references to a central, unified entity node. The accuracy of this mapping is asymmetric. Let A be the set of proposed mappings generated by the preprocessing pipeline, and let R be the reference set (the true, ground-truth mappings). Precision is defined as (how many of our proposed links are correct), and recall is |A∩R|/|R| (how many of the true links we successfully found).

In portfolio construction and factor modeling, false positive entity mappings contaminate risk exposure matrices. If the preprocessing layer relies on aggressive stemming and stopword removal, it might collapse “Apple Inc.” (the technology company) and “Apple Hospitality REIT” (the real estate investment trust) into a unified root token “appl”. The downstream trading model then erroneously assigns technology sector sentiment to a real estate asset, corrupting the sector-neutral hedging logic.

Therefore, entity resolution must utilize a phased approach. Phase 1 preprocessing (tokenization and casing normalization) standardizes spacing and character sets, increasing recall safely without merging distinct roots. Phase 2 preprocessing (stemming and stopword deletion) removes distinguishing tokens, thereby collapsing precision. For entity resolution, Phase 2 should be disabled. Instead, resolution requires dictionary-based normalization (gazetteers) verified by secondary contextual attributes (e.g., checking if the surrounding text mentions “software” versus “hotels” before linking the entity “Apple”).

Log parsing presents a different infrastructural challenge. It converts unstructured machine output into structured data frames suitable for time-series analysis. Trading systems emit millions of execution logs, latency metrics, and error traces every minute. These text strings consist of static templates (the invariant part of the message) interspersed with dynamic variables (IP addresses, specific order quantities, execution prices, microsecond timestamps). A log parser’s job is to extract the static template to group identical event types.

Generic preprocessing often applies a sledgehammer to these logs, masking numerical digits using a simple regex like s/\d+/<NUM>/g. If a parser applies this generic replacement, it destroys the critical distinction between a static error code and a latency measurement. The distinct events “Order rejected: Code 404” and “Order routed: Latency 404ms” both collapse into the identical structural template “Order <WORD>: <WORD> <NUM>”. The system loses the ability to track routing latency because the metric has been grouped with a functional error.

Log preprocessing must execute typed variable extraction. A sequential regular expression framework must categorize variables before masking them. IPv4 addresses, file paths, specific hex error codes, and duration metrics are identified and extracted into separate, typed data columns. The original log string is then updated with a typed placeholder (e.g., <IPV4>, <DURATION>, <ERROR_CODE>).

This approach preserves the structural uniqueness of the event template (preventing the merging of errors and latency logs), while structuring the data for downstream time-series anomaly detection. If latency suddenly spikes, the structured <DURATION> column allows for immediate statistical queries, rather than requiring a secondary, slow text-mining pass over the raw logs.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

def identity(x):
    """Pass-through function since text is already tokenized."""
    return x

# Sklearn pipeline construction for TF-IDF representations.
# It expects documents already tokenized by `protected_tokenize`
# and does not apply its own default lowercasing or tokenization.
model = Pipeline([
    ("tfidf", TfidfVectorizer(
        tokenizer=identity,
        preprocessor=identity,
        token_pattern=None,
        lowercase=False,
        ngram_range=(1, 2),
        min_df=3,
        max_df=0.90
    )), ("clf", LogisticRegression(max_iter=200, class_weight="balanced"))])

def walk_forward_splits(dates: List[int], train_days: int=252, test_days: int=21):
    """
    Generates indices for strict chronological walk-forward validation 
    to avoid look-ahead bias in vocabulary construction and stopword optimization.
    """
    unique_days = sorted(set(dates))
    start = 0
    while start + train_days + test_days <= len(unique_days):
        train_set = set(unique_days[start:start+train_days])
        test_set = set(unique_days[start+train_days:start+train_days+test_days])
        
        train_idx = [i for i, d in enumerate(dates) if d in train_set]
        test_idx = [i for i, d in enumerate(dates) if d in test_set]
        yield train_idx, test_idx
        start += test_days

The script quantifies the improvements in parsing metrics achieved by implementing these typed preprocessing frameworks, showing that statistical parsers like Drain or IPLoM see significant accuracy boosts when the preprocessing layer respects the variable types.

Every token rule, normalization choice, stopword filter, entity map, embedding, and log parser decides what the model is allowed to know about the market. When those decisions preserve economic meaning, the model receives cleaner, more faithful signals. When they destroy boundaries, erase negation, collapse entities, or leak future context, the strategy may still look statistically valid while trading on a distorted version.

The central lesson is simple, optimize transformations for market fidelity and linguistic clarity together. Protect financial entities. Preserve directional language. Respect token boundaries. Validate preprocessing with drawdown, turnover, latency, and information coefficient alongside generic NLP accuracy. Use contextual models where they add insight, and keep live trading paths deterministic, auditable, and chronologically clean.

Okay! Great job today, guys. Solid work! And remember, the full code is waiting for you in the appendix, ready to be dissected, stress-tested, and tortured as much as you like. Time to wrap it up. Stay sharp, stay bold, stay unstoppable. 📈

PS: Would you rather have a high win rate or strong risk-reward?

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full code

import re
import numpy as np
import matplotlib.pyplot as plt
from dataclasses import dataclass
from typing import List, Dict, Set, Tuple
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

# Lexical canonicalization 
NEGATION = {"not", "no", "never", "without", "neither", "nor"}
UNCERTAINTY = {"may", "might", "could", "expects", "guides", "sees"}

@dataclass(frozen=True)
class ProcessedText:
    tokens: List[str]
    entities: List[str]
    flags: Dict[str, bool]

def normalize_market_text(text: str, known_entities: Set[str]) -> ProcessedText:
    """
    Standardizes text while preserving case-sensitive market entities and 
    extracting logical flags before irreversible lowercasing.
    """
    text = re.sub(r"<[^>]+>", " ", text)
    text = re.sub(r"\s+", " ", text).strip()
    
    # Extract entities preserving case
    entities = [e for e in known_entities if e in text]
    
    # Tokenize broadly, capturing percentages and punctuation
    raw_tokens = re.findall(r"[A-Za-z][A-Za-z.'\-]*|\$?\d+(?:\.\d+)?%?|[!?]", text)
    tokens = []
    
    for tok in raw_tokens:
        if tok in entities:
            tokens.append(tok)
        elif tok.lower() in NEGATION or tok.lower() in UNCERTAINTY:
            tokens.append(tok.lower())
        else:
            tokens.append(tok.lower())
            
    flags = {
        "has_negation": any(t in NEGATION for t in tokens),
        "has_uncertainty": any(t in UNCERTAINTY for t in tokens),
        "has_percent": any(t.endswith("%") for t in tokens),
    }
    return ProcessedText(tokens=tokens, entities=entities, flags=flags)

# Token boundary specification 
def protected_tokenize(text: str) -> List[str]:
    """
    Applies regex to extract specific market boundaries (money, percentages, ratings)
    before standard tokenization shatters them into unanchored digits.
    """
    patterns = {
        "MONEY": r"\$\d+(?:\.\d+)?(?:\s?-\s?\$?\d+(?:\.\d+)?)?",
        "PCT": r"[-+]?\d+(?:\.\d+)?%",
        "RATING": r"\b(?:AAA|AA|A|BBB|BB|B|CCC|CC|C|D)[+-]?\b",
        "FX": r"\b[A-Z]{3}/[A-Z]{3}\b",
        "FILING": r"\b(?:10-K|10-Q|8-K|S-1)\b",
    }
    
    spans = []
    for label, pat in patterns.items():
        for m in re.finditer(pat, text):
            spans.append((m.start(), m.end(), label, m.group()))
            
    spans = sorted(spans, key=lambda s: (s[0], -(s[1]-s[0])))
    
    tokens = []
    i = 0
    for start, end, label, raw in spans:
        if start < i:
            continue
        tokens.extend(re.findall(r"[A-Za-z][A-Za-z'\-]*|\d+|[!?]", text[i:start]))
        tokens.append(f"<{label}:{raw}>")
        i = end
        
    tokens.extend(re.findall(r"[A-Za-z][A-Za-z'\-]*|\d+|[!?]", text[i:]))
    return tokens

# Stopword removal integrity 
BASE_STOPWORDS = {
    "the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
    "this", "that", "these", "those", "it", "its", "he", "she", "they", "in", "of"
}
TRADING_KEEP = {
    "not", "no", "never", "without", "despite", "against", "under", "over",
    "before", "after", "from", "to", "above", "below", "between", "near", "will", "has"
}

def remove_stopwords_for_trading(tokens: List[str]) -> List[str]:
    """
    Filters stopwords while strictly retaining logical and directional modifiers.
    """
    cleaned = []
    for tok in tokens:
        low = tok.lower()
        if low in TRADING_KEEP:
            cleaned.append(low)
        elif low in BASE_STOPWORDS:
            continue
        else:
            cleaned.append(low)
    return cleaned

# Morphological reduction validation 
def economic_validation_loss(ic: float, feature_turnover: float, 
                             drawdown_contrib: float, latency_ms: float,
                             lam_turn: float=0.20, lam_dd: float=0.50, lam_lat: float=0.01) -> float:
    """
    Calculates the loss function penalized by latency, turnover, and drawdown.
    """
    return -ic + (lam_turn * feature_turnover) + (lam_dd * drawdown_contrib) + (lam_lat * latency_ms)


# Feature pipeline and walk-forward governance 
def identity(x):
    return x

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        tokenizer=identity,
        preprocessor=identity,
        token_pattern=None,
        lowercase=False,
        ngram_range=(1, 2),
        min_df=3,
        max_df=0.90
    )),
    ("clf", LogisticRegression(max_iter=200, class_weight="balanced"))
])

def walk_forward_splits(dates: List[int], train_days: int=252, test_days: int=21):
    """
    Generates indices for strict chronological walk-forward validation.
    """
    unique_days = sorted(set(dates))
    start = 0
    while start + train_days + test_days <= len(unique_days):
        train_set = set(unique_days[start:start+train_days])
        test_set = set(unique_days[start+train_days:start+train_days+test_days])
        
        train_idx = [i for i, d in enumerate(dates) if d in train_set]
        test_idx = [i for i, d in enumerate(dates) if d in test_set]
        yield train_idx, test_idx
        start += test_days

# Log-template parsing 
LOG_REGEX = {
    "IPV4": r"(?:\d{1,3}\.){3}\d{1,3}(?::\d+)?",
    "DURATION_MS": r"\b\d+(?:\.\d+)?ms\b",
    "ORDER_ID": r"\border[_-]?id=[A-Za-z0-9_-]+\b",
    "PRICE": r"\bpx=\d+(?:\.\d+)?\b",
    "QTY": r"\bqty=\d+\b",
    "PATH": r"/(?:[\w.-]+/)*[\w.-]+",
}

def parse_log_template(line: str) -> Tuple[str, List[Tuple[str, str]]]:
    """
    Masks specific dynamic variables in execution logs to generate static templates.
    """
    variables = []
    template = line
    for label, pat in LOG_REGEX.items():
        for m in re.finditer(pat, template):
            variables.append((label, m.group()))
        template = re.sub(pat, f"<{label}>", template)
    return template, variables

# Main
if __name__ == "__main__":
    print(" Testing normalization")
    sample_text = "Company ABC expects Q3 revenue to hit $45.5M. They are not guiding higher."
    pt = normalize_market_text(sample_text, {"ABC"})
    print("Tokens:", pt.tokens)
    print("Flags:", pt.flags)
    
    print("\n Testing protected tokenization")
    sample_text_2 = "AAPL reported a 5.2% increase, raising target to $150 - $155."
    print("Tokens:", protected_tokenize(sample_text_2))
    
    print("\n Testing safe stopword removal")
    sample_tokens = ["the", "fund", "will", "not", "be", "liquidating", "in", "Q4"]
    print("Cleaned tokens:", remove_stopwords_for_trading(sample_tokens))
    
    print("\n Testing log template parser")
    log_line = "router order_id=XYZ987 qty=500 px=102.50 sent to 192.168.1.1 in 12.5ms"
    template, extracted_vars = parse_log_template(log_line)
    print("Template:", template)
    print("Variables:", extracted_vars)
    
    print("\n Evaluating policy")
    loss = economic_validation_loss(ic=0.045, feature_turnover=0.15, drawdown_contrib=0.05, latency_ms=1.2)
    print(f"Calculated economic loss: {loss:.4f}")

Table of contents:

1. Introduction.

2. Global macro regime after the February rupture.

3. Trade, tariffs, and the AI capital cycle.

4. Financial conditions, emerging markets, and the new debt geometry.

5. Europe between inflation, industry, and rearmament.

6. Chokepoints, rare earths, and the structure of strategic rivalry.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The central feature of this moment is that macroeconomics can no longer be separated from geopolitics. Oil routes, LNG flows, shipping insurance, semiconductor supply, rare-earth processing, defence procurement, and AI infrastructure now sit inside the same transmission system. A military clashes can move inflation expectations. A tariff decision can alter capital spending. A mineral export license can affect vehicle production, renewable deployment, and defence readiness. The channels that once belonged to separate policy worlds now reinforce one another.

The AI capital cycle is still creating demand for chips, servers, data centres, power systems, and advanced equipment. Fiscal support and defence spending are cushioning parts of the cycle. Services trade remains more resilient than goods trade. But the composition of growth has changed. Expansion increasingly depends on investment linked to security, technology, energy resilience, and state-backed industrial strategy.

That shift creates a new hierarchy among economies. Countries with energy supply, fiscal space, reserve buffers, advanced industrial capacity, or control over strategic inputs gain room to maneuver. Countries dependent on imported fuel, external financing, weak currencies, or narrow export bases face a tighter constraint. Financial markets are beginning to price that difference. The old distinction between advanced and emerging markets matters less than exposure to chokepoints, debt rollover, energy costs, and industrial depth.

Europe sits at the center of this adjustment because it faces all of these pressures at once. Imported energy inflation, weak industrial momentum, rising defence commitments, and the need to rebuild competitiveness without abandoning decarbonisation, all at the same time. The United States sits in a different position, supported by technology investment and capital inflows but still exposed to inflation persistence and strategic overextension. China retains industrial scale and export strength, while also relying on external demand and commanding key processing stages in critical supply chains.

You can go deeper by reviewing this:

OECD Economic Outlook, Interim Report March 2026

1.66MB ∙ PDF file

Download

Download

World Economic Outlook, April 2026

1.1MB ∙ PDF file

Download

Download

Global Trade Outlook And Statistics March 2026

1.44MB ∙ PDF file

Download

Download

The result is a world economy still moving forward, but no longer under the assumptions that defined the previous era. Efficiency is giving way to resilience. Open access is giving way to managed access. Supply chains are being judged not only by cost, but by survivability. The rest of 2026 will therefore depend less on whether growth exists and more on what kind of growth remains possible when war risk, tariffs, energy security, AI investment, debt pressure, and strategic bottlenecks all operate at the same time.

Subscribe now

Global macro regime after the February rupture

The macro setting in April carries a dual character with unusual clarity. Growth continues, yet the safety margin around that growth has narrowed. Productive capacity still expands, yet the cost of keeping that expansion on course has risen. The world economy entered the year with stronger momentum than many institutions had expected in late 2025, then moved into a new phase when the Middle East conflict. That event arrived while the international trade system was still adjusting to the earlier shift in United States trade policy, which means the present environment combines war risk, tariff uncertainty, supply-chain rewiring, and a tighter premium on energy security inside a single macro regime. The result!? The economy still moves forward…

Even under that path, the global outlook loses altitude. World growth is projected at 3.1 percent in 2026 and 3.2 percent in 2027, while global headline inflation rises to 4.4 percent in 2026 before easing to 3.7 percent in 2027. It confirms continued global expansion, yet it also confirms a return of the energy channel as a first-order driver of inflation, monetary timing, household purchasing power, and corporate margins. The downside scenarios reinforce the message. In the adverse case, growth falls to 2.5 percent and inflation rises to 5.4 percent in 2026. In the severe case, growth falls to 2 percent in both 2026 and 2027 while inflation rises above 6 percent. That spread between the reference path and the tail scenarios defines the new regime.

This chart gives the best opening frame for this because it shows that the February rupture belongs to a longer rise in geopolitical tension across regions. The 2020s stand above the 2000s and 2010s in every major region shown. The Middle East and Africa carry the highest level, which fits the immediate conflict geography. Europe and Asia-Pacific also register a clear step-up, which speaks to the wider strategic setting that already included the war in Ukraine, technology rivalry, sanctions, and maritime frictions before the latest shock. North America and South America show a lower level than the conflict core, yet they also sit above earlier decades.

That structural shift affects the transmission channels because it moves through the entire macro system. There are three central channels. The first is the direct commodity-price shock, which raises energy and input costs, disrupts transport, and weakens real incomes. The second is the expectations channel, where firms and workers seek to recover losses and thereby give price persistence a stronger foothold. The third is the financial channel, where higher risk premia, weaker asset valuations, tighter credit conditions, and capital outflows reduce demand and complicate policy. These channels reinforce one another. Oil and gas feed into fertilizer, freight, aviation fuel, and industrial inputs. Higher shipping and insurance costs pass into goods prices. Exchange-rate depreciation amplifies imported inflation, especially in commodity-importing emerging economies. In that setting, macroeconomics and statecraft operate in one field of transmission. Growth, inflation, shipping security, energy access, sanctions, and industrial policy act together.

The regional pattern adds another layer of meaning because the large macro blocs entered this point with different structures. The activity in China and the United States had been stronger than expected, yet that strength carried an uneven internal composition. In China, exports ran ahead while domestic activity, especially housing, remained softer. In the United States, strong activity coincided with low employment growth and slower labor-force expansion. At the same time, the trade regime itself remained in motion. United States statutory tariff rate stood about 5.3 percentage points below the level assumed after court rulings and executive actions, while other countries advanced trade negotiations and new partnerships of their own.

The United States enters this year with firmer demand conditions than most advanced peers and with a continuing technology-investment wave that still supports capital spending. AI-related technology investment added about 0.5 percentage point to US GDP growth, with sizeable spillovers into Asia through import demand for equipment and components. At the same time, inflation pressure in the United States remains higher than expected. Core personal consumption expenditure inflation at 3.1 percent year over year, and the wider WEO discussion presents a path in which US core inflation returns to target during 2027 rather than through an immediate or automatic easing sequence. In the case of China, export performance and industrial scale still carry the expansion. China is projected to grow 4.4 percent in 2026 and 4.0 percent in 2027, with external demand and some fiscal support offsetting a weaker domestic side where consumption remains soft. The euro area enters the year with modest growth and a higher sensitivity to imported energy and industrial input costs. The WEO places euro area growth at 1.2 percent in 2026 and 1.5 percent in 2027, while the euro area headline inflation rises above 2 percent in 2026 and remains above target in 2027.

In the United States, the issue centers on second-round price pressure because domestic demand has held up and inflation remains elevated. In the euro area, the issue centers on imported-price exposure because energy and other tradable inputs still form a key external channel into headline inflation. In China, the policy mix carries a different balance in which low inflation, export competitiveness, the exchange rate, and fiscal support weigh more heavily than overheating fears. The IMF press briefing therefore recommends patience from central banks under the reference case, together with clear communication and readiness to act if inflation expectations shift. On the fiscal side, the Fund emphasizes targeted and temporary support over broad subsidies or price caps, since public debt burdens have already reduced fiscal room.

The world economy still retains pillars of support, and these pillars explain why the baseline remains one of slower expansion rather than a broad contraction. The first pillar is the AI capital cycle. Technology investment remains a real source of demand for semiconductors, network equipment, power systems, and digital infrastructure, while the import intensity of that wave supports trade and investment spillovers across Asia. The second pillar is fiscal support. Fiscal support in various economies and the role of public spending in cushioning economic activity. The third pillar is adaptation through trade and inventories. Global trade remained robust because technology-related exports offset slower momentum in other product categories, and the IMF expects world trade volume growth to slow from 5.1 percent in 2025 to 2.8 percent in 2026 before improving to 3.8 percent in 2027. Goods trade faces a steeper deceleration, while services trade shows greater resilience. That distinction favors economies with stronger positions in digital, financial, and business services and gives the global system a partial buffer against fragmentation in goods.

This other chart deepens the argument because it shows that the present shock lands on an international system already shaped by long-running external asymmetries. Global imbalances may decline only modestly over the medium term. Some cyclical narrowing comes from expansionary fiscal packages in surplus economies, yet that narrowing faces countervailing forces. A technology-driven investment surge continues to attract capital toward the United States, and is expected stronger US productivity and high returns in technology-related sectors to keep capital inflows elevated. The same idea states that positive wealth effects and sustained inflows keep the US current account deficit wider than the level seen during the decade before the pandemic. At the same time, China’s continued reliance on export-led growth, together with limited rebalancing toward domestic consumption, sustains another major source of external imbalance.

That is important for geopolitics due to external imbalances influence power, and vulnerability. Current-account surpluses provide room for reserve accumulation, outward finance, and shock absorption. Large debtor positions sharpen sensitivity to financing conditions and exchange-rate moves. In the presence of higher geopolitical risk, those balance-sheet realities shape coalition behavior, sanctions exposure, reserve management, and the capacity to absorb commodity shocks.

Here the estimation is one-standard-deviation. Geopolitical risk leads to a decline in real GDP of about 0.8 percent one year after the initial shock. The main channels run through weaker private consumption and weaker investment, while consumer prices rise and the nominal effective exchange rate falls. Besides average geopolitical risk in Europe rose by about 1.2 standard deviations after Russia’s invasion of Ukraine and remained elevated at about 0.5 standard deviation now. This chart shows that geopolitical shocks reduce output through spending decisions made by households and firms, lift inflation through price channels, and weaken currencies through financial and external adjustment.

The broader implication is that supply security now carries a direct macro price. Maritime corridors, tanker insurance, refinery access, data infrastructure, and cross-border finance sit much closer to the center of the business cycle than they did under the earlier assumption of a largely rules-based and low-friction global system. Fertilizer markets, food security, reversals in capital flows, and abrupt adjustments in asset prices are all part of the chain of downside risks, especially for economies with weaker fiscal frameworks and lower external reserves.

Trade, tariffs, and the AI capital cycle

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

What a trading hypothesis is not

The lecture explains that charts, indicators, and backtests can all be useful, but none of them are complete hypotheses on their own. Their value appears only when they are connected to mechanism, scope, timing, variable design, and evidence, so the researcher can move from surface regularity to a claim that can be tested, challenged, and refined.

What’s inside:

1. What a trading hypothesis is not. The lecture begins by separating a genuine hypothesis from three incomplete objects (a chart pattern, an indicator, and a backtest result).

2. A chart pattern without explanatory content. A visual formation may reveal tension, compression, rupture, or recovery, but it remains only an empirical clue until the researcher explains what market process could have produced it.

3. Visual regularity without mechanism. Repeated shapes on a chart can create the impression of structure, but research begins only when those shapes are translated into participants, frictions, information, liquidity, or structural relations.

4. Repetition without causal interpretation. A pattern that appears many times does not automatically become a hypothesis. Repetition gains research value only when it is linked to a stable cause and a clear condition under which it should persist or disappear.

5. Description without research value. Statements such as “price tends to bounce after strong declines” are useful starting observations, but they need scope, variables, comparison design, and evidence criteria before they become researchable.

6. An indicator without market logic. Indicators compress data into signals, thresholds, scores, or ratios, but they matter scientifically only when they represent a real market condition such as crowding, strain, dislocation, liquidity withdrawal, or delayed adjustment.

7. A backtest result without a prior claim. A good historical result is not enough. The lecture emphasizes that a backtest should examine a claim stated in advance, rather than becoming a machine for discovering attractive performance and inventing explanations afterward.

8. The importance of forward logic. Strong research moves from observation to mechanism, from mechanism to formal claim, and from claim to evidence. This order gives meaning to both success and failure, allowing the researcher to learn from the market rather than simply collect historical artifacts.

Sample

1.87MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Data-Shape transformation.

3. Identifying the risks and limitations.

4. Data shape transformations for predictive features.

   1. State-space approach.

   2. Dynamic benchmark neutralization.

   3. Stochastic trend removal via cointegration.

   4. Multiscale isolation approach.

   5. Bounded state encoding.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Imagine that a team downloads a price series, defines a target, applies a transformation, and moves on to signal design, model fitting, validation, and execution. That sequence looks efficient. However, the transformation of the data is is the first act of model construction.

That is why data-shape transformation sits at the true front line of feature engineering. The problem is whether the operator used to create that series preserves the economic structure that generates predictive edge. A feature can look impeccable under statistical diagnostics and still be useless in live trading. It can pass every formal test of stability while removing the component that contained the alpha.

This article examines that hidden fault line. It argues that data-shape transformation must be treated as a structural modeling choice rather than a preprocessing convenience. We will identify the main risks introduced by transformation operators, explain how they distort the topology of predictive information, and then study five targeted transformations designed for distinct market nuisances: state-space innovations, rolling benchmark residuals, cointegration spreads, multiscale Haar details, and rolling range positions. The objective is not to make features look cleaner. The objective is to ensure that the shape of the data remains faithful to the market mechanism the model is supposed to learn.

Every transformation enforces a decision about relevance. It selects one part of the market path and suppresses another. It may emphasize short-horizon variation, eliminate common drift, compress local extremes, or redefine the position of the current price relative to its recent history. Before any model is trained, the researcher has already imposed a theory of what matters.

By the way, you can go deeper in this topic by exploring this paper.

Canonical time-series characteristics

1.3MB ∙ PDF file

Download

Download

That choice is decisive because financial data are layered. A single price series contains slow structural movement, temporary dislocations, benchmark comovement, volatility clustering, and random noise at the same time. A predictive model is never interested in all these layers equally. It depends on a narrow component that matches the strategy horizon and economic logic. The transformation either isolates that component or removes it.

Data-Shape transformation

The starting phase of feature engineering contains a severe trap because it masquerades as a trivial data processing step. A quant team downloads a time series of prices, computes a target variable, and faces the decision of whether the predictive model should ingest the raw level, a discrete difference, a standardized statistical score, a regression residual, an equilibrium spread, or a bounded state variable. The standard academic and industry reflex is to treat this decision as a technical prerequisite, often labeled preprocessing or data normalization. That classification is a mistake.

In many pipelines, this step is outsourced to data engineers or automated via standard library defaults before the alpha researchers even begin their core modeling. This assumes that data transformation is a lossless translation of information. It is not. Once a specific transformation is applied to the raw time series, the researcher has declared what specific variation in the market path constitutes actionable information and what variation constitutes noise. If the chosen transformation is misaligned with the economic mechanism the strategy intends to exploit, the resulting model can exhibit perfect statistical behavior while remaining economically blind.

In our domain, this situation is unforgiving because the price is a cumulative, high-memory object. It aggregates historical microstructure shocks, macroeconomic regime shifts, discrete splits in volatility, slow reallocations of institutional capital, structural risk premia, and continuous benchmark movements.

A raw price level carries a massive amount of historical baggage that the short-term or medium-term target variable is invariant to. Assume the core research problem is to predict the magnitude and direction of a short-horizon log return y_t+h. The actual feature matrix the learning algorithm receives is rarely the raw price p_t. Instead, the model ingests a transformed object x_t = T(F_t), where F_t represents the filtration generated by the history of prices and exogenous covariates up to time t. The selection of the transformation operator T is the actual front line of the research design. This operator dictates whether the vector x_t represents a local surprise, a relative displacement, a structurally bounded state, etc.

Identifying the risks and limitations

The risk surface introduced by the operator T is broader and more destructive than standard statistical literature admits. Researchers frequently inherit standard transformation recipes without interrogating their geometric implications or their effect on the data topology. They compute discrete log returns r_t = \ln(p_t) - \ln(p_t-1) because financial libraries expect stationary inputs. They build rolling z-scores because the resulting chart oscillates symmetrically around zero and fits neatly into the activation functions of deep neural networks. These choices impose heavy structural assumptions. A standard return calculation deletes the absolute price level and the cumulative memory of the path. A residual calculation deletes the variance explained by the specified reference model.

The first specific risk is false statistical comfort. A transformed time series can be manipulated to exhibit perfect weak-sense stationarity, zero mean, and constant variance, while simultaneously failing to express any tradable market state. Any researcher can run an Augmented Dickey-Fuller or KPSS test, achieve a spectacular p-value, and incorrectly assume the feature is ready for predictive modeling. Extracting white noise from a time series guarantees stability, but white noise contains zero predictive alpha. The researcher has converted the price into a stable sequence that is devoid of economic value.

The second risk is structural signal “amputation”. A transformation can neutralize the exact slow-moving component that carried the predictive edge. This failure mode occurs when researchers impose rigid discrete differencing, rolling normalization, or arbitrary short lookback windows without first establishing the exact holding period the systematic strategy is designed to exploit. If a macro trend-following strategy relies on three-month momentum, applying a 5-day rolling z-score to the input features will delete the low-frequency eigenvalue that the model requires to identify the trend.

The third risk is benchmark confusion and specification transfer. When a transformation operator incorporates a secondary series—such as a benchmark ETF, a sector index, or a paired asset—the resulting feature inherits the specification risk of that external asset. A regression residual is only valid if the variable against which it is residualized represents a stable economic factor. If a researcher isolates a tech stock by residualizing it against the QQQ index, and the index undergoes a massive constituent rebalance or suffers a liquidity shock, the resulting feature vector will exhibit severe volatility that has nothing to do with the target stock’s idiosyncratic state. The feature becomes contaminated by the reference frame.

The fourth risk is causality leakage. This is the most common fatal error in systematic trading. When the transformation relies on parameters estimated using information outside the filtration F_t, the feature becomes predictive in the research environment and false in the production environment. Using centered rolling windows, applying Principal Component Analysis eigenvectors derived from the full target matrix, computing full-sample mean vectors for standardization, or extracting post-hoc equilibrium coefficients creates a matrix that cannot exist in live trading. Forward-looking bias is adept at hiding within the scaling constants of data-shape transformations.

The fifth risk is geometric distortion. This transformation alters local curvature, variance persistence, variance clustering, and the relative distance metric of excursions. Consider a neural network optimizer relying on Euclidean distance to compute gradients. Mapping a log-normal price path into a bounded state space between zero and one forces the distance metric near the boundaries to behave differently from the distance metric in the center of the distribution. A raw price move that represents a massive four-standard-deviation tail event might only translate to a 0.01 shift in the bounded feature space if the state was already sitting at 0.98. Extracting a state space model collapses a continuous smooth drift into a localized sequence of forecast errors clustered around zero. This alters the fundamental definition of what a market burst represents to the downstream algorithm.

The conflict between raw data and transformed geometry culminates in a specific, observable failure in the research lifecycle. A model is built on a transformed feature set. The initial teardown looks immaculate. The in-sample backtest “demonstrates” it works. The cross-validation metrics are stable across distinct market regimes. The feature importance vectors are dense and logical. The Sharpe ratio appears robust across multiple chronological subsamples, and the drawdowns are well contained.

Then… the model is advanced to paper trading or live execution, and the predictive relationship collapses. The alpha decay instantaneous. The post-mortem analysis reveals that the algorithm never learned the underlying market mechanism. The optimization process learned the geometry introduced by the transformation operator T. It fit itself to sample-specific trend removal artifacts, spurious benchmark dependencies, bound-rejection behavior dictated by rolling window mechanics rather than market participants, and local scaling normalization patterns that possessed no forward-looking stability.

Subscribe now

Data shape transformations for predictive features

If the strategy is engineered to trade benchmark-relative mispricing, then the absolute market benchmark exposure is the primary nuisance structure. If the strategy capitalizes on statistical arbitrage and pair reversion, the common stochastic trend is the primary nuisance structure. If the strategy dictates short-horizon momentum reactions, slow multi-month macroeconomic drift is the primary nuisance structure. If the strategy operates as a local-state mean-reversion classifier, the absolute nominal price level is the primary nuisance structure.

This perspective enforces a precise operational taxonomy of data-shape transformations. The state space algorithm eliminates predictable local trends governed by state-space transition matrices, leaving the prediction surprise. The rolling_regression_residual algorithm eliminates time-varying benchmark exposure, yielding a relative-value deviation vector. The cointegration_spread algorithm eliminates common stochastic drift based on equilibrium vector autoregression logic, resulting in a stationary spread conducive to error-correction modeling. The haar_detail_same_length algorithm eliminates low-frequency power at a selected scale, leaving a targeted multiscale shock component. The rolling_range_position algorithm eliminates absolute nominal levels and local variance scaling, yielding a bounded coordinate variable.

These five methods represent interventions designed to blind the model to specific, targeted market nuisances. However without the right measures, the same mistakes could continue in some of them. Now that you’re aware of them, we can move forward.

State-space approach

The first targeted transformation is the state-space. In technical literature, this is defined as the one-step-ahead prediction error derived from a local linear trend state-space model. This transformation alters the time-series geometry by removing the exact variation that the internal state estimate already expects the system to produce at time t. This isolation mechanism is required when the raw log-price is cumulative, exhibiting a smooth drift that masks the actual high-frequency surprise component necessary for short-horizon prediction.

The formulation relies on setting up the observed log-price y_t as the output of a local linear trend model. The system requires an observation equation and state transition equations:

Here, the unobserved latent state consists of the level μ_t and the slope β_t. The disturbances ε_t, η_t, and ζ_t are mutually independent Gaussian white noise processes governing the measurement error, the level shift, and the slope shift. The state space model operates recursively. Given the filtration up to t-1, it produces the optimal linear one-step-ahead prediction E[y_t | F_t-1]. The feature we extract is the sequence v_t:

It is crucial to understand that v_t represents the precise segment of the current price movement that could not be predicted by the prior dynamic state. In the context of algorithmic trading, this means the feature measures orthogonal surprise rather than raw directional motion.

The rationale for this transformation is embedded in market microstructure. A high-frequency algorithm processing order flow must differentiate between a market that ticks upward at a steady, anticipated velocity and a market that prints an identical upward tick against a declining latent trend. The nominal difference y_t -y_t-1 will record an identical positive value in both scenarios. The state space v_t will output a minor value for the first scenario and a massive positive value for the second. Innovation-based feature engineering forces the learning algorithm to respect the conditional context of the regime.

def state_space(price_series: pd.Series) -> pd.Series:
    """
    Extracts the one-step-ahead forecast error from a local linear trend
    state-space model.
    """
    lp = price_series.rename("log_price")

    model = UnobservedComponents(lp, level="local linear trend")
    result = model.fit(disp=False)

    space = pd.Series(
        result.filter_results.forecasts_error[0],
        index=lp.index,
        name="state_space",)

    space.iloc[0] = np.nan
    return space

However, the researcher must decide if local drift should be neutralized. For execution timing, short-term liquidity provision, and algorithmic burst-detection, neutralizing drift is necessary. Utilizing v_t ensures the resulting feature matrix is robust across shifting volatility regimes because the state-space covariance matrices adapt to changing latent slopes. The downstream classifier is thus relieved of the computational burden of discovering that adaptation matrix from raw inputs.

The primary failure mode of this transformation is specification error, which occurs when the assumed dynamics of the state-space model fail to map onto the conditions of the asset’s microstructure. The standard local linear trend model relies on continuous Gaussian diffusion. If the true data generation process is instead dominated by discrete Poisson jumps—such as overnight gaps, scheduled FOMC macroeconomic releases, or sudden liquidity vacuums—the linear Gaussian assumptions break down. When a structural price jump occurs, the linear filter cannot snap to the new regime. Instead, it lags, requiring multiple timesteps for the state space gain matrix to fully correct the latent state components. During this structural catch-up phase, the sequence v_t will output an artificial cluster of large values. These values no longer represent an orthogonal market surprises but a deterministic filter correction error.

Researchers are tempted by the aesthetic stability of the a posteriori residual, often extracted via two-sided smoothing algorithms provided by default in many statistical libraries. Because these smoothers utilize future price information from t+1 to T to optimize the state estimate at time t, utilizing them injects terminal forward-looking leakage into the feature matrix. The resulting in-sample backtest will generate flawless, profitable metrics, while the live production system will bleed money.

Dynamic benchmark neutralization

The second transformation points to a different geometric annoyance known as time-varying external beta. The label is the rolling Ordinary Least Squares residual. In systematic portfolio management, this is recognized as the dynamic hedge residual. The core logic dictates that the variance of the target asset y_t is contaminated by the variance of an external factor x_t, which could represent a sector ETF, a broad index future, or a principal component basket. To model the asset’s idiosyncratic state, the researcher must regress the asset against the reference factor and isolate the residual error.

The model specifies:

Because the relationship between financial assets is never static, α_t and β_t cannot be estimated over the full sample. They must be approximated over a rolling window of length w. The parameter vector is computed utilizing the local design matrix X_t-w+1:t. The resulting transformed feature is the out-of-sample or end-of-sample deviation:

This specific subtraction is geometrically distinct from subtracting simple returns or computing a ratio. The dynamic residual projects the asset into the null space of the benchmark over the specified lookback window. The output geometry abandons the absolute coordinate system and adopts a relative, benchmark-neutral coordinate system.

def rolling_regression_residual(
    price_series: pd.Series,
    benchmark_series: pd.Series,
    window: int = 80,) -> pd.Series:
    """
    Computes the dynamic rolling residual of the asset relative to the benchmark.
    """
    y = price_series.rename("log_price")
    x = benchmark_series.rename("log_benchmark")

    X = sm.add_constant(x)
    result = RollingOLS(y, X, window=window).fit()
    params = result.params

    fitted = params["const"] + params["log_benchmark"] * x
    residual = (y - fitted).rename(f"rolling_regression_residual_{window}")

    return residual

In algorithmic trading, raw price vectors deceive predictive models by presenting sector-wide beta expansion as idiosyncratic momentum. A single equity instrument may display severe overbought technical geometry when the underlying condition is uniform sector strength. A cryptocurrency token may exhibit severe downward drift when the entire beta complex is repricing. Feeding raw prices into a deep neural network forces the network to allocate hidden layers to the task of inferring the benchmark matrix and computing the subtraction. By passing û_t, the transformation executes the subtraction analytically, presenting the network with the pure, actionable relative-value deviation.

The geometric shift transforms a trend-dominated price path into a sequence that continually crosses zero, widening when the beta relationship degrades and compressing when the hedge is tight. This sequence describes relative-value dislocations that are primed for statistical arbitrage.

The implementation contains a critical stability trade-off governed by the window parameter w. A minimal window length allows the matrix inversion (X^TX)^-1 to adapt to microstructural shifts in beta, but exposes the parameters to severe estimation noise and matrix ill-conditioning. A maximum window length stabilizes the eigenvalue structure of the covariance matrix but renders the hedge ratio unresponsive to macroeconomic events.

Stochastic trend removal via cointegration

The third data-shape transformation addresses the phenomenon of common stochastic trends. Referred to as the Engle-Granger residual or the cointegration spread, this method supersedes local benchmark neutralization by enforcing an equilibrium hypothesis. It relies on the proof that while two distinct log-price series y_t and x_t may both possess unit roots, a specific linear combination of the two series may be integrated of order 0, yielding a stationary process.

If there exists a parameter vector [α, β] such that the linear combination:

is a stationary process, the variables are defined as cointegrated, and s_t represents the equilibrium spread. The Engle-Granger two-step procedure first estimates the scalar β via ordinary least squares on a causally valid training set, and applies an Augmented Dickey-Fuller test to the residual sequence s_t to reject the null hypothesis of a unit root.

The concept of the cointegration spread over a simple rolling OLS residual lies in its equilibrium framework and the physical market mechanics that enforce it. A dynamic rolling residual states that a local, temporary variance explanation exists. It makes no claims about the terminal behavior of the asset pair. A cointegration differential spread that the components of the two assets are structurally linked to each other by an exogenous economic force. This force could be identical underlying cash flows (in the case of dual-listed equities), arbitrage boundaries enforced by Exchange Traded Fund creation and redemption mechanisms, or capitalized statistical arbitrage desks trading the structural mean. The raw log-prices are permitted to drift into uncharted coordinate space, but the distance metric between them, s_t, is constrained to revert to its historical mean.

The standard Error Correction Model representation models the short-term dynamics of the system as a direct response to the current magnitude of the equilibrium deviation, while controlling for localized high-frequency shocks:

In this structural equation, λ represents the speed-of-adjustment coefficient, and it is the most critical parameter in the relative-value strategy. The predictive alpha is not contained in the absolute nominal level of y_t or x_t. The actionable alpha is isolated within s_t-1, the exact geometric distance from equilibrium. By executing this transformation, the researcher compresses the shared macroeconomic drift and aligns the input feature with the logic of convergence.

The magnitude of λ dictates the interactions of the trading book. A high absolute value of λ implies rapid mean reversion. This creates a high-turnover strategy that requires minimal capital lockup but demands extreme execution efficiency, as the profit margin per trade is small and sensitive to latency. Conversely, a low absolute value of λ implies slow, structural reversion. This dictates a low-turnover, capacity-rich strategy that suffers from capital lockup and extended periods of mark-to-market drawdown when the spread temporarily widens beyond historical norms.

def cointegration_spread(
    price_series: pd.Series,
    partner_series: pd.Series,) -> pd.Series:
    """
    Estimates the Engle-Granger equilibrium spread.
    """
    y = price_series.rename("log_price")
    x = partner_series.rename("log_partner")

    X = sm.add_constant(x)
    fit = sm.OLS(y, X, missing="drop").fit()

    spread = (y - fit.predict(X)).rename("cointegration_spread")
    return spread

Transforming non-stationary prices into a stationary spread state variable provides relief to machine learning algorithms. If a random forest or a gradient boosting machine is fed two raw, drifting I(1) sequences, it will attempt to construct arbitrary, overfit split points in coordinate spaces that the asset has never historically visited. When the asset prices drift beyond the maximum values observed in the training set, the tree algorithm can only output a constant terminal leaf value, breaking the model. Similarly, feeding unbounded I(1) sequences into deep neural networks causes severe gradient instability and forces activation functions into extreme saturation zones. By feeding the optimization algorithm the bounded sequence s_t, the algorithm is restricted to evaluating the magnitude and velocity of the equilibrium error. The model focuses its entire computational capacity on timing the reversion and sizing the risk optimally, rather than wasting matrix operations attempting to subtract non-stationary trends.

Remember that correlation does not imply cointegration. Two assets can exhibit a Pearson correlation coefficient of 0.95 over a multi-year horizon while failing unit root tests on their residuals. Trading this spurious pattern leads to the classic relative-value failure where the spread widens, the model increases the position size assuming mean reversion, and the error variance expands until you face a margin call.

Furthermore, a statistically verified stationary spread is not guaranteed to be economically tradable. The theoretical spread s_t ignores market friction. If the half-life of mean reversion exceeds the margin constraints of the portfolio, the position will be forcibly liquidated before the equilibrium error corrects. Additionally, if the borrow costs on the short leg of the spread exceed the expected mathematical yield of the reversion, the statistical feature is invalid in production.

Send me a copy

Multiscale isolation approach

The fourth transformation steps away from reference benchmarks and equilibrium partners to address internal scale contamination. The algorithm extracts the Haar wavelet detail coefficient. In signal processing, this operator isolates a multiscale high-frequency component while decimating low-frequency drift. This shape transformation is deployed when the researcher has established that the predictive alpha resides within a very specific time scale, and is masked by the simultaneous presence of microstructural noise at lower scales and macro-level cumulative drift at higher scales.

Financial order books process an amalgamation of participants operating at different frequencies. Tick-by-tick is dominated by market makers canceling and replacing quotes. Intraday minutes to hours is dominated by algorithmic execution engines slicing large institutional blocks via VWAP algorithms. Days to weeks is dominated by fundamental portfolio reallocation. A standard differencing operator (calculating a basic log return) compresses all of these distinct horizons into an identical, localized logic. It forces the learning algorithm to separate these overlapping frequency bands using raw function approximation.

The discrete Haar wavelet transform operates as an orthogonal filter bank designed to decouple these scales. It applies a low-pass filter to compute local averages, pushing the slow trend information into a lower-frequency approximation space. Simultaneously, it applies a high-pass filter to compute local differences, capturing the exact detail variation occurring uniquely at that specific scale. For adjacent discrete observations a and b, the fundamental operations are:

Applied recursively in a cascaded architecture, the algorithm generates multiscale approximation coefficients and detail coefficients. A detail coefficient extracted at Level 2 represents the difference of the local averages computed at Level 1. This distinction allows to target a specific frequency band—a precise temporal horizon—rather than defaulting to the single-step increment forced by standard return calculations.

The selection of the Haar wavelet over continuous alternatives (such as the Morlet or Daubechies wavelets) is intentional in financial time series analysis. Continuous wavelets are optimized for smooth signals like acoustics or electromagnetics. Market prices are noisy. The Haar basis function is defined as a step function. It matches the market dislocation, preserving the exact magnitude of the discrete jump without introducing the artificial ringing artifacts or smoothed boundaries common in continuous wavelet convolutions.

To utilize this transformed output in a predictive feature matrix alongside standard, daily-sampled covariates, the implementation requires a length-preserving operation. The standard decimated wavelet transform halves the array length at each level. To align the sequence with the original time index t, the system must upsample the coefficients. In live trading systems, this upsampling must be causal. Utilizing standard linear interpolation imports future values into the current state. The correct engineering solution relies on zero-order hold upsampling, repeating the most recent coefficient forward to prevent any look-ahead bias.

The resulting geometric intervention transforms a drifting, multi-layered log-price path into a sequence clustered around zero, where the amplitude of the oscillations represents the magnitude of the targeted scale-specific burst. The lower-frequency trend components are annihilated.

def haar_detail_same_length(price_series: pd.Series, level: int = 2) -> pd.Series:
    """
    Extracts the multiscale high-frequency detail coefficient using a discrete
    Haar wavelet filter bank and returns it with the same length as the input.
    """
    x = np.asarray(price_series, dtype=float)
    n = len(x)
    coeffs = x.copy()
    current_len = n
    detail = None
    block = 1

    for _ in range(level):
        if current_len < 2:
            break

        used_len = current_len if current_len % 2 == 0 else current_len - 1
        current = coeffs[:used_len]

        avg = (current[0:used_len:2] + current[1:used_len:2]) / np.sqrt(2)
        det = (current[0:used_len:2] - current[1:used_len:2]) / np.sqrt(2)

        coeffs[: len(avg)] = avg
        current_len = len(avg)
        detail = det
        block *= 2

    if detail is None:
        return pd.Series(
            np.nan,
            index=price_series.index,
            name=f"haar_detail_L{level}",)

    upsampled = np.repeat(detail, block)
    if len(upsampled) < n:
        upsampled = np.r_[upsampled, np.repeat(upsampled[-1], n - len(upsampled))]

    return pd.Series(
        upsampled[:n],
        index=price_series.index,
        name=f"haar_detail_L{level}",)

This transformation is critical for horizon-specific execution logic. Consider a temporary order book imbalance generated by digesting a scheduled macroeconomic data release versus the slow absorption of a multi-day block trade. The macro release generates a visible, high-amplitude signature. However, this precise signature remains undetectable within the raw log-price sequence due to the overwhelming magnitude of the concurrent multi-month macro trend drift. By passing the multiscale detail coefficient D_t^(j) to the predictive model, the signal-to-noise ratio is maximized at the exact frequency the strategy is designed to hold the position. The optimization algorithm can recognize the structural footprint of the transient shock.

The limitations of wavelet transformations revolve around boundary conditions, asynchronous sampling, and noise. When applied to raw tick data characterized by bid-ask bounce and irregular timestamps, fine-scale detail coefficients will isolate and amplify the microstructural noise rather than the intended economic event. Wavelet transforms assume a uniform sampling grid, violating this assumption distorts the frequency targeting.

Furthermore, handling the right-hand boundary (the actual live time t) is hazardous. Wavelet filters require data points that straddle the center point of the convolution. At the live edge of the time series, the filter lacks the future data points necessary to complete the calculation. Utilizing symmetric boundary extension—a common default in open-source Python libraries—folds the array back on itself, utilizing data points from t-k as proxies for t+k. While elegant, this corrupts the state variable at the exact moment the trading system requires it. Researchers must enforce causal padding or utilize specialized boundary wavelets to ensure the feature matrix remains uncontaminated.

Bounded state encoding

The fifth and final transformation algorithm in this framework is the rolling range position. While computationally less complex than state-space estimators or orthogonal wavelets, it introduces non-linear geometric modifications. In fact, it converts a non-stationary price path into a bounded state occupancy variable.

Let P_(1),t denote the minimum order statistic and P_(n),t denote the maximum order statistic of the price sequence over a causal trailing window of length w. The rolling range position r_t is defined as:

Assuming the local variance is positive, the output space of r_t is bounded within the closed interval [0, 1]. (The addition of a microscopic constant ε in the denominator is a production requirement to prevent zero-division errors during market halts, extended limit-up/limit-down states, or extreme microstructural illiquidity where the local maximum exactly equals the local minimum).

A feature vector evaluating to 0.05 indicates that the current market price is compressed against its recent lower extreme. A value of 0.98 indicates extreme upper-bound occupancy. This transformation kills the nominal price leveles by using the local variance.

Breakout momentum models, dynamic stop-loss placement, and urgency-based execution routing evaluate expected return based on whether the local boundary conditions are being tested or rejected. Furthermore, bounded interval features are immune to cross-sectional scaling issues. If a portfolio must evaluate a penny stock trading at 1.50 and a mega-cap equity trading at 3,500, passing the raw levels or even standard deviations will warp the loss function. The variable r_t normalizes the topological space, making the states comparable across disparate asset classes.

def rolling_range_position(series: pd.Series, window: int = 60) -> pd.Series:
    """
    Maps the series into [0, 1] based on the rolling min-max range.
    """
    lo = series.rolling(window).min()
    hi = series.rolling(window).max()

    pos = (series - lo) / (hi - lo).replace(0, np.nan)
    return pos.rename(f"range_position_{window}")

The interaction between bounded variables and tree-based learning algorithms (Random Forests, Gradient Boosted Trees) is pretty interesting. These algorithms show great performance when provided with bounded state features. Trees partition coordinate space via orthogonal hyperplanes. If the raw price continues to trend upwards into uncharted numerical area, the tree algorithm can only output the constant value of its terminal leaf node. By mapping the price into r_t, the price drift is recursively folded back into the [0, 1] interval, allowing the decision tree to apply its historically learned logic regarding bound-rejection.

The danger of the rolling range position lies in the non-differentiability at the boundaries and the assumption of local support. The trailing window w enforces a boundary that the market does not acknowledge. The range is an artifact of local sample memory. If w is excessively short, standard Brownian noise will trigger extreme readings of 0 or 1, causing the classifier to generate false breakout signals. If w is excessively long, the range minimum and maximum become anchored to stale macro-regimes, rendering the current fluctuations invisible as the variable hovers near 0.5. The parameter w must be mapped to the temporal memory of the specific market participants the strategy is attempting to exploit.

Okay! Great job today, guys! Solid work. And remember, the full code is in the appendix, sitting there for you to dissect and torture as much as you like. Time to say goodbye. Stay sharp, stay bold, stay unstoppable 📈

PS: What would you do if you found a product with a drawdown of <1% and more than 1 year performance?

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full code

import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.tsa.statespace.structural import UnobservedComponents
from statsmodels.regression.rolling import RollingOLS
from matplotlib.ticker import MaxNLocator


# Data
def generate_synthetic_market_data(n_samples: int = 1000) -> pd.DataFrame:
    """
    Generates realistic log-price data exhibiting stochastic trend,
    benchmark beta exposure, and pair cointegration.
    """
    np.random.seed(42)
    dates = pd.date_range(start="2020-01-01", periods=n_samples, freq="D")

    benchmark_returns = np.random.normal(loc=0.0002, scale=0.01, size=n_samples)
    benchmark_log_price = np.cumsum(benchmark_returns)

    beta_true = 1.2
    idiosyncratic_shocks = np.random.normal(loc=0.0, scale=0.015, size=n_samples)
    idiosyncratic_shocks[400:450] += np.random.normal(loc=0.02, scale=0.05, size=50)

    latent_trend = np.cumsum(
        np.random.normal(loc=-0.0001, scale=0.005, size=n_samples)
    )

    asset1_log_price = (
        beta_true * benchmark_log_price
        + latent_trend
        + idiosyncratic_shocks
    )

    equilibrium_error = np.zeros(n_samples)
    for i in range(1, n_samples):
        equilibrium_error[i] = (
            0.85 * equilibrium_error[i - 1] + np.random.normal(0, 0.008)
        )

    asset2_log_price = asset1_log_price - 0.5 - equilibrium_error

    df = pd.DataFrame(
        {
            "log_benchmark": benchmark_log_price,
            "log_price": asset1_log_price,
            "log_partner": asset2_log_price,
        },
        index=dates,)

    return df



# Transformation algorithms
def state_space(price_series: pd.Series) -> pd.Series:
    """
    Extracts the one-step-ahead forecast error from a local linear trend
    state-space model.
    """
    lp = price_series.rename("log_price")

    model = UnobservedComponents(lp, level="local linear trend")
    result = model.fit(disp=False)

    space = pd.Series(
        result.filter_results.forecasts_error[0],
        index=lp.index,
        name="state_space",)

    space.iloc[0] = np.nan
    return space


def rolling_regression_residual(
    price_series: pd.Series,
    benchmark_series: pd.Series,
    window: int = 80,) -> pd.Series:
    """
    Computes the dynamic rolling residual of the asset relative to the benchmark.
    """
    y = price_series.rename("log_price")
    x = benchmark_series.rename("log_benchmark")

    X = sm.add_constant(x)
    result = RollingOLS(y, X, window=window).fit()
    params = result.params

    fitted = params["const"] + params["log_benchmark"] * x
    residual = (y - fitted).rename(f"rolling_regression_residual_{window}")

    return residual


def cointegration_spread(
    price_series: pd.Series,
    partner_series: pd.Series,) -> pd.Series:
    """
    Estimates the Engle-Granger equilibrium spread.
    """
    y = price_series.rename("log_price")
    x = partner_series.rename("log_partner")

    X = sm.add_constant(x)
    fit = sm.OLS(y, X, missing="drop").fit()

    spread = (y - fit.predict(X)).rename("cointegration_spread")
    return spread


def haar_detail_same_length(price_series: pd.Series, level: int = 2) -> pd.Series:
    """
    Extracts the multiscale high-frequency detail coefficient using a discrete
    Haar wavelet filter bank and returns it with the same length as the input.
    """
    x = np.asarray(price_series, dtype=float)
    n = len(x)
    coeffs = x.copy()
    current_len = n
    detail = None
    block = 1

    for _ in range(level):
        if current_len < 2:
            break

        used_len = current_len if current_len % 2 == 0 else current_len - 1
        current = coeffs[:used_len]

        avg = (current[0:used_len:2] + current[1:used_len:2]) / np.sqrt(2)
        det = (current[0:used_len:2] - current[1:used_len:2]) / np.sqrt(2)

        coeffs[: len(avg)] = avg
        current_len = len(avg)
        detail = det
        block *= 2

    if detail is None:
        return pd.Series(
            np.nan,
            index=price_series.index,
            name=f"haar_detail_L{level}",
        )

    upsampled = np.repeat(detail, block)
    if len(upsampled) < n:
        upsampled = np.r_[upsampled, np.repeat(upsampled[-1], n - len(upsampled))]

    return pd.Series(
        upsampled[:n],
        index=price_series.index,
        name=f"haar_detail_L{level}",)


def rolling_range_position(series: pd.Series, window: int = 60) -> pd.Series:
    """
    Maps the series into [0, 1] based on the rolling min-max range.
    """
    lo = series.rolling(window).min()
    hi = series.rolling(window).max()

    pos = (series - lo) / (hi - lo).replace(0, np.nan)
    return pos.rename(f"range_position_{window}")

Table of contents:

1. Introduction.

2. Global macro after the energy shock.

3. The United States, the axis of growth, finance, and military escalation.

4. The main price-maker in the global economy.

5. China and Asia are the first receivers of the Hormuz shock.

6. The dollar, bond markets, and the return of geopolitical duration risk.

7. NATO, military burden-sharing, and the remapping of Western power.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The global backdrop has entered a harder regime. Energy, shipping, fiscal capacity, military posture, and capital allocation now trade through the same set of constraints. Variables that once sat in separate analytical buckets now belong to a single pricing framework. Oil routes shape inflation. Alliance commitments shape issuance. Industrial policy shapes defence readiness. Currency strength shapes import costs and external adjustment. The market is pricing all of it together.

The central question concerns the kind of growth that can endure when energy friction rises, logistics tighten, and states absorb a larger strategic burden. That shift changes the hierarchy of variables. Security of supply carries more weight. Duration risk carries more weight. Fiscal credibility carries more weight. The ability to convert spending into usable capacity also carries more weight, whether in semiconductors, power systems, refining, shipping, or defence production.

This piece follows that chain from the macro layer into the strategic layer and back again. It begins with the global economy after the energy shock, where an active investment cycle now meets a harsher cost structure. It then turns to the United States, the core balance sheet in the system, where growth leadership, Treasury funding, monetary policy, and force projection converge. From there, the focus shifts to the conflict itself as the marginal driver of price formation across oil, gas, freight, and risk premia.

The analysis then moves to Asia, where direct exposure to Gulf energy flows makes the region the first receiver of the shock in physical, financial, and policy terms. It then examines the dollar and sovereign debt markets, where the length of the shock has become a key variable for yields, term premium, and global liquidity demand. The final section studies NATO and the wider Western security architecture, where rising military spending points to a deeper redistribution of industrial capacity, operational responsibility, and strategic weight.

Check more about all of this here.

US employment situation

491KB ∙ PDF file

Download

Download

Economic bulletin 2026

3.17MB ∙ PDF file

Download

Download

Shorter energy outlook

6.84MB ∙ PDF file

Download

Download

The aim is to frame the current environment the way markets are beginning to price it, as one integrated system. Growth, inflation, war, energy, debt, and alliance structure now move through the same circuit. The edge comes from seeing the connections early, before consensus fully absorbs their implications.

Subscribe now

Global macro after the energy shock

We can say we have two engines that pull in opposite directions. One engine comes from the capital cycle that formed through 2025 and carried into 2026 through technology investment, private-sector adaptation, and supportive demand. The other engine comes from the war-driven energy shock, the tightening of transport routes, and the rise in geopolitical risk premia across commodities, shipping, rates, and equities. The world economy is treated as a system where energy access, market confidence, military escalation, and asset pricing move together. The IMF and OECD now describe the same architecture in institutional language. The IMF’s update still presented a world economy with resilient growth, projecting global output growth at 3.3% in 2026 and 3.2% in 2027, driven in part by technology investment, supportive financial conditions, and private-sector adaptability. By late March, the OECD had redrawn that picture. It projected global growth at 2.9% in 2026 and 3.0% in 2027, with the Middle East conflict, higher energy prices, and supply-chain disruption pushing costs higher and demand lower.

The tension between those two engines defines the whole macro regime. The first engine still matters because technology investment remains large, especially in sectors tied to artificial intelligence, semiconductors, cloud infrastructure, industrial automation, and the digital build-out of the energy and security complex. The IMF highlighted strong performance in exports of semiconductors and related equipment across Asian economies and linked that strength to the rise in information and technology investment. That matters because a technology-led capex cycle creates growth through many channels at once. It supports equipment orders, lifts productivity expectations, raises corporate spending, improves cash-flow visibility for firms near the center of the build-out, and attracts capital toward the economies that lead those value chains. This is one reason the United States has strong relative support. It is also one reason parts of East Asia carried stronger industrial momentum into the year than many analysts expected at the end of 2025.

At the same time, the second engine has become the dominant marginal force in pricing. The OECD says the conflict in the Middle East is testing the resilience of a global economy that had been supported by financial and fiscal conditions and growing demand for artificial intelligence technologies. It states that the halt in shipments through the Strait of Hormuz and the closure or damage of energy infrastructure generated a surge in energy prices and disrupted the global supply of energy and other key commodities, including fertilisers. This is central because it shifts the macro discussion away from the classic debate about whether demand remains firm and toward a more strategic debate about how long the energy shock lasts, how broad the shipping disruption becomes, and how much of the rise in commodity prices passes into wages, services, industrial margins, and inflation expectations.

This leaves the world economy in a narrower corridor than the one investors saw at the beginning of the year. In January, the baseline still looked like a world of steady growth with disinflation in progress. Now that baseline became conditional on the moderation of energy-market disruption from mid-2026 onward. The OECD states that its projections rely on that technical assumption. The economic meaning is clear. A shorter disruption preserves a large part of the prior macro script. A longer disruption transforms the outlook into one shaped by persistent cost pressure, weaker real income growth, broader financial repricing, and stronger dependence on fiscal triage and central-bank credibility.

The inflation story reflects this shift with unusual clarity. The IMF’s update expected global inflation to fall, projecting headline inflation at 3.8% in 2026 and 3.4% in 2027. That forecast came from a world in which supply conditions improved, commodity pricing eased from earlier extremes, and private adaptation softened the impact of trade-policy friction. The OECD’s update already incorporates the war shock. It projects G20 inflation at 4.0% in 2026, which is 1.2 percentage points higher than it expected before the conflict escalated, before easing to 2.7% in 2027 under the assumption that energy pressures fade. This difference between the January IMF view and the March OECD view is one of the cleanest expressions of the regime change. It shows how quickly a global disinflation narrative can bend when energy, logistics, and risk premia move together.

The inflation process has also split into layers. Headline inflation is the first recipient of the shock because oil, gas, fuel transport, chemicals, and fertilisers transmit rapidly into measured prices. Core inflation then becomes the field of contest. If households and firms treat the energy shock as a passing event, core inflation stabilises with less secondary impact. If firms push higher input costs through broader price categories and workers seek compensation for weaker real incomes, core inflation becomes more persistent. The OECD says medium-term inflation expectations already rose following the energy-price spike. That point matters because expectations serve as the bridge between a commodity event and a more durable inflation process. The shape of that bridge has become one of the decisive macro variables.

Growth now rests on dispersion rather than on one shared global rhythm. The United States still benefits from strong AI-related investment, and the OECD projects U.S. growth at 2.0% in 2026 before easing to 1.7% in 2027. China remains a large growth pole at 4.4% in 2026 and 4.3% in 2027 in the OECD baseline, with state support and industrial strength still carrying weight. The euro area carries a weaker profile. The OECD projects euro area growth at 0.8% in 2026 before a move to 1.2% in 2027, helped in part by stronger defence spending. These numbers describe more than regional variation. They describe the way energy exposure, trade composition, industrial structure, and policy room now shape macro outcomes. Economies close to the technology capex wave retain momentum. Economies more exposed to imported energy and external demand face a tighter balance.

That regional dispersion also changes the meaning of policy credibility. In a calmer cycle, policy credibility often refers to a central bank’s ability to anchor inflation and a finance ministry’s ability to sustain debt. In the current cycle, the idea becomes wider. Credibility now includes the ability to secure energy, preserve transport flows, design temporary support measures without weakening the fiscal base, and maintain enough strategic coherence that firms can plan capital expenditure with confidence. The OECD writes that government measures to cushion higher energy prices should be targeted, preserve incentives to lower energy use, and carry clear expiry mechanisms. It also stresses limited fiscal space and the need for credible medium-term adjustment paths. This means macro policy has become an exercise in war-era selectivity rather than a broad stimulus exercise.

The capital-market layer follows the same logic. In the technology engine, markets still reward growth visibility, margin durability, and long-duration cash flows linked to infrastructure, semiconductors, power systems, software, automation, and data architecture. In the energy-shock engine, markets demand compensation for transport friction, raw-material scarcity, rate volatility, and policy uncertainty. The OECD says financial conditions tightened after the outbreak of hostilities, with volatility rising and the previous degree of accommodation shrinking. The important point lies in the combination rather than in a single asset class. Equities, bonds, currencies, commodities, and credit spreads are all trying to price the same unknown at once, namely the duration and reach of the conflict shock. That creates a macro system in which every bond curve carries an energy assumption, every central-bank path carries a shipping assumption, and every equity multiple carries an embedded judgment about geopolitical durability.

The structural dimension runs deeper than the current quarter. The OECD highlights that persistent disruptions to Middle East exports could aggravate shortages of key commodities and produce a broader repricing in financial markets. It also notes that countries in the Gulf serve as an important source of aluminium, helium, bromine, fertiliser inputs, and logistics services. This widens the macro significance of the current conflict. The issue extends beyond oil and gas. It reaches into food systems through fertilisers, into industrial production through metals and specialty materials, and into advanced manufacturing through chemical and memory-chip supply chains. A prolonged disruption therefore reshapes more than headline inflation. It reshapes the cost map of the industrial economy.

The United States, the axis of growth, finance, and military escalation.

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Why a trading hypothesis?

A trading hypothesis is presented as the first scientific form of a market idea, one that transforms intuition into a structured claim with an activating condition, an expected response, a time horizon, and a proposed mechanism. The lecture develops this framework as the bridge between market logic and empirical evidence, and explains how restriction, precision, and coherence turn a vague observation into a research object that can be tested, challenged, and refined.

What’s inside:

1. Why a trading hypothesis matters. A trading hypothesis is the first formal research object in quantitative trading because it gives direction, causal meaning, and empirical consequences to an otherwise informal market observation.

2. Hypothesis as a conditional claim about market behavior. The lecture defines a genuine hypothesis as a structured conditional statement that specifies when a market condition appears, what response should follow, over what horizon, and because of what underlying process.

3. The activating condition. A hypothesis begins with a trigger that identifies when the market enters a meaningful state, such as a volatility expansion, spread dislocation, order flow imbalance, or liquidity shock, so the sample becomes selective in a principled way.

4. The expected market response. The claim must state what should happen after activation by defining the object of movement, the direction of adjustment, and the scale or relevance of that movement, so the idea acquires a clear empirical identity.

5. The horizon over which the response should emerge. Time is treated as part of the hypothesis itself because different mechanisms operate on different clocks, and the validity of the research depends on matching the response window to the real rhythm of the market process.

6. A bridge between market logic and empirical evidence. The lecture shows that a good hypothesis connects two domains at once, the logic of participants, frictions, and structural forces on one side, and the language of variables, samples, outcomes, and comparisons on the other.

7. Empirical signature and supporting evidence. A strong hypothesis does not predict profitability alone, but a specific patterned trace in the data, including direction, structure, timing, conditional strength, and comparative contrast, so support comes from alignment rather than from one backtest statistic.

8. The hypothesis as a restricted object. The lecture explains that every valid hypothesis must be restricted to a coherent asset universe, a defined market state, and a proper time horizon, because restriction is what gives the claim scientific shape and protects it from vague breadth.

Sample

1.89MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Why transform data?

3. The risks of data transformation.

4. The memory dilemma and distributional shifts.

   1. The fractional differencing paradigm.

   2. Higher moments and the Cornish-Fisher expansion.

5. Time and variance.

   1. Time deformations and information clocks.

   2. Structural volatility standardization.

6. Cross-sectional geometry and noise mitigation.

   1. Entropy-based outlier mitigation without information loss.

   2. Cross-sectional orthogonalization.

7. Non-linear embedding and phase-space reconstruction.

8. The unified pipeline.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Raw market data is the closest thing we have to the market’s observable dynamics, so the instinct to leave it untouched feels sensible. But predictive models consume numerical representations. Between the tape and the optimizer there is always a translation step, whether explicit or hidden, and that translation determines which structures remain visible, which ones are attenuated, and which ones disappear. In financial machine learning, representation is part of the model itself. Prices, returns, clocks, scales, and cross-sectional coordinates all define different statistical objects, and each one presents a different geometry to the learning algorithm.

The literature on volume clocks showed that market activity is better understood in an event-driven metric than in a chronological one. The literature on fractional differencing showed that differencing need not be an all-or-nothing choice, but can instead vary continuously to preserve long-range structure while moving toward stationarity. The literature on conditional heteroskedasticity showed that variance itself has memory and must be modeled as a dynamic object rather than treated as a fixed nuisance parameter.

Here an old one about volume clock:

The volume clock

661KB ∙ PDF file

Download

Download

Yet every transformation carries a cost. A bad transformation can alter the topology of the series, flatten its tails, suppress its regime structure, and leak future information into the past. The goal here is to show that each transformation answers a specific failure mode of financial data. Some methods regularize the clock. Others preserve memory while controlling non-stationarity. Others stabilize conditional variance, correct tail asymmetry, reduce cross-sectional noise, or reconstruct hidden structure from a single observable. When these operations are applied in the right order, transformation stops being a destructive preprocessing ritual and becomes a way of making market structure legible to statistical and machine-learning models.

This is the real question is “how to make data look cleaner, but how to modify it without amputating the very dependencies we hope to trade.”

Why transform data?

If raw data represents the perfect economic reality of the market, why do we need to transform it at all? Why not feed the raw sequence of events, the actual execution prices and sizes, directly into our predictive algorithms?

The answer lies in the rigid constraints of linear algebra and numerical optimization. Machine learning algorithms, from basic Ordinary Least Squares to stochastic gradient descent solvers, don’t understand economics, market microstructure, or the concept of fiat currency. They only understand geometry and topology. They evaluate the distance between points in a high-dimensional space.

Let X∈R^TxN be a feature matrix of raw asset prices over T time steps for N assets. Raw prices are strictly non-stationary I(1) processes, their mean and variance scale continually with time. If we attempt to fit a linear model to minimize the empirical risk, we must compute the inverse covariance matrix (X^TX)^-1 to solve for the coefficient vector. Because the raw prices share a common, unbounded macro trend, the vectors in X are collinear. The matrix X^TX becomes nearly singular. Its condition number explodes, meaning the matrix inversion becomes numerically unstable. The optimizer can’t find a unique global minimum. Instead, it yields spurious regressions. The model will report a massive R²X value and highly significant t-statistics, suggesting it has found a relationship, when in reality, it has correlated two independent random walks that happen to drift in the same general direction.

Furthermore, deep learning architectures—specifically Recurrent Neural Networks and Long Short-Term Memory networks—require bounded, zero-centered inputs to propagate gradients efficiently through time. If we feed raw, un-scaled prices into a network utilizing sigmoid or hyperbolic tangent (tanh) activation functions, the architecture breaks down immediately. If an asset’s price has drifted from $10 to $400 over a decade, feeding the value $400 into a tanh function yields an output that is computationally indistinguishable from $1.0. The local derivative of the activation function at this asymptote is zero. When the backpropagation algorithm attempts to calculate the chain rule to update the network’s weights, it multiplies by this zero derivative, causing the vanishing gradient problem. The network stops learning on the very first epoch. The weights freeze.

Subscribe now

The risks of data transformation

For some models transformation is a necessity, however it is the highest-risk component of the entire pipeline. Every operation applied to a temporal series acts as a filter. Applying an operator T(·) to a raw series X_t involves a manipulation of entropy. You are altering the properties of the dataset. If the transformation isn’t an exact bijection tailored to the specific distributional properties of the underlying asset, it is a destructive process that deletes alpha.

1. The primary risk is geometric distortion. Quants attempt to force non-Gaussian market data into Gaussian topologies using standardized scalers. Let μ_t and σ_t be the rolling mean and standard deviation. The standard Z-score transformation is Z_t = (X_t - μ_t) / σ_t. This operation is ubiquitous in basic data science, but it is catastrophic in finance. As established by empirical market data, equity returns exhibit severe negative skewness and massive excess kurtosis. By applying this linear transformation, we assume the data is symmetric around the mean. We geometrically compress the left tail and stretch the right tail. When the optimizer processes Z_t, it assigns incorrect probabilistic weights to the downside risk. The transformation has lied to the algorithm about the probability of ruin, mapping a 6σ market crash into a normalized space that makes it look like a mild, acceptable deviation.

2. The second critical risk is information destruction via arbitrary numerical thresholds. Common quantitative practices like hard-clipping outliers or dropping low-volume trading hours manually alter the sequence of the time series to make the data cleaner for the optimizer. If a quant runs a standard Winsorization protocol and drops an outlier return of -12% down to a hard cap of -3% simply because the extreme value “ruins the scale of the loss function,” they have committed a severe theoretical error. They have deleted the exact moment of maximum market inefficiency. That -12% print is the precise data point where algorithmic alpha is generated. It represents a moment where traders capitulated and forced liquidations occurred. Transforming the data to make it look smooth and continuous for an optimizer strips out the true signal, leaving behind highly stationary, useless white noise.

3. The third, and most fatal, risk is structural data leakage, also known as look-ahead bias. This occurs when a transformation operator inadvertently uses information from the future to scale or center the data of the past. If a quant runs a Principal Component Analysis (PCA) on the entire matrix X spanning from 2010 to 2024 to find orthogonal risk factors, and then trains a predictive model sequentially on the data from 2015, the model is compromised. The eigenvectors calculated by the global PCA contain variance data from the 2020 pandemic market crash. The transformation has leaked future volatility structures into the historical training set.

This leakage happens with basic functions like global mean subtraction or min-max scaling if the boundaries aren’t rolling. The model will appear profitable during out-of-sample backtesting because the transformed features implicitly “know” the future basis vectors and maximum bounds of the market. In live trading, where the production operator only has access to historical data up to time t, this false performance collapses instantly.

The memory dilemma and distributional shifts.

The fundamental dilemma in quantitative modeling revolves around the requirement for stationarity and the predictive requirement for memory. To train any robust inferential model, whether a vector autoregression or a neural architecture, the input data must be stationary. The statistical properties of the series—mean, variance, and autocorrelation structure—can’t vary over time. If they do, the model learns a regime that ceases to exist the moment it is deployed in live trading.

When you feed non-stationary price levels into a learning algorithm, you violate the core assumptions of the optimizer. As previously mentioned, in linear models like Ordinary Least Squares, non-stationarity leads to spurious regressions where the model identifies high R² values and significant t-statistics between independent random walks. In deep learning architectures like Recurrent Neural Networks, non-stationary variance causes the gradient descent process to either vanish into zero or explode to infinity, preventing convergence.

However, the standard mathematical transformations used to enforce this required stationarity destroy the predictive power of the dataset. Financial time series, specifically price levels, are mostly I(1) processes, meaning they possess a unit root. The conventional approach to solving this is applying an integer differentiation, calculating the first-order difference or the standard log-return. This transformation achieves stationarity, yielding a series with a constant mean and finite variance. Yet, it erases the long-term memory of the process. Prices retain the entire history of market shocks, but returns only remember the shock of the immediately preceding period. By integer differencing, we obtain stationary noise that is impossible to predict.

Furthermore, integer differencing often leads to over-differencing. When you subtract X_t-1 from X_t to create a return series, you inject a moving average MA(1) component with a coefficient of -1 into the resulting noise. This artificially creates a strong negative autocorrelation at the first lag. Algorithms trained on this over-differenced data will discover short-term mean reversion that doesn’t exist in the market microstructure, leading to systematic trading losses.

The risks of relying on models built atop these conventional transformations are severe. We force non-stationary data through linear, memoryless filters, creating an illusion of predictability. We evaluate stationarity using binary heuristics like the Augmented Dickey-Fuller test, treating it as an absolute state rather than a continuous spectrum. Models trained on first-differenced data often exhibit high backtest performance due to noise-fitting but fail rapidly out-of-sample because the underlying structural dependencies of the market have been severed.

Do you rememenber the Quant Meltdown? Statistical arbitrage funds, operating with massive leverage, relied on mean-reversion models trained on orthogonalized, first-differenced equity returns. These models assumed that the residuals of their transformations were stationary, memoryless white noise, safely confined to narrow bands. When sudden, massive liquidations hit the market due to subprime mortgage exposure, the assumed independence of these time series broke down.

The residuals exhibited extreme, long-term memory. A portfolio liquidation creates a directional order flow imbalance that persists across days, resurrecting the unit root that the quants thought they had differenced away. The standard transformations had hidden it under normal, high-liquidity market conditions. When the regime shifted, the models failed because the foundation of their data transformation was inadequate. The algorithms kept doubling down on mean-reverting bets while the market displayed a persistent, memory-driven structural break.

The fractional differencing paradigm

We must abandon the misconception that differentiation is an integer operator. In the context of trading, forcing a time series into either a raw price format (where d=0) or a returns format (where d=1) is an arbitrary constraint. The correct approach is to treat differentiation as a continuous domain operator, allowing us to find the exact fractional value of d that achieves stationarity while preserving the maximum possible amount of memory.

We define the backshift operator B such that BX_t = X_t-1. For any real number d, the fractional difference operator (1-B)^d can be expanded using the infinite binomial series:

This expansion yields an infinite sequence of weights ω_k that we apply to past observations.

Notice the recursive property of these weights:

When d=1, the math is trivial: ω₀=1, ω₁=-1, and all subsequent weights are zero. This describes the standard first-order difference. The memory of the series is truncated at one period.

But consider what happens when d is a fraction, say 0.5. The sequence of weights becomes ω₀=1, ω₁=-0.5, ω₂=-0.125, ω₃=-0.0625, and so on. The weights decay asymptotically to zero rather than abruptly terminating. This means X_t is transformed using an exponentially decaying window of its entire history. The current value of the transformed series is heavily influenced by yesterday’s price, moderately influenced by last week’s price, and influenced by the price a year ago. This try to mirror the market information dissemination.

In practice, financial time series like equity indices or major fiat currencies achieve ADF stationarity at d values ranging between 0.3 and 0.4. This is an empirical fact. It proves that by using standard log-returns (d=1), the quantitative finance industry is discarding 60% to 70% of the useful memory contained in the price data.

To implement algorithm, we calculate the weights iteratively and apply a fixed-width window to prevent data leakage and memory overflow. We establish a tolerance threshold τ (often set to 10^-4 or 10^-5) and drop weights where |ω_k| < τ. This generates a fixed lookback window, typically spanning a few thousand observations depending on the chosen threshold.

We must use a fixed-width window rather than an expanding window. If we apply fractional differencing using the entire available history of the series starting from t=0, the number of weights applied to the current observation grows as time progresses. This causes the variance of the transformed series to drift, violating the finite variance requirement of stationarity. The fixed-width window ensures that the exact same mathematical operation is applied to every observation, maintaining a constant variance profile.

Let’s implement this.

import numpy as np
import pandas as pd
from statsmodels.tsa.stattools import adfuller

def get_weights_ffd(d, length, threshold=1e-5):
    """
    Calculate weights for fractional differentiation using a fixed-width window.
    Weights iteratively decay, preserving continuous memory constraints.
    """
    weights = [1.0]
    k = 1
    while True:
        weight_k = -weights[-1] * (d - k + 1) / k
        if abs(weight_k) < threshold and k >= length:
            break
        weights.append(weight_k)
        k += 1
        if k > 10000:  # Safety break for massive arrays
            break
    return np.array(weights[::-1])

def fractional_diff(series, d, threshold=1e-5):
    """
    Applies fractional differencing to a pandas Series to maintain memory 
    while achieving strict mathematical stationarity.
    """
    weights = get_weights_ffd(d, length=1, threshold=threshold)
    width = len(weights) - 1
    
    df = pd.Series(index=series.index, dtype=float)
    
    # Apply the dot product across the fixed-width rolling window
    for i in range(width, len(series)):
        window = series.iloc[i - width : i + 1]
        df.iloc[i] = np.dot(weights, window)
        
    return df.dropna()

def optimize_fractional_d(series, d_range=np.arange(0.1, 1.0, 0.1), p_value_threshold=0.05):
    """
    Finds the minimum fractional d that successfully passes the Augmented 
    Dickey-Fuller (ADF) test for stationarity, maximizing retained memory.
    """
    for d in d_range:
        diffed = fractional_diff(series, d)
        if len(diffed) > 10:
            adf_stat = adfuller(diffed, maxlag=1, regression='c', autolag=None)
            if adf_stat[1] < p_value_threshold:
                return d
    return 1.0

This gives us the orange series you see below. Do you notice the difference compared to a simple differentiation?

Higher moments and the Cornish-Fisher expansion

Quants frequently take log-returns, subtract the rolling mean, and divide by the rolling standard deviation, assuming the resulting series z follows a standard normal distribution N(0,1). Sound familiar? This is the foundation of the Z-score, a metric that is built into the base functionality of almost every data science library.

Financial returns don’t follow a Gaussian distribution, they exhibit significant skewness and excess kurtosis. Equity markets, for example, display structural negative skew, prices drop much faster than they rise due to the mechanics of margin calls, stop-loss triggering, and fear-driven liquidity vacuums. Besides, the phenomenon of volatility clustering generates excess kurtosis, creating a distribution with a sharp central peak and fat tails.

Standardizing without accounting for these higher moments leads to severe underestimation of tail risk and generates false trading signals. Consider a standard normal distribution. A 5-standard-deviation event has a probability of approximately 2.8x10^-7, which translates to an expected occurrence of once every 13,800 years of daily trading. Yet, in live equity markets, 5-sigma moves happen every few years. If an algorithm feeds a raw 5-sigma Z-score into a neural network, the model interprets it as an impossible anomaly and the assigned weights will skew, breaking the loss function.

To transform a time series into a Gaussian-equivalent state, we must incorporate its empirical skewness (S) and excess kurtosis (K). We achieve this via the Cornish-Fisher expansion, which provides a framework to estimate the quantiles of a non-normal distribution based on its specific cumulants.

If Z_α is the α-quantile of the standard normal distribution, the corresponding quantile W_α of our financial time series can be approximated as:

However, in feature engineering for predictive modeling, we often need the inverse transformation. We have an observed standardized return w coming from the live market, and we want to map it back to a normal variable z so that our downstream linear models and gradient descent algorithms function correctly.

We compute the rolling skewness and rolling excess kurtosis of the time series. Estimating the third and fourth moments requires significant data to achieve statistical stability, as outliers heavily distort these metrics. Therefore, the lookback window for S and K must be longer than the window used for the mean and variance—often spanning two to three years of daily data.

Once we establish stable estimates for the cumulants, we apply the inverse Cornish-Fisher expansion to normalize the data. We map the empirical quantiles of the fat-tailed data to the theoretical quantiles of the standard normal distribution.

This adjustment guarantees that a severe market shock is processed correctly by the predictive architecture. A drop that registers as a 5-sigma event in raw standard deviations might map to a much more manageable 2.5-sigma event in the Cornish-Fisher adjusted space, reflecting its true probabilistic reality in a fat-tailed environment.

import scipy.stats as stats
import pandas as pd

def cornish_fisher_transform(returns):
    """
    Normalizes higher moments (skewness and kurtosis) mapping an empirically 
    fat-tailed financial series back to a strictly Gaussian space.
    """
    # Standardize the data first based on initial 2 moments
    z = (returns - returns.mean()) / returns.std()
    
    # In a strict ML feature engineering pipeline, we map the empirical quantiles 
    # of the fat-tailed data directly to the theoretical quantiles of the standard normal distribution
    
    ranks = z.rank(pct=True)
    
    # Map back to standard normal quantiles. We use the PPF (Percent Point Function)
    normalized = stats.norm.ppf(ranks)
    
    return pd.Series(normalized, index=returns.index)

I admit I like this change in silhouette. In the next post, we’ll talk about changes in shapes, I have some ideas that might be quite interesting.

As can be seen, our distribution now looks more like a normal distribution than the classic silhouette of financial returns.

Time and variance

Data transformation involves addressing how we sample the data itself. The most pervasive obstacle in modeling financial markets is the unquestioned reliance on chronological time. Across the industry, the default behavior is to extract open, high, low, and close prices at fixed chronological intervals—minute bars, hourly bars, daily bars. This framework is a legacy artifact of print newspapers and early charting software, and it is disjointed from the reality of market micro-structure.

Market information doesn’t arrive in a continuous, uniform flow. It arrives in distinct bursts driven by human reaction and algorithmic execution. The central limit order book operates on an event-driven basis. The matching engine processes incoming FIX messages—adds, cancels, and modifications—sequentially. It has no internal concept of a “minute.” During the opening auction, or in the milliseconds following a critical macroeconomic data release like the Non-Farm Payrolls, thousands of trades occur per second. These specific moments carry massive amounts of market-moving information, representing true price discovery.

During the mid-day lull or an overnight electronic session, hours can pass with negligible volume and absolute zero new information entering the system. When we sample data chronologically, we force these disparate states into sized computational intervals. This creates a severe statistical anomaly known as a zero-inflated distribution. If you sample an illiquid asset using one-second bars, 90% of your dataset will consist of zero-returns because no trades occurred in those specific seconds. When you feed a neural network a continuous string of zero-returns, the gradient vectors collapse. The algorithm determines that the optimal prediction is always zero, stalling the learning process.

The obstacle this creates for data transformation is systemic. Chronological sampling guarantees heteroskedasticity—the variance of the data is unstable. The standard variance of a one-minute bar at 09:30 AM is an order of magnitude larger than the variance of a one-minute bar at 12:30 PM. This phenomenon creates the well-documented U-shaped intraday volatility curve. The beginning and end of the trading session contain massive variance, while the middle contains almost none.

Furthermore, chronological sampling leads to massive oversampling of low-information periods and dangerous undersampling of high-information periods. If we attempt to apply technical indicators like moving averages, momentum oscillators, or machine learning models to chronological bars, we are feeding the algorithms a distorted geometry of the market.

Time deformations and information clocks

To resolve the obstacle of chronological heteroskedasticity, we must change the domain of the temporal series. We must transition from chronological time to information time. Information can be approximated by the volume of shares traded or the total fiat value exchanged. We implement this through Time Deformations, specifically by constructing Volume Clocks or Dollar Clocks.

While some practitioners use Tick Bars (sampling every N transactions) or Volume Bars (sampling every V shares), these are suboptimal over long horizons. Tick bars are vulnerable to order fragmentation. A large institutional order sliced into 100 smaller algorithmic executions via a Time-Weighted Average Price algorithm will accelerate a tick clock without actually injecting new economic information.

You can go deeper about this kind of methods here:

Volume bars are better but remain vulnerable to corporate actions like stock splits or long-term price appreciation. Consider a multi-year backtest on a high-growth technology equity. A 10,000-share bar of a stock trading at $10 in year one represents $100,000 of market participation. If that same stock appreciates to $200 by year five, a 10,000-share bar now represents $2,000,000 of market participation. By using volume bars, the implicit information threshold has unknowingly shifted by a factor of twenty. The statistical properties of the bars from year one are incomparable to the bars from year five.

Dollar Bars correct these deficiencies, instead of sampling a new data point every time t seconds pass, we sample a new data point every time a cumulative fiat threshold θ is reached. Let v_i be the volume of the i-th tick and p_i be its price. The cumulative dollar volume traded at tick t is

We define the sampling times T_k such that:

When we sample the time series at these intervals T_k, we generate Dollar Bars. The threshold θ is often calculated dynamically, set to a fraction of the 50-day moving average of the daily dollar volume, ensuring the clock ticks at a consistent daily frequency even across decades of data.

The transformation from chronological space to dollar-volume space has statistical implications that directly improve model performance. Empirical studies demonstrate that the returns of dollar bars exhibit lower serial correlation. Indeed, they reduce the excess kurtosis found in chronological data. By sampling based on fiat flow, we eliminate the zero-return intervals of the overnight session and parse the massive volatility spikes of the open into multiple, manageable bars. They recover the properties of an independent and identically distributed.

By defining the series through the Dollar Clock, we adjust for both intraday seasonality and macro-regime shifts. A single Dollar Bar might span exactly 1.2 seconds during the market open, capturing the rapid influx of high-density information with perfect granularity. The subsequent Dollar Bar might span 45 minutes during the lunch hour. The variance of the returns across these bars becomes more stable.

import pandas as pd
import numpy as np

def generate_dollar_bars(ticks_df, dollar_threshold):
    """
    Transforms chronological tick data into Dollar-Volume bars to eliminate 
    chronological heteroskedasticity.
    """
    # Calculate exact fiat commitment per tick
    ticks_df['dollar_value'] = ticks_df['price'] * ticks_df['volume']
    ticks_df['cumulative_dollars'] = ticks_df['dollar_value'].cumsum()
    
    # Determine discrete bar groupings based on cumulative flow
    ticks_df['bar_id'] = (ticks_df['cumulative_dollars'] // dollar_threshold).astype(int)
    
    # Aggregate raw ticks into clean OHLCV structural geometry
    bars = ticks_df.groupby('bar_id').agg(
        open=('price', 'first'),
        high=('price', 'max'),
        low=('price', 'min'),
        close=('price', 'last'),
        volume=('volume', 'sum'),
        dollars=('dollar_value', 'sum')
    )
    
    return bars

Let’s take an image from one of the previous articles. If you look closely, you’ll see how sampling results in fewer bars, but also less noise.

Structural volatility standardization

Once the time series is sampled correctly using an information clock, we face the next critical transformation hurdle: volatility clustering. Even within time-deformed series, prolonged macroeconomic regimes—such as quantitative tightening cycles, global liquidity crises, or prolonged zero-interest-rate environments—cause the baseline variance of the asset to expand and contract over weeks and months. The standard quantitative practice to normalize this behavior is to standardize returns by dividing them by a simple rolling standard deviation.

This naive approach is flawed and introduces dangerous artifacts into the data matrix. A rolling standard deviation applies an equal, flat weight to all observations within the lookback window and absolute zero weight to everything outside it. Consider the exact mechanics of this operation when a massive exogenous shock enters the window. Suppose we use a 30-period rolling window. On period 1, the market drops by 10%. The estimated volatility spikes and remains elevated at a constant plateau for the next 29 periods. During this time, the true market volatility may have completely subsided, but our denominator remains massive, compressing the transformed returns and suppressing valid trading signals.

Then, exactly 30 periods later, when that specific 10% shock falls out of the lookback window, the estimated volatility drops sharply and discontinuously. This creates a mathematically induced ghost effect. The algorithm perceives a sudden collapse in market volatility, generating spurious trading signals and artificial regime shifts. A normal 0.5% return on period 31 is suddenly divided by a tiny denominator, creating a massive standardized value. In reality, absolutely nothing happened in the live market on that specific day. The anomaly was purely a structural failure of the flat rolling window equation.

The classical transformation for volatility standardization relies on Generalized Autoregressive Conditional Heteroskedasticity ak GARCH models. Rather than relying on a flat, memoryless rolling window, we use a structural probabilistic model to estimate the conditional variance σ_t² at each specific time step. For a standard GARCH(1,1) specification, the conditional variance is modeled as:

Where ε is the squared residual of the previous period (the ARCH term, representing immediate short-term market panic), and σ is the variance of the previous period (the GARCH term, representing long-term structural persistence). The constant ω defines the baseline variance level.

The power of this formulation lies in the interplay between α and β. A high α indicates a market that reacts to new information, while a high β indicates a market where volatility shocks persist for extended periods. To transform the temporal series, we fit these parameters ω, α, and β using maximum likelihood estimation over an expanding or long-term rolling window. We then filter the raw returns r_t by dividing them by the modeled conditional volatility:

This GARCH-standardized series reacts to volatility because the GARCH model incorporates an exponential autoregressive decay component (β), the influence of a massive price shock diminishes over time. It never drops off a cliff. The denominator decays according to the empirical persistence of the market, eliminating the ghost effects caused by naive rolling windows.

This transformation contextualizes price action for the downstream inferential models. In a quiet market with low σ_t, a 2% directional move results in a value that correctly identifies an anomaly. On the other hand, in a volatile market where VIX is trading above 40, a 2% move is divided by σ_t.

The transformation strips out the predictable component of the volatility clustering, leaving behind a standardized, stationary residual series. It forces the predictive algorithm to focus on directional alpha of the price movement. By factoring out the structural volatility, the features fed into the final machine learning layer have a linear relationship with the future expected returns, improving the optimizer’s convergence rate.

from arch import arch_model
import warnings

def garch_standardize(returns):
    """
    Standardizes returns using structural conditional volatility estimated 
    from a GARCH(1,1) model, completely bypassing the 'ghost effects' 
    of naive flat rolling windows.
    """
    # Rescale returns to avoid numerical precision errors in the MLE optimizer
    scaled_returns = returns * 100 
    
    # Define a pure volatility model assuming a zero-mean return 
    am = arch_model(scaled_returns, vol='Garch', p=1, q=1, mean='Zero')
    
    # Fit parameters via Maximum Likelihood Estimation
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        res = am.fit(disp='off')
    
    # Extract conditional volatility, scale back down, and apply standardizer
    conditional_volatility = res.conditional_volatility / 100
    standardized_returns = returns / conditional_volatility
    
    return standardized_returns

Remember that these are just simple examples to give you a complete picture of how and why to transform data. There are many more approaches and techniques for transforming data. In this case, if we continued with the example and used toy data, we would see something like this, and if we wanted market data, we would see something like the image above.

Cross-sectional geometry and noise mitigation

Up to this point, our transformations have focused on the isolated temporal dynamics of a single series—adjusting its memory, clock, and variance. However, institutional algorithmic trading doesn’t operate on single assets in isolation. We evaluate massive, dense feature matrices spanning hundreds of correlated instruments, exposing our optimizers to severe geometric hazards. The two most destructive issues in this cross-sectional space are the gradient-exploding impact of extreme microstructure shocks and the matrix-destabilizing curse of multicollinearity. To prepare our data for complex inferential models without erasing the underlying market signal, we must implement noise mitigation. In the following sections, we introduce two key approaches to solve outlier compression that preserves topological rank without triggering numerical overflow. Then, we apply Random Matrix Theory to orthogonally decouple and denoise the cross-sectional feature space, ensuring our covariance matrices remain mathematically stable.

Send me a copy

Entropy-based outlier mitigation without information loss

Financial time series contain severe outliers. Broken trades, massive exogenous shocks, and unexpected central bank interventions create data points that lie far beyond the expected bounds of any standard probability distribution. Leaving these unaddressed destroys the gradient descent processes. Most optimization functions, particularly those relying on Mean Squared Error, penalize large deviations. A 10σ price shock creates a gradient update that is a hundred times larger than a standard 1σ move. If this raw data is fed into a neural network, the weights will update so violently that the model diverges, forgetting previously learned market structures.

The traditional quantitative response to this problem is Winsorization: setting a static threshold (e.g., the 1st and 99th percentiles) and hard-capping any values beyond those limits. If a return exceeds the 99th percentile, it is rewritten as the 99th percentile value.

This approach is destructive and theoretically unsound. In algorithmic trading, extreme events contain the most critical alpha. Hard-capping an extreme move flattens the local gradient of the price action, deleting information about market capitulation, liquidity voids, and directional conviction. If we cap the data, a +5% move and a +15% move become identical to the algorithm. The algorithm loses the ability to distinguish between high volatility and a fundamental structural break. We need a transformation that mitigates the numerical impact of the outlier on the model’s loss function without erasing the topological information and rank order it provides.

We achieve this via trimming based on Shannon Entropy. Instead of an arbitrary, static percentile, we compress the tails of the distribution using a soft-clipping function parameterized by the entropy of the rolling lookback window.

Let H(X) be the Shannon Entropy of the time series X over a recent window. We calculate this by discretizing the continuous return space into a set of B fixed bins and computing the empirical probability p(x_i) of a return falling into the i-th bin:

Shannon Entropy measures the unpredictability or noise of the local market regime. If prices are bouncing randomly within a tight, mean-reverting range, the probability mass is spread across many bins, resulting in high entropy. If the market is experiencing a persistent, directional trend with low noise, the probability mass concentrates in specific bins, resulting in low entropy.

We define a threshold τ that scales inversely with this local entropy. When entropy is high (noise is elevated), the threshold tightens to compress spurious microstructure spikes. When entropy is low (a clear, persistent trend is forming), the threshold widens to capture the full magnitude of the directional move without interference.

We then apply a non-linear squashing function, the hyperbolic tangent, to the data points that exceed this dynamic τ_t.

Where λ controls the asymptotic limit of the compression. This entropy-based transformation acts as a differential shock absorber. The tanh function guarantees monotonicity. A +15% move will always compute to a higher transformed value than a +5% move, preserving the exact sequential logic and rank order of the time series. However, the numerical value is compressed into a safe bound (bounded by τ_t + λ). We preserve the exact directional information of the capitulation while neutralizing the numerical hazard that causes gradient explosion.

import numpy as np
import pandas as pd

def shannon_entropy(series, bins=50):
    """Calculates the Shannon Entropy of a time series chunk."""
    hist, _ = np.histogram(series, bins=bins, density=True)
    hist = hist[hist > 0] # Avoid log(0)
    return -np.sum(hist * np.log2(hist))

def entropy_winsorization(series, window=100, base_threshold=3.0, lambda_val=1.5):
    """
    Soft-clips outliers based on dynamic rolling entropy.
    Preserves topological information while guarding against gradient explosion.
    """
    cleaned_series = series.copy()
    
    for i in range(window, len(series)):
        chunk = series.iloc[i-window:i]
        entropy = shannon_entropy(chunk)
        
        dynamic_tau = base_threshold * (1.0 / (entropy + 0.1)) 
        current_val = series.iloc[i]
        
        if abs(current_val) > dynamic_tau:
            # Apply tanh soft clipping
            sign = np.sign(current_val)
            clipped = dynamic_tau + lambda_val * np.tanh((abs(current_val) - dynamic_tau) / lambda_val)
            cleaned_series.iloc[i] = clipped * sign
            
    return cleaned_series

This is starting to look more like a series that you can trade on.

Cross-sectional orthogonalization

Quants don’t usually model a single time series in isolation. It’s pretty common to evaluate dense feature matrices spanning dozens of correlated assets, hundreds of technical indicators, or entire statistical arbitrage baskets. When we transform these multiple temporal series into a unified dataset, we immediately encounter the curse of multicollinearity.

If we feed highly correlated time series—for example, the minute-by-minute returns of SPY, QQQ, and IWM—into a linear predictive model, the covariance matrix X^TX becomes nearly singular. Because the vectors are almost parallel in geometric space, the determinant of X^TX approaches zero. When the algorithm attempts to invert this matrix to calculate coefficients via (X^TX)^-1, the condition number explodes. The resulting feature weights become hyper-sensitive to microscopic noise. The model might assign a weight of +1500 to SPY and -1499 to QQQ just to fit a tiny basis point of random variance. This leads to catastrophic overfitting and immediate out-of-sample failure.

The required transformation is Cross-Sectional Orthogonalization, executed via PCA or Gram-Schmidt orthonormalization across the time domain. By transforming the raw, correlated feature space into an orthogonal basis, we geometrically decouple the variance.

Let X be our T ∈ N matrix of standardized time series. We compute the empirical covariance matrix

Because Σ is symmetric and positive semi-definite, we can perform an eigendecomposition to find the orthogonal eigenvectors W and their corresponding eigenvalues λ:

The eigenvectors represent the orthogonal directions of maximum variance in the market. The time series are then transformed into Principal Components Z:

The resulting matrix Z consists of N uncorrelated time series. The dot product of any two distinct columns in Z is zero. In quantitative finance, these components have direct, tradable economic interpretations. The first principal component (Z₁), associated with the largest eigenvalue, universally represents the underlying market beta—the systemic risk factor moving all assets. The second and third components represent sector rotations, value-growth spreads, or distinct macroeconomic risk premia.

By applying this transformation, we solve the multicollinearity problem. The optimizer can now assign weights to independent risk factors. Furthermore, we leverage this orthogonalization to denoise the entire system using Random Matrix Theory, applying the Marchenko-Pastur distribution.

Not all computed eigenvectors contain actual signal. Eigenvectors associated with eigenvalues near zero represent idiosyncratic microstructure bounce—pure noise. According to the Marchenko-Pastur theorem, we can analytically calculate a threshold eigenvalue below which the variance is statistically indistinguishable from a random matrix.

We discard these low-variance principal components. We set their eigenvalues to zero and reconstruct the time series using only the top K components. This generates a synthetic, orthogonalized data matrix that captures the true underlying drivers of the market while deleting the highly correlated, overfitted noise. Algorithms trained on this subspace converge rapidly and display superior out-of-sample Sharpe ratios.

from sklearn.decomposition import PCA
import pandas as pd
import numpy as np

def pca_orthogonalize(df_features, variance_threshold=0.95):
    """
    Transforms cross-sectional time series into orthogonal Principal Components,
    filtering out multicollinearity and idiosyncratic noise.
    """
    # Standardize features to ensure fair variance extraction
    standardized = (df_features - df_features.mean()) / df_features.std()
    
    # Initial fit to evaluate the variance explained by each orthogonal factor
    pca = PCA()
    pca.fit(standardized)
    
    # Determine the number of principal components to keep (Marchenko-Pastur proxy)
    cumulative_variance = np.cumsum(pca.explained_variance_ratio_)
    n_components = np.argmax(cumulative_variance >= variance_threshold) + 1
    
    # Re-fit and transform the data using only the strictly valid components
    pca_reduced = PCA(n_components=n_components)
    components = pca_reduced.fit_transform(standardized)
    
    # Reconstruct the feature matrix in the new, uncorrelated topological space
    columns = [f'PC_{i+1}' for i in range(n_components)]
    return pd.DataFrame(components, index=df_features.index, columns=columns)

In this plot we can see how there are a few components that explain the variance of the data we have used.

Non-linear embedding and phase-space reconstruction

The preceding six transformations focus on structural adjustment, stationarity, and linear geometry. However, financial markets also display complex nonlinear dynamics driven by interacting algorithmic agents, endogenous liquidity feedback loops, and other factors.

In dynamical-systems terms, this behavior arises from a deterministic system with extreme sensitivity to initial conditions, a property that can be quantified through a positive maximal Lyapunov exponent. To model this kind of temporal behavior, analyzing a one-dimensional sequence x₁, x₂, …, x_t is insufficient. The time series obtained from an exchange API is only a single observable projection of a far more complex multidimensional system.

When we track the price of an asset, we observe only one coordinate of a much larger state space. The hidden variables remain unobserved: aggregate market maker inventory, institutional limit orders resting off-book, options-driven delta-hedging pressure, and the intraday collateral constraints of leveraged funds. Since explicit measurement of all these variables is impossible, we must transform the 1D time series into a multidimensional geometric representation.

This is precisely the role of Takens’ Delay Embedding Theorem. Under broad regularity conditions, the theorem shows that a single observed time series can be used to reconstruct a geometry that is diffeomorphic, or smoothly equivalent, to the underlying hidden attractor of the full dynamical system. We don’t need direct access to the latent variables. Their influence is already encoded in the sequential history of the observable series.

We transform the scalar time series x_t into an m-dimensional vector x_t using a time delay τ:

This mapping lifts the time series into phase space. The choice of the delay τ and the embedding dimension m is critical because the validity of the reconstructed geometry depends on both.

If τ is too small, the coordinates x_t and x_t-τ remain highly correlated, so the embedded points collapse near a diagonal and provide almost no new structural information. If τ is too large, the coordinates become nearly independent, the deterministic linkage weakens, and the reconstructed geometry loses coherence. The standard way to select τ is to compute the mutual information between x_t and x_t-τ across candidate lags. Unlike Pearson correlation, mutual information is entropy-based and captures nonlinear dependence. We choose the first local minimum, which identifies the delay where the next coordinate contributes maximum new structural information while still preserving dynamical dependence.

We determine the embedding dimension m through the False Nearest Neighbors algorithm. If a genuinely high-dimensional system is projected into a space of insufficient dimension, trajectories that are actually far apart can appear artificially close. The FNN procedure increases the dimension step by step. For each point, it identifies its nearest neighbor in dimension m, then measures how that same pair separates when lifted into dimension m+1. If the distance expands beyond a prescribed threshold, the neighbor was false. We stop increasing m when the proportion of false nearest neighbors approaches zero. At that point, the attractor has been properly unfolded and the geometric trajectories no longer overlap because of projection error.

Once we plot the transformed matrix X in two or three dimensions, we no longer observe a jagged price trace indexed by time. We observe the geometry of market regimes. A mean-reverting regime can generate a torus-like structure. A persistent directional regime can form a stretched manifold. A volatility expansion can appear as an outward spiral.

By transforming the time series into phase space, we allow models such as convolutional neural networks and support vector machines to analyze the data as a spatial and topological object rather than as a raw sequence of values. We can further map the embedded trajectories into Gramian angular fields. A CNN can then scan the resulting matrix as it would scan an image, learning the geometric signature of a volatil regime instead of memorizing a noisy run of negative returns. The task shifts from forecasting a scalar value to recognizing a geometric configuration.

import numpy as np

def takens_embedding(series, m, tau):
    """
    Reconstructs the multi-dimensional phase space of a chaotic dynamical system 
    from a single 1D temporal series using Takens' Delay Embedding Theorem.
    """
    N = len(series)
    
    # Mathematical guardrail: ensure we have enough data points to populate the geometry
    if N < (m - 1) * tau + 1:
        raise ValueError("Time series is mathematically too short for these embedding parameters.")
        
    # Initialize the phase-space matrix
    embedded = np.zeros((N - (m - 1) * tau, m))
    
    # Vectorized mapping of the delayed coordinates into the topological space
    for i in range(m):
        # We slice the array with the necessary delays and assign to the specific dimension
        embedded[:, i] = series.iloc[i * tau : N - (m - 1) * tau + i * tau]
        
    return embedded

The reconstructed phase-space projection forms a dense cloud centered near the origin, with no clear low-dimensional manifold, closed orbit, or recurrent geometric skeleton. This suggests that, at the selected delay, the observable is dominated by noise or weak dependence rather than by a visually recoverable deterministic structure.

The unified pipeline

The core conflict of data transformation in quantitative finance lies in the attempt to apply singular, naive adjustments to a complex system. The quantitative finance industry is littered with the remnants of funds that treated data transformation as a menu of independent choices, or worse, engaged in kitchen-sink feature engineering by feeding 500 un-transformed indicators into a gradient boosting model. If a quant applies a standard scaler without adjusting for non-normality, or differentiates integer-wise without considering memory, the resulting models are doomed to fail in production.

The correct chronological execution of this pipeline resolves the fundamental obstacles:

1. Deformation: We first eliminate chronological heteroskedasticity by sampling the raw tick data using a Dollar-Volume clock. The series moves into information-time.

2. Stationarity: We apply Fractional Differencing to the dollar bars. We optimize the d parameter to achieve stationarity while maintaining the maximum theoretical memory. This step requires the stable variance profile created by the time deformation in step one.

3. Outlier mitigation: We apply the entropy-based soft-clipping function to neutralize extreme prints without destroying the topological footprint of the shock. We do this before volatility modeling to prevent a single extreme outlier from biasing the GARCH parameter estimation. If a extreme volatility hits a GARCH model without prior entropy clipping, the conditional variance explodes and suppresses the transformed returns for the next N periods.

4. Volatility standardization: We fit a GARCH(1,1) model and divide the fractionally differenced, clipped series by the conditional volatility to remove autoregressive variance clustering.

5. Distributional adjustment: We calculate the empirical skewness and kurtosis and apply the inverse Cornish-Fisher expansion to normalize the higher moments of the data. This must occur after GARCH standardization. GARCH removes the conditional, short-term volatility bursts, leaving behind the true, unconditional structural non-normality of the asset for the Cornish-Fisher expansion to correct.

6. Orthogonalization: Across multiple parallel assets, we apply PCA to extract the uncorrelated risk factors, dropping the noise-heavy eigenvectors via Marchenko-Pastur thresholding. We can only compute valid covariance matrices for PCA because the prior five steps guaranteed that our individual series are stationary, outlier-free, and distributionally stable. If you run PCA on raw returns, the principal components align with the noise.

7. Embedding: Finally, if required by the predictive architecture, we map the clean, stationary, orthogonalized sequence into an m-dimensional phase space using Takens’ Theorem. This topological mapping requires pristine data. Otherwise, Takens’ theorem simply maps the microstructure noise instead of the system’s actual attractor.

In a live trading environment, these seven steps must be computed sequentially on streaming tick data with sub-millisecond latency. This demands optimized C++ implementations or JAX-compiled Python arrays, bypassing the overhead of standard pandas dataframes.

Cool guys! Good job today! Remember that you have the full code in the appendix so you can torture it. Time to say good bye! Stay sharp, stay bold, stay unshakeable 📈

PS: What do you prefer to trade, big caps or small caps?

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full code:

import warnings
import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt

from scipy.optimize import minimize
from statsmodels.tsa.stattools import adfuller
from sklearn.decomposition import PCA

warnings.filterwarnings("ignore")

class TimeSeriesTransformer:
    def get_weights_ffd(self, d, length, threshold=1e-5):
        weights = [1.0]
        k = 1
        while True:
            weight_k = -weights[-1] * (d - k + 1) / k
            if abs(weight_k) < threshold and k >= length:
                break
            weights.append(weight_k)
            k += 1
            if k > 10000:
                break
        return np.array(weights[::-1])

    def fractional_diff(self, series, d, threshold=1e-5):
        weights = self.get_weights_ffd(d, length=1, threshold=threshold)
        width = len(weights) - 1
        out = pd.Series(index=series.index, dtype=float)
        vals = series.to_numpy(dtype=float)
        for i in range(width, len(series)):
            out.iloc[i] = np.dot(weights, vals[i - width:i + 1])
        return out.dropna()

    def optimize_fractional_d(self, series, d_range=np.arange(0.1, 1.0, 0.1), p_value_threshold=0.05):
        best = None
        rows = []
        for d in d_range:
            diffed = self.fractional_diff(series, d)
            if len(diffed) > 10:
                adf_stat = adfuller(diffed, maxlag=1, regression="c", autolag=None)
                rows.append((d, adf_stat[0], adf_stat[1], len(diffed)))
                if best is None and adf_stat[1] < p_value_threshold:
                    best = d
        return (1.0 if best is None else best), pd.DataFrame(rows, columns=["d", "adf_stat", "p_value", "n_obs"])

    def cornish_fisher_transform(self, returns):
        z = (returns - returns.mean()) / returns.std(ddof=0)
        ranks = z.rank(pct=True).clip(1e-6, 1 - 1e-6)
        return pd.Series(stats.norm.ppf(ranks), index=returns.index)

    def generate_dollar_bars(self, ticks_df, dollar_threshold):
        df = ticks_df.copy()
        df["dollar_value"] = df["price"] * df["volume"]
        df["cumulative_dollars"] = df["dollar_value"].cumsum()
        df["bar_id"] = (df["cumulative_dollars"] // dollar_threshold).astype(int)
        return df.groupby("bar_id").agg(
            open=("price", "first"),
            high=("price", "max"),
            low=("price", "min"),
            close=("price", "last"),
            volume=("volume", "sum"),
            dollars=("dollar_value", "sum"),
        )

    def _garch11_negloglik(self, params, r_scaled):
        omega, alpha, beta = params
        if omega <= 1e-10 or alpha < 0 or beta < 0 or alpha + beta >= 0.999:
            return 1e12
        h = np.empty(len(r_scaled))
        h[0] = np.var(r_scaled, ddof=1)
        for t in range(1, len(r_scaled)):
            h[t] = omega + alpha * (r_scaled[t - 1] ** 2) + beta * h[t - 1]
            if h[t] <= 1e-12 or not np.isfinite(h[t]):
                return 1e12
        return 0.5 * np.sum(np.log(2 * np.pi) + np.log(h) + (r_scaled ** 2) / h)

    def garch_standardize(self, returns):
        r = returns.dropna().to_numpy(dtype=float)
        scale = 100.0
        r_scaled = r * scale
        var0 = np.var(r_scaled, ddof=1)
        x0 = np.array([0.01 * var0, 0.05, 0.90])
        bounds = [(1e-10, None), (0.0, 0.999), (0.0, 0.999)]
        constraints = ({"type": "ineq", "fun": lambda x: 0.999 - (x[1] + x[2])},)
        res = minimize(
            self._garch11_negloglik,
            x0,
            args=(r_scaled,),
            method="SLSQP",
            bounds=bounds,
            constraints=constraints,
            options={"maxiter": 500},
        )
        if res.success:
            omega, alpha, beta = res.x
            h = np.empty(len(r_scaled))
            h[0] = var0
            for t in range(1, len(r_scaled)):
                h[t] = omega + alpha * (r_scaled[t - 1] ** 2) + beta * h[t - 1]
        else:
            lam = 0.94
            h = np.empty(len(r_scaled))
            h[0] = var0
            for t in range(1, len(r_scaled)):
                h[t] = (1 - lam) * (r_scaled[t - 1] ** 2) + lam * h[t - 1]
        cond_vol = np.sqrt(h) / scale
        return (
            pd.Series(r / cond_vol, index=returns.dropna().index),
            pd.Series(cond_vol, index=returns.dropna().index),
            {"success": bool(res.success), "params": res.x if res.success else np.array([np.nan, np.nan, np.nan])},
        )

    def shannon_entropy(self, series, bins=50):
        hist, _ = np.histogram(series, bins=bins, density=True)
        hist = hist[hist > 0]
        return -np.sum(hist * np.log2(hist))

    def entropy_winsorization(self, series, window=100, base_threshold=3.0, lambda_val=1.5):
        cleaned = series.copy()
        taus = pd.Series(index=series.index, dtype=float)
        for i in range(window, len(series)):
            chunk = series.iloc[i - window:i]
            entropy = self.shannon_entropy(chunk)
            tau = base_threshold * (1.0 / (entropy + 0.1))
            taus.iloc[i] = tau
            x = series.iloc[i]
            if abs(x) > tau:
                sign = np.sign(x)
                clipped = tau + lambda_val * np.tanh((abs(x) - tau) / lambda_val)
                cleaned.iloc[i] = clipped * sign
        return cleaned, taus

    def pca_orthogonalize(self, df_features, variance_threshold=0.95):
        z = (df_features - df_features.mean()) / df_features.std(ddof=0)
        pca_full = PCA()
        pca_full.fit(z)
        cum_var = np.cumsum(pca_full.explained_variance_ratio_)
        n_components = np.argmax(cum_var >= variance_threshold) + 1
        pca_reduced = PCA(n_components=n_components)
        comps = pca_reduced.fit_transform(z)
        cols = [f"PC_{i+1}" for i in range(n_components)]
        return pd.DataFrame(comps, index=df_features.index, columns=cols), pd.Series(
            pca_full.explained_variance_ratio_,
            index=[f"PC_{i+1}" for i in range(len(pca_full.explained_variance_ratio_))]
        )

    def takens_embedding(self, series, m, tau):
        N = len(series)
        if N < (m - 1) * tau + 1:
            raise ValueError("Time series is too short for these embedding parameters.")
        embedded = np.zeros((N - (m - 1) * tau, m))
        for i in range(m):
            embedded[:, i] = series.iloc[i * tau:N - (m - 1) * tau + i * tau]
        return embedded

Table of contents:

1. Introduction.

2. Escalation of war in the Gulf.

3. Hormuz as the strategic hinge.

4. Kharg island is the energy chokepoint.

5. Energy shock and the macro transmission channel.

6. Second-order economic effect of war.

7. Egypt external war pressure.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

The Gulf war now sits at the center of the global economic system. What once looked like a regional military confrontation now shapes the price of oil, the cost of maritime insurance, the movement of LNG, the availability of fertilizers, the outlook for inflation, and the fiscal stability of import-dependent states. The conflict has crossed the boundary between battlefield dynamics and macroeconomic reality. Every strike on infrastructure, every disruption in transit, and every shift in force posture now feeds directly into trade flows, industrial costs, sovereign risk, and household pressure across regions far from the Gulf.

At the center of that system stands the Strait of Hormuz. This narrow maritime corridor connects armed confrontation with global energy supply in a direct and immediate way. It is the hinge between missiles and inflation, between naval pressure and industrial costs, between regional violence and worldwide repricing across oil, gas, freight, food, and sovereign debt. Around it, Kharg Island emerges as a second critical node, a concentrated export point whose strategic value lies in its ability to convert Iranian production into global market pressure and fiscal survival.

What follows traces that chain with a clear line of sight. It begins with the expansion of war across the Gulf export system and then moves through Hormuz as the decisive chokepoint where military risk becomes macroeconomic shock. From there, the focus shifts to Kharg Island and the wider energy channel, where the conflict enters inflation, trade, growth, and monetary policy. The later sections turn to the deeper economic consequences of prolonged confrontation, including alliance strain, defence spending, fiscal burdens, and the exposure of vulnerable import-dependent economies such as Egypt.

You can go deeper by checking this report:

Gobal trade outlook

1.44MB ∙ PDF file

Download

Download

This moment also carries a strategic message for markets and states alike. The most important risks sit in the infrastructure that keeps economies supplied, in the narrow corridors that keep trade alive and domestic stability. Once those channels come under pressure, the effects spread with speed from ports to prices, from commodities to currencies, and from regional confrontation to global adjustment.

Subscribe now

Escalation of war in the Gulf

The war now sits inside the core of the global economy. The opening phase concentrated on Iranian leadership nodes, missile capacity, nuclear assets, and military infrastructure. The present phase covers the Gulf export system, maritime transit, shipping insurance, alliance bargaining, and market pricing across oil, gas, freight, food, and sovereign risk. Attacks hit major oil and gas sites across Iran, Saudi Arabia, Kuwait, Qatar, the United Arab Emirates, Bahrain, and Iraq. Bahrain declared force majeure after damage at the Sitra refinery, Iraq cut output as Hormuz disruption blocked crude flows, and Qatar reported the loss of around one-sixth of LNG capacity after damage to Ras Laffan. That sequence transformed the conflict into a regional energy and trade shock with global reach.

Persian Gulf oil shipments has lifted energy prices. Its adjusted high-energy scenario lowers 2026 world merchandise trade volume growth to 1.4% from 1.9% and trims global GDP growth to 2.5% from 2.8%. The IMF added that a persistent 10% increase in oil prices can raise global headline inflation by around 40 basis points and reduce global output by 0.1 to 0.2 percentage points. Together, those estimates show that the conflict now acts through the central channels of inflation, trade, and growth rather than through a narrow regional risk premium.

The Strait of Hormuz sits at the center of this transmission mechanism. UNCTAD estimated that the strait carries around one quarter of global seaborne oil trade along with major LNG and fertilizer flows. Its briefing shows ship transits through the strait falling by 97% from the February average, while oil and gas prices surged between 27 February and 9 March. In that setting, Hormuz becomes the hinge that connects military escalation with industrial input costs, European gas pricing, Asian energy security, fertilizer markets, freight rates, bunker fuel, and food pressure across import-dependent economies.

Washington is sending thousands of additional Marines and sailors to the region, adding to an existing U.S. military presence of about 50,000 personnel. At the same time, the White House has pressed allies to help secure passage through Hormuz, while European partners have adopted a posture centered on safe passage, inflation control, and de-escalation. This combination places deterrence, burden sharing, maritime security, and macro stabilization inside the same strategic field.

Outside powers have entered the same equation. China warned that the war threatens global energy supplies, trade routes, and economic stability. Russia declared political support for Tehran through a Kremlin message that described Russia as a loyal friend and reliable partner. These positions matter because they influence shipping expectations, sanctions politics, commodity diplomacy, and the pricing of geopolitical persistence across global markets.

Financial transmission has also accelerated. A U.S.-backed maritime reinsurance plan led by Chubb seeks to restore commercial shipping through Hormuz, which shows how finance and logistics now function as strategic infrastructure alongside naval and air power. UNCTAD’s assessment also tracks sharp rises in tanker costs, bunker fuel, and bond yields across exposed Gulf economies, confirming that the shock now moves through transport costs, refinancing conditions, and imported inflation as well as through crude and LNG benchmarks.

The confrontation therefore belongs to the category of wider-system conflict. Geography, energy, trade, shipping, insurance, alliance politics, inflation, and growth forecasts now move inside one connected field. Each strike on infrastructure or transit routes now feeds into industrial costs, consumer prices, sovereign funding conditions, and policy choices across regions far from the battlefield.

By the way, before we move to next section, a new TACO just happned today because of:

Hormuz as the strategic hinge

The closure of the strait has stopped the passage of about 20% of the world’s oil and liquefied natural gas since 28 February. The International Maritime Organization said around 20,000 seafarers on nearly 2,000 ships west of the strait are caught in the disruption, and Iraq has already declared force majeure because Hormuz disruption halted most crude exports and cut Basra output from 3.3 million barrels per day to 900,000. The narrow channel now links missile risk and naval pressure with global energy supply, shipping capacity, and state revenue.

The U.S. Energy Information Administration reported that Hormuz carried about 20 million barrels per day in 2024, equal to about 20% of global petroleum liquids consumption and more than one-quarter of seaborne oil trade. Around one-fifth of global LNG trade also moved through the strait. EIA estimated bypass capacity in Saudi and Emirati pipelines at about 2.6 million barrels per day, so rerouting covers only a small share of the volume at risk. UNCTAD’s adds that, one week before the conflict, crude oil represented 38% of the relevant seaborne trade categories moving through Hormuz, LPG 29%, LNG 19%, refined oil products 19%, and chemicals, including fertilizers, 13%. Asia receives 84% of Hormuz crude and 83% of Hormuz LNG, which gives the chokepoint immediate reach across the main import centers of the global economy.

Traffic data show how fast military pressure becomes market shock. UNCTAD recorded an average of 141 daily ship transits during 1–27 February, followed by a 97% collapse. Brent had risen 27% and Dutch TTF gas 74% from then. Chubb joined a $20 billion maritime reinsurance plan because shipowners, cargo interests, and financiers need war-risk cover before normal commercial flows can return. Passage through Hormuz therefore sets the marginal cost of energy movement, insurance, and cargo finance for economies far beyond the Gulf.

The macroeconomic spillover now reaches trade volumes, food costs, inflation expectations, and sovereign financing. Severe curtailment of Persian Gulf oil shipments could cut world merchandise trade volume growth in 2026 to 1.4% from 1.9% and lower world GDP growth to 2.5% from 2.8% under its high-energy-price scenario. This connects higher Gulf energy prices with fertilizer stress and broader food-system pressure. Besides, government bond yields in the United States and Europe jumped as investors priced a more persistent inflation shock, with the UK 10-year yield rising above 5% and markets shifting from rate-cut expectations toward rate-hike expectations. Hormuz therefore acts through central-bank reaction functions and sovereign borrowing costs as well as through the oil curve.

Kharg island is the energy chokepoint

Kharg Island now stands as one of the clearest points where battlefield logic and macroeconomic leverage meet. Kharg handles 90% of Iran’s oil exports, and some reports showed that Iran continued to ship between 1.1 million and 1.5 million barrels per day during the war. Iran exported about 1.7 million barrels per day so far in 2026, with about 1.55 million barrels per day moving through Kharg, after exports had reached about 2.17 million barrels per day in February. That concentration gives the island a dual role. It functions as Iran’s main export valve and as a target whose condition shapes global expectations for supply, escalation, and endurance.

The strategic value of Kharg comes from concentration of flow, storage, and loading capacity in a single hub. The island holds storage capacity of roughly 30 million barrels and held about 18 million barrels of crude. Its offshore position also matters. Kharg as sitting 16 miles from Iran’s coast in waters deep enough for very large tankers that cannot approach the shallow mainland coast. That gives Kharg a special role in converting upstream production into export revenue at scale. A strike, blockade, or sustained interdiction at that one node would reach revenue, shipping schedules, tanker availability, buyer confidence, and physical balances at the same time. Market analysis are pointing to a potential loss of around 2 million barrels per day from the market if Kharg infrastructure were taken out.

Kharg also carries the fiscal weight of Iran’s external energy system. The U.S. Energy Information Administration’s 2025 SHIP Act report estimated Iran’s crude oil and condensate export revenue at $43 billion in 2024, with exports at 1.483 million barrels per day. The same report estimated exports to China at 1.444 million barrels per day in 2024, which shows how strongly Iran’s export realization depends on a narrow buyer corridor linked to Asian demand. Iranian oil accounted for 11.6% of China’s seaborne imports so far in 2026. Kharg therefore sits at the intersection of Iranian fiscal resilience and Chinese energy procurement. The island matters because it converts production into cash flow, and it does so through a commercial route that remains central to a major importing power.

U.S. forces struck more than 90 Iranian military targets on Kharg while preserving the oil infrastructure. That decision revealed the hierarchy of escalation inside the war. Military assets on the island formed the first target set, while the export system itself remained the larger coercive lever. Trump administration was considering plans to occupy or blockade Kharg in order to pressure Iran to reopen the Strait of Hormuz. Kharg thus moved from vulnerable infrastructure into active strategic bargaining space. In this setting, the island serves as a pressure point whose control, impairment, or threatened seizure can alter both the battlefield and the oil market.

Energy shock and the macro transmission channel

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Sufficient statistics and minimal signals

This lecture defines a trading signal through sufficiency rather than through a rule, threshold, or pattern. A signal is a compression of the decision-time information set that preserves the conditional law of the declared outcome, so predictive meaning remains tied to the future law rather than to a single backtest metric. The lecture then develops minimality, bottleneck design, and representation stability as the conditions that make that compressed state usable under finite data, drift, and noise.

What’s inside:

1. Sufficiency as the definition of signal. A signal is a representation that preserves the conditional distribution of the outcome, which separates signal from strategy and makes feature relevance a property of conditional-law preservation rather than raw correlation.

2. Signal as compression with law preservation. Compression becomes signal construction when the reduced state carries the same predictive content as the full decision-time information set, which lets the system operate on a smaller and more stable representation without changing the law it conditions on.

3. Approximate sufficiency in practice. Exact sufficiency appears as the ideal case, while practical sufficiency is defined through a declared tolerance using divergence, proper-score regret, and the residual predictive content that remains outside the bottleneck.

4. Minimal sufficient sets. The lecture shows that a minimal signal is the smallest representation that still preserves the full conditional law, removes nuisance variation, lowers selection pressure, and keeps the signal portable across operators and model classes.

5. Non-uniqueness of sufficient representations. Many different representations can preserve the same predictive law, so the scientific question is whether preservation exists, while the engineering question is which representation remains most stable under noise, drift, and deployment constraints.

6. Bottleneck trade-offs. Bottleneck design becomes an information-budget problem that balances relevance against representational capacity, because too little compression increases variance and selection pressure, while too much compression removes predictive fragments the outcome still needs.

7. Overcompression and undercompression. The lecture explains that overcompression distorts the conditional law by blending distinct regimes, while undercompression creates a high-capacity overfitting surface where sharp in-sample structure often fails under recurrence and proper scoring.

8. Representation stability. A minimal signal must retain stable meaning under measurement noise, drift in state occupancy, and regime mixtures, so conditional beliefs, calibration, and divergence structure remain coherent when the signal moves from research into live deployment.

Sample

1.93MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. The API layer.

3. Protocol semantics, transport choice, and system state.

4. Risk and financial APIs limitations.

5. Historical-live non-equivalence and data reconstruction error.

6. Execution state machines and order routing.

7. Simulation and live trading.

8. Creating the architecture for a trading API.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

Before moving on to the next series, there’s an important point that any algorithmic trader needs to know: the point about APIs.

A quant can state the issue in simple terms. Let s_t​ denote latent market state and let x_t​ denote observed state at decision time. The strategy trades on x_t​ rather than s_t​. The transformation from s_t​ to x_t​ includes vendor capture, transport, buffering, schema validation, clock alignment, and local state reconstruction. When that transformation changes, the strategy changes with it, even if the signal formula stays fixed.

The API layer belongs to the hypothesis itself because it determines the admissible information set on which the strategy acts.

This point matters because the gap between research and deployment tends to emerge at the boundary where information becomes software state. Bars in a notebook arrive sorted and complete. Live messages arrive through sessions, quotes, reconnect paths, and transport rules. Orders in a backtest pass from signal to fill in one step. Orders in production pass through acceptance, routing, queue interaction, etc. In a systematic process, those choices carry mathematical weight because they alter the state space on which the strategy operates.

The live strategy therefore admits a fuller description than the research signal alone. A useful expression is

Many teams place their energy on features, models, and optimization while the interface layer receives less formal treatment. That habit carries a cost. A signal derived from one information regime and deployed through another regime becomes a different object. A simulator that treats execution as an atomic event studies a market process distinct from the one live code encounters.

The API layer is where these fractures surface. It is also where discipline can restore coherence. If you want a stack to start with, here it is:

Algorithmic Trading in the Global FX Market

176KB ∙ PDF file

Download

Download

An API is an information contract, a timing contract, and an execution contract. That’s to say, data fields need causal meaning, timestamps need clear authority, order states need legal transitions, internal services need shared schemas across research, simulation, and live trading. Do you get the ida right? Under those conditions, performance claims acquire stronger footing because the path from observation to action remains explicit.

The subject here is the form through which market enters the strategy and the form through which strategy intent reaches the market. Code style, framework choice, and endpoint count matter through their effect on that form. A live trading stack earns coherence when the objects that move across it stay explicit, typed, and stable across the full path from research to execution. That coherence is a design choice and it begins at the API layer.

The API layer

In a trading system, the central question asks what is known, when it becomes known, and which representation carries it. The API layer lives at that boundary. It defines the admissible information set available to the strategy at decision time. That framing captures the practical source of failure with precision.

Let the latent market state at event index t be s_t. The strategy never accesses that state in raw form. It receives an encoded view x_t, produced through venue generation, vendor capture, normalization, transport, buffering, decoding, and local deserialization. If those operations are compressed into a single operator H, then the strategy acts on

In live systems, H belongs to a family of transformations shaped by endpoint choice, session state, codec, throttling behavior, aggregation policy, and recovery logic. The research question therefore asks two things at once: whether a mapping f(x_t) carries predictive value, and whether the x_t used in research belongs to the same family as the x_t available in deployment.

A useful decomposition is

The term H_t(s_0:t) shows that observed state may depend on a path rather than a single contemporaneous latent state. Corrections, delayed packets, local aggregation, and dropped messages create that path dependence. The residual term ε_t collects the user-facing uncertainty generated by packet loss, schema surprises, decoder failure, clock mismatch, and hidden vendor behavior. In a calm hourly-bar strategy, ε_t may occupy a small role. In a short-horizon or execution-sensitive strategy, it often defines the core challenge.

This perspective dissolves a common misconception. Many quants treat the API as a latency channel and connect latency risk with high-frequency trading. A richer view sees the API as a state-visibility mechanism. Consider a breakout rule triggered when price crosses a level and volume in the same interval exceeds a threshold. One endpoint may publish aggregated bars with finalized volume, while another may publish incremental trades that require local accumulation. The rule then sees different objects.

Speed alone does not resolve that difference. One representation is final. The other is provisional. One is bounded by bar closure. The other evolves through the interval. If research evaluates the final object while deployment trades the provisional one, then backtest and live trading inhabit different information worlds.

The official protocol and framework documentation supports this view from the software side. HTTP semantics are defined as stateless at the application-protocol level. That property makes HTTP well suited for resource retrieval, explicit commands, and idempotent inspection, while state continuity moves into higher layers. The OpenAPI specification makes HTTP service capabilities machine-readable through paths, parameters, schemas, and responses. FastAPI builds on that structure by generating OpenAPI descriptions and interactive documentation from Python declarations. Pydantic treats typed data models as executable contracts and emits JSON Schema from those models. Finance gives special weight to this alignment because state integrity sits at the center of the problem.

For a systematic trading system, the internal data model becomes a compact statement of admissible knowledge. A well-defined Bar object can enforce monotonic timestamps, nonnegative volume, high-low consistency, a declared timezone, and provenance metadata. A Signal object can enforce a causal timestamp, a horizon, a generation identifier, and a confidence field interpreted consistently by the risk layer. An OrderIntent object can separate intent from broker acknowledgment, so the system preserves a clean distinction between desired action, accepted action, and executed action. These structures carry operational value because they keep portfolio state, execution state, and research state aligned across time.

Subscribe now

Protocol semantics, transport choice, and system state

A familiar hierarchy pairs REST with slower interactions, WebSocket with streaming interactions, and FIX with institutional workflows. A stronger formulation sees each protocol as an organizer of interaction and, through that role, as a designer of local architecture.

Start with HTTP. Under RFC 9110, HTTP defines resource-oriented request-response semantics, methods, status codes, headers, and representations. It serves clear operations with strong structure: fetching historical bars, querying account state, placing an order, canceling an order, requesting configuration, and obtaining a health report. It also supports documentation with high clarity.

For internal trading services, that structure has high value. The stack gains clear contracts, reproducible input schemas, and rapid client generation.

HTTP also channels interaction through discrete exchanges. Applications that consume a continuous stream of trade events, order updates, heartbeats, or PnL deltas benefit from WebSocket. RFC 6455 defines a bidirectional protocol over TCP with an opening handshake and framed messages, built for two-way communication.

For example, Alpaca’s streaming documentation—I mean, the broker—presents trade, account, and order updates over WebSocket and highlights frame types across endpoints, plus authorization and stream subscription after connection. For a quant system, the implication is clear: WebSocket provides continuity, and continuity assigns responsibility. A persistent connection requires reconnect handling, duplicate control, interval recovery, subscription replay, sequence validation, and local buffering.

{
  "action": "auth",
  "key": "{YOUR_API_KEY_ID}",
  "secret": "{YOUR_API_SECRET_KEY}"
}

FIX addresses a different problem. The FIX Trading Community describes FIX as:

A standardized transaction language across the securities trade cycle, with message types aligned to steps such as quote request and new order, and with standardized fields as the building blocks.

That matters because institutional trading depends on precise business semantics across a workflow that spans indication, order, execution report, allocation, and post-trade reporting. FIX draws strength from a protocol that already speaks the language of the transaction. A modern retail or semi-professional quant stack may begin on other connectivity paths, yet it can still absorb the lesson: trading systems gain stability when message meanings stay explicit, durable, and tied to the lifecycle they govern.

The mathematics of transport stays simple. Let T_total denote end-to-end delay from the moment a signal becomes ready to the moment a fill event reaches local state. Then

A robust analysis evaluates the whole sum. In many implementations, T_broker and T_reconcile generate more variation than raw network travel time. Serialization, authentication, request signing, broker throttling, internal queuing, and local reconciliation can dominate the tails. Tail behavior matters more than averages because losses tend to concentrate there. A mean delay of 20 ms with a 95th percentile of 60 ms may fit one strategy and damage another whose edge concentrates in short-lived bursts.

The histogram decomposes end-to-end order latency into plausible components and shows the aggregate distribution. The plot reveals that total latency is a sum of heterogeneous terms and that the tail can widen even when most components remain well behaved.

Risk and financial APIs limitation

Once the API is treated as a formal component, the next task is classification. Risk in financial APIs represents a family of failure modes that interact. A useful taxonomy separates at least seven categories: data risk, temporal risk, schema risk, session risk, throughput risk, execution risk, and governance risk.

1. Data risk concerns what is transmitted. This includes missing fields, inconsistent bar construction, delayed corrections, undocumented transformations, and ambiguity about what a timestamp refers to. Is the bar timestamp the interval open, the interval close, or publication time? Are trades condition-filtered? Are odd lots included? Is volume consolidated at the same cadence in historical and live endpoints? Vendor documentation sometimes answers these questions, sometimes not. The point is that the risk exists even when the feed looks clean. A clean feed can still describe the wrong object for the strategy.

2. Temporal risk concerns when the data is available and in which order. Clock drift between client, broker, and data vendor is the obvious part. More subtle is the distinction between event time and arrival time. A message may encode one time but arrive at another. Streaming systems can also deliver temporary disorder. If the strategy assumes a total order while the transport offers only a best-effort order plus reconnect recovery, then the local state machine must restore order or at least detect that order was broken.

3. Schema risk concerns how the data is structured. An endpoint that changes a field name, adds a nullable attribute, changes a numeric type from integer to string, or alters nesting may be perfectly legal at the API level and still break live trading. This is why type validation libraries matter.

4. Session risk is operational but critical. Authentication expiry, refresh tokens, signed requests, environment separation, and broker session exclusivity determine whether the trading path remains live. Some APIs are quite clear that trading-enabled sessions have different restrictions than generic access, including brokerage-level single-session limits.

5. Throughput risk includes rate limits, quotas, pagination, backpressure, and resource exhaustion. This category is often underestimated by quants who work first in notebooks. Yet the feasible research surface is shaped by what the API allows. If a cross-sectional study requires more requests per minute than the provider permits, then the signal either needs a different collection design, a local cache, a licensed feed, or a narrower universe. Throughput is a constraint on the set of admissible strategies.

6. Execution risk concerns the gap between request acceptance and actual market interaction. Here a good example of why this matters: “the order object has identifiers, statuses, and a lifecycle that can be queried after placement. That is useful, but it also confirms the obvious point that the initial placement call is only one transition in a larger process. SEC routing and execution disclosures exist because the path from broker to venue materially affects outcomes”. In trading, the request path and the execution path should never be conflated.

7. Governance risk covers versioning, deprecation, environment drift, and organizational ambiguity. An internal API that is undocumented, lacks version tags, or mixes research-only fields with live-only fields can corrupt an otherwise sound strategy. Governance is what keeps the contract stable long enough for experiments to be comparable.

A convenient formalization is to model observed system risk as the union of these categories:

Each component can be associated with observables: gap counts, duplicate counts, validation failures, authentication renewals, request rejections, order rejects, and contract-version mismatches. The important point is that API risk becomes measurable only after categories are explicit.

The table below compresses this pitfalls:

Throughput deserves one extra comment because it changes research conclusions. Assume that trade opportunities survive for a characteristic time constant τ and that the system polls every Δ units of time rather than consuming a push stream. A rough first-order miss model is:

This formula is for illustrative purposes. It says that when opportunity duration is short relative to the observation interval, miss probability rises quickly. Coarser polling changes the distribution of observed opportunities. The picture below visualizes that relationship:

Quants often frame API problems as reliability problems. They are also selection problems. The feed, the protocol, and the quota choose which events you ever get the chance to model. That means a strategy discovered under one data-access regime may not even be defined under another. Once this is understood, the response becomes sharper.

Historical-live non-equivalence and data reconstruction error

Historical endpoints and live endpoints are seldom equivalent in the strict sense required by causal research. They may deliver related objects, but related is not enough. Historical data is often cleaned, deduplicated, repaired, aligned to bar boundaries, and packaged into stable records. Live data is incremental, provisional, and susceptible to temporary disorder. A strategy that learns on one and trades on the other inherits a reconstruction problem.

There are at least four sources of non-equivalence:

1. The first is finalization: Historical bars tend to be final objects. Live bars, unless the provider specifically publishes only completed intervals, may be in progress.

2. The second is correction policy: Historical vendors can revise prior records after exchange corrections or internal data-quality routines. Live systems receive the pre-correction path and may or may not later receive explicit correction messages.

3. The third is aggregation method: A historical endpoint may aggregate on server-side rules that differ from a client’s local reconstruction from trade ticks.

4. The last one is delivery semantics: Historical data is fetched as a completed array. Live data is delivered as a stream in which messages can be delayed, duplicated, or arrive after reconnect.

Any research process should ask a simple but uncomfortable question: is the historical object used in research reproducible from the live stream using only information available at the time? If the answer is no, then the strategy is at risk of hidden lookahead or deployment drift. Sometimes the mismatch is harmless. Often it is not. Opening-range logic, event-triggered strategies, and execution-sensitive systems are especially vulnerable because they depend on exact temporal boundaries.

One clean way to formalize the problem is to define a reconstruction operator G that maps a live stream path into a bar or state record:

The historical dataset supplies b_t^hist. The research assumption that often goes unstated is that b_t^hist = b_t^live_recon. This is rare. The true object of concern is the reconstruction error:

A strategy that is stable under small δ_t may survive deployment. A strategy that changes sign, entry timing, or position size under small δ_t may lose money even if the backtest looks good. This is one reason why mature groups insist on replay tests. They record live messages, reconstruct the internal state offline, and compare it with both the production decision log and the historical vendor view.

Out-of-order delivery is a particularly sharp example. Suppose messages carry event timestamps e_i but arrive at times a_i. A client that sorts by arrival time instead of event time assumes that sign(a_i - a_j) equals sign(e_i - e_j) for all pairs. That assumption is false in many real networks. Once it breaks, local bars, rolling indicators, or event-triggered conditions may shift. The correct response is to build reordering logic with bounded buffers and explicit policy for what to do when order cannot be restored with confidence.

Here is a minimal sketch of such a guard:

from dataclasses import dataclass
from heapq import heappush, heappop


@dataclass(order=True)
class MarketEvent:
    seq: int
    ts_event_ns: int
    payload: dict


class ReorderBuffer:
    def __init__(self, max_gap: int = 32):
        self.expected = 0
        self.max_gap = max_gap
        self.heap = []

    def push(self, event: MarketEvent):
        if event.seq < self.expected:
            return []  # duplicate or stale replay

        heappush(self.heap, event)
        out = []

        while self.heap and self.heap[0].seq == self.expected:
            out.append(heappop(self.heap))
            self.expected += 1

        if self.heap and self.heap[0].seq - self.expected > self.max_gap:
            raise RuntimeError("sequence gap too large")

        return out

The point of the snippet is to show the shape of the invariant. Local state should advance only when sequence integrity is acceptable. If the gap exceeds a declared tolerance, the client should raise an incident, resubscribe, or rebuild state from a safer source.

The next plot shows the operational consequence of delayed observations on a simple synthetic crossover-style entry. The delayed observation sees the same price path later and enters later. The lesson is that even modest observation delay can transform entry timing in a way that a clean historical backtest never exposed.

A common response is to smooth the signal until these issues disappear. That can help, but it does not solve the identification problem. If the strategy only works when final corrected bars are used, then the strategy depends on a state that may not exist in live trading.

First, define decision-time objects. Second, test historical-live parity by replaying captured live streams into the same internal schemas used in production. Third, store provenance metadata with every derived bar or indicator: source, reconstruction mode, timezone, sequence completeness, and whether the object is provisional or finalized. Fourth, make non-equivalence visible in research by introducing uncertainty envelopes. If a signal flips under plausible reconstruction perturbations, that signal should not be promoted.

Execution state machines and order routing

The industry still tolerates expressions like: “send the order”, “the order is in”, “the order got filled”. For software, these phrases hide critical distinctions. For PnL, they hide money. However, execution should be modeled as a state machine.

A request to place an order is only a statement of intent until the broker acknowledges it. The acknowledgment is not yet a fill or the fill may be partial. The remaining quantity may rest, cancel, expire, or be modified by subsequent logic. Some APIs reflect these transitions in a REST resource that can be queried after placement; others also emit asynchronous trade or order updates over a stream; sometimes users place, monitor, and cancel orders through the Trading API, and that order identifiers and status objects support later inspection. The best approach is to maintain two separate layers: account and order routing. That separation is healthy because it prevents one channel from pretending to be the whole lifecycle.

Okay, let’s talk about the second one: routing. SEC rules on order execution and order routing disclosure exist because the path an order takes after leaving the customer-facing system affects economic results. A retail API client sees only part of the venue-level journey, and the system designer should treat that journey as relevant. Even if the internal API abstracts away venue detail, it should preserve enough state to distinguish user intent, broker acceptance, downstream routing outcome, and final position reconciliation.

A minimal lifecycle can be written as:

with additional branches for rejected, canceled, expired, and replaced. Not every broker surfaces each branch in the same way. The internal system should anyway. Doing so has three advantages. First, it prevents false certainty. Second, it enables reconciliation. Third, it supports simulation. A sound backtest represents the full state machine between signal and filled position, because those intermediate transitions reveal failure modes that materially shape PnL.

Consider expected execution price. A crude but useful decomposition is:

Here P_ref is a chosen reference price, S_queue captures queue-position effects and spread interaction, S_latency captures adverse movement while the order is in flight, S_impact captures self-induced price movement for non-negligible size, and S_fees captures commissions, exchange fees, rebates, and financing where relevant. Many prototype systems focus on S_latency and ignore the rest. That is tolerable for a toy simulator, but not for a research workflow that wants to claim tradability. Even a simple simulator should at least model partial fills, expiry, and side-dependent slippage under volatility.

An internal order model should therefore separate three objects:

1. OrderIntent: what the strategy wants.

2. BrokerOrderState: what the broker has acknowledged and how it has evolved.

3. PositionLedgerEntry: what the portfolio believes happened after reconciliation.

Conflating them creates the classic bug where the strategy believes it is flat because its own intent was to exit, while the broker still shows a partially open quantity.

A minimal type sketch might look like this:

from enum import Enum
from pydantic import BaseModel, Field, field_validator
from typing import Optional


class OrderSide(str, Enum):
    BUY = "BUY"
    SELL = "SELL"


class OrderState(str, Enum):
    INTENT = "INTENT"
    ACCEPTED = "ACCEPTED"
    WORKING = "WORKING"
    PARTIALLY_FILLED = "PARTIALLY_FILLED"
    FILLED = "FILLED"
    REJECTED = "REJECTED"
    CANCELED = "CANCELED"
    EXPIRED = "EXPIRED"


class OrderIntent(BaseModel):
    client_order_id: str
    symbol: str
    side: OrderSide
    qty: float = Field(gt=0)
    limit_price: Optional[float] = Field(default=None, gt=0)


class BrokerOrderState(BaseModel):
    client_order_id: str
    broker_order_id: Optional[str] = None
    state: OrderState
    filled_qty: float = Field(ge=0)
    avg_fill_price: Optional[float] = None

    @field_validator("avg_fill_price")
    @classmethod
    def positive_price_if_present(cls, v):
        if v is not None and v <= 0:
            raise ValueError("avg_fill_price must be positive")
        return v

The exact fields can be adjusted. Once these objects are defined, live logic becomes cleaner. Reconciliation compares the latest broker state with the internal ledger. Simulation advances through legal states instead of jumping from signal to fill. Risk logic defines kill-switch conditions in terms of state divergence, not only PnL.

Quants often focus on price improvement as the hard part of execution. In many systems, the hard part is state certainty. A robust system determines with high confidence whether an order is live, whether a cancel request succeeded, and whether a partial fill changed net exposure. Alpha only matters when the process itself is reliable. The real objective is to build an execution engine whose states are clear enough that later improvements rest on solid assumptions.

This leads to prototype API design. Even at the prototyping stage, order endpoints benefit from idempotency, accept client-generated identifiers, distinguish acceptance from fill, and expose a machine-readable state model. Internal consumers rely on structured state as the source of truth. Human-readable logs still help, but they support the contract rather than define it.

The external side of the problem therefore becomes clear: financial APIs present data and orders as evolving states carried through channels with meaningful semantics. The constructive response is to assign those states formal names and transitions. Once that structure is in place, the internal API becomes the stable surface across which research, simulation, and live code remain aligned.

Send me a copy

Simulation and live trading

One script uses the same field names as the next, timezones remain consistent across services, the execution engine works with normalized order states, the backtester uses the same fill object as live trading, and the monitoring service reads position exposure from a shared contract rather than reconstructing it from logs. That is manageable at small scale and essential at medium scale. The solution is to design an internal API.

The main purpose of an internal API is contract preservation. Research, paper trading, backtesting, and live services should consume and emit the same conceptual objects even when their data sources differ. If one system produces Signal(direction, generated_at, horizon, confidence) and another expects alpha_score, entry_ts, and weight_hint, the boundary needs stronger alignment. The internal API creates that alignment by enforcing shared names, units, valid ranges, and temporal interpretation.

A sensible contract design starts with five invariants:

1. Causality: Every object that can influence a decision carries a timestamp or event key that identifies when it became admissible. This includes bars, features, signals, risk checks, and order updates. With that structure in place, replay testing becomes meaningful.

2. Provenance: A bar is more than OHLCV. It carries information about whether it came from vendor history, live reconstruction, a synthetic resampler, or a replay archive. That small field turns future debugging into straightforward engineering.

3. State separability: Data objects, signal objects, order-intent objects, broker-state objects, and portfolio-ledger objects each remain distinct. This separation keeps assumptions from leaking across modules and preserves clarity at the interfaces.

4. Versioning: Schemas evolve, and they should evolve explicitly. A schema_version field together with artifacted OpenAPI snapshots provides inexpensive and reliable protection.

5. Idempotency: Commands that change state, especially order-related commands, should support replay without accidental duplication. Client order IDs provide the standard mechanism.

These invariants can be supported with concise Pydantic models. Here is a small example:

from datetime import datetime, timezone
from pydantic import BaseModel, Field, field_validator
from typing import Literal


class Bar(BaseModel):
    schema_version: str = "1.0"
    source: Literal["vendor_history", "live_recon", "replay"]
    symbol: str
    ts_open: datetime
    ts_close: datetime
    open: float = Field(gt=0)
    high: float = Field(gt=0)
    low: float = Field(gt=0)
    close: float = Field(gt=0)
    volume: float = Field(ge=0)

    @field_validator("ts_open", "ts_close")
    @classmethod
    def enforce_timezone(cls, v: datetime):
        if v.tzinfo is None:
            raise ValueError("timestamps must be timezone-aware")
        return v.astimezone(timezone.utc)


class Signal(BaseModel):
    schema_version: str = "1.0"
    generated_at: datetime
    symbol: str
    direction: int
    horizon_s: int = Field(gt=0)
    confidence: float = Field(ge=0.0, le=1.0)
    model_tag: str

The snippet already delivers three concrete benefits. Invalid states fail early, schemas become inspectable by both humans and machines and, client libraries can be generated or written against a stable description rather than inherited through oral tradition.

The internal API also makes research and live mode comparable by design. A strong pattern places all market-data acquisition behind adapter interfaces and exposes only canonical internal models to the rest of the stack. The rest of the code then works with the same objects whether bars came from Alpaca, IBKR, a CSV archive, a replay file, or MetaTrader. Sources are not interchangeable, but their specific oddities belong at the boundary, where they can be normalized, instead of spreading through the strategy codebase.

Do you know what is premature? To build a large platform before the contracts are understood. A lean internal API can live inside a single FastAPI application with a few strict models and a small set of endpoints. What matters is that the contracts are defined before the complexity arrives.

A practical minimal endpoint set for a quant prototyping API includes /health, /bars, /signal/run, /orders/simulate, /orders/submit, /positions, /portfolio/state, and /backtest/run. Each one benefits from typed inputs and outputs, deterministic validation, and a precise statement of time semantics.

The internal API also supports replay, allowing a team to verify that the object evaluated in research matches the object the live system would have seen. An endpoint that consumes archived messages and emits canonical internal states often adds more value than a second optimization endpoint. Little by little we will be building things like this:

Creating the architecture for a trading API

Let’s build a contracts, domain state, adapters, services, routers, lifecycle, and local entrypoint. That structure fits FastAPI well because bigger apps are meant to be split into multiple files and grouped with APIRouter, then assembled with include_router(). FastAPI also recommends lifespan for startup/shutdown work, and it supports WebSockets as a first-class transport beside HTTP. Pydantic’s Field(...) and field_validator(...) are the right tools for constraints and cross-field validation in the schema layer.

The target layout is this:

app/
├── __init__.py
├── main.py
├── dependencies.py
├── core/
│   ├── __init__.py
│   └── lifespan.py
├── domain/
│   ├── __init__.py
│   ├── enums.py
│   └── state.py
├── schemas/
│   ├── __init__.py
│   ├── market.py
│   └── trading.py
├── adapters/
│   ├── __init__.py
│   └── synthetic_market.py
├── services/
│   ├── __init__.py
│   └── trading_engine.py
└── routers/
    ├── __init__.py
    ├── health.py
    ├── market.py
    ├── trading.py
    └── stream.py

What we are trying to do is:

1. To keep the market contracts separate from the mutable engine state.

2. To keep the data source separate from the trading logic.

3. And to keep the trading logic separate from HTTP and WebSocket transport.

This first part is intentionally tiny. Side is a trading concept, so it belongs in the domain layer. That keeps the meaning of BUY and SELL independent from request parsing or response formatting.

from enum import Enum


class Side(str, Enum):
    BUY = "BUY"
    SELL = "SELL"

The next defines the public market-data contract. A Bar is the canonical market object that moves through the API. BarsResponse is the public envelope returned by the market-data route. Pydantic models are the right place for this because they give you typed structure, field constraints, and schema metadata, and FastAPI uses them for parsing and automatic docs.

from __future__ import annotations

from typing import List

from pydantic import BaseModel, Field


class Bar(BaseModel):
    ts: int = Field(..., description="Unix timestamp in seconds")
    open: float
    high: float
    low: float
    close: float
    volume: float


class BarsResponse(BaseModel):
    symbol: str
    bars: List[Bar]

The important point here is that from now on the rest of the code deals with one well-defined object.

The next chunk defines the public trading contract by covering signal requests, order requests, portfolio views, and backtest results. The validators matter because they stop invalid states before they reach the engine. For example, slow <= fast is malformed input, and Pydantic field validators are made for exactly that kind of check.

from __future__ import annotations

from typing import List, Optional

from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator

from app.domain.enums import Side


class SignalRequest(BaseModel):
    symbol: str
    fast: int = Field(5, ge=2, le=500)
    slow: int = Field(20, ge=3, le=1000)

    @field_validator("slow")
    @classmethod
    def validate_windows(cls, v: int, info: ValidationInfo) -> int:
        fast = info.data.get("fast")
        if fast is not None and v <= fast:
            raise ValueError("slow must be greater than fast")
        return v


class SignalResponse(BaseModel):
    symbol: str
    signal: int
    fast_sma: float
    slow_sma: float
    last_close: float
    ts: int


class OrderRequest(BaseModel):
    model_config = ConfigDict(validate_assignment=True)

    symbol: str
    side: Side
    qty: float = Field(..., gt=0)
    price: Optional[float] = Field(None, gt=0)


class PositionModel(BaseModel):
    symbol: str
    qty: float
    avg_price: float
    market_price: float
    unrealized_pnl: float


class PortfolioResponse(BaseModel):
    cash: float
    equity: float
    positions: List[PositionModel]


class OrderResponse(BaseModel):
    order_id: int
    symbol: str
    side: Side
    qty: float
    fill_price: float
    status: str
    ts: int


class BacktestRequest(BaseModel):
    symbol: str
    fast: int = Field(10, ge=2, le=500)
    slow: int = Field(30, ge=3, le=1000)
    qty: float = Field(1.0, gt=0)
    fee_bps: float = Field(1.0, ge=0.0, le=1000.0)
    slippage_bps: float = Field(0.5, ge=0.0, le=1000.0)

    @field_validator("slow")
    @classmethod
    def validate_windows(cls, v: int, info: ValidationInfo) -> int:
        fast = info.data.get("fast")
        if fast is not None and v <= fast:
            raise ValueError("slow must be greater than fast")
        return v


class TradeModel(BaseModel):
    entry_ts: int
    exit_ts: int
    side: Side
    qty: float
    entry_price: float
    exit_price: float
    pnl: float


class BacktestResponse(BaseModel):
    symbol: str
    trades: int
    win_rate: float
    total_pnl: float
    avg_pnl: float
    sharpe_like: float
    equity_curve: List[float]
    trade_log: List[TradeModel]

The rest of the architecture will work around these contracts.

In the next snippet we deal with internal machine memory. The schema layer tells clients what the API accepts and returns. The state layer tells the engine what it holds. The async lock is also part of this state because the app has concurrent flows: background market updates and incoming requests. That shared state must be protected.

from __future__ import annotations

import asyncio
from collections import deque
from dataclasses import dataclass, field
from typing import Deque, Dict

from app.schemas.market import Bar


@dataclass
class PositionState:
    qty: float = 0.0
    avg_price: float = 0.0


@dataclass
class EngineState:
    bars: Dict[str, Deque[Bar]] = field(default_factory=dict)
    positions: Dict[str, PositionState] = field(default_factory=dict)
    cash: float = 100_000.0
    next_order_id: int = 1
    lock: asyncio.Lock = field(default_factory=asyncio.Lock)

This is one of the most important boundaries in the whole design. It prevents the internal engine memory from getting confused with the external API contract.

Now the adapter becomes the place where the market data is normalized into the internal Bar contract. Here we used synthetic data but later it can be any data provider. In this example, the engine should only consume Bar objects. That is the point of the adapter boundary.

from __future__ import annotations

import time
from collections import deque
from typing import Deque, Dict, List

import numpy as np

from app.schemas.market import Bar


class SyntheticMarketAdapter:
    def __init__(
        self,
        symbols: List[str],
        max_bars: int = 2000,
        history_seed: int = 7,
        live_seed: int = 42,
    ):
        self.symbols = symbols
        self.max_bars = max_bars
        self.history_seed = history_seed
        self.live_rng = np.random.default_rng(live_seed)

    def seed_history(self) -> Dict[str, Deque[Bar]]:
        bars_by_symbol: Dict[str, Deque[Bar]] = {
            s: deque(maxlen=self.max_bars) for s in self.symbols
        }

        now = int(time.time()) - 300
        rng = np.random.default_rng(self.history_seed)

        for symbol in self.symbols:
            base = 100.0 if symbol == "SPY" else 400.0 if symbol == "QQQ" else 1.10
            prices = [base]

            for _ in range(300):
                shock = rng.normal(0.0, 0.25 if symbol != "EURUSD" else 0.002)
                prices.append(max(0.1, prices[-1] + shock))

            for i in range(1, len(prices)):
                o = prices[i - 1]
                c = prices[i]
                spread = abs(c - o) + abs(rng.normal(0.0, 0.15))
                h = max(o, c) + spread * 0.5
                l = min(o, c) - spread * 0.5
                v = float(abs(rng.normal(1000, 150)))

                bars_by_symbol[symbol].append(
                    Bar(
                        ts=now + i,
                        open=float(o),
                        high=float(h),
                        low=float(l),
                        close=float(c),
                        volume=v,
                    )
                )

        return bars_by_symbol

    def next_bar(self, symbol: str, last: Bar) -> Bar:
        ts = int(time.time())

        drift = 0.02 if symbol == "SPY" else 0.03 if symbol == "QQQ" else 0.0
        sigma = 0.35 if symbol != "EURUSD" else 0.003

        ret = drift + self.live_rng.normal(0.0, sigma)
        close = max(0.1, last.close + ret)
        spread = abs(close - last.close) + abs(self.live_rng.normal(0.0, sigma * 0.5))

        return Bar(
            ts=ts,
            open=float(last.close),
            high=float(max(last.close, close) + spread * 0.25),
            low=float(min(last.close, close) - spread * 0.25),
            close=float(close),
            volume=float(abs(self.live_rng.normal(1200, 200))),
        )

The two methods play different roles. seed_history() creates an initial in-memory history so the app has something to work with. next_bar() generates the next event in the stream.

The next snippet is the center of the prototype. It is the service layer. Its job is to hold the actual trading logic and state transitions: retrieve bars, compute the latest price, generate a signal, simulate orders, compute portfolio state, and run the backtest. It should know nothing about HTTP status codes, route decorators, or WebSocket handshakes.

from __future__ import annotations

import asyncio
import math
from typing import List

import numpy as np

from app.adapters.synthetic_market import SyntheticMarketAdapter
from app.domain.enums import Side
from app.domain.state import EngineState, PositionState
from app.schemas.market import Bar
from app.schemas.trading import (
    BacktestRequest,
    BacktestResponse,
    OrderRequest,
    OrderResponse,
    PortfolioResponse,
    PositionModel,
    SignalResponse,
    TradeModel,
)


class TradingEngine:
    """
    Prototype trading engine:
    - stores bars in memory
    - computes a simple SMA signal
    - simulates orders
    - maintains portfolio state
    - runs a basic next-open backtest
    """

    def __init__(self, adapter: SyntheticMarketAdapter, starting_cash: float = 100_000.0):
        self.adapter = adapter
        self.symbols = adapter.symbols

        bars = self.adapter.seed_history()
        self.state = EngineState(
            bars=bars,
            positions={s: PositionState() for s in self.symbols},
            cash=starting_cash,
        )

    async def update_market(self) -> None:
        while True:
            async with self.state.lock:
                for symbol, bars in self.state.bars.items():
                    last = bars[-1]
                    new_bar = self.adapter.next_bar(symbol, last)
                    bars.append(new_bar)

            await asyncio.sleep(1.0)

    def _check_symbol(self, symbol: str) -> None:
        if symbol not in self.state.bars:
            raise ValueError(f"unsupported symbol: {symbol}")

    def get_bars(self, symbol: str, limit: int = 200) -> List[Bar]:
        self._check_symbol(symbol)
        return list(self.state.bars[symbol])[-limit:]

    def latest_price(self, symbol: str) -> float:
        self._check_symbol(symbol)
        return float(self.state.bars[symbol][-1].close)

    def signal_sma(self, symbol: str, fast: int, slow: int) -> SignalResponse:
        bars = self.get_bars(symbol, limit=slow + 5)
        closes = np.array([b.close for b in bars], dtype=np.float64)

        if closes.size < slow:
            raise ValueError("not enough bars")

        fast_sma = float(np.mean(closes[-fast:]))
        slow_sma = float(np.mean(closes[-slow:]))
        signal = 1 if fast_sma > slow_sma else -1 if fast_sma < slow_sma else 0

        return SignalResponse(
            symbol=symbol,
            signal=signal,
            fast_sma=fast_sma,
            slow_sma=slow_sma,
            last_close=float(closes[-1]),
            ts=bars[-1].ts,
        )

    async def simulate_order(self, request: OrderRequest) -> OrderResponse:
        self._check_symbol(request.symbol)

        async with self.state.lock:
            fill_price = float(request.price or self.latest_price(request.symbol))
            notional = request.qty * fill_price
            position = self.state.positions[request.symbol]

            if request.side == Side.BUY:
                if self.state.cash < notional:
                    raise ValueError("insufficient cash")

                new_qty = position.qty + request.qty
                if new_qty <= 0:
                    raise ValueError("invalid resulting quantity")

                position.avg_price = (
                    (position.qty * position.avg_price + request.qty * fill_price) / new_qty
                    if position.qty > 0
                    else fill_price
                )
                position.qty = new_qty
                self.state.cash -= notional

            else:
                if position.qty < request.qty:
                    raise ValueError("not enough inventory to sell")

                position.qty -= request.qty
                self.state.cash += notional

                if math.isclose(position.qty, 0.0, abs_tol=1e-12):
                    position.qty = 0.0
                    position.avg_price = 0.0

            order_id = self.state.next_order_id
            self.state.next_order_id += 1

        return OrderResponse(
            order_id=order_id,
            symbol=request.symbol,
            side=request.side,
            qty=request.qty,
            fill_price=fill_price,
            status="FILLED",
            ts=self.state.bars[request.symbol][-1].ts,
        )

    def portfolio(self) -> PortfolioResponse:
        positions_out: List[PositionModel] = []
        equity = self.state.cash

        for symbol, pos in self.state.positions.items():
            if pos.qty <= 0:
                continue

            market = self.latest_price(symbol)
            unreal = (market - pos.avg_price) * pos.qty
            equity += pos.qty * market

            positions_out.append(
                PositionModel(
                    symbol=symbol,
                    qty=pos.qty,
                    avg_price=pos.avg_price,
                    market_price=market,
                    unrealized_pnl=unreal,
                )
            )

        return PortfolioResponse(
            cash=self.state.cash,
            equity=equity,
            positions=positions_out,
        )

    def backtest_sma(self, request: BacktestRequest) -> BacktestResponse:
        bars = self.get_bars(request.symbol, limit=1000)
        closes = np.array([b.close for b in bars], dtype=np.float64)
        opens = np.array([b.open for b in bars], dtype=np.float64)
        ts = np.array([b.ts for b in bars], dtype=np.int64)

        if closes.size <= request.slow + 2:
            raise ValueError("not enough bars for backtest")

        fast_ma = self._rolling_mean(closes, request.fast)
        slow_ma = self._rolling_mean(closes, request.slow)

        offset_fast = request.fast - 1
        offset_slow = request.slow - 1

        signal = np.zeros(closes.size, dtype=np.int8)

        for i in range(closes.size):
            if i < offset_fast or i < offset_slow:
                continue

            f = fast_ma[i - offset_fast]
            s = slow_ma[i - offset_slow]
            signal[i] = 1 if f > s else -1 if f < s else 0

        position = 0
        entry_price = 0.0
        entry_ts = 0
        trade_log: List[TradeModel] = []
        equity_curve = [0.0]
        cumulative_pnl = 0.0

        fee = request.fee_bps / 10_000.0
        slip = request.slippage_bps / 10_000.0

        for i in range(1, closes.size - 1):
            prev_sig = signal[i - 1]
            curr_sig = signal[i]
            next_open = opens[i + 1]

            bullish_cross = prev_sig <= 0 and curr_sig > 0
            bearish_cross = prev_sig >= 0 and curr_sig < 0

            if position == 0 and bullish_cross:
                position = 1
                entry_price = next_open * (1.0 + slip + fee)
                entry_ts = int(ts[i + 1])

            elif position == 1 and bearish_cross:
                exit_price = next_open * (1.0 - slip - fee)
                pnl = (exit_price - entry_price) * request.qty
                cumulative_pnl += pnl
                equity_curve.append(cumulative_pnl)

                trade_log.append(
                    TradeModel(
                        entry_ts=entry_ts,
                        exit_ts=int(ts[i + 1]),
                        side=Side.BUY,
                        qty=request.qty,
                        entry_price=float(entry_price),
                        exit_price=float(exit_price),
                        pnl=float(pnl),
                    )
                )

                position = 0
                entry_price = 0.0
                entry_ts = 0

        if position == 1:
            exit_price = closes[-1] * (1.0 - slip - fee)
            pnl = (exit_price - entry_price) * request.qty
            cumulative_pnl += pnl
            equity_curve.append(cumulative_pnl)

            trade_log.append(
                TradeModel(
                    entry_ts=entry_ts,
                    exit_ts=int(ts[-1]),
                    side=Side.BUY,
                    qty=request.qty,
                    entry_price=float(entry_price),
                    exit_price=float(exit_price),
                    pnl=float(pnl),
                )
            )

        pnls = np.array([t.pnl for t in trade_log], dtype=np.float64)
        trades = int(pnls.size)

        win_rate = float(np.mean(pnls > 0)) if trades > 0 else 0.0
        avg_pnl = float(np.mean(pnls)) if trades > 0 else 0.0
        sharpe_like = float(np.mean(pnls) / (np.std(pnls) + 1e-12)) if trades > 1 else 0.0

        return BacktestResponse(
            symbol=request.symbol,
            trades=trades,
            win_rate=win_rate,
            total_pnl=float(np.sum(pnls)) if trades > 0 else 0.0,
            avg_pnl=avg_pnl,
            sharpe_like=sharpe_like,
            equity_curve=[float(x) for x in equity_curve],
            trade_log=trade_log,
        )

    @staticmethod
    def _rolling_mean(x: np.ndarray, window: int) -> np.ndarray:
        w = np.ones(window, dtype=np.float64) / window
        return np.convolve(x, w, mode="valid")

The constructor receives the adapter and uses it to build the initial state. That means the engine never decides how the data source works. update_market() runs forever and appends one new bar per symbol, but the logic for generating that bar lives in the adapter. signal_sma() turns market state into a typed signal response. simulate_order() mutates positions and cash under the lock so state transitions stay consistent. portfolio() turns internal state into a public response model. backtest_sma() uses the same market objects and similar decision surface as the live part of the prototype, which is a good habit because it keeps the research surface and the API surface aligned.

Now we link the app and the routers. Instead of importing a mutable global engine, routers ask FastAPI for the engine through a dependency. That is a cleaner pattern because it makes the engine app-scoped rather than module-scoped.

from fastapi import Request

from app.services.trading_engine import TradingEngine


def get_engine(request: Request) -> TradingEngine:
    return request.app.state.engine

This router exposes the smallest possible operational surface: is the app alive, which symbols does the engine know about, and what time is it on the server. APIRouter exists to group related path operations into separate files and include them later in the main app.

import time

from fastapi import APIRouter, Depends

from app.dependencies import get_engine
from app.services.trading_engine import TradingEngine

router = APIRouter(tags=["health"])


@router.get("/health")
async def health(engine: TradingEngine = Depends(get_engine)):
    return {
        "status": "ok",
        "symbols": engine.symbols,
        "server_time": int(time.time()),
    }

The important thing is to know how the market is updated.

This router exposes the market-data endpoint. It receives the symbol and limit from the HTTP request, asks the engine for data, translates an internal ValueError into an HTTP exception, and returns a typed response. That is the right shape for a transport layer: receive, delegate, map errors, return.

from fastapi import APIRouter, Depends, HTTPException

from app.dependencies import get_engine
from app.schemas.market import BarsResponse
from app.services.trading_engine import TradingEngine

router = APIRouter(tags=["market"])


@router.get("/bars/{symbol}", response_model=BarsResponse)
async def get_bars(symbol: str, limit: int = 200, engine: TradingEngine = Depends(get_engine)):
    try:
        bars = engine.get_bars(symbol, limit=limit)
    except ValueError as e:
        raise HTTPException(status_code=404, detail=str(e)) from e

    return BarsResponse(symbol=symbol, bars=bars)

This covers signal generation, order simulation, portfolio state, and backtest execution. Each route receives a typed request, obtains the engine through the dependency layer, delegates the logic, and returns a typed response. The routers stay thin because all financial meaning lives in the service layer.

from fastapi import APIRouter, Depends, HTTPException

from app.dependencies import get_engine
from app.schemas.trading import (
    BacktestRequest,
    BacktestResponse,
    OrderRequest,
    OrderResponse,
    PortfolioResponse,
    SignalRequest,
    SignalResponse,
)
from app.services.trading_engine import TradingEngine

router = APIRouter(tags=["trading"])


@router.post("/signal/run", response_model=SignalResponse)
async def run_signal(request: SignalRequest, engine: TradingEngine = Depends(get_engine)):
    try:
        return engine.signal_sma(request.symbol, request.fast, request.slow)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e)) from e


@router.post("/orders/simulate", response_model=OrderResponse)
async def simulate_order(request: OrderRequest, engine: TradingEngine = Depends(get_engine)):
    try:
        return await engine.simulate_order(request)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e)) from e


@router.get("/portfolio", response_model=PortfolioResponse)
async def get_portfolio(engine: TradingEngine = Depends(get_engine)):
    return engine.portfolio()


@router.post("/backtest/run", response_model=BacktestResponse)
async def run_backtest(request: BacktestRequest, engine: TradingEngine = Depends(get_engine)):
    try:
        return engine.backtest_sma(request)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e)) from e

FastAPI also supports WebSockets as a first-class transport. This route exposes a live stream of the latest bar for a symbol. It reads the engine from websocket.app.state, validates the symbol, accepts the connection, and pushes one JSON payload per second. This is a different transport model from HTTP, so it deserves its own router file.

import asyncio

from fastapi import APIRouter, WebSocket, WebSocketDisconnect

router = APIRouter(tags=["stream"])


@router.websocket("/ws/{symbol}")
async def ws_prices(websocket: WebSocket, symbol: str):
    engine = websocket.app.state.engine

    try:
        engine._check_symbol(symbol)
    except ValueError:
        await websocket.close(code=1008)
        return

    await websocket.accept()

    try:
        while True:
            bar = engine.get_bars(symbol, limit=1)[0]
            await websocket.send_json(
                {
                    "symbol": symbol,
                    "ts": bar.ts,
                    "price": bar.close,
                    "open": bar.open,
                    "high": bar.high,
                    "low": bar.low,
                    "volume": bar.volume,
                }
            )
            await asyncio.sleep(1.0)
    except WebSocketDisconnect:
        return

FastAPI recommends lifespan for startup and shutdown logic, and it makes clear that if you use lifespan, the old startup and shutdown event handlers are no longer called. This is the right place to build the adapter, build the engine, start the background market task, attach both to the app state, and cancel the task when the process shuts down.

from __future__ import annotations

import asyncio
from contextlib import asynccontextmanager

from fastapi import FastAPI

from app.adapters.synthetic_market import SyntheticMarketAdapter
from app.services.trading_engine import TradingEngine


@asynccontextmanager
async def lifespan(app: FastAPI):
    adapter = SyntheticMarketAdapter(symbols=["SPY", "QQQ", "EURUSD"])
    engine = TradingEngine(adapter=adapter)

    market_task = asyncio.create_task(engine.update_market())

    app.state.engine = engine
    app.state.market_task = market_task

    try:
        yield
    finally:
        market_task.cancel()
        try:
            await market_task
        except asyncio.CancelledError:
            pass

This final part assembles the application. It creates the FastAPI app, attaches metadata, includes the routers, and defines the local Uvicorn entrypoint.

from fastapi import FastAPI

from app.core.lifespan import lifespan
from app.routers import health, market, stream, trading


def create_app() -> FastAPI:
    app = FastAPI(
        title="Prototype Quant Trading API",
        description="Reference architecture for a prototyping trading API",
        lifespan=lifespan,
    )

    app.include_router(health.router)
    app.include_router(market.router)
    app.include_router(trading.router)
    app.include_router(stream.router)

    return app


app = create_app()


if __name__ == "__main__":
    import uvicorn

    uvicorn.run("app.main:app", host="127.0.0.1", port=8000, reload=True)

Okay, let’s execute this

INFO:     Started server process [252]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [252]

Cool! This is alive! In practice, the next step for most quant teams is to define a narrow internal API with a few canonical models and a few endpoints. Once those are stable, extra layers can be added without changing the meaning of the core objects.

A final misconception is worth removing. Good API design is quantitative thinking applied to state, observability, and execution. That is why the API layer matters, why financial APIs must be handled with more skepticism than most research pipelines apply, and why designing your own internal API is one of the most practical acts for a systematic trader.

Cool guys! Good job today! Remember that you have the full code in the appendix so you can torture it. Time to say good bye! Stay sharp, stay bold, stay unshakeable 📈

PS: What do you value more? Downloading data from an API or having it plotted?

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full code:

from __future__ import annotations

import asyncio
import math
import time
from collections import deque
from contextlib import asynccontextmanager
from dataclasses import dataclass, field
from enum import Enum
from typing import Deque, Dict, List, Optional

import numpy as np
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator


# DOMAIN
class Side(str, Enum):
    BUY = "BUY"
    SELL = "SELL"

# SCHEMAS
class Bar(BaseModel):
    ts: int = Field(..., description="Unix timestamp in seconds")
    open: float
    high: float
    low: float
    close: float
    volume: float


class BarsResponse(BaseModel):
    symbol: str
    bars: List[Bar]


class SignalRequest(BaseModel):
    symbol: str
    fast: int = Field(5, ge=2, le=500)
    slow: int = Field(20, ge=3, le=1000)

    @field_validator("slow")
    @classmethod
    def validate_windows(cls, v: int, info: ValidationInfo) -> int:
        fast = info.data.get("fast")
        if fast is not None and v <= fast:
            raise ValueError("slow must be greater than fast")
        return v


class SignalResponse(BaseModel):
    symbol: str
    signal: int
    fast_sma: float
    slow_sma: float
    last_close: float
    ts: int


class OrderRequest(BaseModel):
    model_config = ConfigDict(validate_assignment=True)

    symbol: str
    side: Side
    qty: float = Field(..., gt=0)
    price: Optional[float] = Field(None, gt=0)


class PositionModel(BaseModel):
    symbol: str
    qty: float
    avg_price: float
    market_price: float
    unrealized_pnl: float


class PortfolioResponse(BaseModel):
    cash: float
    equity: float
    positions: List[PositionModel]


class OrderResponse(BaseModel):
    order_id: int
    symbol: str
    side: Side
    qty: float
    fill_price: float
    status: str
    ts: int


class BacktestRequest(BaseModel):
    symbol: str
    fast: int = Field(10, ge=2, le=500)
    slow: int = Field(30, ge=3, le=1000)
    qty: float = Field(1.0, gt=0)
    fee_bps: float = Field(1.0, ge=0.0, le=1000.0)
    slippage_bps: float = Field(0.5, ge=0.0, le=1000.0)

    @field_validator("slow")
    @classmethod
    def validate_windows(cls, v: int, info: ValidationInfo) -> int:
        fast = info.data.get("fast")
        if fast is not None and v <= fast:
            raise ValueError("slow must be greater than fast")
        return v


class TradeModel(BaseModel):
    entry_ts: int
    exit_ts: int
    side: Side
    qty: float
    entry_price: float
    exit_price: float
    pnl: float


class BacktestResponse(BaseModel):
    symbol: str
    trades: int
    win_rate: float
    total_pnl: float
    avg_pnl: float
    sharpe_like: float
    equity_curve: List[float]
    trade_log: List[TradeModel]

# INTERNAL STATE
@dataclass
class PositionState:
    qty: float = 0.0
    avg_price: float = 0.0


@dataclass
class EngineState:
    bars: Dict[str, Deque[Bar]] = field(default_factory=dict)
    positions: Dict[str, PositionState] = field(default_factory=dict)
    cash: float = 100_000.0
    next_order_id: int = 1
    lock: asyncio.Lock = field(default_factory=asyncio.Lock)

# ADAPTER
class SyntheticMarketAdapter:
    def __init__(
        self,
        symbols: List[str],
        max_bars: int = 2000,
        history_seed: int = 7,
        live_seed: int = 42,
    ):
        self.symbols = symbols
        self.max_bars = max_bars
        self.history_seed = history_seed
        self.live_rng = np.random.default_rng(live_seed)

    def seed_history(self) -> Dict[str, Deque[Bar]]:
        bars_by_symbol: Dict[str, Deque[Bar]] = {
            s: deque(maxlen=self.max_bars) for s in self.symbols
        }

        now = int(time.time()) - 300
        rng = np.random.default_rng(self.history_seed)

        for symbol in self.symbols:
            base = 100.0 if symbol == "SPY" else 400.0 if symbol == "QQQ" else 1.10
            prices = [base]

            for _ in range(300):
                shock = rng.normal(0.0, 0.25 if symbol != "EURUSD" else 0.002)
                prices.append(max(0.1, prices[-1] + shock))

            for i in range(1, len(prices)):
                o = prices[i - 1]
                c = prices[i]
                spread = abs(c - o) + abs(rng.normal(0.0, 0.15))
                h = max(o, c) + spread * 0.5
                l = min(o, c) - spread * 0.5
                v = float(abs(rng.normal(1000, 150)))

                bars_by_symbol[symbol].append(
                    Bar(
                        ts=now + i,
                        open=float(o),
                        high=float(h),
                        low=float(l),
                        close=float(c),
                        volume=v,
                    )
                )

        return bars_by_symbol

    def next_bar(self, symbol: str, last: Bar) -> Bar:
        ts = int(time.time())

        drift = 0.02 if symbol == "SPY" else 0.03 if symbol == "QQQ" else 0.0
        sigma = 0.35 if symbol != "EURUSD" else 0.003

        ret = drift + self.live_rng.normal(0.0, sigma)
        close = max(0.1, last.close + ret)
        spread = abs(close - last.close) + abs(self.live_rng.normal(0.0, sigma * 0.5))

        return Bar(
            ts=ts,
            open=float(last.close),
            high=float(max(last.close, close) + spread * 0.25),
            low=float(min(last.close, close) - spread * 0.25),
            close=float(close),
            volume=float(abs(self.live_rng.normal(1200, 200))),
        )

# SERVICE LAYER
class TradingEngine:
    """
    Prototype trading engine:
    - stores bars in memory
    - computes a simple SMA signal
    - simulates orders
    - maintains portfolio state
    - runs a basic next-open backtest
    """

    def __init__(self, adapter: SyntheticMarketAdapter, starting_cash: float = 100_000.0):
        self.adapter = adapter
        self.symbols = adapter.symbols

        bars = self.adapter.seed_history()
        self.state = EngineState(
            bars=bars,
            positions={s: PositionState() for s in self.symbols},
            cash=starting_cash,
        )

    async def update_market(self) -> None:
        while True:
            async with self.state.lock:
                for symbol, bars in self.state.bars.items():
                    last = bars[-1]
                    new_bar = self.adapter.next_bar(symbol, last)
                    bars.append(new_bar)

            await asyncio.sleep(1.0)

    def _check_symbol(self, symbol: str) -> None:
        if symbol not in self.state.bars:
            raise ValueError(f"unsupported symbol: {symbol}")

    def get_bars(self, symbol: str, limit: int = 200) -> List[Bar]:
        self._check_symbol(symbol)
        return list(self.state.bars[symbol])[-limit:]

    def latest_price(self, symbol: str) -> float:
        self._check_symbol(symbol)
        return float(self.state.bars[symbol][-1].close)

    def signal_sma(self, symbol: str, fast: int, slow: int) -> SignalResponse:
        bars = self.get_bars(symbol, limit=slow + 5)
        closes = np.array([b.close for b in bars], dtype=np.float64)

        if closes.size < slow:
            raise ValueError("not enough bars")

        fast_sma = float(np.mean(closes[-fast:]))
        slow_sma = float(np.mean(closes[-slow:]))
        signal = 1 if fast_sma > slow_sma else -1 if fast_sma < slow_sma else 0

        return SignalResponse(
            symbol=symbol,
            signal=signal,
            fast_sma=fast_sma,
            slow_sma=slow_sma,
            last_close=float(closes[-1]),
            ts=bars[-1].ts,
        )

    async def simulate_order(self, request: OrderRequest) -> OrderResponse:
        self._check_symbol(request.symbol)

        async with self.state.lock:
            fill_price = float(request.price or self.latest_price(request.symbol))
            notional = request.qty * fill_price
            position = self.state.positions[request.symbol]

            if request.side == Side.BUY:
                if self.state.cash < notional:
                    raise ValueError("insufficient cash")

                new_qty = position.qty + request.qty
                if new_qty <= 0:
                    raise ValueError("invalid resulting quantity")

                position.avg_price = (
                    (position.qty * position.avg_price + request.qty * fill_price) / new_qty
                    if position.qty > 0
                    else fill_price
                )
                position.qty = new_qty
                self.state.cash -= notional

            else:
                if position.qty < request.qty:
                    raise ValueError("not enough inventory to sell")

                position.qty -= request.qty
                self.state.cash += notional

                if math.isclose(position.qty, 0.0, abs_tol=1e-12):
                    position.qty = 0.0
                    position.avg_price = 0.0

            order_id = self.state.next_order_id
            self.state.next_order_id += 1

        return OrderResponse(
            order_id=order_id,
            symbol=request.symbol,
            side=request.side,
            qty=request.qty,
            fill_price=fill_price,
            status="FILLED",
            ts=self.state.bars[request.symbol][-1].ts,
        )

    def portfolio(self) -> PortfolioResponse:
        positions_out: List[PositionModel] = []
        equity = self.state.cash

        for symbol, pos in self.state.positions.items():
            if pos.qty <= 0:
                continue

            market = self.latest_price(symbol)
            unreal = (market - pos.avg_price) * pos.qty
            equity += pos.qty * market

            positions_out.append(
                PositionModel(
                    symbol=symbol,
                    qty=pos.qty,
                    avg_price=pos.avg_price,
                    market_price=market,
                    unrealized_pnl=unreal,
                )
            )

        return PortfolioResponse(
            cash=self.state.cash,
            equity=equity,
            positions=positions_out,
        )

    def backtest_sma(self, request: BacktestRequest) -> BacktestResponse:
        bars = self.get_bars(request.symbol, limit=1000)
        closes = np.array([b.close for b in bars], dtype=np.float64)
        opens = np.array([b.open for b in bars], dtype=np.float64)
        ts = np.array([b.ts for b in bars], dtype=np.int64)

        if closes.size <= request.slow + 2:
            raise ValueError("not enough bars for backtest")

        fast_ma = self._rolling_mean(closes, request.fast)
        slow_ma = self._rolling_mean(closes, request.slow)

        offset_fast = request.fast - 1
        offset_slow = request.slow - 1

        signal = np.zeros(closes.size, dtype=np.int8)

        for i in range(closes.size):
            if i < offset_fast or i < offset_slow:
                continue

            f = fast_ma[i - offset_fast]
            s = slow_ma[i - offset_slow]
            signal[i] = 1 if f > s else -1 if f < s else 0

        position = 0
        entry_price = 0.0
        entry_ts = 0
        trade_log: List[TradeModel] = []
        equity_curve = [0.0]
        cumulative_pnl = 0.0

        fee = request.fee_bps / 10_000.0
        slip = request.slippage_bps / 10_000.0

        for i in range(1, closes.size - 1):
            prev_sig = signal[i - 1]
            curr_sig = signal[i]
            next_open = opens[i + 1]

            bullish_cross = prev_sig <= 0 and curr_sig > 0
            bearish_cross = prev_sig >= 0 and curr_sig < 0

            if position == 0 and bullish_cross:
                position = 1
                entry_price = next_open * (1.0 + slip + fee)
                entry_ts = int(ts[i + 1])

            elif position == 1 and bearish_cross:
                exit_price = next_open * (1.0 - slip - fee)
                pnl = (exit_price - entry_price) * request.qty
                cumulative_pnl += pnl
                equity_curve.append(cumulative_pnl)

                trade_log.append(
                    TradeModel(
                        entry_ts=entry_ts,
                        exit_ts=int(ts[i + 1]),
                        side=Side.BUY,
                        qty=request.qty,
                        entry_price=float(entry_price),
                        exit_price=float(exit_price),
                        pnl=float(pnl),
                    )
                )

                position = 0
                entry_price = 0.0
                entry_ts = 0

        if position == 1:
            exit_price = closes[-1] * (1.0 - slip - fee)
            pnl = (exit_price - entry_price) * request.qty
            cumulative_pnl += pnl
            equity_curve.append(cumulative_pnl)

            trade_log.append(
                TradeModel(
                    entry_ts=entry_ts,
                    exit_ts=int(ts[-1]),
                    side=Side.BUY,
                    qty=request.qty,
                    entry_price=float(entry_price),
                    exit_price=float(exit_price),
                    pnl=float(pnl),
                )
            )

        pnls = np.array([t.pnl for t in trade_log], dtype=np.float64)
        trades = int(pnls.size)

        win_rate = float(np.mean(pnls > 0)) if trades > 0 else 0.0
        avg_pnl = float(np.mean(pnls)) if trades > 0 else 0.0
        sharpe_like = float(np.mean(pnls) / (np.std(pnls) + 1e-12)) if trades > 1 else 0.0

        return BacktestResponse(
            symbol=request.symbol,
            trades=trades,
            win_rate=win_rate,
            total_pnl=float(np.sum(pnls)) if trades > 0 else 0.0,
            avg_pnl=avg_pnl,
            sharpe_like=sharpe_like,
            equity_curve=[float(x) for x in equity_curve],
            trade_log=trade_log,
        )

    @staticmethod
    def _rolling_mean(x: np.ndarray, window: int) -> np.ndarray:
        w = np.ones(window, dtype=np.float64) / window
        return np.convolve(x, w, mode="valid")

# DEPENDENCY
def get_engine(request: Request) -> TradingEngine:
    return request.app.state.engine

# ROUTERS
health_router = APIRouter(tags=["health"])
market_router = APIRouter(tags=["market"])
trading_router = APIRouter(tags=["trading"])
stream_router = APIRouter(tags=["stream"])


@health_router.get("/health")
async def health(engine: TradingEngine = Depends(get_engine)):
    return {
        "status": "ok",
        "symbols": engine.symbols,
        "server_time": int(time.time()),
    }


@market_router.get("/bars/{symbol}", response_model=BarsResponse)
async def get_bars(symbol: str, limit: int = 200, engine: TradingEngine = Depends(get_engine)):
    try:
        bars = engine.get_bars(symbol, limit=limit)
    except ValueError as e:
        raise HTTPException(status_code=404, detail=str(e)) from e

    return BarsResponse(symbol=symbol, bars=bars)


@trading_router.post("/signal/run", response_model=SignalResponse)
async def run_signal(request: SignalRequest, engine: TradingEngine = Depends(get_engine)):
    try:
        return engine.signal_sma(request.symbol, request.fast, request.slow)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e)) from e


@trading_router.post("/orders/simulate", response_model=OrderResponse)
async def simulate_order(request: OrderRequest, engine: TradingEngine = Depends(get_engine)):
    try:
        return await engine.simulate_order(request)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e)) from e


@trading_router.get("/portfolio", response_model=PortfolioResponse)
async def get_portfolio(engine: TradingEngine = Depends(get_engine)):
    return engine.portfolio()


@trading_router.post("/backtest/run", response_model=BacktestResponse)
async def run_backtest(request: BacktestRequest, engine: TradingEngine = Depends(get_engine)):
    try:
        return engine.backtest_sma(request)
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e)) from e


@stream_router.websocket("/ws/{symbol}")
async def ws_prices(websocket: WebSocket, symbol: str):
    engine: TradingEngine = websocket.scope["app"].state.engine

    try:
        engine._check_symbol(symbol)
    except ValueError:
        await websocket.close(code=1008)
        return

    await websocket.accept()

    try:
        while True:
            bar = engine.get_bars(symbol, limit=1)[0]
            await websocket.send_json(
                {
                    "symbol": symbol,
                    "ts": bar.ts,
                    "price": bar.close,
                    "open": bar.open,
                    "high": bar.high,
                    "low": bar.low,
                    "volume": bar.volume,
                }
            )
            await asyncio.sleep(1.0)
    except WebSocketDisconnect:
        return

# LIFESPAN
@asynccontextmanager
async def lifespan(app: FastAPI):
    adapter = SyntheticMarketAdapter(symbols=["SPY", "QQQ", "EURUSD"])
    engine = TradingEngine(adapter=adapter)
    market_task = asyncio.create_task(engine.update_market())

    app.state.engine = engine
    app.state.market_task = market_task

    try:
        yield
    finally:
        market_task.cancel()
        try:
            await market_task
        except asyncio.CancelledError:
            pass

# APP ASSEMBLY
def create_app() -> FastAPI:
    app = FastAPI(
        title="Prototype Quant Trading API",
        description="Single-file trading API for testing",
        lifespan=lifespan,
    )

    app.include_router(health_router)
    app.include_router(market_router)
    app.include_router(trading_router)
    app.include_router(stream_router)

    return app


app = create_app()

# MAIN
if __name__ == "__main__":
    import uvicorn

    uvicorn.run("main:app", host="127.0.0.1", port=8000, reload=True)

Table of contents:

1. Introduction.

2. Conflict and escalation dynamics.

3. Energy shock, inflation pass-through, and logistics friction.

4. Monetary policy, fiscal space, and financial stability channels.

5. Alliance, war, and strategic competition.

6. Defense-industrial economics and the cost curve of modern warfare.

7. Investment implications for multi-asset portfolios.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Today we map the transmission system through which an Iran-centered war in the Gulf becomes a global pricing regime. The shock already extends far beyond the battlefield. It now moves through oil, equities, shipping, insurance, and industrial planning. Brent crude traded above $119 a barrel in early March 2026 as the conflict widened. Businesses and investors across Europe, Asia, and North America repriced energy exposure, freight risk, and supply-chain vulnerability. That response follows the structure of the system itself. The Strait of Hormuz remains one of the world’s decisive energy corridors. In 2025, flows through the strait averaged about 20 million barrels per day, equal to roughly one-fifth of global petroleum liquids consumption. Around one-fifth of global LNG trade also moved through that corridor.

The central frame is escalation through transmission channels. Missile exchanges raise shipping insurance. Higher shipping insurance raises fuel and freight costs. Higher fuel and freight costs strengthen inflation pressure. Stronger inflation pressure tightens monetary conditions, narrows fiscal room, and increases sensitivity in rates and credit markets. Geopolitical violence therefore enters asset prices through a cumulative macro-financial sequence. That sequence now operates in a constrained policy setting. On January 28, 2026, the Federal Reserve kept the federal funds target range at 3.5% to 3.75%. On February 5, 2026, the European Central Bank kept the deposit facility rate at 2.00%. The Congressional Budget Office projected a $1.9 trillion U.S. federal deficit for fiscal 2026. SIPRI reported that world military expenditure reached $2.718 trillion in 2024. Monetary authorities, fiscal states, and defense systems entered this phase with meaningful pre-existing constraints.

If you want to learn more, you’ll find more information here:

Short Term Energy Outlook 2026

7.03MB ∙ PDF file

Download

Download

Short Term Energy Outlook 2026

7.03MB ∙ PDF file

Download

Download

Read through that lens, the conflict functions as a regime test for portfolios, governments, and industrial systems. It tests whether logistics networks can reroute under pressure. It tests whether central banks can preserve credibility under renewed energy inflation. It tests whether fiscal states can absorb a larger strategic bill while debt burdens remain elevated. It also tests whether defense-industrial supply chains can sustain a modern cost curve shaped by expensive interceptors, long lead times, and repeated replenishment cycles. For that reason, the analysis that follows moves from escalation to energy pass-through. Then to monetary and fiscal constraints. Then to alliance management and defense-industrial economics. And finally to the asset-pricing implications that matter most for multi-asset investors in a world where geopolitical shocks increasingly behave like macro transmission systems rather than isolated events.

Subscribe now

Conflict and escalation dynamics

The market narrative around an Iran-centered conflict that expands across the broader Middle East, with spillovers into Gulf targets, regional infrastructure, and allied basing arrangements. That matches the pattern described by major wire services: a U.S.-Israeli campaign against Iran has coincided with missile and drone strikes across multiple Gulf states and with pressure on energy logistics and critical infrastructure.

A core structural feature of this conflict phase is its multi-node geometry. It links air and missile defense networks, energy export platforms, sea lanes, and diplomatic posture into a single operating system. Missile-defense radars and communications sites have emerged as strategic targets because they anchor interception chains for systems deployed by the United States and regional partners. This target set differs from a narrow, territory-bound war and aligns with a systems war logic: degrading sensors can reduce the marginal value of interceptors and raise the per-attack success probability for drones and missiles.

A gap between initial strike objectives and post-strike political end-states, framed as the absence of a defined phase two. A related public signal appeared in reporting from the Financial Times, which cited Senator Mark Warner’s view that administration briefings lacked an articulated next-step plan for the aftermath of the military campaign. In investor terms, that type of gap raises the probability distribution over duration, geographic spread, and infrastructure targeting, which then feeds into commodity risk premia and credit repricing.

Escalation pathways come from three channels that interact. The first is kinetic interaction across dense air-defense coverage, where a high volume of launches produces stress on interceptors, radars, and maintenance cycles. The second is economic coercion via chokepoints and storage constraints, which can force producers to curb output when exports face friction. The third is alliance-management stress, especially when operations touch bases, ports, and treaty obligations for states that seek distance from a campaign.

At the strategic layer, European governments have produced a legal framing that increases political distance from kinetic participation, which raises burden-sharing uncertainty and can bring additional volatility around NATO posture and basing in Southern Europe. This divergence matters for markets because alliance cohesion influences sanction coordination, shipping escorts, and the credibility of de-escalation off-ramps.

Conflict intensity has already shaped equity pricing. Intraday drawdowns in U.S. equities exceeded one thousand Dow points in at least one session during the first week of March, with oil-price acceleration as a primary driver in contemporaneous reporting. That linkage between war-duration uncertainty and inflation risk is central: markets can price tactical battlefield outcomes, yet they price duration and breadth through commodities, rates, and credit spreads.

Energy shock, inflation pass-through, and logistics friction

The dominant macro transmission mechanism in this phase is the energy channel, amplified by chokepoint concentration. The Strait of Hormuz serves as a major conduit for global petroleum liquids and for liquefied natural gas trade. The U.S. Energy Information Administration estimated average oil flows through the strait around twenty million barrels per day in 2024, which aligns with about one-fifth of global petroleum liquids consumption, and it assessed about one-fifth of global LNG trade transiting the strait in 2024, primarily from Qatar. The International Energy Agency provides a parallel framing, with LNG transiting volumes around one-fifth of global LNG trade and seaborne oil trade shares tied to Hormuz flows.

In this conflict phase, shipping disruption and storage constraints have emerged as binding constraints for producers and buyers. Recent reportes indicate producer firms have managed offshore output around storage restrictions and have used alternative export routes, while the broader war environment has pushed some Gulf producers toward output curtailment and force majeure declarations. In parallel, European policymakers convened contingency coordination groups after a sharp rise in regional gas prices.

Price action in crude and refined products has reflected a risk-premium regime. In early March, Brent traded in the mid-to-high eighties and moved into the nineties in subsequent sessions, while U.S. benchmarks also rose with large daily moves. A key market feature is convexity: small changes in expected duration of disruption can produce large changes in expected supply deficit, inventory draw rates, and implied volatility in oil options.

European natural gas provides a second amplification channel because LNG availability and shipping routes link Asia and Europe through spot competition. A surge exceeding fifty percent in European gas prices after the conflict began to disrupt key flows and to affect perceptions of LNG supply security, with storage levels and refill paths becoming a policy focus. This dynamic creates a macro wedge between the United States and the euro area: U.S. gas prices can move on domestic balances, whereas Europe clears at the margin through LNG cargo competition.

Freight and air logistics form a third channel that carries inflation into tradables beyond energy. The Washington Post described broad disruption to air and sea traffic, with airfreight rates between Asia and Europe rising by about forty-five percent in its reporting, alongside stalled shipping around Hormuz and suspended new shipments by major carriers. A shipping surcharge response reflects the same mechanism: higher fuel costs and routing risk raise the delivered price of intermediate and final goods, with the largest impulse for time-sensitive goods.

Inflation pass-through operates through headline energy prices, second-round wage and margin dynamics, and inflation expectations. Energy-driven inflation blocks rate cuts by the Federal Reserve and implies tighter conditions across Europe. A similar dilemma appears in mainstream reporting that highlights central bank hesitation under an oil-driven inflation impulse. Several analysis ahead of the strikes assessed that a prolonged disruption could push oil toward one hundred dollars and add roughly six to seven tenths of a percentage point to global inflation, illustrating how the energy channel can dominate the inflation path when the shock persists.

Monetary policy, fiscal space, and financial stability channels

The monetary side of the macro regime is shaped by a clash between disinflation progress and an energy shock that lifts headline inflation and shifts risk distributions. The Federal Open Market Committee held the target range for the federal funds rate at 3.5% to 3.75% in its January statement. A patient stance among Fed officials, with emphasis on holding policy steady while monitoring inflation dynamics and the war-driven energy impulse. That combination gives the policy path a more state-contingent structure: energy moves compress the room for easing, while labor-market weakening raises the weight on employment objectives.

In the euro area, the European Central Bank kept key rates unchanged in early February, with a deposit facility rate at 2.00% and related corridor rates as published by the ECB. The euro area energy shock risk differs from the U.S. risk because net import dependence remains meaningful and because the marginal LNG price clears through global competition. That asymmetry can widen growth and inflation dispersion within advanced economies and can shape currency and rate differentials.

Fiscal capacity and debt service costs form an additional constraint layer. The U.S. Congressional Budget Office projected a deficit of $1.9 trillion in fiscal year 2026 and a rising path for federal debt, reaching 120 percent of GDP by 2036 in its baseline outlook. Rising net interest costs represent a large share of that story, with outside analyses drawing from the same baseline highlighting a doubling path for net interest outlays across the next decade. These fiscal conditions matter for geopolitics because large, persistent deficits reduce political room for sustained high-cost expeditionary operations and raise the sensitivity of sovereign financing conditions to inflation shocks.

Private-sector balance sheet stress and credit repricing tighten the feedback loop between rates and growth. U.S. private credit defaults reached a record 9.2% in 2025 in a Fitch dataset, with floating-rate structures linking cash-flow stress to policy rates and to credit spreads. Household stress measures show a parallel dynamics: the Federal Reserve Bank of New York reported aggregate delinquency worsening in late 2025, with 4.8% of outstanding household debt in some stage of delinquency, and a rise in serious delinquency flows for several categories. Consumer balance sheets carry fatigue from prior inflation cycles, which raises the probability that an energy shock translates into demand compression rather than into a clean growth-through-inflation regime. The combination of household stress and private credit defaults suggests a financial system that prices duration risk in the real economy with greater sensitivity to energy-driven inflation.

Markets have begun to price this regime as a volatility state change rather than as a single event shock. The VIX moved above 20 amid the conflict, with attention to shock-absorber trades and hedging demand. Equity selloffs and partial rebounds have reflected a tug-of-war between the energy/inflation channel and the expectation that logistics protection or de-escalation steps might cap the duration of disruption.

Alliance, war, and strategic competition

Read more

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Calibration and scoring under uncertainty

This chapter turns an inefficiency claim into a measurable probabilistic object. A model earns credibility when its output remains a truthful conditional description of the future under the same decision-time information that defines the claim. Proper scoring rules provide that bridge. They test whether probabilities, quantiles, or densities align with realized outcomes, and they do so inside the exact state regions where the strategy intends to deploy capital.

What’s inside:

1. Scoring for conditional evidence. The chapter defines prediction as a conditional statement about the future and explains why scoring must reward truthful reporting of the conditional law rather than flattering in-sample behavior.

2. Proper scoring fundamentals. It establishes the role of strict propriety and separates probabilistic description from downstream action, so learning targets the conditional mechanism itself rather than the trade rule built on top of it.

3. Log score and likelihood geometry. Full conditional densities are evaluated through log score, which measures how much probability mass the model assigns to realized outcomes and turns distributional mismatch into an empirical cost.

4. Brier score and probability geometry. Binary event probabilities are scored through squared probability error, which makes calibration visible and tests whether reported event likelihoods match realized frequencies.

5. Pinball loss and quantile geometry. Conditional quantile boundaries are evaluated through asymmetric loss that checks whether reported boundaries retain the correct crossing behavior under the declared state and horizon.

6. Calibration under state selection. The chapter shows why a model can look acceptable on pooled data while failing in the narrow states where capital is actually deployed, so calibration must be checked on the same conditional sample that drives execution.

7. Conditional calibration. Local inefficiency claims require local validation, so probabilities and quantiles must retain their interpretation inside the same state regions where divergence suggests conditional structure.

8. Miscalibration as false inefficiency. The chapter explains how overconfident outputs can manufacture the illusion of edge when threshold-based decision rules amplify extreme signals that lack stable probabilistic meaning.

9. Decision-aware scoring. Evaluation is aligned with the architecture of risk, exposure, and constraints, so errors matter most where the portfolio carries the highest operational sensitivity.

Sample

1.94MB ∙ PDF file

Download

Download

Read more

Table of contents:

1. Introduction.

2. Global macro outlook.

3. Regional breakdown.

4. Major geopolitical flashpoints.

5. Commodity markets and supply chains.

6. Financial markets and capital flows.

7. Policy and regulatory trends.

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Growth can print while the distribution of outcomes widens, because the drivers that move expectations arrive as policy decisions sanction packages, court rulings, or any other events. Markets price that distribution every day through curves, basis, dispersion, and volatility surfaces, often faster than macro releases can confirm the regime shift.

Large refinancing calendars and ongoing portfolio runoff change who absorbs duration and at what price, which means the term premium becomes more active. In that setting, the same growth print can produce different outcomes depending on auction demand, collateral conditions, and the willingness of balance sheets to intermediate. This is why the discussion of central banks sits alongside issuance and liquidity: the reaction function only matters if the market can clear the supply without repricing risk.

Trade policy and regulation matter in the same way: they are rule changes that modify cash-flow boundaries and execution constraints. Tariff schedules alter landed costs, sourcing, and inventory timing. Legal pathways and exemption design shape how durable the tariff environment is, which in turn shapes how firms time capex and how markets price margins. Sanctions and enforcement innovations act through shipping, insurance, classification, and payments, changing the set of compliant routes and the economics of delivery. These are microstructure channels with macro consequences, and they move faster than quarterly accounts. If you trade cross-asset, you already know the pattern: the first clean signal often appears in basis, time spreads, freight, and vol before it reaches earnings or CPI.

But there are much more, indeed, if you want to go deeper don’t miss this:

Gas market report

3.83MB ∙ PDF file

Download

Download

Global Economic Prospects, January 2026

4.77MB ∙ PDF file

Download

Download

Economic bulletin

2.78MB ∙ PDF file

Download

Download

Financial markets and capital flows tie the regime together by showing who finances whom and under what conditions. Fund flows reveal risk appetite and breadth, while cross-border data anchors the foreign bid that matters for rate clearing and dollar dynamics. In 2026, these flow signals interact with policy stability, issuance pressure, and market depth in a way that can flip the sign of traditional relationships.

Subscribe now

Global macro outlook

The World Bank describes resilience amid persistent trade tensions and uncertainty, and it points to a mild step-down in growth as demand for traded goods cools after a tariff-related pull-forward in 2025. The UN’s 2026 outlook reinforces this texture by forecasting subdued global growth relative to the pre-pandemic average and by linking trade tensions and fiscal strains to the outlook. The IMF’s update frames the offsetting forces that keep growth steady, with technology investment and private-sector adaptability acting as counterweights to trade policy headwinds. In that IMF framing, technology-linked capex, including investment tied to artificial intelligence, plays a role in sustaining activity even when goods trade momentum softens.

A services-led expansion favors domestic demand signals, labor income dynamics, and pricing power inside non-tradable segments. A cooling goods cycle shifts the focus toward inventories, freight rates, and margin compression in trade-exposed firms. Policy uncertainty adds a layer of option-like convexity to macro outcomes: tariff announcements, legal rulings, and negotiated carve-outs create discrete jumps in expectations that propagate through FX basis, forward points, and equity dispersion. Reporting on tariff policy illustrates this channel by showing how legal constraints and policy substitutions sustain uncertainty even when headline measures change.

The front-loading mechanism matters because it alters the timing of realized activity. When firms advance imports ahead of higher tariff schedules, measured trade volumes and goods output pull demand from future quarters into the present. World Bank communication around Global Economic Prospects links this pattern to the 2025 trade impulse and expects that tailwind to fade, leaving a softer goods backdrop in 2026. The key step involves separating genuine final-demand strength from calendar-shifted flows. A model that treats the 2025 goods surge as durable can overstate cyclical beta and underweight defensive carry. A model that recognizes the inventory-and-tariff cycle can frame 2026 as a transition from goods momentum toward services endurance, with factor leadership that rotates across value chains and across regions in different phases.

Trade policy shifts influence external balances and currency pressures through re-routing of goods flows. When tariff schedules change, exporters redirect volume toward markets with lower barriers, and importers shift sourcing toward jurisdictions that preserve margin. Reporting on tariff developments highlights a setting where tariff policy remains a durable feature of the landscape, and it points to diversion of Chinese exports toward Europe and other regions as part of the adjustment. ECB analysis of China’s widening goods trade surplus since the pandemic provides a complementary lens on how global goods trade patterns evolve amid geopolitical tension and shifts in trade policy. These forces matter for quant strategies because external balances drive medium-horizon FX drift, while trade shocks drive short-horizon FX volatility, and the interaction between the two shapes carry-to-trend transitions in currency markets.

ECB analysis describes services activity that continues to expand while industry remains muted, using PMI measures as a signal of a recovery that runs through services first. This pattern appears in survey releases that show services readings holding above the expansion threshold while manufacturing moves toward stabilization. PMIs in Germany and France captures cross-country dispersion inside the euro area, with services carrying the composite in Germany and weaker demand weighing on French activity. These survey dynamics matter for trading research because PMI surprises map into short-horizon moves in rates, cyclicals, and currencies, while the level and composition of the PMI can guide medium-horizon positioning in curve steepeners, credit carry, and sector rotation.

PMI data provides a bridge between growth narrative and microstructure. A services-led composite often coincides with sticky wage dynamics, steadier cash-flow expectations in domestic-oriented sectors, and a tighter link between labor-market prints and rate pricing. A manufacturing stabilization phase tends to lift sensitivity to new orders, inventories, and global demand proxies. In systematic research, this implies that the same headline PMI level can support different trade decisions depending on whether services or manufacturing drives the move. It implies that signal design benefits from separating diffusion indexes by sector and by subcomponents such as new orders and employment, then aligning those readings with the asset classes that transmit the information with the least delay. The ECB’s characterization of a dual-speed recovery gives a qualitative anchor for this decomposition.

Central-bank decisions in the opening months of 2026 translate these macro signals into a policy-rate path that markets price each day. The Federal Reserve’s statement kept the target range unchanged and emphasized assessment of incoming data and the balance of risks. The ECB’s decision kept key rates unchanged and pointed to resilience supported by labor-market conditions and the rollout of public spending on defense and infrastructure. The Bank of England’s minutes show an internal split, with a narrow majority favoring an unchanged policy rate and a sizable minority favoring a cut. These decisions describe a global reaction function that prices a wide corridor for cuts, with emphasis on inflation evidence, services pricing, and the durability of disinflation.

Policy communication matters alongside the decision itself because it governs how markets map the same data to different paths for rates. A steady policy rate with language that emphasizes confidence in convergence to target supports carry regimes, where realized volatility remains low relative to the level of yields. A steady policy rate with language that emphasizes uncertainty around inflation drivers supports volatility regimes, where rates option markets price a wider distribution of paths. The ECB’s communication points to medium-term stabilization of inflation at target while describing a challenging global environment that includes policy uncertainty. This combination encourages a research posture that treats rate cuts as a function of inflation composition and of the persistence of growth in services, rather than as a function of headline growth alone.

Inflation dynamics form the hinge between macro and policy caution, and energy sits at the center of that hinge at the start of 2026. ECB analysis attributes the expected decline in headline inflation to base effects linked to energy prices, and separate ECB reporting shows a large negative reading for energy inflation alongside softer underlying inflation measures. Oil markets move in a direction that reduces the disinflation impulse that came from earlier price declines. As crude prices rise and year-over-year declines narrow, which pushes oil’s impact toward inflation pressure and raises the hurdle for policy easing. World Bank commodity data shows a rise in the energy price index driven by moves in natural gas and crude oil, reinforcing the picture of an energy complex that feeds into near-term inflation prints through transport and input costs. IEA market reporting adds a supply-side layer, with a large supply response projected that interacts with geopolitics and weather to drive volatility around the inflation path.

Energy-driven inflation pressure interacts with tariffs through pass-through channels that vary across economies and across consumption baskets. When tariff schedules rise, tradable-goods prices can reflect a combination of higher landed cost and of shifts in sourcing. When energy prices rise, freight and input costs amplify that pass-through. The World Bank’s emphasis on trade policy shifts and uncertainty outlook highlights the macro setting where these channels matter for inflation forecasts and for policy paths. The practical implication involves treating inflation as a composite of distinct drivers rather than as a single series. Energy, goods import prices, and services inflation each respond to different shocks and each maps into different parts of the curve and different equity sectors.

The euro-area inflation channel adds a second force that pulls in the opposite direction through goods prices. The ECB policymaker Fabio Panetta on the role of Chinese imports in the sharper-than-forecast drop in euro area inflation, linking disinflation in exposed goods categories to price and volume dynamics in import competition. An ECB Economic Bulletin focus box provides context by describing the rise in Chinese competitive pressure across export markets and its implications for European firms. This combination produces a cross-category split where goods disinflation can coexist with services stickiness, which keeps the inflation profile two-sided even when headline prints move below target for a time. This split carries direct implications for quant models that use inflation surprises as a single factor, since the market response can differ when the surprise comes from tradable goods versus domestic services.

Energy uncertainty and supply disruptions remain the second half of the euro-area inflation story. ECB communication frames geopolitical tensions as a source of two-sided inflation risks through energy markets, confidence, and business investment. Panetta’s comments point to energy instability and supply-chain disruptions as upside risks alongside the disinflationary pressure from import competition. This configuration supports an inflation process that behaves like a mixture of mean reversion in goods and jump risk in energy-sensitive components. That structure supports stress testing portfolios against energy spikes and supply-chain events, while mapping the influence of import prices on goods-sensitive sectors and on inflation-linked instruments.

Fiscal capacity shapes the medium-horizon risk premium landscape because it governs sovereign issuance, refinancing calendars, and the credibility of backstops. As refinancing needs rise and central bank balance sheets shrink through runoff, the marginal buyer of duration shifts toward price-sensitive private investors, which places emphasis on term premia as a driver of long yields. An ECB speech on safe asset abundance argues that large fiscal deficits and balance-sheet normalization reduce the safety and liquidity premia embedded in government bonds, compressing the convenience yield that investors pay for scarce safe assets and changing the forces that shaped real long-term rates during the prior decade.

This fiscal and balance-sheet backdrop matters for pricing in a way that reaches beyond government bonds. A rise in term premia raises discount rates that feed into equity valuations, increases the hurdle for private credit origination, and shifts the relative appeal of carry strategies versus volatility strategies. It places a premium on liquidity, since price-sensitive investors demand compensation for balance-sheet use during periods of heavy issuance. This argues for explicit monitoring of issuance calendars, auction tails, and measures of market depth in core sovereign futures, because those observables act as the proximate mechanism through which a fiscal narrative becomes a PnL driver. The IMF emphasis on public debt dynamics and the ECB focus on convenience yield provide the macro foundation for building those monitors.

Financial stability adds another layer because valuation and liquidity conditions shape the transmission mechanism from macro shocks into asset prices. The IMF’s October 2025 Global Financial Stability Report frames financial stability risks as elevated amid stretched asset valuations, sovereign bond market pressures, and the rising influence of nonbank financial institutions, and it highlights structural shifts in foreign exchange and emerging market bond markets that reshape both risk and resilience. This matters for systematic strategies that lean on historical diversification patterns, since structural market plumbing determines the speed and shape of drawdowns and the path of margin dynamics. The IMF’s discussion of stock-bond diversification complements this framing by linking bond-market dynamics to the absorption of supply by price-sensitive investors under balance-sheet runoff.

Major geopolitical flashpoints

Read more

Table of contents:

1. Introduction.

2. What is web scraping in a quantitative context?

3. Where does it come from?

4. How web scraping actually works.

5. Risks and limitations of the protocol.

6. Screener network topography.

7. Scraping fund positions.

Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.

Introduction

In fund research, the input that matters most is simple: what stocks are inside the funds right now, and in what weight. Without that look-through layer, fund momentum, category rotation, or risk exposure becomes a label-driven proxy.

We are going to construct and execute a systematic fund scraping operation. The explicit objective of this architecture is to expose what underlying stocks these otherwise opaque mutual funds and ETFs are holding.

If you intend to calculate true portfolio variance, isolate specific factor exposures, or detect hidden beta overlap across multiple assets, you must possess the exact equity constituents and their precise percentage weightings for every fund in your cross-section. Without ticker-level resolution, you might allocate capital to two distinct Diversified Value funds that both hold massive, highly correlated positions in the exact same mega-cap technology stock.

So, we will engineer a deterministic data pipelineto get a clean, structured JSON. The architecture we are deploying will execute in distinct, logical phases:

1. We will intercept the provider’s APIs to extract the core fund metadata and historical return vectors.

2. We will utilize concurrent, threaded I/O operations to query the global holdings API.

3. We will map this extracted data into memory-contiguous C-arrays.

4. We will apply algorithmic deduplication using tokenized lexical analysis and tuple signatures to eliminate share class multicollinearity.

The final output of this operation will be a sanitized, lightweight matrix of unique portfolios, ready for immediate ingestion into our engine.

What is web scraping in a quantitative context?

To understand the architecture we build for trading, we must first define what web scraping is at a fundamental, computational level. In the broader software engineering industry, web scraping is often defined as the automated gathering of information from the internet. In quantitative finance, we require a technical definition.

Web scraping is the automated, programmatic execution of HTTP network protocols to request, receive, and extract specific data structures from external servers, translating unstructured or semi-structured web payloads into strict, localized, memory-contiguous matrices for immediate algorithmic ingestion.

To know more about scraping check that pdf:

Web Scrapping: The basics explained

236KB ∙ PDF file

Download

Download

When a quantitative researcher builds a cross-sectional momentum strategy or a fundamental mean-reversion model, the model requires historical price vectors, dividend yields, expense ratios, constituent weights, etc. Historically, this data was locked behind expensive, proprietary data terminals operated by a few institutional vendors.

Web scraping bypasses this vendor lock-in. It allows a trading desk to aggregate fragmented public data from asset managers, regulatory filings, and financial portals directly into a local array or a structured SQL database. It is the direct acquisition of the raw material required for alpha generation.

Furthermore, standard vendor data is often commoditized. If every quant receives the exact same fundamental dataset at the exact same millisecond via an identical API feed, the alpha decays. Constructing custom extraction architecture allows a quantitative team to ingest alternative data—such as retail sentiment from localized forums, unstructured XBRL data from SEC 10-Q filings, or dynamically updating ETF composition lists directly from the issuer’s backend.

Subscribe now

Where does it come from?

The methodology of data extraction has evolved in parallel with the evolution of web architecture itself. We can trace this lineage through distinct technological eras, driven by the need to parse complex data delivery mechanisms. Understanding this history is critical to understanding why modern quantitative infrastructure rejects certain parsing libraries.

Era 1: Static transfer and regular expressions

In the early days of the internet, web pages were static HTML documents stored directly on server hard drives. When a client sent an HTTP GET request, the server responded with the exact text file. Scraping in this era involved basic command-line tools executing standard network fetches.

The data was extracted using Regular Expressions (Regex). Regex operates as a finite state machine that matches specific string patterns. If a researcher wanted a stock price, they wrote a strictly defined string-matching algorithm to locate a sequence of numbers immediately following a specific HTML text block.

This was efficient computationally. A regex search over a 10-megabyte text file resolves in milliseconds. However, it was incredibly brittle from an operational standpoint. Consider the following HTML string: <td class="price">150.25</td>. A regex designed to capture the float inside the tags (r'class="price">(\d+\.\d+)<') works fine. But if the webmaster updates the website and injects a bold tag for visual styling—<td class="price"><b>150.25</b></td>—the regex logic fails. It returns a null value. In a trading pipeline, a null value propagates through the covariance matrix, zeroes out the optimization weights, and halts the entire trading execution.

Era 2: The Document Object Model and structural parsing

As web pages evolved into generated templates mapping to backend databases, relying on raw string matching became impossible. The industry shifted to structural parsing. Browsers render HTML into a hierarchical tree structure known as the Document Object Model (DOM).

Scrapers evolved to replicate this exact rendering process in memory. Python libraries, such as BeautifulSoup utilizing lxml parsers, were developed to download the HTML string, parse it into an internal tree data structure, and allow engineers to search the tree using XPath or CSS selectors. We stopped searching for text strings and started searching for specific structural nodes, such as the third cell of the second row within a specific table ID.

While more robust than Regex, structural parsing introduces severe computational overhead. Converting a 5-megabyte HTML string into a fully traversable DOM tree requires significant CPU cycles and RAM allocation. The search algorithm must execute O(N) traversal operations across thousands of nested nodes. When running high-frequency data ingestion across thousands of financial instruments, the CPU time spent building these DOM trees severely limits the concurrency capacity of the extraction server.

Era 3: Single Page Applications and JavaScript rendering

The critical inflection point occurred when web development shifted to reactive JavaScript frameworks (React, Angular, Vue). The server no longer returned HTML containing the financial data. Instead, the server returned an empty HTML shell and a massive JavaScript file. The client’s browser was expected to execute the JavaScript, which would then dynamically fetch the data and build the DOM locally on the user’s machine.

Standard HTML parsers broke; they would download the empty shell, parse it, and find zero data. To combat this, scrapers evolved into headless browsers. Infrastructure had to spin up entirely automated, invisible instances of Chromium in the background, load the page, wait for the V8 JavaScript engine to execute the rendering logic, wait for the network requests to resolve, and then extract the data from the rendered DOM.

This was disastrous. Running a single headless browser instance requires hundreds of megabytes of RAM. If a quantitative pipeline requires 100 concurrent workers to update the daily pricing matrix before the market opens, the server must allocate over 50 gigabytes of RAM strictly for browser overhead. This is inefficient and unscalable for certain types of trading systems.

Era 4: API interception

This brings us to the modern quantitative standard. Because Single Page Applications must retrieve their dynamic data from somewhere, they utilize Application Programming Interfaces (APIs)—typically REST or GraphQL architectures.

Instead of rendering a heavy headless browser, we utilize network monitoring tools to observe the traffic generated by the web application. We identify the exact API endpoint the JavaScript is querying, and we replicate that precise HTTP request from our Python architecture. We bypass the HTML entirely. We bypass the JavaScript rendering engine. We request the raw, serialized JSON data directly from the provider’s backend database. This is the optimal state of data extraction: maximum fidelity, minimum computational overhead.

How web scraping actually works

To extract data, you must understand the exact sequence of network events that occur during a programmatic request. Web scraping is the programmatic automation of the standard client-server architecture, executing at speeds unachievable by human interaction.

Step 1: The network handshake and request formulation

The process begins when our Python architecture instantiates a TCP connection. We formulate a strict HTTP request, this request consists of a specific method (GET for retrieving data, POST for submitting query parameters), the target URL, and a dictionary of Headers.

Headers are critical, they dictate the exact rules of engagement with the target server. We set the User-Agent header to declare what type of client we are. By default, Python networking libraries broadcast their standard identity (e.g., python-requests/2.31.0). Modern Web Application Firewalls flag this string and drop the TCP connection. We must inject standard browser headers to bypass basic heuristic checks.

Furthermore, we must configure advanced headers such as Accept-Encoding: gzip, deflate, br to inform the server that we can handle compressed byte streams, reducing network transit time.

Modern security also involves TLS (Transport Layer Security) fingerprinting. Servers analyze the specific cryptographic cipher suites our client proposes during the initial connection setup (the Client Hello packet). Standard Python networking libraries propose a different sequence of ciphers than a standard Google Chrome browser. If the server detects a mismatch between the User-Agent claiming to be Chrome and a TLS fingerprint matching Python, it blocks the request. Advanced quantitative architecture utilizes specialized network adapters to mimic exact browser TLS signatures, guaranteeing connection acceptance.

Before we even introduce anything, let’s start here with our practical example.

import requests

url = "https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener?page=1&pageSize=50&sortOrder=ReturnM120%20desc&outputType=json&version=1&languageId=es-ES&currencyId=EUR&universeIds=FOESP%24%24ALL&securityDataPoints=SecId%7CName%7CPriceCurrency%7CTenforeId%7CLegalName%7CClosePrice%7CYield_M12%7CCategoryName%7CAnalystRatingScale%7CStarRatingM255%7CQuantitativeRating%7CSustainabilityRank%7CReturnD1%7CReturnW1%7CReturnM1%7CReturnM3%7CReturnM6%7CReturnM0%7CReturnM12%7CReturnM36%7CReturnM60%7CReturnM120%7CFeeLevel%7CManagerTenure%7CMaxDeferredLoad%7CInitialPurchase%7CFundTNAV%7CEquityStyleBox%7CBondStyleBox%7CAverageMarketCapital%7CAverageCreditQualityCode%7CEffectiveDuration%7CMorningstarRiskM255%7CAlphaM36%7CBetaM36%7CR2M36%7CStandardDeviationM36%7CSharpeM36%7CTrackRecordExtension&filters=&term=&subUniverseId="

r = requests.get(url, timeout=20)
print("status:", r.status_code)
print("content-type:", r.headers.get("content-type"))
print("content-encoding:", r.headers.get("content-encoding"))
print("head:", r.text[:200])

Step 2: Server-side resolution and response

Upon receiving the formulated request, the target server evaluates it against its internal security protocols. It checks for IP reputation, rate limit thresholds (Token Bucket algorithms), and geographic blocking rules. If the request passes these validation checks, the server queries its internal SQL or NoSQL database, formats the result into the requested format, and transmits the payload back to our server over the open connection.

The server prefaces the payload with an HTTP Status Code. We evaluate this code before reading the data buffer. A 200 OK indicates absolute success. A 403 Forbidden indicates our IP or TLS signature was blocked by the WAF. A 429 Too Many Requests indicates we have exceeded the permissible concurrency limit. When a 429 is encountered, our infrastructure immediately halts execution on that specific thread and initiates a deterministic exponential backoff algorithm, calculating a precise wait time before re-attempting the connection.

status: 200
content-type: application/json
content-encoding: gzip
head: {"total": 49441,"page": 1,"pageSize": 50,"rows": [{"SecId": "F00000S8RT","Name": "Multipartner Konwave Gold Equity C USD","PriceCurrency": "USD","TenforeId": "52.8.LU1001014080","LegalName": "Multipar

Step 3: Payload deserialization and matrix mapping

Once the payload bytes are received locally via the network interface card, they must be transformed into operational data. In modern scraping (API interception), the response header Content-Type is typically application/json. The data arrives as a serialized, UTF-8 encoded byte string.

We execute a high-speed deserialization function. In environments prioritizing microseconds, standard libraries are replaced by C-compiled alternatives like orjson or ujson that map JSON strings to memory structures faster. The deserializer parses the string and maps the key-value pairs into native Python dictionaries and lists.

From this state, we execute our extraction logic. We iterate through the arrays, filtering out the specific financial metrics we require, and allocating them into contiguous memory blocks. We actively cast data types at this stage—ensuring strings representing numbers are cast directly to float64 objects in a structured array, preparing the data for immediate ingestion into linear algebra operations.

To demonstrate the evolutionary mechanics, observe the explicit difference in code execution between legacy DOM parsing and modern high-speed JSON deserialization:

import json
from bs4 import BeautifulSoup
import orjson
import numpy as np

# Legacy Mechanics: Structural DOM Parsing (Era 2) 
# The server returns a heavy string of markup logic mixed with data.
html_payload = """
<div class="financial-table">
    <div class="row">
        <span class="ticker">AAPL</span>
        <span class="price">150.25</span>
    </div>
</div>
"""
# We must allocate significant memory to build the entire tree structure just to find one float.
# This requires CPU cycles to parse the entire string, handle malformed tags, and build node relationships.
tree = BeautifulSoup(html_payload, "html.parser")
legacy_price = float(tree.find("span", class_="price").text)

# Modern Mechanics: API Deserialization (Era 4) 
# The server returns strictly serialized data with zero HTML rendering logic.
json_payload_bytes = b'{"data": [{"ticker": "AAPL", "price": 150.25}]}'

# We execute a direct memory mapping using a C-optimized parser (orjson).
# This bypasses standard Python object creation overhead where possible.
parsed_data = orjson.loads(json_payload_bytes)

# We extract the specific float and immediately allocate it into a NumPy array type.
modern_price = np.float64(parsed_data["data"][0]["price"])

The difference is structural and computational. Legacy scraping reads documents; modern scraping queries databases. By understanding the lineage and the strict network mechanics, we engineer scraping pipelines that don’t simulate human browsing. Instead, they operate as direct, high-speed, database-to-database communication protocols executing over the public internet.

Risks and limitations of the protocol

The execution of thousands of sequential HTTP requests introduces specific, quantifiable, and systemic risks to the trading infrastructure. These risks are guaranteed events in any large-scale network operation. They manifest as missing data rows, unhandled exceptions, TCP timeouts, and truncated JSON payloads, which directly corrupt the backtesting matrix.

The primary operational risks are categorized by standard HTTP response codes. We anticipate HTTP 429 (Too Many Requests) errors when our extraction loops exceed the provider’s token bucket algorithm limits. We expect TCP connection timeouts when network routing degrades. We will encounter 502 Bad Gateway responses during dynamic load balancer reallocations, and 503 Service Unavailable responses when the backend database cluster is executing routine locking operations or snapshots.

To mitigate these risks, we construct a session adapter using the urllib3 library. We implement a deterministic exponential backoff algorithm. When a 429 or 5xx status code is returned, the system pauses before executing a retry, reducing the concurrency pressure on the provider’s server. We target the 429, 500, 502, 503, and 504 codes. We exclude 400 (Bad Request) and 404 (Not Found) codes, as those represent structural errors in our query parameters that a retry will never resolve.

Here is the implementation of the retry adapter used in our pipeline:

import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def _make_retry_adapter() -> HTTPAdapter:
    retry = Retry(
        total=6,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=("GET",),
        raise_on_status=False,
    )
    return HTTPAdapter(max_retries=retry, pool_connections=50, pool_maxsize=50)

def _lt_session() -> requests.Session:
    # We precisely replicate standard browser headers to bypass basic WAF heuristic checks.
    LT_HEADERS = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "es-ES,es;q=0.9,en;q=0.8",
        "Referer": "[https://lt.morningstar.com/](https://lt.morningstar.com/)",
    }
    s = requests.Session()
    s.headers.update(LT_HEADERS)
    adapter = _make_retry_adapter()
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    return s

Notice the specific performance parameters initialized in the adapter: pool_connections=50 and pool_maxsize=50. This keeps 50 distinct TCP connections open in the background, bound to the Session object. By utilizing the HTTP Keep-Alive header within the requests.Session architecture, we completely bypass the severe latency introduced by repeated Transport Layer Security handshakes.

Every new HTTPS request requires DNS resolution, a 3-way TCP handshake (SYN, SYN-ACK, ACK), and an expensive TLS cryptographic key exchange. This sequence requires 150 to 300 milliseconds of network overhead before a single byte of actual data is transmitted. This is the protocol tax. If we initiate a new session for 5,000 distinct requests, we incur 25 minutes of pure dead time negotiating connections. By pooling connections and persisting them, we reduce network overhead to the strict physical transfer time of the payload itself, dropping the marginal connection time to zero.

To visualize the progression of the exponential backoff risk mitigation, consider the following plot. It models the exact delay duration sequence based on the specified backoff_factor, demonstrating how the system slows down to prevent triggering a permanent IP ban.

This is the precise moment the entire architecture upgrades. We stop looking at the rendered web page, and we start interrogating the network layer.

By opening the developer tools within a controlled Chromium environment and monitoring the Fetch/XHR network traffic generated by the web application during standard user interaction, we discover and intercept the underlying API. We isolate the exact network request executing the database search. We identify the explicit endpoint serving the raw data to the frontend UI:

https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener.

This endpoint doesn’t return complex, nested HTML documents. It accepts defined query parameters that dictate the SQL-equivalent operations on the backend: pagination offsets, sorting directives, universe limiters, and projection columns (securityDataPoints). In return, it provides a raw JSON payload.

By interfacing with this REST API, the extraction process transitions from a heuristic, error-prone text-parsing operation into a deterministic database query. We send a formalized HTTP GET request, and we receive a serialized data structure that maps into our operational memory. A float value is transmitted as a float. A string is transmitted as a string. There is no string-manipulation required to strip percentage signs or currency symbols, which eliminates an entire class of data engineering bugs.

This single event renders all traditional web scraping techniques—BeautifulSoup HTML parsing, XPath selection, and resource-heavy headless browsers like Selenium or Playwright— obsolete within our infrastructure.

Screener network topography

Having established the foundational infrastructure and intercepted the correct REST API endpoint, we now face the challenge of extracting the financial data. The provider’s backend database contains tens of thousands of mutual funds, exchange-traded funds, and institutional share classes. This dataset is vastly larger than the maximum permitted payload size for a single HTTP response.

The endpoint enforces a strict pageSize limit. This is a direct mechanism to prevent memory exhaustion on their internal database cluster. When a client requests data, the provider’s server must execute a SQL or NoSQL query, allocate a continuous block of RAM to hold the result set, serialize that data into a JSON string, and transmit it over the network interface. Requesting 10,000 rows creates a massive serialization overhead. To bypass these hard limits, we define our extraction parameter as pageSize=50.

Extracting the complete cross-sectional universe requires mapping the network topography and iterating over a paginated endpoint recursively. We must calculate the required number of requests based on the total record count returned in the initial response. We can’t hardcode the total page count, as the universe size fluctuates daily due to fund liquidations, mergers, and new product launches.

The algorithmic execution proceeds sequentially for the initialization phase. We fetch page 1. We parse the initial JSON payload to read the total integer field, which dictates the absolute size of the dataset for that specific query. We calculate the ceiling of the division of total by pageSize (math.ceil(total / page_size)). This yields the exact integer value of total pages required to complete the extraction. If the total is 0, we default to 1 to avoid ZeroDivisionError exceptions in downstream processing logic.

Besides, we pass specific query parameters in the HTTP request: universeIds=FOESP$$ALL specifies the precise fund universe routing key on the provider’s backend. We inject the parameter securityDataPoints=SecId|LegalName|CategoryName|ReturnM120 to isolate only the necessary relational columns.

We request the SecId, the LegalName, the CategoryName, and the ReturnM120 (the 120-month historical return, which is the foundational variable for long-term quantitative momentum analysis).

If we omit the securityDataPoints parameter, the API defaults to returning every available column—over 100 data points including tracking error, manager tenure, and daily standard deviation. This inflates the JSON payload from 50 kilobytes to over 5 megabytes per page. Requesting unnecessary data points saturates network bandwidth, increases TLS decryption time, and degrades the JSON parsing speed. We limit the payload to our exact algorithmic requirements.

import math
import numpy as np
import requests

LT_BASE = "[https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener](https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener)"
SECURITY_POINTS = "SecId|LegalName|CategoryName|ReturnM120"

def fetch_lt_page(session: requests.Session, page: int, page_size: int) -> dict:
    params = {
        "page": int(page),
        "pageSize": int(page_size),
        "sortOrder": "ReturnM120 desc",
        "outputType": "json",
        "version": 1,
        "languageId": "es-ES",
        "currencyId": "EUR",
        "universeIds": "FOESP$$ALL",
        "securityDataPoints": SECURITY_POINTS,
        "filters": "",
        "term": "",
        "subUniverseId": "",
    }
    # Notice the exact tuple for the timeout: (4, 20)
    # 4 seconds allocated for the TCP SYN/ACK connection handshake.
    # 20 seconds allocated for the server to process the query and return the first byte.
    r = session.get(LT_BASE, params=params, timeout=(4, 20))
    ct = (r.headers.get("content-type") or "").lower()
    if r.status_code != 200 or "json" not in ct or not r.content:
        raise RuntimeError(f"LT bad response: {r.status_code} ct={ct} head={r.text[:200]!r}")
    return r.json()

def _normalize_pages(pages, total_pages: int):
    # This logic dictates the strict operational scope of the extraction matrix.
    if pages is None:
        # None implies a full universe extraction. WARNING: Huge data pull spanning gigabytes of memory.
        return range(1, total_pages + 1)
    if isinstance(pages, (int, np.integer)):
        # Integer implies pulling the top N pages strictly based on the sortOrder parameter.
        n = int(pages)
        return range(1, min(max(n, 0), total_pages) + 1)
    # List implies pulling specific, explicit page indices, useful for error-handling retry loops.
    # We apply a set comprehension to mathematically eliminate duplicate page requests.
    lst = sorted({int(p) for p in pages if 1 <= int(p) <= total_pages})
    return lst

The pagination handler _normalize_pages is engineered to accept three distinct input states, providing the system with deterministic flexibility. If a researcher is running a lightweight daily update to assess the top decile of momentum performers, they pass an integer pages=10 to scan only the top 500 funds. If the infrastructure team is bootstrapping the initial quantitative database from a blank state, they pass None to fetch the absolute entirety of the universe. If a previous extraction loop dropped specific network packets, the error handler passes a list pages=[14, 22, 105] to specifically target the missing vectors. The use of a set comprehension {int(p) ...} guarantees that duplicate page indices are stripped before execution, preventing redundant network I/O.

We append these scalar values into localized lists. Python lists operate with O(1) amortized append time complexity because they over-allocate memory buffers under the hood. We can’t initialize the final, rigid structured array at this specific stage because we don’t yet know the exact valid row count; we filter out invalid JSON objects—such as funds missing a critical SecId—during the loop.

The primary obstacle at the screener stage is managing the sequential nature of the pagination discovery. Because we don’t possess the total integer until the very first request resolves and the JSON is parsed, the initial discovery phase is linear. However, once the total_pages parameter is acquired, the subsequent page requests become entirely stateless and independent. This sets the theoretical stage for future parallelization. Yet, for this specific screener layer, executing sequential HTTP requests over an established, persistent TLS connection pool is efficient, completing the initial metadata extraction in a matter of a few seconds, preparing the matrix for the much heavier, threaded Global API processing phase.

Send me a copy

Scraping fund positions

The screener endpoint provides the fund-level metadata, consisting of basic identifying parameters and historical return vectors. However, a fund’s surface-level metadata is insufficient for risk management and alpha generation. To construct a look-through portfolio, assess true factor exposure, or run Barra risk decomposition models, we require the exact equity holdings inside each fund. We need the underlying constituents, and we need them weighted.

This data requirement introduces a separate, more complex endpoint into our architecture: the Global API (api-global.morningstar.com). Unlike the public screener interface, this endpoint demands an explicit authentication mechanism. We must inject an apikey (it’s free) header into every request.

# 2) Direct holdings 
APIKEY = "HERE YOUR API KEY"

Furthermore, the endpoint requires the specific SecId extracted from the first phase to be formatted directly into the URL path.

The primary architectural obstacle is the sheer volume of network requests. The screener API supports bulk pagination; the Global API doesn’t. If the screener returns an array of 5,000 distinct funds, we must execute 5,000 separate, individual HTTP GET requests to the Global API to retrieve the constituent holdings for each individual fund.

A standard, sequential execution loop operates at approximately 0.5 seconds per request. This delay is a function of the physical network latency—the round-trip time between our trading servers and the provider’s data center, plus the provider’s internal database query execution time. Executing 5,000 requests sequentially requires 2,500 seconds, equating to over 40 minutes of pure execution time. This is unacceptable for a daily production pipeline that must process data, run optimizations, and generate execution orders before the market open.

We resolve this processing bottleneck through concurrent execution. We implement a ThreadPoolExecutor from the concurrent.futures module. Because HTTP network requests are I/O-bound tasks—meaning the CPU sits idle while waiting for the network interface card to receive the inbound packet from the remote server—the GIL doesn’t inhibit parallel execution in this specific scenario. The interpreter releases the GIL during standard I/O blocking operations. Therefore, we can launch 20 concurrent threads to process the requests without migrating to a complex multiprocessing architecture.

We chose thread pooling over an asynchronous event loop (like asyncio) because our existing network infrastructure relies on the requests library and custom urllib3 retry adapters. Threaded execution provides the exact same I/O concurrency benefits without requiring a complete rewrite of the underlying synchronous network stack.

import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
import numpy as np

APIKEY = "HERE YOUR API KEY"
MS_BASE = "[https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2](https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2)"
MS_HEADERS = {
    "apikey": APIKEY,
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
    "Accept": "application/json, text/plain, */*",
}

_thread_local = threading.local()

def _ms_session() -> requests.Session:
    # We must ensure strict thread safety across concurrent network operations.
    s = getattr(_thread_local, "session", None)
    if s is None:
        s = requests.Session()
        s.headers.update(MS_HEADERS)
        # Assuming _make_retry_adapter is defined as in Part 1 for exponential backoff
        # s.mount("https://", _make_retry_adapter())
        _thread_local.session = s
    return s

def fetch_holdings_us_topN(secid: str, country: str = "United States", top: int = 10):
    s = _ms_session()
    url = f"{MS_BASE}/{secid}/data"
    params = {
        "clientId": "MDC",
        "version": "4.71.0",
        "premiumNum": 10000,
        "freeNum": 10000,
    }
    r = s.get(url, params=params, timeout=(4, 25))
    if r.status_code != 200 or not r.content:
        return []
    
    data = r.json()
    page = (data.get("equityHoldingPage") or {})
    hlist = (page.get("holdingList") or [])
    
    out = []
    for h in hlist:
        c = h.get("country") or h.get("countryName")
        if c == country:
            t = h.get("ticker") or h.get("tradingSymbol")
            if t:
                out.append(t)
                if len(out) >= top:
                    break
    return out

def attach_holdings_all(secids, max_workers: int = 20, country: str = "United States", top: int = 10):
    holdings = [None] * len(secids)
    with ThreadPoolExecutor(max_workers=max_workers) as ex:
        futs = {ex.submit(fetch_holdings_us_topN, sid, country, top): i for i, sid in enumerate(secids)}
        for fut in as_completed(futs):
            i = futs[fut]
            try:
                holdings[i] = fut.result()
            except Exception:
                # We enforce strict failure handling to maintain array dimensions.
                holdings[i] = []
    return np.asarray(holdings, dtype=object)

Observe the use of threading.local(). This is an essential architectural detail that prevents systemic failure. We instantiate a unique requests.Session() object for every single active thread. When multiple concurrent connections interact with a single, shared urllib3 connection pool, race conditions occur at the socket level, leading to corrupted TCP streams and dropped connections. By isolating the session object to local thread memory storage, we guarantee that each thread maintains its own independent pool of Keep-Alive HTTP sockets. This ensures deterministic network execution.

We limit max_workers=20 as calculated threshold based on the provider’s implicit rate limits. Exceeding this thread count doesn’t yield linear performance gains; instead, it triggers the provider’s Web Application Firewall, which responds with HTTP 429 status codes, forcing our exponential backoff adapter into extended sleep cycles. Twenty workers provide the optimal equilibrium between maximum extraction speed and minimal server-side rejection.

The next part is to map the extracted data into a NumPy structured array. A structured array allocates a single, contiguous block of memory in C. We define exact byte boundaries for each field in the matrix. When the CPU executes operations on this array, it loads sequential memory blocks directly into the L1/L2 cache, executing instructions with zero pointer-chasing overhead.

We allocate a 16-character unicode string (U16) for the SecId, a 256-character unicode string (U256) for the Fund name, an 8-byte float (f8) for the return metric, and an object pointer (O) for the variable-length list of constituent stock holdings.

# Execution assembly demonstrating the structured array initialization
import numpy as np

# Assuming secids, names, rets, and stocks are already populated lists/arrays from the extraction phase
# We allocate exactly the required memory footprint before population.
arr = np.empty(len(secids), dtype=[("Id", "U16"), ("Fund", "U256"), ("ReturnM120", "f8"), ("Stocks", "O")])
arr["Id"] = np.asarray(secids, dtype="U16")
arr["Fund"] = np.asarray(names, dtype="U256")
arr["ReturnM120"] = rets
arr["Stocks"] = stocks

We acknowledge the use of the object pointer ("O") for the stocks column breaks strict C-contiguity for the holding string arrays themselves. However, this is computationally acceptable because our heavy numerical operations—the sorting and ranking logic—operate on the f8 float array, which remains contiguous and hardware-optimized.

Furthermore, we must handle missing data points at the numerical level. Web scraped data contains holes; some funds lack a 10-year track record. We temporarily replace NaN values with negative infinity (-np.inf) prior to sorting. Standard sorting functions in C don’t possess financial logic. Mathematical comparisons against NaN fail silently (NaN > 5 evaluates to False, NaN < 5 evaluates to False). If we don’t execute this specific substitution, NaN values sort unpredictably, injecting null pointers into the highest momentum deciles and breaking the downstream portfolio construction models. This substitution ensures that funds lacking complete return histories are forced to the absolute bottom of the generated matrix.

def sort_by_return_desc(arr: np.ndarray) -> np.ndarray:
    if arr.size == 0:
        return arr
    r = arr["ReturnM120"]
    
    # Isolate NaN values and push them to the lowest possible finite float boundary (-inf).
    # This prevents incomplete track records from ranking highly in momentum strategies.
    r2 = np.where(np.isfinite(r), r, -np.inf)
    
    # Execute the C-level sort on the substituted float array, and reverse it ([::-1]) for descending order.
    idx = np.argsort(r2)[::-1]
    
    # Return the newly aligned, completely contiguous memory view.
    return arr[idx]

The final, most complex important barrier in systematic fund scraping is the share class multiplicity problem. Asset managers construct a single underlying portfolio of equities, but they distribute that identical portfolio across dozens of distinct administrative wrappers. They do this to target different regulatory jurisdictions, implement varying management fee structures, and separate retail investors from institutional capital. Consequently, you will see Institutional classes, Retail classes, Accumulating (Acc) classes, Distributing (Inc) classes, GBP hedged variants, and EUR unhedged variants—all holding the exact same underlying assets in the exact same proportions.

A standard database unique constraint operating on the SecId is insufficient here because each distinct share class possesses a unique Morningstar SecId. If we include all share classes in a model without aggressive deduplication, we introduce multicollinearity into the output.

We must reduce the dataset to a single representative vector per underlying portfolio. We achieve this by cross-referencing two distinct topographical vectors: the exact list signature of the underlying top holdings, and a strict lexical analysis of the fund’s string name.

First, we define a tokenization function to strip punctuation and structural noise from the fund name strings. We use regular expressions to explicitly remove parentheses and brackets, as share classes are frequently denoted by suffixes like “(USD)” or “[Acc]”. Stripping these allows us to isolate the core lexical identity of the fund. We then define an algorithmic function to calculate the strict length of the common prefix between two tokenized arrays.

def _name_tokens(name: str) -> list[str]:
    s = (name or "").strip().lower()
    # Normalize common typographic variations in fund prospectuses
    s = s.replace("–", "-").replace("—", "-")
    # Strip isolating punctuation typically used for share class designations
    s = re.sub(r"[()\[\],]", " ", s)
    return [t for t in s.split() if t]

def _common_prefix_len(a: list[str], b: list[str]) -> int:
    n = min(len(a), len(b))
    k = 0
    # Calculate exact sequential token matches from index 0
    while k < n and a[k] == b[k]:
        k += 1
    return k

The clustering logic dictates a two-factor rule: two rows are identified as overlapping if and only if their top 10 holding tickers match exactly (creating a deterministic tuple signature) AND their fund names share a strong lexical prefix.

Why both? Relying solely on the holdings signature is dangerous. Two different passive index funds managed by different firms might hold the exact same top 10 mega-cap technology stocks. If we only checked holdings, we would collapse two distinct operational funds into one.

By enforcing the lexical prefix check, we ensure accuracy. If ‘Global Equity Fund Class A USD’ and ‘Global Equity Fund Class B EUR’ possess the exact same stocks, their holding tuple signature matches. Their tokenized prefix ['global', 'equity', 'fund'] also matches based on our defined threshold (e.g., 60% of the token length). Therefore, they are safely clustered together as a single entity.

Once clustered, we invoke a selection algorithm. We iterate over the array indices within the cluster, evaluate the ReturnM120 float vector, and retain the exact index containing the maximum finite return value.

We select for the maximum 120-month return because older, institutional share classes exhibit longer, uninterrupted historical track records and lower fee drags than newer retail classes. Retaining the asset with the longest valid track record is optimal for backtesting; it maximizes our sample size and statistical significance across different historical market regimes.

def _pick_best_index(arr: np.ndarray, idxs: list[int], keep: str = "max_return") -> int:
    if keep == "first":
        return idxs[0]
    
    best = idxs[0]
    best_r = arr["ReturnM120"][best]
    
    for i in idxs[1:]:
        r = arr["ReturnM120"][i]
        r_ok = np.isfinite(r)
        b_ok = np.isfinite(best_r)
        
        # We strictly prioritize finite, measurable returns over NaN or infinite values.
        # If the current best is NaN, any finite return replaces it immediately.
        if (r_ok and not b_ok) or (r_ok and b_ok and r > best_r):
            best = i
            best_r = r
            
    return best

def dedupe_unnecessary_share_classes(arr: np.ndarray, min_prefix_tokens=8, prefix_ratio=0.60, keep="max_return") -> np.ndarray:
    if arr.size == 0:
        return arr
        
    groups: dict[tuple, list[int]] = {}
    unique_empty_keys: list[int] = []
    tokens_cache = [None] * arr.size
    
    for i in range(arr.size):
        tokens_cache[i] = _name_tokens(str(arr["Fund"][i]))
        stocks = arr["Stocks"][i]
        try:
            # Convert the list of tickers into an immutable, hashable tuple for O(1) dictionary lookups
            sig = tuple(stocks) if stocks else tuple()
        except TypeError:
            sig = tuple()
            
        if len(sig) == 0:
            unique_empty_keys.append(i)
        else:
            groups.setdefault(sig, []).append(i)
            
    kept = set(unique_empty_keys)
    
    for sig, idxs in groups.items():
        # Sort by name to guarantee deterministic execution across multiple pipeline runs
        idxs_sorted = sorted(idxs, key=lambda i: arr["Fund"][i])
        clusters: list[list[int]] = []
        reps: list[int] = []
        
        for i in idxs_sorted:
            ti = tokens_cache[i]
            placed = False
            for c_idx, rep_i in enumerate(reps):
                tr = tokens_cache[rep_i]
                cpl = _common_prefix_len(ti, tr)
                
                # Dynamic threshold calculation ensures short names and long names are evaluated fairly
                threshold = max(min_prefix_tokens, int(math.ceil(prefix_ratio * min(len(ti), len(tr)))))
                if cpl >= threshold:
                    clusters[c_idx].append(i)
                    placed = True
                    break
            if not placed:
                reps.append(i)
                clusters.append([i])
                
        for cl in clusters:
            kept.add(_pick_best_index(arr, cl, keep=keep))
            
    kept_idx = np.array(sorted(kept), dtype=int)
    return arr[kept_idx]

So let’s see what we got from this. Time to test!

Oh! This looks good to me!

Screener Extract: 498 funds in 11.567s | Holdings Intercept: 7.711s | Total Architecture Execution: 19.278s | Matrix Dedupe Reduction: 498 -> 346 strict portfolios
[('F00000S8RT', 'Multipartner SICAV - Konwave Gold Equity Fund C USD', 25.92, list(['AU']))
 ('0P0000OMTB', 'Polar Capital Funds PLC - Polar Capital Global Technology Fund I Income EUR', 25.56, list(['NVDA', 'AVGO', 'AMD', 'META', 'LRCX', 'KLAC', 'LITE', 'COHR', 'SNDK', 'STX']))
 ('F0GBR04H0X', 'Multipartner SICAV - Konwave Gold Equity Fund B USD', 25.38, list(['AU']))
 ('F00000VI1X', 'Bakersteel Global Funds SICAV - Precious Metals Fund D EUR', 24.62, list(['NEM', 'AU', 'CDE']))
 ('F00001S8MC', 'CT (Lux) - Global Technology Class ZE (EUR Accumulation Shares)', 23.5 , list(['BE', 'LRCX', 'NVDA', 'AVGO', 'WDC', 'GOOGL', 'AMAT', 'MSFT', 'MRVL', 'AAPL']))]

Ticker    Count   PctFunds
--------------------------
NVDA        146      59.11
MSFT        132      53.44
AAPL        119      48.18
AVGO        113      45.75
GOOGL       110      44.53
AMZN        110      44.53
META        104      42.11
GOOG         68      27.53
NEM          44      17.81
LLY          40      16.19
MA           36      14.57
LRCX         33      13.36
MU           33      13.36
CRM          29      11.74
TSLA         28      11.34
AMAT         23       9.31
V            22       8.91
CDE          20       8.10
AMD          18       7.29
MRVL         18       7.29
FSLR         18       7.29
DC           16       6.48
PWR          16       6.48
RGLD         15       6.07
NFLX         15       6.07

Okay! No surprises here! This output matrix is formatted, computationally lightweight, and ready to be serialized to a Parquet file or whatelse. The architecture is resolved and ready for daily production scheduling.

Alright crew—good session today. Until the next article! And remember, may liquidity is with you! 💵​

PS: Help me to know what you think about the newsletter, give me feedback and rate it!

This is an invitation-only access to our QUANT COMMUNITY, so we verify numbers to avoid spammers and scammers. Feel free to join or decline at any time. Tap the WhatsApp icon below to join

Appendix

Full script:

"""
Morningstar fund screener (LT) + direct holdings (global API) + de-dup share classes
- No pandas in your code (NumPy structured array output)
- pages:
    * None  -> all pages (WARNING: huge)
    * int   -> 1..pages
    * list  -> explicit pages
- De-dup logic:
    * Only de-dups when TOP holdings signature matches AND fund names share a strong common prefix.
    * Keeps the class with max ReturnM120 by default.
"""

import math
import time
import re
import numpy as np
import requests
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# 1) LT Screener
LT_BASE = "https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener"

LT_HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "es-ES,es;q=0.9,en;q=0.8",
    "Referer": "https://lt.morningstar.com/",
}

SECURITY_POINTS = "SecId|LegalName|CategoryName|ReturnM120"


def _make_retry_adapter() -> HTTPAdapter:
    retry = Retry(
        total=6,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=("GET",),
        raise_on_status=False,
    )
    return HTTPAdapter(max_retries=retry, pool_connections=50, pool_maxsize=50)


def _lt_session() -> requests.Session:
    s = requests.Session()
    s.headers.update(LT_HEADERS)
    adapter = _make_retry_adapter()
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    return s


def fetch_lt_page(session: requests.Session, page: int, page_size: int) -> dict:
    params = {
        "page": int(page),
        "pageSize": int(page_size),
        "sortOrder": "ReturnM120 desc",
        "outputType": "json",
        "version": 1,
        "languageId": "es-ES",
        "currencyId": "EUR",
        "universeIds": "FOESP$$ALL",
        "securityDataPoints": SECURITY_POINTS,
        "filters": "",
        "term": "",
        "subUniverseId": "",
    }
    r = session.get(LT_BASE, params=params, timeout=(4, 20))
    ct = (r.headers.get("content-type") or "").lower()
    if r.status_code != 200 or "json" not in ct or not r.content:
        raise RuntimeError(f"LT bad response: {r.status_code} ct={ct} head={r.text[:200]!r}")
    return r.json()


def _normalize_pages(pages, total_pages: int):
    if pages is None:
        return range(1, total_pages + 1)
    if isinstance(pages, (int, np.integer)):
        n = int(pages)
        return range(1, min(max(n, 0), total_pages) + 1)
    lst = sorted({int(p) for p in pages if 1 <= int(p) <= total_pages})
    return lst


def scrape_lt_screener(pages=None, page_size: int = 50, category_contains: str = "RV", max_funds: int | None = None):
    secids, names, rets = [], [], []

    with _lt_session() as s:
        first = fetch_lt_page(s, 1, page_size)
        total = int(first.get("total", 0))
        total_pages = int(math.ceil(total / page_size)) if total > 0 else 1

        count = 0
        for pg in _normalize_pages(pages, total_pages):
            data = first if pg == 1 else fetch_lt_page(s, pg, page_size)
            for row in (data.get("rows") or []):
                cat = str(row.get("CategoryName") or "")
                if category_contains in cat:
                    sid = row.get("SecId")
                    if not sid:
                        continue
                    secids.append(sid)
                    names.append(row.get("LegalName") or "")
                    v = row.get("ReturnM120")
                    try:
                        rets.append(float(v) if v is not None else np.nan)
                    except Exception:
                        rets.append(np.nan)

                    count += 1
                    if max_funds is not None and count >= int(max_funds):
                        break
            if max_funds is not None and count >= int(max_funds):
                break

    return secids, names, np.asarray(rets, dtype="f8")

# 2) Direct holdings 
APIKEY = "HERE YOUR API KEY"

MS_BASE = "https://api-global.morningstar.com/sal-service/v1/fund/portfolio/holding/v2"
MS_HEADERS = {
    "apikey": APIKEY,
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
}

_thread_local = threading.local()


def _ms_session() -> requests.Session:
    s = getattr(_thread_local, "session", None)
    if s is None:
        s = requests.Session()
        s.headers.update(MS_HEADERS)
        adapter = _make_retry_adapter()
        s.mount("https://", adapter)
        s.mount("http://", adapter)
        _thread_local.session = s
    return s


def fetch_holdings_us_topN(secid: str, country: str = "United States", top: int = 10):
    s = _ms_session()
    url = f"{MS_BASE}/{secid}/data"
    params = {
        "clientId": "MDC",
        "version": "4.71.0",
        "premiumNum": 10000,
        "freeNum": 10000,
    }
    r = s.get(url, params=params, timeout=(4, 25))
    if r.status_code != 200 or not r.content:
        return []

    data = r.json()
    page = (data.get("equityHoldingPage") or {})
    hlist = (page.get("holdingList") or [])

    out = []
    for h in hlist:
        c = h.get("country") or h.get("countryName")
        if c == country:
            t = h.get("ticker") or h.get("tradingSymbol")
            if t:
                out.append(t)
                if len(out) >= top:
                    break
    return out


def attach_holdings_all(secids, max_workers: int = 20, country: str = "United States", top: int = 10):
    holdings = [None] * len(secids)
    with ThreadPoolExecutor(max_workers=max_workers) as ex:
        futs = {ex.submit(fetch_holdings_us_topN, sid, country, top): i for i, sid in enumerate(secids)}
        for fut in as_completed(futs):
            i = futs[fut]
            try:
                holdings[i] = fut.result()
            except Exception:
                holdings[i] = []
    return np.asarray(holdings, dtype=object)

# 3) De-dup share classes (fixes your repeated funds)
#    Rule: same holdings signature AND strong common name prefix.
def _name_tokens(name: str) -> list[str]:
    s = (name or "").strip().lower()
    s = s.replace("–", "-").replace("—", "-")
    s = re.sub(r"[()\[\],]", " ", s)
    return [t for t in s.split() if t]


def _common_prefix_len(a: list[str], b: list[str]) -> int:
    n = min(len(a), len(b))
    k = 0
    while k < n and a[k] == b[k]:
        k += 1
    return k


def _pick_best_index(arr: np.ndarray, idxs: list[int], keep: str = "max_return") -> int:
    if keep == "first":
        return idxs[0]
    # keep == "max_return" (nan-safe)
    best = idxs[0]
    best_r = arr["ReturnM120"][best]
    for i in idxs[1:]:
        r = arr["ReturnM120"][i]
        r_ok = np.isfinite(r)
        b_ok = np.isfinite(best_r)
        if (r_ok and not b_ok) or (r_ok and b_ok and r > best_r):
            best = i
            best_r = r
    return best


def dedupe_unnecessary_share_classes(
    arr: np.ndarray,
    min_prefix_tokens: int = 8,
    prefix_ratio: float = 0.60,
    keep: str = "max_return",
) -> np.ndarray:
    """
    Dedup only when:
      - Stocks tuple matches exactly (same signature)
      - AND fund names share a long common prefix (token-based)

    This fixes cases like:
      '... Fund I Income GBP' vs '... Fund R Income'
      '... Fund B USD' vs '... Fund B EUR'
    """
    if arr.size == 0:
        return arr

    # Group by holdings signature; empty holdings => never dedup (keep all)
    groups: dict[tuple, list[int]] = {}
    unique_empty_keys: list[int] = []

    tokens_cache = [None] * arr.size

    for i in range(arr.size):
        toks = _name_tokens(str(arr["Fund"][i]))
        tokens_cache[i] = toks

        stocks = arr["Stocks"][i]
        try:
            sig = tuple(stocks) if stocks else tuple()
        except TypeError:
            sig = tuple()

        if len(sig) == 0:
            unique_empty_keys.append(i)
        else:
            groups.setdefault(sig, []).append(i)

    kept = set(unique_empty_keys)

    # For each signature group, cluster by name-prefix similarity
    for sig, idxs in groups.items():
        # sort by name (stable) to make clustering deterministic
        idxs_sorted = sorted(idxs, key=lambda i: arr["Fund"][i])

        clusters: list[list[int]] = []
        reps: list[int] = []  # representative index per cluster

        for i in idxs_sorted:
            ti = tokens_cache[i]
            placed = False

            for c_idx, rep_i in enumerate(reps):
                tr = tokens_cache[rep_i]
                cpl = _common_prefix_len(ti, tr)

                threshold = max(min_prefix_tokens, int(math.ceil(prefix_ratio * min(len(ti), len(tr)))))
                if cpl >= threshold:
                    clusters[c_idx].append(i)
                    placed = True
                    break

            if not placed:
                reps.append(i)
                clusters.append([i])

        # keep one per cluster
        for cl in clusters:
            best = _pick_best_index(arr, cl, keep=keep)
            kept.add(best)

    kept_idx = np.array(sorted(kept), dtype=int)
    return arr[kept_idx]


def sort_by_return_desc(arr: np.ndarray) -> np.ndarray:
    if arr.size == 0:
        return arr
    r = arr["ReturnM120"]
    r2 = np.where(np.isfinite(r), r, -np.inf)
    idx = np.argsort(r2)[::-1]
    return arr[idx]

# 4) Runner
def run(
    pages=10,
    page_size: int = 50,
    max_workers: int = 20,
    category_contains: str = "RV",
    holdings_country: str = "United States",
    holdings_top: int = 10,
    max_funds: int | None = None,  # recommended if pages=None
    dedupe: bool = True,
):
    t0 = time.perf_counter()

    secids, names, rets = scrape_lt_screener(
        pages=pages,
        page_size=page_size,
        category_contains=category_contains,
        max_funds=max_funds,
    )
    t1 = time.perf_counter()

    stocks = attach_holdings_all(secids, max_workers=max_workers, country=holdings_country, top=holdings_top)
    t2 = time.perf_counter()

    arr = np.empty(len(secids), dtype=[("Id", "U16"), ("Fund", "U256"), ("ReturnM120", "f8"), ("Stocks", "O")])
    arr["Id"] = np.asarray(secids, dtype="U16")
    arr["Fund"] = np.asarray(names, dtype="U256")
    arr["ReturnM120"] = rets
    arr["Stocks"] = stocks

    arr = sort_by_return_desc(arr)

    before = arr.size
    if dedupe:
        arr = dedupe_unnecessary_share_classes(
            arr,
            min_prefix_tokens=8,   # if you still see duplicates, increase to 10–12
            prefix_ratio=0.60,
            keep="max_return",
        )
        arr = sort_by_return_desc(arr)
    after = arr.size

    print(
        f"Screener: {len(secids)} funds in {t1 - t0:.3f}s | "
        f"Holdings in {t2 - t1:.3f}s | Total {t2 - t0:.3f}s | "
        f"Dedupe: {before}->{after}"
    )
    return arr


import numpy as np
import matplotlib.pyplot as plt
from collections import Counter

def topk_stock_table_and_plot(arr: np.ndarray, topk: int = 20, *, title: str = "Top holdings frequency"):
    """
    Counts how often each ticker appears in arr["Stocks"] (per-fund top holdings list),
    returns a NumPy table with (Ticker, Count, PctFunds) and shows a barplot in BLACK + CYAN.

    Percentage = Count / (# funds with non-empty Stocks) * 100
    """
    if arr.size == 0:
        out = np.empty(0, dtype=[("Ticker", "U16"), ("Count", "i4"), ("PctFunds", "f8")])
        return out

    counter = Counter()
    funds_with_holdings = 0

    for stocks in arr["Stocks"]:
        if not stocks:
            continue
        funds_with_holdings += 1
        # count presence within a fund once (avoid double-counting duplicates inside the list)
        seen = set()
        for t in stocks:
            if t is None:
                continue
            t = str(t).strip().upper()
            if not t:
                continue
            seen.add(t)
        counter.update(seen)

    if funds_with_holdings == 0 or not counter:
        out = np.empty(0, dtype=[("Ticker", "U16"), ("Count", "i4"), ("PctFunds", "f8")])
        return out

    items = counter.most_common(topk)
    tickers = [k for k, _ in items]
    counts = np.array([v for _, v in items], dtype=np.int32)
    pct = counts / float(funds_with_holdings) * 100.0

    table = np.empty(len(tickers), dtype=[("Ticker", "U16"), ("Count", "i4"), ("PctFunds", "f8")])
    table["Ticker"] = np.array(tickers, dtype="U16")
    table["Count"] = counts
    table["PctFunds"] = pct

    fig, ax = plt.subplots(figsize=(10, 6))
    fig.patch.set_facecolor("black")
    ax.set_facecolor("black")

    y = np.arange(len(tickers))
    ax.barh(y, pct, color="cyan", edgecolor="cyan")

    ax.set_yticks(y)
    ax.set_yticklabels(tickers)
    ax.invert_yaxis()

    ax.set_xlabel(f"% of funds (N={funds_with_holdings})")
    ax.set_title(title)

    ax.tick_params(colors="cyan")
    ax.xaxis.label.set_color("cyan")
    ax.yaxis.label.set_color("cyan")
    ax.title.set_color("cyan")
    for spine in ax.spines.values():
        spine.set_color("cyan")
    ax.grid(True, axis="x", linestyle="--", linewidth=0.6, color="cyan", alpha=0.35)

    plt.tight_layout()
    plt.show()

    return table


def print_topk_table(table: np.ndarray, decimals: int = 2):
    """Pretty-prints the output table (Ticker, Count, PctFunds)."""
    if table.size == 0:
        print("Empty table.")
        return
    w_t = max(6, int(max(len(t) for t in table["Ticker"])))
    header = f"{'Ticker':<{w_t}}  {'Count':>7}  {'PctFunds':>9}"
    print(header)
    print("-" * len(header))
    for row in table:
        print(f"{row['Ticker']:<{w_t}}  {int(row['Count']):>7}  {row['PctFunds']:>9.{decimals}f}")

if __name__ == "__main__":
    matrix = run(pages=10, max_workers=20)  # pages=None --> all pages (very large)
    print(matrix[:5])

    table = topk_stock_table_and_plot(matrix, topk=25, title="Most-held stocks (top holdings lists)")
    print_topk_table(table)

Table of contents:

1. Introduction.

2. The United States coercive diplomacy.

3. Iran as a cost imposer.

4. Nuclear thresholds and monitoring.

5. Israel, Gaza, and the legitimacy environment.

6. Saudi Arabia and Syria as pivots.

7. Ukraine load, Caucasus exposure, Middle East spillover.

8. Hormuz and Red Sea.

9. Taiwan pressure and U.S. allocation constraints

10. Sudan and multilateral bandwidth competition

11. Europe risk multiplier

Audio Note: Before we begin, remember that if you’re accessing this article through the Substack app, you can listen to it instead of reading it. The MarketOps section is best suited for this format.

Before you begin, remember that you have an index with the newsletter content organized by clicking on the image below.

Introduction

Regional crises function as interconnected nodes within a broader architecture of geopolitical, economic, and military friction. From the contested maritime chokepoints of the Middle East to the industrial load bearing down on Europe and the persistent readiness burdens in the Indo-Pacific, the international system is experiencing simultaneous, compounded stress.

The expiration of the New START treaty in February 2026 serves as a structural inflection point for this new era. The loss of top-level arms control norms trickles down into regional theaters, replacing established escalation management with reciprocal fear dynamics. This vacuum elevates the importance of physical, measurable guardrails. Nuclear verification has transitioned from a technical non-proliferation tool into the central currency for de-escalation. The ability to monitor and verify intent—whether tracking Iranian breakout timelines or securing the fragile safety parameters at Ukraine’s Zaporizhzhya facility—now dictates the risk premiums applied by global financial markets and the preventive strike incentives of regional powers.

Within this volatile environment, statecraft has fundamentally shifted toward continuous force design and structural cost imposition. The dynamics analyzed in this report illustrate how major actors manipulate risk:

• The United States employs a visible escalation ladder, leveraging maritime dominance and advanced defense systems to enforce declarative realism, deter regional hegemons, and reassure exposed partners.

• Actors like Iran utilize dispersed proxy networks, drone swarms, and missile architectures to saturate defenses and impose systemic economic costs, altering global freight spreads and insurance premiums in critical corridors like the Red Sea and the Strait of Hormuz.

• The sheer volume of global flashpoints generates competition for diplomatic and military bandwidth. Legitimacy pressures emanating from the Gaza theater strain coalition durability, while severe humanitarian crises drain the diplomatic resources required for effective sanctions enforcement and crisis mediation.

• Geopolitical conflict now operates as financial architecture: Sanctions, export controls, and payment rails are weaponized, making the resilience of domestic industrial bases and the protection of strategic supply chains key.

As the geopolitical center of gravity shifts, regional pivots like Saudi Arabia and Syria determine the dynamics of these conflicts, influencing everything from global energy spare capacity to the friction of proxy routing across the Levant. Meanwhile, the persistent pressure around Taiwan forces a continuous recalibration of U.S. asset allocation, demonstrating that a surge in one theater degrades deterrence perceptions in another.

Subscribe now

The United States coercive diplomacy

The United States current application of coercive diplomacy reflects a transition toward force design as a primary tool for crisis management. Recent military movements in the Middle East illustrate this shift, where the deployment of the USS Abraham Lincoln carrier strike group and advanced missile defense systems like THAAD serves as a deliberate signal of readiness for high-intensity operations. This posture functions through a ladder logic where every deployment and public statement represents a specific rung on an escalatory scale. Officials monitor these thresholds to ensure that adversaries perceive a unified and unwavering message. The strategy relies on strict message to prevent the ambiguity that arises from distributed agency among local commanders and regional maritime actors.

Force design serves three critical functions within this philosophy. First, it establishes visible capability to extract concessions from Tehran regarding its nuclear enrichment and ballistic missile programs. Second, it provides physical assurance to regional partners who face the risk of direct exposure during heightened tensions. Third, it shapes domestic leadership perceptions by demonstrating capacity within an accelerating crisis loop. This internal-to-external cycle acts as a structural engine: domestic political pressure necessitates swift action, which leads to increased military visibility abroad. The outcomes of these crises then flow back into the domestic sphere, influencing legislative cycles and public media narratives.

A deeper structural layer guides these institutional choices beyond individual political identities. The maintenance of global hegemony and the stability of the economic regime create persistent incentives that survive changing administrations. This systemic position requires the United States to prioritize maritime dominance and the protection of strategic chokepoints like the Strait of Hormuz. The Maritime Action Plan reinforces this by seeking to revitalize domestic shipbuilding and imposing fees on foreign-built vessels to fund a Maritime Security Trust Fund. This focus on industrial capacity and robotic autonomous systems ensures that the American military remains the dominant force in contested waters.

Check the document here:

Restoring Americas Maritime Dominance

981KB ∙ PDF file

Download

Download

The 2026 National Defense Strategy formalizes this shift by replacing integrated deterrence with a clear hierarchy of priorities. This strategy explicitly identifies the defense of the American homeland and the containment of regional hegemons as the primary missions. By moving away from democratic idealism and open-ended nation-building, the current approach emphasizes declarative realism. This involves setting firm red lines and making the cost of aggression infeasible through denial. The use of force has become more prominent, as seen in recent operations in the Western Hemisphere and the intensified pressure on Iranian leadership. These actions suggest that diplomacy now serves as a prelude to or a partner of military coercion rather than a standalone alternative.

Iran as a cost imposer

The strategic posture of Iran emphasizes its role as a cost imposer through an integration of military dispersion and diplomatic signaling. This approach utilizes an expansive geography of exposure where missile and drone units maintain a mobile presence. These forces create a wide planning surface for any defensive entity because launch platforms distribute across vast terrains and reconstitute after operations. The sheer volume of these assets aims to saturate point defenses, which forces an opponent to coordinate responses across a multitude of nodes and decision authorities. This increase in coordination demand leads to inherent friction in execution and expands the window for adversary opportunity.

The leverage held by Tehran manifests through three synchronized layers of influence. Direct strike capabilities place regional bases and essential infrastructure under constant risk of engagement. Parallel to this, proxy depth extends the potential map of conflict across Iraq, Syria, Lebanon, and critical maritime zones. Narrative framing serves as the third layer, as Iranian officials craft messages centered on legitimacy, sovereignty, and proportionality to justify their defensive stance. These elements work together to manage escalation, where proxy incidents provide a means of signaling intent while maintaining a level of flexible attribution. In the maritime domain, these actions impose systemic economic costs by triggering a repricing of insurance and widening freight spreads.

Recent diplomatic activity reinforces this deterrent logic through formal communications to the United Nations. These messages frame potential retaliation as a defensive response tied to the protection of regional facilities and assets. This use of legal language seeks to establish international legitimacy while providing enough clarity to raise the overall effect of deterrence. By defining the scope of their claims, Iranian leaders constrain the space for miscalculation among partners and adversaries.

Information regarding indirect negotiations through the Omani Foreign Ministry highlights a parallel track of engagement. While military drills influence the pricing channel of global commodities, the cadence of these consultations suggests a deliberate attempt to balance pressure with dialogue. The uncertainty surrounding the safety of transit corridors carries convex effects on market demand and insurance terms. This dual-track strategy of maritime signaling and diplomatic mediation ensures that every tactical movement translates into a tangible economic or political cost for the international community.

Nuclear thresholds and monitoring

Nuclear verification functions as the essential medium for modern de-escalation because the transformation of political intent into measurable data provides the only reliable baseline for regional stability. This process replaces abstract estimates of breakout timelines with concrete measurements, thereby altering the incentive structure for preventive military action. The current bargaining environment centers on the specific modalities of access, including the physical sampling of materials, the continuous presence of international inspectors, and the robust protection of monitoring hardware against physical or cyber interference.

Developments this month illustrate the vulnerability of this verification infrastructure within active conflict zones. The International Atomic Energy Agency has drawn specific attention to the Zaporizhzhya Nuclear Power Plant, where operational safety relies on a single remaining external power line. This precarious situation demonstrates that monitoring capabilities remain tethered to local security conditions, which in turn depend on the shifting incentives of combatants. Any disruption to these essential links triggers a surge in uncertainty and fuels intense credibility contests between opposing parties. The resulting feedback loop shows that perceived threats to monitoring equipment directly influence the probability of kinetic strikes, as reduced visibility encourages more aggressive defensive postures.

Markets now interpret the continuity of nuclear monitoring as a leading indicator for escalation risks. Financial actors price the stability of these verification regimes rather than treating conflict as a binary outcome. The preservation of inspector access bounds the potential paths toward military confrontation, as it provides a cooling mechanism for political tensions. This economic dimension reinforces the status of verification as a central currency in international relations, where the transparency of a nuclear program serves as a prerequisite for broader trade and investment stability.

A secondary node of verification significance has emerged through the expansion of civil nuclear cooperation in the Gulf. Recent discussions regarding Saudi Arabian nuclear arrangements have introduced new variables concerning regional safeguards and enrichment constraints. This wave of civil nuclear development shifts the traditional balance of leverage and forces a recalibration of alliance commitments. The integration of advanced monitoring norms into these new programs serves as a stabilizing force, ensuring that the proliferation of technology aligns with established security equilibria. The pursuit of these verification standards remains the primary tool for managing the complex relationship between energy independence and regional strategic balance.

Israel, Gaza, and the legitimacy environment

Legitimacy pressure within the Gaza dictates the durability of international coalitions and the scope of regional policy. Data from February 2026 indicates that humanitarian reporting functions as a variable that shifts domestic opinion and legal forum activity in partner capitals. The United Nations human rights system confirms that displacement and demographic changes across Gaza and the West Bank create sustained pressure on diplomatic consent for basing and sanctions enforcement. The first fifteen months of the conflict, report mortality figures that exceed earlier counts. These findings increase litigation risk and impact coalition cohesion by altering the political cost of alignment.

The connection between the Gaza conflict and the broader regional landscape remains a central feature of current strategic planning. Narratives centered on Gaza mobilize support for maritime action and proxy activation, drawing energy from the humanitarian situation to shape the Iran file. This dynamic creates a chain where combat outcomes generate data on civilian harm, which then drives activity within international legal forums. Such legal developments shift domestic opinion, which in turn defines the space available for coalition partners to maintain sanctions and offer logistics support.

The International Committee of the Red Cross issued statements emphasizing obligations under international law regarding detainee and civilian protection. These communications affect the risk tolerance of partners for participation in kinetic operations. Regional actors monitor these signals to calibrate their level of cooperation with external powers. The relationship between humanitarian metrics and strategic leverage ensures that the legitimacy of military presence depends on the management of civilian protection norms. Coalition space remains tied to the perception of adherence to these standards, as seen in the current diplomatic discourse surrounding base access and crisis bargaining.

Saudi Arabia and Syria as pivots

Read more