Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.
From raw market series to stationary model inputs
Raw price series carry unbounded drift, non-constant variance, and heavy tails that break numerical optimizers through singular matrices and vanishing gradients. Effective time series preparation replaces naive preprocessing with targeted mathematical operators that make market structure legible to learning algorithms without destroying predictive memory or leaking future data.
What’s inside:
Controlling transformation risks: Standard Z-scores distort skewed downside risks, hard Winsorization erases capitulation signals, and global scaling leaks future variance into historical training sets.
Preserving long-range memory: Integer differencing (d=1) discards over 60% of predictive memory, whereas fractional differencing uses exponentially decaying weights to achieve stationarity while retaining historical path memory.
Mapping fat-tailed distributions: Converting skewed returns through Cornish-Fisher expansions or rank-to-Gaussian transforms prevents optimizers from treating real market tail shocks as mathematically impossible anomalies.
Sampling on information time: Chronological time bars generate zero-inflated distributions and intraday volatility spikes, while Dollar Bars sample data on dynamic fiat thresholds to produce more stable return properties.
Eliminating volatility ghost effects: Standardizing returns with GARCH(1,1) conditional volatility adjusts for autoregressive variance clustering, avoiding the abrupt artificial drops caused by flat rolling lookback windows.
Mitigating noise and multicollinearity: Combining Shannon entropy soft-clipping with Marchenko-Pastur PCA thresholding compresses extreme shocks without losing rank order and strips out correlated noise vectors.
Reconstructing phase-space geometry: Applying Takens’ Delay Embedding Theorem lifts 1D price series into multi-dimensional phase space, exposing hidden market regime structures to spatial classifiers.
This is truly a very useful article, not just for reading but to put in practice.
I have a question regarding the Cornish Fisher Transform. You talk about calculating rolling skew, kurtosis, but don't see that in the code.
The transform should be applied on a rolling basis, right? Otherwise, this will create a look-ahead bias. or is this supposed to be applied like a StandardScaler for preprocessing?
Great article QB.
I have a question-- before orthogonalization, would you align all assets to the same time points by using each asset’s most recently completed bar, so every asset ends up with the same number of rows even though volume bars occur at different times?