[WITH CODE] Data transformations: Time series preparation
The hidden statistical problems inside untransformed market series
Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.
From raw market series to stationary model inputs
Raw price series carry unbounded drift, non-constant variance, and heavy tails that break numerical optimizers through singular matrices and vanishing gradients. Effective time series preparation replaces naive preprocessing with targeted mathematical operators that make market structure legible to learning algorithms without destroying predictive memory or leaking future data.
What’s inside:
Controlling transformation risks: Standard Z-scores distort skewed downside risks, hard Winsorization erases capitulation signals, and global scaling leaks future variance into historical training sets.
Preserving long-range memory: Integer differencing (d=1) discards over 60% of predictive memory, whereas fractional differencing uses exponentially decaying weights to achieve stationarity while retaining historical path memory.
Mapping fat-tailed distributions: Converting skewed returns through Cornish-Fisher expansions or rank-to-Gaussian transforms prevents optimizers from treating real market tail shocks as mathematically impossible anomalies.
Sampling on information time: Chronological time bars generate zero-inflated distributions and intraday volatility spikes, while Dollar Bars sample data on dynamic fiat thresholds to produce more stable return properties.
Eliminating volatility ghost effects: Standardizing returns with GARCH(1,1) conditional volatility adjusts for autoregressive variance clustering, avoiding the abrupt artificial drops caused by flat rolling lookback windows.
Mitigating noise and multicollinearity: Combining Shannon entropy soft-clipping with Marchenko-Pastur PCA thresholding compresses extreme shocks without losing rank order and strips out correlated noise vectors.
Reconstructing phase-space geometry: Applying Takens’ Delay Embedding Theorem lifts 1D price series into multi-dimensional phase space, exposing hidden market regime structures to spatial classifiers.
