Table of contents:
Introduction.
Do we really need low-Latency data structures?
Risk and limitations of the data structures.
Low latency, reframed: Risk distribution.
Memory and allocation.
Core optimization.
Tail-latency budgeting as a first-class risk metric (quantile engineering).
State and flag encoding under memory-bandwidth constraints.
SoA vs AoS for feature pipelines.
Preallocation and out=.
Queueing primitives.
Zero-copy paths and multiprocess topologies.
Before you begin, remember that you have an index with the newsletter content organized by clicking on “Read the newsletter index” in this image.
Introduction
You can have the right signal, the right risk limits, and the right order type, and still lose money because your system expresses that decision at the wrong moment. That failure mode is especially cruel because it doesn’t look like bad alpha in a backtest; it looks like slippage, adverse selection, and fills that degrade exactly when you most need precision. The strategy survives on paper while the execution layer taxes it in production.
Most teams treat that tax as an engineering inconvenience: optimize later, refactor later, rewrite later. But latency is not a single number you improve. The market doesn’t care about your median. It cares about the rare stalls that occur during stress, when queues build, spreads widen, and your signal’s half-life collapses. Those stalls are where good research becomes a losing business.
Today we take a different stance: the execution pipeline is a risk book, and data structures are positions inside it. Each allocation, copy, branch, cache miss, and serialization step is exposure—not in dollars, but in timing error. If your system occasionally becomes unavailable for milliseconds, you are effectively holding stale orders with unpriced optionality against faster participants.
If you want to go deeper, don’t forget to check that:
So the goal here is not to make Python fast in the abstract. The goal is to make the latency profile boring: predictable under load, stable under bursts, and engineered around hard budgets. Think of it as deadline-driven execution:
A trade has value only while it arrives before a moving boundary. Past that boundary, correctness decays into noise.
What follows is a practical framework for designing that boringness: treating tail latency as a first-class risk metric, treating memory bandwidth as a scarce resource, and treating the allocator as a hazard to be removed from the hot path.
Do we really need low-Latency data structures?
In algorithmic trading, low latency is frequently fetishized as a vanity metric—a badge of honor for engineers who enjoy optimizing C++ templates, shaving nanoseconds off serialization protocols, or debating the merits of kernel bypass networking. This is an error in categorization that pervades the industry. Latency is not a stylistic preference, a marketing bullet point, or a nice-to-have feature for the roadmap; it is a measurable, deterministic property of your execution pipeline:
Market event → State update → Signal generation → Risk check → Order transmission.
Every discrete step in this chain introduces delay. More critically, every step introduces variance.
The uncomfortable dilemma facing quantitative development teams—particularly those utilizing Python for live execution—is that low latency is often treated as an optional optimization phase, something to be bolted on once the strategy demonstrates alpha.
I know what you’re going to say: Python is for prototyping and C++ is for running code. But many retailers, and even small businesses, can’t do that, due to time and cost constraints. Yes, I know, maybe they’ve done it wrong from the start, maybe they should invest more in migrating their entire infrastructure. But for whatever reason, they can’t.
So this creates a dangerous, often fatal, path dependency. Research code, optimized for vectorized throughput and batch processing (e.g., pandas DataFrames loaded from Parquet files), becomes production code. This architecture functions adequately in quiet markets where arrival rates are Poisson-distributed and manageable. But when the regime changes—a volatility spike, a venue microburst, or a specific symbol becoming the locus of global attention—the infrastructure reveals its true nature.
The senior quant’s version of this dilemma is not the binary, simplistic question “Are we fast?” It is the statistical question regarding the distribution of outcomes:
Are we optimizing Expected Latency E[L] or Tail Latency Q99.9(L)?
If you optimize for the average case (E[L]), you are implicitly accepting that in the moments of highest market stress—when the opportunity cost is highest, liquidity is most fleeting, and spreads are widening—your system will perform at its worst. This is the definition of negative convexity. If you operate strategies where the marginal expected return is a function of queue position, adverse selection avoidance, or reactive hedging, latency is not an engineering detail. It is a core parameter of the execution model. It directly impacts fill probabilities (Pfill), slippage (S), and effective spread capture.
Consider the implications of average latency in a winner-takes-all auction mechanism like a limit order book. If your system is faster than 90% of the market 90% of the time, but slower than the market during the 10% of intervals where 80% of the volume trades, your effective capture rate is near zero. You are winning races that have no prize money. Therefore, the necessity of low-latency data structures is not about speed in the abstract. It is about reducing the probability of wrong timing outcomes that invalidate the alpha model. We do not need to be fast to feel good; we need to be fast to reduce the timing error term in our PnL equation. A strategy that generates 2 bps of edge but surrenders 3 bps due to tail-latency-induced slippage during volatility is not a good strategy with bad tech; it is a losing strategy.
Risk and limitations of the data structures
The primary risk in Python-based trading systems is not the interpreter overhead itself (though that exists and sets a theoretical floor of perhaps 10-20 microseconds), but the memory access patterns dictated by standard data structures. Just use NumPy is an insufficient heuristic; in fact, naive NumPy usage can be detrimental in event-driven loops. A data structure implies a rigid contract with the hardware. It dictates:
Memory access patterns: Modern CPUs depend heavily on hardware prefetchers to predict which data to load into the L1 cache before the instruction requests it. Prefetchers thrive on linearity. Are we reading contiguous blocks, or are we chasing pointers across the heap? Random access into a fragmented heap results in Cache Misses (L1/L2/L3). A single L3 cache miss can stall the CPU pipeline for 300+ cycles. If your data structure requires three pointer dereferences to read a price (e.g.,
OrderBook→Level→Price), you are guaranteeing execution stalls.
Copy semantics: Where is data duplicated? Is the copy operation visible to the developer, or hidden behind a convenient API like
fancy_indexingor slicing that triggers a deep copy? In Python, ease of use often correlates inversely with memory efficiency. For example, calling a C-compiled function that expects a C-contiguous array with a non-contiguous slice forces NumPy to silently allocate a new buffer, copy all data, execute the function, and then potentially copy results back. This is invisible in the code but devastating to the memory bus.
Allocation behavior: Does this operation require finding new memory pages? Does it trigger the OS allocator (
malloc/free)? Does it invoke the Python Garbage Collector? Allocation is non-deterministic; it depends on the fragmentation state of the heap at that exact moment. Requesting 64 bytes when the heap is clean might take 50ns. Requesting 64 bytes when the heap is Swiss-cheesed by fragmentation might take 5µs as the allocator searches for a contiguous block or requests new pages from the OS kernel.
Latency distribution: Does the system exhibit consistent service times, or is it spiky? In trading, spiky latency is structurally worse than slower-but-smooth latency. A system with a median tick-to-trade time of 50μs and a p99 of 5ms is significantly more dangerous than a system with a median of 100μs and a p99 of 120μs. The former implies that during a burst, the system effectively collapses. This collapse creates stale orders resting in the book, waiting to be picked off by faster competitors who have already updated their valuation models using the new market data you haven’t processed yet.
The specific risks introduced by naive structure choices include:
Hidden copying: Implicit casting (e.g., performing arithmetic between
float32andfloat64resulting in an upcasted temporary array) ornp.ascontiguousarraycalls in hot paths. A copy operation is O(N) bandwidth consumption; doing this on every tick saturates the memory bus (DRAM bandwidth), which is often the tightest bottleneck in modern multi-core systems.Allocator jitter: Repeated temporary array creation in chained expressions (e.g.,
A = B * C + D) forces the allocator to thrash. This involves not just the time to allocate, but the time to zero-out memory for security, and the degradation of cache locality. It fragments the heap, increasing the time complexity of finding a free memory block for subsequent allocations.Bandwidth saturation: Using
np.bool_(1 byte) for binary flags wastes 8x memory bandwidth compared to bit-packed representations. In a bandwidth-constrained environment—which describes all modern HFT servers sharing Last Level Cache (LLC) across cores—wasting bandwidth is equivalent to throttling your CPU. Reading 8 bytes to get 1 bit of information is a 6300% efficiency loss.Cache thrashing: Array-of-Structures layouts (e.g., a list of
Tradeobjects) force the CPU to load cache lines containing data (e.g.,exchange_order_id,client_tag,flags) that are irrelevant to the current calculation (e.g.,mid_price). This reduces the effective cache size available for the relevant data, causing higher eviction rates. If your working set for a calculation exceeds L2 cache size because of structure bloat, your performance falls off a cliff.
The core problem here manifests during a pivotal event. Consider a mean-reversion strategy that appears stable during backtesting and low-volatility paper trading. The strategy relies on comparing the order book imbalance of a correlated ETF against the futures contract.
A macro news release (e.g., non-farm payrolls) triggers an auction imbalance burst. The feed handler rate jumps from a manageable 10,000 messages/sec to a torrential 500,000 messages/sec. This is not a gradual ramp; it is a step function change in load.
In this scenario, the alpha model remains valid. The signal is clear: the futures have moved, the ETF is lagging, and there is a theoretical arbitrage. But the realized fills degrade sharply. You are getting hit on your passive quotes after you intended to cancel them, and your aggressive orders are landing in the queue after the liquidity has been taken. A postmortem analysis typically reveals:
Median tick-to-trade time remained within tolerance (e.g., 50µs).
The p99.9 latency blew out by orders of magnitude (e.g., from 80µs to 15ms).
Correlation: This blowout correlates perfectly with bursts of memory allocation and Garbage Collection (GC) pauses.
Why does the GC wake up exactly then? Because the high message rate generated a high rate of temporary object creations (allocations)—parsing new price levels, creating new order objects, instantiating log messages—filling the generation 0 heap and triggering a collection cycle. The GC is a stop-the-world event. While the GC is walking the object graph to find unreachable objects, your network card is buffering packets. When the GC finishes, you process those packets, but the market has moved.
This is the event that forces the issue. We do not lose because we are slow on average. We lose because we are effectively offline for milliseconds at a time due to structural inefficiencies, exactly when the market is moving the fastest. The conflict is no longer theoretical; it is PnL-negative. The infrastructure has become the primary source of risk, eclipsing market risk.
Low latency, reframed: Risk distribution
To solve this, we must reframe our definition of success. A misconception that persists in the industry—often imported from web development or big data—is that low latency is equivalent to low mean latency. This is a rookie mistake in the context of stochastic control and competitive games.
We must treat latency L as a random variable with a probability density function f(L). We are not minimizing the expectation E[L]. We are solving a constrained optimization problem where the tail is the constraint:
Where l* is the maximum tolerable tick-to-trade latency (the cliff where the alpha decays to zero) and ϵ is a strictly bounded probability (e.g., 10-3 or 10-4). In high-frequency strategies, l* might be the time it takes for a signal to traverse the colocation center cross-connect.
This aligns with standard risk management frameworks. Just as we calculate Value at Risk to understand tail financial loss, we must calculate Latency at Risk to understand tail execution failure. If the cost of latency c(L) is convex—meaning a 10ms delay costs more than 10x a 1ms delay (common in momentum bursts where the price moves away exponentially)—then reducing the variance of L yields a significantly higher improvement in Expected PnL (E[π]) than reducing the mean.
Furthermore, we must consider the conditional probability of latency given volatility: P(L>x | σ>high). If this probability is high, our system is anti-correlated with opportunity. We are slow exactly when we need to be fast. If c(L) spikes during market bursts (when L also naturally rises due to load), the correlation implies that tail latency reduction is the single highest-leverage engineering activity available. A robust system must have a flat latency profile regardless of input throughput, up to the saturation point of the physical hardware.
Memory and allocation
Python is, in many respects, fast enough for the logic layer of mid-frequency strategies. The obstacles are not the language syntax, but the underlying machinery. The interpreter loop is optimized for flexibility, not predictability.
The specific obstacles are physical and architectural:
Python object overhead: Every
intorfloatin a standard Python list is a fullPyObjectC-struct. This includes a reference count (ob_refcnt), a type pointer (ob_type), and the value itself. A list of integers is not a contiguous block of memory (like a C array); it is a contiguous block of pointers to objects scattered across the heap. Iterating over this list requires double dereferencing (pointer to list → pointer to object → value) and destroys locality of reference. This causes the CPU to stall constantly waiting for data to arrive from main memory.Implicit allocations: High-level NumPy expressions optimize for readability, not memory reuse. The expression
C = A + Ballocates a new buffer forCevery time it is executed. In a loop running 1,000 times a second, this creates 1,000 allocations and 1,000 deallocations per second. This places immense pressure onmalloc/free, increasing the likelihood of locking contention in the memory allocator (glibc malloc has per-arena locks). Even worse, it churns the CPU cache, as new memory is fundamentally cold memory.Layout mismatches: We often compute feature vectors (columnar operations: mean of price, standard deviation of size), but market data arrives as discrete events (row-based: Update for symbol X). Converting between these layouts at high frequency creates friction. Naively appending rows to a NumPy array is an O(N) operation (copying the whole array), which is catastrophic for latency. Storing data as a list of rows creates the cache thrashing issues mentioned earlier. We need a hybrid approach that allows row-based updates into column-based storage without reallocation.
The GIL and GC: The Global Interpreter Lock (GIL) ensures thread safety by allowing only one thread to execute Python bytecodes at a time, effectively serializing execution and preventing true parallelism on multicore systems. More dangerously, the Garbage Collector pauses execution to mark and sweep objects. These pauses are not scheduled; they occur when the allocation threshold is breached—which, per Murphy’s Law, is always during the busiest market interval because that’s when you are generating the most data. The duration of a GC pause is proportional to the number of live objects, meaning as your internal state grows (more orders, more signals), your latency spikes get worse.
The challenge, therefore, is to build data structures in Python that minimize three physical costs:
Allocator involvement: We want zero allocations in the hot path. All memory must be claimed at startup. We want to treat Python as a static memory language within the critical section.
Memory bandwidth waste: We want to move only the bits we read. We cannot afford to load 64 bytes of cache line to read 1 byte of data. We need dense, packed representations.
Copy operations: We want to mutate data in place or use pre-existing buffers. We need to manage the lifecycle of our buffers explicitly, rather than relying on the interpreter to clean up after us.
This requires a departure from Pythonic code (idiomatic, concise) and an embrace of systems thinking within Python (explicit, memory-aware). It means writing code that looks more like C or Fortran, wrapped in Python syntax.
