Why high-frequency trading systems demand array languages

There is a particular moment in every market-data project where someone asks, reasonably, why the whole thing is not just a Postgres table with an index on (symbol, timestamp). It is a good question, and the honest answer is not "because Postgres is slow". Postgres is not slow. The answer is that the shape of the data and the shape of the questions asked of it are both columnar, and a row-oriented system spends most of its energy converting between the two.

This post is about what that conversion costs, why it is structural rather than incidental, and what an array language does instead.

A trade is not a row

Consider a single print on the consolidated tape. It has a symbol, an exchange timestamp, a participant timestamp, a price, a size, an aggressor side, a venue, a sale condition, and a sequence number. Nine fields. A row-oriented system stores those nine fields adjacently, because the mental model is that a trade is a thing and its fields belong together.

Now ask a question. Any question.

What was the volume-weighted average price of AAPL between 09:30 and 10:00?

That question touches three of the nine fields: sym, time, px, sz. Four, if you are being precise. It touches none of the venue, the sale condition, the sequence number, or either of the two timestamps you did not filter on. In a row-oriented layout, reading the four you want means reading the five you do not, because they are interleaved in the same cache lines.

On a day with 38 million prints — a modest US-equity session in the simulator that ships with amber-tick — that is not a rounding error. A trade record is roughly 64 bytes packed. The four columns you actually want are about 24. You have just read 2.7× the bytes you needed, and every one of those bytes moved through the same memory bus and evicted something from the same L2.

Columnar storage fixes that by inverting the layout: nine arrays, each holding one field for every row. A query for px and sz reads two arrays and never touches the other seven. In amber-tick's splayed store this is literal — each column is its own file:

store/2026.08.21/trades/
  .d  sym  time  px  sz  side  venue  cond  seq  extime

A query that needs px and sz opens two file descriptors. That is the whole mechanism, and it is the least interesting part of what follows.

The overhead that survives the layout change

Suppose you fix the storage and keep the execution model. You now have columns on disk and a runtime that, for each row, materialises an object, calls a method on it, and accumulates into another object. You have moved the bottleneck, not removed it.

Here is what per-row execution costs that per-column execution does not:

• An allocation, or at minimum a boxed header, per value. Amber's own query layer had exactly this problem before 1.9.1: select … by … from boxed one K object per row on the group-and-probe path. Fixing it — grouping and probing on raw column vectors instead — made group-by 24.7× faster and inner join 19.3× faster, on unchanged data, with unchanged answers. That factor of twenty was not algorithmic. It was allocation and pointer chasing.
• A dispatch per element. If the runtime does not know until it looks at the value whether it is adding two int64s or two float64s, it cannot emit an add instruction — it has to branch. Branching per element in a ten-million-element loop is a branch predictor's worst day.
• No vectorisation, ever. AVX2 processes four doubles per instruction; NEON processes two. Neither is reachable from a loop whose body is a virtual call. Amber's src/simd.c supplies simd_add_i64/f64, simd_mul_i64/f64 and simd_sum_i64/f64 over plain int64_t* / double* arrays precisely because that is the only shape a SIMD kernel can consume.
• No parallelism without a per-row locking story. Splitting a contiguous array of ten million doubles across eight threads is one line of arithmetic. Splitting a linked structure of ten million objects is a research project. Amber's src/parallel.c splits arrays above 100,000 elements into one contiguous chunk per POSIX thread and hands each chunk to the SIMD kernel. Below that threshold it calls the kernel directly with no thread overhead, because for small inputs the pthread_create is the cost.

None of these are exotic. They are the standard consequences of a runtime whose unit of work is a scalar and whose unit of data is an object.

What the array model is, precisely

An array language makes the vector the unit of both. a+b where a and b are ten-million-element float vectors is one operation, dispatched once, over two contiguous payloads, into a kernel that knows both types statically.

The syntax is downstream of that. Terse notation is what you get when the primitive operations are whole-array and there are enough of them to need short names — not the other way round.

Here is a complete intraday VWAP in Amber:

select vwap:wavg[sz;px] by time:1m xbar time from trades

Read it right to left as a pipeline: bucket time onto one-minute boundaries, group by the bucket, and within each group compute the size-weighted average of price. Every step is a whole-column operation. On five million trades it takes 151 milliseconds.

The equivalent in a row-oriented system is a GROUP BY date_trunc('minute', ts) with a SUM(px*sz)/SUM(sz), and a competent columnar SQL engine will do it in a comparable time — DuckDB is genuinely good at this. The difference shows up when the operation is not one SQL has a name for.

The operation SQL does not have a name for

Every serious question about execution quality starts with the same join: for each trade, what was the quote standing at the moment it printed?

This is the as-of join, and it is the load-bearing operation of market microstructure. Expressing it in standard SQL means a correlated subquery or a window function over a union of two tables, and either way the planner has to discover, from the shape of the query, that what you meant was "walk both sequences in order". It usually does not.

In an array language, it is a primitive:

m:aj[`sym`time; trades; quotes]

Five million trades against ten million quotes: 704 milliseconds. The kernel is a branch-free lower_bound over each symbol group's sorted nanosecond timestamp slice. It is branch-free because a binary search with a data-dependent branch mispredicts about half the time, and five million times half a mispredict is the whole runtime.

Three things make that number possible, and only one of them is the kernel:

What attributes actually buy

The `s attribute is worth dwelling on, because it is the clearest case of "the same answer, four orders of magnitude apart". bench.k measures ? (find) on identical data, with and without it:

Note the shape of the speedup column: it grows with n, because the scan is O(n) and the search is O(log n). This is the difference between a system that degrades gracefully as the tape grows and one that falls off a cliff at the exact moment a busy day makes it matter.

All four kdb-style attributes are implemented in C: `s sorted, `u unique, `p parted, `g grouped. Parted gives you O(log n) group slicing; grouped plus the group index gives O(1) per-symbol slicing.

An honest accounting

Array languages are not universally faster and it does nobody any good to pretend otherwise. Here is Amber measured against nine other engines on four workloads, with a correctness gate that compares every answer bit-exactly against a C reference:

CBQN beats Amber on three of four. NumPy beats it on two. DuckDB — a mature, vectorised, JIT-free columnar SQL engine with a real planner — is competitive everywhere and better on group-by. Amber's reduction row is 0.70× of plain C, which is the four-accumulator float sum vectorising, and its join row is where the index build pays off.

The point is not that the array language wins. The point is which row it wins and by how much: the join, which is the operation the whole domain is built on, and the reduction, which is the operation you do ten thousand times a day. It loses on element-wise arithmetic against libraries that have spent twenty years on element-wise arithmetic, and that is a fair trade.

The part nobody benchmarks: the distance from question to query

Here is a complete execution-quality analysis. Not pseudocode — this runs:

gentq 5000000                                   / a session
trades:xasc[`sym`time; trades]                  / sort + stamp attributes
quotes:xasc[`sym`time; quotes]

m:aj[`sym`time; trades; quotes]                 / TAQ
m:update mid:0.5*bid+ask from m
m:update side:tsign m from m                    / Lee-Ready sign
m:update eff:effspread m from m                 / 2 * |px - mid|

select n:#px, vol:sum sz, effbps:10000*wavg[sz; eff%mid] by sym from m

Nine lines from nothing to a per-symbol effective-spread table over five million prints. There is no schema migration, no ORM, no serialisation format, no intermediate materialisation, and no point at which the data leaves the process.

That last property compounds. When the analysis needs to become a chart, it stays in the same process: python-amber hands you the columns as NumPy views pointing at the engine's own buffers. When it needs to become a dashboard, amberd serialises it column-oriented over a length-prefixed line protocol. When it needs to become four hundred million rows for a Polars pipeline, amber-arrow streams record batches that are windows onto one export rather than slices of it.

The array model is what makes all three of those cheap, because in each case the thing being handed across the boundary is already a contiguous typed buffer. There is nothing to convert.

When not to reach for this

An array language is the wrong tool when your access pattern is genuinely a single row by primary key, when your workload is transactional, when your data does not fit the rectangular model, or when the operation you need is a graph traversal. It is also, bluntly, the wrong tool if nobody on the team wants to learn it — a terse notation is a real cost and pretending otherwise is how these projects die.

But if the question you keep asking is some variation of "across all of today's prints, grouped by something, aligned in time against something else" — then the array model is not an optimisation you are applying to the problem. It is the shape the problem already had.

Everything in this post is runnable. The engine is at github.com/BonucciAndrea/amber; the tick architecture is at amber-tick; benchmark methodology is in the benchmarks page and bench/SPEC.md. If you want to try the notation without installing anything, the browser scratchpad runs the real C interpreter compiled to WebAssembly.