The array language
markets run on.
Amber is a small, fast, self-contained engine written in portable C99: columnar,
vectorised, in-memory, with the working vocabulary of q/kdb+ — dictionaries,
tables and keyed tables, the full join family including a native as-of kernel, qSQL
select/by, and column attributes implemented in C that turn search from
O(n) into O(log n).
Built for the shape data actually has
A tick is not a row. It is one value in each of nine columns, and every question you ask of a trading day is a question about columns. Amber is columnar all the way down — storage, evaluation, the wire, and the handoff to Arrow.
Zero-copy memory layout
A column is one contiguous, cache-line-aligned payload. amber_get_vector_ptr()
hands you that pointer — not a copy of it. NumPy's base, Arrow's data buffer and the
engine's column are the same address, asserted by pointer equality in the test suites.
Native TCP wire protocol
amberd is ~450 lines of C: one engine, many connections, a one-line
request and a length-prefixed reply in text, json, jsonc or
raw. Small enough to reimplement in an afternoon — which Go, TypeScript and Python
clients all have.
Array evaluation, not row loops
Every expression compiles to a flat opcode array and runs on a real stack VM.
Vector ops dispatch into AVX2 / NEON kernels and split across cores above 100,000 elements.
\disasm decodes the bytecode the compiler actually emitted.
Columnar storage & attributes
All four kdb-style attributes are implemented in C — `s sorted,
`u unique, `p parted, `g grouped. Sorted turns
? into a binary search: 1.73 s → 1.4 ms on 2M rows, same answer.
Native as-of join
aj matches each trade to its most recent quote with a branch-free
lower_bound over each symbol group's sorted nanosecond slice, off a thread-local
16 MB bump arena that keeps malloc jitter out of the hot path.
Diagnostics that read like Rust
An error[E0104] line, a --> locator, a gutter-aligned
source line, ^^^ underlines and a = help: note. One report, never two —
and switchable at runtime so a trap loop stays quiet.
q vocabulary, K notation
If you have written q, you already know most of Amber: select … by … from … where …,
aj, wj, xbar, wavg, meta,
([]…) table literals and keyed tables all mean what you expect. Bare qSQL works at the
prompt with no sel"…" wrapper.
The notation underneath is terse array grammar. Any two-argument function can be applied
infix (sz wavg px, 2 3 in 2 4 5) or with brackets
(wavg[sz;px]); three-argument functions like aj[c;x;y] use brackets.
There is no >= or <=, and symbols cannot
contain _ — because _ is a verb.
/ tables are first-class and render without `show`
t:([]sym:`AAPL`MSFT`AAPL; px:187.3 411.2 187.4; sz:100 250 50)
/ qSQL, typed bare at the prompt
select vwap:wavg[sz;px] by sym from t
/ 1-minute OHLCV bars — the classic tickerplant query
gentq 100000 / sets the globals: trades, quotes
tb:+@[+trades; ,`time; minbar[1]@]
qby[tb; `sym`time;
`o`h`l`c`v!({first x`px};{max x`px};{min x`px};{last x`px};{sum x`sz})]
/ TAQ: the prevailing quote for every print
m:aj[`sym`time; trades; quotes]
/ sorted attribute => O(log n) lookups
v:asc 2000000?1000000000
`at v / `s
v ? 12345 67890 / binary search, ~1244x faster
/ zero-copy Arrow C Data Interface, no libarrow linkage
p:arrow.export t / (schemaAddr; arrayAddr)A whole market, generated and stored
amber-tick writes a partitioned, splayed column store the engine reads natively — 38M prints, 89M quotes, ten-level book snapshots and LULD halts across five sessions and 500 names, from a simulator whose realism is checked: twenty-one published stylized facts, all passing on every generated store.
- arrivals
- discrete-time Hawkes process — bursts, not flat Poisson
- quotes first
- so a print can never fall outside the NBBO
- epoch
- nanoseconds since 2000-01-01 — Amber's own, not Unix
- files
- each column is one
-8!value, byte-identical to the engine's
Simulated tape: mean-reverting tick returns (bid-ask bounce) with clustered arrivals.
One engine, three seams
Everything that consumes Amber lives
outside the engine repository and reaches it through libamber.so, the in-process
ext/ registry, or a TCP socket. The engine gained one build flag, one export map and one
header section for all of it.
Measured against C, and against everyone else
Seven engines, four workloads, one specification. Every answer is
an integer that fits float64 exactly, so summation order cannot change it — the harness compares
answers bit-exactly against the C reference and prints WRONG instead of a
time for any engine that disagrees.
| Workload (10M rows unless noted) | C -O3 | Amber | Amber qSQL | CBQN | NumPy | DuckDB | ngn/k |
|---|---|---|---|---|---|---|---|
| Vector arithmetic + mask | 10.06 | 22.06 | 52.78 | 35.73 | 32.02 | 29.00 | 320.10 |
| Reductions — sum + max + dot | 28.27 | 13.46 | 69.08 | 5.62 | 10.21 | 38.00 | 299.92 |
| Group-by — 100 groups | 8.27 | 76.67 | 210.14 | 38.28 | 14.98 | 18.00 | 348.73 |
| Inner join — 1M × 1,000 sparse keys | 1.07 | 4.39 | 8.41 | 2.38 | 14.69 | 11.00 | 452.92 |
Median of 5 timed runs after 2 warm-ups, milliseconds, kernel time only; the 2.1.0 build, regenerated by CI (the group-by row still spells the aggregate with the pre-2.1 idiom and refreshes on the next run; the fused gsum path measures 21 ms on the scout matrix).
Amber is reported twice on purpose: array-primitive code is the fair peer of ngn/k and CBQN;
the qSQL layer is the fair peer of DuckDB's planner. The gap between the rows is the query
layer's overhead, and it is meant to be visible.
Full results →
Written down as it was built
Amber started as a 10,000-line C interpreter and became an engine one layer at a time. The build log follows the work in the order it happened, and the docs carry what came out of it — the cost model, the internals, and the failures worth knowing about.
The build log →
The interpreter underneath, teaching K to speak q, one byte at offset −13, down to the metal, the platform seams, and 2.0. In implementation order.
Writing fast Amber →
A primitive call costs 60–100 ns before it touches an element. The whole cost model follows from that, including a kernel that got 10× faster on layout alone.
Engine internals →
The C source map, how a primitive is dispatched and specialised by type, the 16 MB scratch arena, and where to look when something is slow.
Gotchas →
Seven ways Amber returns a plausible wrong answer instead of an error, and the ten-second check that catches each one.
Nothing here patches the engine
Each satellite reaches Amber through a published seam and nothing else. Delete any of them and the engine is byte-for-byte what it was.
python-amber
libamber.soThe engine in your Python process. pip install amber, then NumPy views
that point at the engine's own column buffers — no server, no socket, no serialisation.
amber-arrow
libamber.soAn ArrowArrayStream whose batches are windows onto one export,
not slices of it. Plus amberd, the TCP query server, and an Arrow Flight daemon.
amber-jupyter
python-amberAmber cells and %%python cells in one process, so a column crosses the
boundary as a pointer. HTML grids with type tags, capped at 500 rows on purpose.
amber-tick
store + runtimeA US-equity simulator, a splayed partitioned store, TAQ analytics, and a live tickerplant → RDB → HDB pipeline with a replayable sequenced log.
grafana-amber
amberd socketLive dashboards over bare qSQL, column-oriented on the wire. A Go backend, a
mage build, and time macros that speak Amber's epoch rather than Unix's.
vscode-amber
LSPqSQL-aware completion where the clause decides the vocabulary, hovers that explain
idioms like x@<x, and diagnostics that never evaluate your code.
amber-ai
ext/ seamA schema-aware co-pilot that reads your live tables before it answers, talks only to
127.0.0.1:11434, and cannot hang or break the REPL.
amber-flame
profilerA visual profiler: flamegraphs, Speedscope and Chrome tracing, built on the
engine's own \trace.
Tutorials
start hereThree end-to-end walkthroughs: a 5-million-row tick store, a Grafana dashboard pipeline, and Arrow record batches into Python without a copy.
One command. No install.
Amber compiles from source on first run and
installs nothing system-wide. There is no bin/, no QHOME, no dotfile —
deleting the folder uninstalls it completely.
git clone https://github.com/BonucciAndrea/amber.git
cd amber
chmod +x a build.sh install.sh
./a # builds with -O3, then opens the REPL