Benchmarks

Every number on this page comes out of bench/run_comparative.py, which CI re-runs on every push. The workloads, the data model and the fairness rules are specified once in bench/SPEC.md; every engine implements that document and nothing else.

Correctness gate

Every answer in the suite is an integer that fits in float64 exactly, and every sum is over such integers, so the result is independent of summation order — SIMD pairwise, Kahan-compensated and naive left-fold summation all produce the identical bit pattern. The harness therefore compares answers exactly against the C reference and prints WRONG in place of a time for any engine that disagrees. Each engine also emits a checksum of its input data, so a divergence in the generator is caught separately (BADDATA). You cannot win a cell in this table by computing something cheaper.

Cross-engine results

Median of 5 timed runs after 2 warm-up passes, milliseconds. Kernel time only — process startup is excluded.

Benchmark	C (-O3)	Amber	Amber qSQL	ngn/k	CBQN	J	Uiua	NumPy	Julia	DuckDB
Vector arithmetic + mask — `sum((x*2.5)+y where x>50)`, 10M	8.22	54.56	65.73	254.82	30.91	109.90	795.16	27.47	7.82	22.00
Reductions — `sum + max + dot`, 10M elements	21.92	15.40	52.46	237.93	4.09	102.74	738.85	8.62	21.98	29.00
Group-by aggregation — 100 groups over 10M rows	6.41	61.45	119.63	276.51	28.83	136.47	1,545.94	11.54	5.69	14.00
Inner join — 1M left rows against 1,000 sparse keys	0.83	3.42	9.85	364.13	1.80	93.46	735.79	11.34	10.60	9.00

Relative to the C baseline

Lower is better; 1.00× means it matched plain C.

Benchmark	C	Amber	Amber qSQL	ngn/k	CBQN	NumPy	Julia	DuckDB
Vector arithmetic + mask	1.00×	6.64×	8.00×	31.01×	3.76×	3.34×	0.95×	2.68×
Reductions	1.00×	0.70×	2.39×	10.86×	0.19×	0.39×	1.00×	1.32×
Group-by aggregation	1.00×	9.59×	18.67×	43.15×	4.50×	1.80×	0.89×	2.18×
Inner join	1.00×	4.13×	11.88×	439.40×	2.17×	13.68×	12.79×	10.86×

Amber is reported twice, on purpose

Amber is array-primitive code — the fair peer of ngn/k, CBQN, J and Uiua. Amber qSQL routes the same workloads through the select … by … from layer — the fair peer of DuckDB's SQL planner. Publishing only the faster of the two would mean picking whichever comparison flatters Amber; the gap between the rows is the query layer's overhead and it is meant to be visible.

Timing mode per engine

kernel means the engine timed its own kernel with a monotonic clock; net means it has no usable in-language clock and was measured as total process time − a measured startup baseline. The table labels which mode produced each cell, so the two are never silently mixed.

Engine	Peer group	Mode	Startup baseline
C (-O3)	baseline	kernel	—
Amber	array primitives	kernel	—
Amber qSQL	query layer	kernel	—
ngn/k	array primitives	net	1.56 ms
CBQN	array primitives	kernel	3.02 ms
J	array primitives	net	44.07 ms
Uiua	array primitives	net	9.38 ms
NumPy	array primitives	kernel	—
Julia	scalar loops (JIT)	kernel	—
DuckDB	query layer	kernel	12.20 ms

Two shortcuts that were removed

Both were in the previous suite, both are documented in SPEC.md, and both are the kind of thing a benchmark table quietly relies on unless someone goes looking.

+/!10000000 is O(1) in Amber. src/3.c's arf constant-folds a sum over a range into the closed form n(n-1)/2. The old vecsum benchmark was exactly that expression, so Amber "won" it by never touching 10M elements while every other engine ran a real reduction. All data is now materialised before the clock starts.
A dense-key "join" is just an array index. With right keys 0..K-1, every array language answers the join with a single gather while DuckDB still builds a hash table. Right keys are now sparse and unsorted, forcing a genuine key lookup everywhere.

Attributes — why they matter

bench.k measures ? (find) on identical data, sorted-attributed versus not. Results are identical; only the time differs.

rows	linear scan	binary (`s)	speedup
100 k	87 ms	0.6 ms	141×
500 k	417 ms	0.9 ms	470×
2 M	1.73 s	1.4 ms	1244×
5 M	4.23 s	1.9 ms	2261×

Self-benchmarks — what each release moved

Release	Change	Effect
1.9.1	the `select … by … from` layer groups and probes on raw column vectors instead of boxing one K object per row	group-by 24.7× inner join 19.3×
1.9.2	integer `?` (find) builds an index over its left argument instead of scanning it per probe	180.95 ms → 5.66 ms 32×
1.9.2	float `+/` uses four independent accumulators so it vectorises; array payloads are cache-line aligned	—
1.9.3	`peach` ships worker results over the `-8!` binary wire instead of formatting and reparsing text	—
1.9.5	sliding windows + radix sort	—

After 1.9.1 both group-by and inner join sit within ~1.1–1.5× of hand-written Amber array code.

A real HFT run

demo/hft_demo.k, on 500,000 trades and a million quotes. Row counts and timings vary by machine — the script prints its own table every time.

./demo.sh

== benchmark summary =======================================
stage      ms
----------
gentq   2381.9
vwap     151.2
ema        0.1
asof     704.0
------------------------------------------------------------
total: ~3.2 s end-to-end for 500,000 trades

Running them yourself

./amber bench.k                  # attribute / find speedups
./amber bench-fin.k              # the finance module
./amber bench-std.k              # moving windows, sorts
bash bench/run.sh                # cross-engine sanity + speed vs numpy / pandas / polars / duckdb
python bench/run_comparative.py  # the ten-engine CI table

The per-engine query files live in bench/queries/ and are separate, independently tuned scripts — not the same file reused. Each amber_*.k documents in its header what optimisation was tried, what was measured, and why.