From language to platform: -8!, one-report diagnostics, owning the terminal, and libamber.so

Six point releases, and between them they added almost no vocabulary. No new verbs worth mentioning, no new joins, nothing you would put on a feature list. What they added instead was everything that lets other software exist next to Amber without being inside it: a wire format, an error report you can act on, a terminal the REPL actually owns, a place to plug things in, and a shared library.

This is the stretch where Amber stopped being a language I was writing and became something other projects could be written against.

1.9.1 — the query layer was allocating ten million objects

The vector kernels from post four made one thing very visible: the query layer was now, by a wide margin, the slowest part of the system. A select … by … over ten million rows took 8.4 seconds. The native vector group-by over the very same data took 87 milliseconds.

The cause was one function. qby, xgroup and ij all grouped and probed through rows:{+. x} — flip the column dictionary into one boxed K value per row, so the rows can be hashed. Ten million rows meant ten million transient heap objects, allocated and refcounted and freed, purely so they could be compared and thrown away. 7.9 of the 8.4 seconds.

The fix is to hand the raw column vectors straight to the kernel's own vector = (group), ? (find) and @ (index) primitives, which is what the array language was for in the first place. Three pieces:

• qgrp[t;b] returns (key-value columns; group row-index vectors). A single-column by groups the column itself. A multi-column by is rank-encoded per column and mixed radix-style into one dense integer key — with a 2^53 cardinality guard that falls back to the old path rather than risk a collision. Key values are recovered by indexing the raw columns with each group's first row: one gather per by-column, instead of a flip over every row.
• ij rank-encodes both sides over one shared code space, so the probe is a native vector ? on a flat integer vector. Single-key joins skip the encoding entirely.
• qproj / qrefs narrow the table to the columns an aggregate can actually reach, before qby materialises per-group sub-tables. The cost now scales with columns used, not columns present.

The last column is the one I care about: after the change, select … by … sits within 1.1–1.5× of what you would get writing the array code by hand. The bottom two rows never touched qby, so their gap is qSQL's fixed per-call parse-and-compile cost — a per-query cost, not a per-row cost, and therefore not worth optimising.

1.9.2 — reductions, and an honest note about floats

The integer-find rewrite from this release is in post three, and the cache-line alignment fix is in post four. The third change is worth its own paragraph because of what it costs.

+/ over floats was a serialised v += p[i] chain running at about 3.5 cycles per element — the latency of a single addsd. The compiler is not allowed to reassociate it, because IEEE addition is not associative. So sumF now keeps four independent partial sums, which breaks the dependency chain and lets the vectoriser issue one wide add per group. Measured on 10M elements: +/ 8.9 → 6.3 ms, +/x*y 18.7 → 15.8 ms.

This changes results for inexact float data by one or two ulp. Always in the direction of more accuracy, since pairwise summation beats a left fold:

before:  10000.000000018848
after:    9999.999999995287
true:    10000.0

Exactly-representable data is bit-identical, as are 0n, 0w and -0w. It is the same trade-off the SIMD sum already made. But it is a semantic change, it is in the changelog as one, and anyone who needs strict left-fold semantics deserves to find that out from a release note rather than from a reconciliation break.

1.9.3 — -8! and -9!: a wire format

Until this release Amber's only wire format was text. `k rendered a value and . reparsed it. peach shipped every worker's result that way, which meant a multi-core map paid a full format-then-parse round trip per chunk — and, worse, any value whose printed form does not reparse to itself could not survive the trip at all. Attributes were lost. Some 0n / 0w edge cases were lost. Nested empties were lost.

src/ser.c is a compact binary encoder: -8!x to bytes, -9!y back, byte-exact.

b:-8!+`a`b!(1 2 3;4 5 6)        / table -> one contiguous byte vector
(-9!b)~+`a`b!(1 2 3;4 5 6)      / 1b — exact, attributes and nulls included

The format is four bytes of magic and version, then one recursive node. Packed atoms carry their value inline with no header at all. Heap types carry an attribute byte, a 64-bit count, and the raw payload. Because the payload is copied verbatim and the width is implied by the type tag, bit vectors (one bit per element) and every temporal width come out exact with no special cases — and nulls and infinities are preserved because they are just their bit patterns. Nothing is routed through a decimal formatter that could round.

Two design points that took thinking:

Symbols are written as names, not ids

Symbol ids are process-local. A forked peach worker can intern a symbol the parent has never seen, so shipping raw 32-bit ids would decode to the wrong name — or to garbage — in the parent. Every symbol goes over the wire as its name and is re-interned on the way in. Correct across processes and across separate runs, at the cost of the bytes.

-9! is a parser for untrusted input

So every read is bounds-checked against the input length, the recursion is depth-limited to 256, and a malformed or truncated buffer yields a clean 'domain error — never a read past the end and never a partially built object left un-freed. Lambdas and projections are deliberately not supported: serialising a closure means serialising its captured environment and its bytecode, which is a much larger feature than a data wire format, and nothing needs it. They raise a clean 'type rather than emitting bytes that would not decode.

Switching peach onto the binary wire also surfaced three bugs in its parent collection loop that the text path had been hiding: a per-chunk leak, an ignored worker exit status, and an unvalidated decode. All three fixed in the same release. Replacing a subsystem is the most reliable way I know to find out what was wrong with it.

1.9.4 — one error, one report

Amber had two error printers. The core's terse caret line, and — since 1.9 — a Rust-style visual diagnostic. Both fired. You got the report, then you got the legacy block underneath it, for every single error.

1.9.4 deleted the duplicate and made the good one universal. Every category — parse, type, domain, rank, length, undefined name — now has a code, a title, a token-spanning underline, an inline label and actionable help:

error[E0101]: Undefined variable `prices`
 --> <amber>:1:3
  |
1 | y:prices+1
  |   ^^^^^^ not found in this scope
  |
  = help: Verify that the variable is defined in the current scope or check for typos.

Underlines span tokens and name them, which is the difference between a caret that says "here" and a report that says "this name". The parser reports through the same path as everything else, so a syntax error and a runtime error look alike rather than coming from two different eras of the codebase.

And because the report is rendered when the error is created, code that catches an error with .[f;args;handler] would still print to stderr. For a test suite or a retry loop that provokes errors on purpose, that is noise. So it is switchable at run time:

prev:`diag 0                    / suppress; returns the previous setting
.[{1+`a};,0;{"caught"}]         / no output at all
`diag prev                      / restore

The compact 'type caret line is not suppressed — it is buffered and still handed to the trap handler and to `err, so a caught error can always be inspected. Quiet is not the same as gone.

1.9.5 — owning the terminal

For years the answer to "Amber has no line editing" was rlwrap. It worked, in the sense that arrow keys moved the cursor. It also did this, mid-session, usually right after an error:

rlwrap: warning: rlwrap appears to do nothing for amber, which asks for
single keypresses all the time. Don't you need --always-readline
and possibly --no-children? (cf. the rlwrap manpage)

dumped across stdout and stderr, followed by a garbled redraw from two editors fighting over one cursor. The diagnosis is exact and it is rlwrap's, not mine: rlwrap runs the wrapped program on a pty and speaks readline on its behalf, which only works for a program that reads whole lines in canonical mode. Amber puts the terminal into raw mode and reads single keypresses. rlwrap genuinely is doing nothing, and it says so.

The fix was to stop wrapping and own the terminal. src/ln.c is a single-file line editor in the linenoise tradition: raw termios, one visible line, ANSI refresh, plain C99 plus POSIX. About 700 lines. No readline, no curses, no terminfo, and nothing allocated on the keystroke path beyond the line buffer.

The Tab row is the one that changes how the REPL feels. Completion sources are lexical and live: your actual globals, and the actual columns of the actual table you are typing about. Not a static keyword list.

Two properties I made non-negotiable

Terminal state is restored on every exit path. Normal exit, Ctrl-C, error, signal. A REPL that leaves your terminal in raw mode when it dies is a REPL you stop trusting, and the failure is invisible until you notice your shell has stopped echoing.

Batch behaviour is unchanged, by construction. The editor degrades to a plain line read whenever stdin or stdout is not a terminal, so echo '2+2' | ./a, here-docs and CI runs behave byte for byte as they always did.

Both are asserted by tests/test_repl_term.py, seventeen tests driven over a real pty: that no rlwrap: diagnostic can reach a session on any path, that termios is byte-for-byte restored after a normal exit and after Ctrl-C, that the editing keys really edit, and that piped behaviour is unchanged. You cannot test a terminal without a terminal.

./a now execs the interpreter directly. The only path that still touches rlwrap is the deliberate AMBER_NO_EDIT=1 fallback for a dumb terminal or a screen reader — and there it is passed -n -a unconditionally, so no rlwrap diagnostic can reach your session on any path whatsoever.

1.9.5 — the extension seam

The same release added the first of Amber's two seams. ext/ is an empty directory in a stock checkout. An out-of-tree package installs itself by dropping .c files there and re-running ./build.sh; they compile with the same flags, link into the same binary, and register themselves from a constructor through the hooks in src/ext.h.

The reason it exists: pulling a new Amber release must never conflict with a package you installed, and a user who installs nothing must pay nothing. Every hook is a null pointer and every call site is a predictable branch. tests/ext_probe.c is a complete worked example and tests/test_ext_seam.sh installs it, checks every hook fires, uninstalls it, and checks the engine is back to stock.

Two of the hooks — am_ext_hint and am_ext_complete — only make sense because the editor is now mine. That is the shape of most of this post: each piece unlocks the next.

1.9.6 — libamber.so

The ext/ seam is in-process and compile-time. The last release added the out-of-process one: the same engine, built as a shared library, with a documented C API on the front of it.

./build.sh                 # ./amber          (unchanged; identical binary, zero new cost)
./build.sh --shared        # ./amber  +  libamber.so
./build.sh --shared-only   # libamber.so only

#include "ext.h"                       /* section 6 -- nothing else from src/ */

amber_init("/path/to/amber");          /* boots the engine, loads the .k stdlib */
amber_value t = amber_eval_qsql("select vwap:wavg[sz;px] by sym from trades");

int type; long long n; int bits;
const void *px = amber_get_vector_ptr(amber_table_column(t, 2), &type, &n, &bits);
/* `px` IS the engine's column payload. Not a copy of it. */

About sixty entry points, all named amber_*: boot and evaluate, reference counting, tables and dictionaries, constructors for pushing data back in, the Arrow C Data Interface, rendering, and amber_plugin_load. And amber_get_vector_ptr, which is the one that matters, because it returns the payload pointer that post one explained is already exactly the shape a foreign consumer wants.

Three decisions that were not obvious

The executable is untouched. ./build.sh with no flags produces byte-for-byte what it produced before. The shared library is a separate object set (-fPIC -Dshared): position-independent code and the global-dynamic TLS model that a dlopen'd library needs both change code generation, and sharing objects between the two would silently pessimise the binary everyone else uses.

Only amber_* and am_ext_* are exported. The engine's internal C is written in K-derived shorthand: its globals are called mr, su, us, err, run, add, sub, pk, cpl. Perfect inside one static binary. Actively dangerous inside a library loaded next to NumPy, libarrow and libpython. An export map (src/libamber.map) makes everything else genuinely absent from the dynamic symbol table — not private by convention, absent — so it cannot be bound to by accident and cannot interpose on a host's symbol of the same name.

The library records a SONAME. A satellite that dlopens a second copy of libamber.so gets a second engine: two heaps, two symbol tables, two global namespaces, and values from one that are meaningless to the other — with no error at any point, because nothing is technically wrong. The SONAME is what lets the loader recognise an already-loaded copy and reuse it.

The test that defines the API

tests/test_capi.c is the only consumer of libamber.so inside the engine repository, and it is written the way a satellite would write it: it includes src/ext.h and nothing else from src/, never dereferences an amber_value, and links the shared library rather than the objects.

If it ever needs a second -I, the API is wrong.

Seventy-eight assertions, run once at -O2 and once under AddressSanitizer and UndefinedBehaviorSanitizer with leak detection on — because a C API whose ownership rules are only documented is a C API whose ownership rules are wrong, and LeakSanitizer is what checks the prose.

What the seams made possible

Everything that consumes Amber now lives outside the engine repository:

The engine gained one build flag, one export map and one section of a header for all of it. There is no AI code and no network code anywhere in src/ — grep -r socket src/ finds only the client-side connect() that hopen has always used. A listener is a policy decision (which port, which framing, which authentication, which concurrency model) and the engine's job is to be a runtime, so amberd lives in amber-arrow where it can grow.

What I would tell myself at 1.0

Three things, and none of them are about performance.

The seam is the feature. Every satellite in that table exists because there was a documented place to attach it. None of them required a change to src/. That was not luck — it is the direct result of spending 1.9.5 and 1.9.6 building nothing users could see.

Replace a subsystem to find out what was wrong with it. Switching peach to the binary wire found three bugs that had been latent for four releases. Writing the disassembler is the only reason I know the compiler constant-folds. Rewriting the error path found the duplicate report that everyone had learned to scroll past.

Say what you did not do. The \hl command I deleted, the NEON path I have not run on real hardware, the fusion that 1.9.2 did not implement, the two shortcuts I found in my own benchmark suite and removed — every one of those is in the changelog with the reason. A project that only publishes its wins is a project whose numbers you cannot use.

That is the series. A 264-kilobyte interpreter, a q vocabulary, one byte at offset -13, five C kernels, and six point releases of plumbing. The engine is at github.com/BonucciAndrea/amber; if you want to type at it without installing anything, the browser scratchpad is the real C interpreter compiled to WebAssembly.