Wire protocol

The engine speaks sockets as a client — ipc.k's hopen/hsync, on the connect() in src/i.c — but has no listener, and should not grow one. A listener is a policy decision: which port, which framing, which authentication, which concurrency model. The engine's job is to be a runtime. So the listener lives outside it, in amberd.

Running the daemon

git clone https://github.com/BonucciAndrea/amber-arrow
cd amber-arrow
./build.sh --amberd

./build/amberd --home ../amber --port 5010 --eval "gentq 10000000" -v
./build/amberd --home ../amber --load /srv/hdb/load.k --mode jsonc --readonly

Framing

A request is one line of UTF-8 terminated by \n. A reply is a header line followed by exactly that many bytes.

+<nbytes>\n<payload>     success
-<nbytes>\n<payload>     error — the payload is the engine's own diagnostic text

That is the entire framing. There is no handshake, no version negotiation, no compression and no sequence number — the protocol is small enough to reimplement in an afternoon in any language, which is the point. The Grafana backend implements it in Go, the language server implements it in TypeScript, and amber_arrow.client.AmberClient is the reference implementation in Python.

Requests

The four reply encodings

Why jsonc exists

Amber stores a table as a dictionary of columns and a Grafana data frame is a set of columns. Serialising row-oriented JSON means transposing on the way out and transposing back on the way in — twice the work, twice the allocation, and a per-row object header for every one of a hundred thousand ticks. Column-oriented is not an optimisation here, it is the absence of a pessimisation.

{
  "sym":  ["AAPL", "MSFT", "GOOG"],
  "vwap": [187.2841, 411.1930, 141.2255],
  "n":    [184203, 92117, 51884]
}

Concurrency

A single-threaded event loop over poll(): many connections, one query at a time, shared global state.

That is not a compromise — it is the model kdb+ uses, and it is the only one where two clients can see each other's writes. Forking per connection would buy parallelism and lose the shared table you connected in order to query. A long query blocks the loop; that is visible, predictable, and fixed by running more amberd processes over a partitioned database, which is also what kdb+ does.

Clients

Python — the reference implementation

from amber_arrow.client import AmberClient

with AmberClient("localhost", 5010) as c:
    c.ping()
    rows  = c.json("select vwap:wavg[sz;px] by sym from trades")
    cols  = c.columns("select from trades where sym=`AAPL")
    frame = c.pandas("select n:#px by sym from trades")

Any language, from scratch

import socket

def query(host, port, src, mode="jsonc"):
    s = socket.create_connection((host, port))
    f = s.makefile("rwb")
    f.write(("\\mode %s\n" % mode).encode()); f.flush(); _read(f)
    f.write((src + "\n").encode()); f.flush()
    return _read(f)

def _read(f):
    head = f.readline()                      # b"+123\n" or b"-45\n"
    if not head:
        raise ConnectionError("amberd closed the connection")
    status, n = head[:1], int(head[1:])
    payload = f.read(n)                      # exactly n bytes — never guess
    if status == b"-":
        raise RuntimeError(payload.decode())
    return payload.decode()

Amber itself — ipc.k

The engine's own client is a raw-socket text protocol, and an in-process tickerplant sits beside it in the same module.

h:hopen `::5010                       / connect
hsync[h; "select from trades where sym=`AAPL"]
hsend[h; "t:1"]                       / fire and forget
hclose h

/ in-process pub/sub
u.def[`trades; cols trades]           / declare a table
u.sub[`trades; `]                     / subscribe (all symbols)
u.pub[`trades; batch]                 / publish a batch
u.end[date]                           / end of day

Streaming: Arrow Flight and ArrowArrayStream

The line protocol is right for dashboards and editors — small results, human-readable, trivially implementable. It is the wrong shape for four hundred million rows. For that, amber-arrow offers two streaming paths that never serialise to text at all:

import amber as am, amber_arrow as aa

am.gentq(400_000_000)
for batch in aa.query("select from trades", batch_rows=1_000_000):
    process(batch)            # 400 batches, 0 copies — each is a window on one export

python -m amber_flight --home ../amber --port 8815 --eval "gentq 10000000"

import pyarrow.flight as fl
c = fl.connect("grpc://localhost:8815")
t = c.do_get(fl.Ticket(b"select vwap:wavg[sz;px] by sym from trades")).read_all()

A Flight ticket is an Amber query. There is no query language of its own to learn and nothing to translate; bare qSQL works exactly as it does at the amber> prompt. See amber-arrow for the batching semantics and the lifetime rules.

Security

Testing against a real server

cd amber-arrow
./build.sh --amberd
python -m pytest tests -q        # 31 tests, against a real amberd on a real socket

The suite covers the full protocol — including that two connections see each other's writes, which is the property the single-threaded shared-state model exists to provide. The Go backend has its own 26 tests against a real amberd, in pkg/amberd, written with nothing but the standard library so a protocol bug reproduces without a Grafana instance anywhere near it.