Wire protocol

This page is amberd: a line protocol for dashboards, editors and anything that wants JSON back, with its own framing and concurrency. Since 2.9.0 the engine also speaks q's wire format itself, both ways: \p, hopen, the .z handlers, .z.pw. That's on talking to q.

Running the daemon

git clone https://github.com/BonucciAndrea/amber-arrow
cd amber-arrow
./build.sh --amberd

./build/amberd --home ../amber --port 5010 --eval "gentq 10000000" -v
./build/amberd --home ../amber --load /srv/hdb/load.k --mode jsonc --readonly

Framing

A request is one line of UTF-8 terminated by \n. A reply is a header line followed by exactly that many bytes.

+<nbytes>\n<payload>     success
-<nbytes>\n<payload>     error — the payload is the engine's own diagnostic text

That is the entire framing. There is no handshake, no version negotiation, no compression and no sequence number. Anyone can reimplement it in an afternoon in any language, which is the idea. The Grafana backend implements it in Go, the language server implements it in TypeScript, and amber_arrow.client.AmberClient is the reference implementation in Python.

Requests

The four reply encodings

Why jsonc exists

Amber stores a table as a dictionary of columns and a Grafana data frame is a set of columns. Serialising row-oriented JSON means transposing on the way out and transposing back on the way in: twice the work, twice the allocation, and a per-row object header for every one of a hundred thousand ticks. Sending columns is not an optimisation. It is the absence of a pessimisation.

{
  "sym":  ["AAPL", "MSFT", "GOOG"],
  "vwap": [187.2841, 411.1930, 141.2255],
  "n":    [184203, 92117, 51884]
}

Concurrency

A single-threaded event loop over poll(): many connections, one query at a time, shared global state.

Nothing was compromised to get there. Shared state is the only model where two clients can see each other's writes. Forking per connection would buy parallelism and lose the shared table you connected in order to query. A long query blocks the loop; that is visible, predictable, and fixed by running more amberd processes over a partitioned database, which is also what q does.

Clients

Python: the reference implementation

from amber_arrow.client import AmberClient

with AmberClient("localhost", 5010) as c:
    c.ping()
    rows  = c.json("select vwap:wavg[sz;px] by sym from trades")
    cols  = c.columns("select from trades where sym=`AAPL")
    frame = c.pandas("select n:#px by sym from trades")

Any language, from scratch

import socket

def query(host, port, src, mode="jsonc"):
    s = socket.create_connection((host, port))
    f = s.makefile("rwb")
    f.write(("\\mode %s\n" % mode).encode()); f.flush(); _read(f)
    f.write((src + "\n").encode()); f.flush()
    return _read(f)

def _read(f):
    head = f.readline()                      # b"+123\n" or b"-45\n"
    if not head:
        raise ConnectionError("amberd closed the connection")
    status, n = head[:1], int(head[1:])
    payload = f.read(n)                      # exactly n bytes — never guess
    if status == b"-":
        raise RuntimeError(payload.decode())
    return payload.decode()

Amber itself

From Amber, talk to another Amber or to q with q's own hopen (talking to q). ipc.k also has an in-process pub/sub that needs no server at all:

u.def[`trade; ([]sym:0#`; px:0#0.0)]  / declare a table by its schema
u.sub[`trade; {[nm;d]`0:"got ",($#d)," rows\n"}]   / subscribe: a callback
u.pub[`trade; ([]sym:`a`b; px:10 20.0)]            / publish a batch
u.end[`trade]                         / end-of-day snapshot: the accumulated table

Streaming: Arrow Flight and ArrowArrayStream

The line protocol is right for dashboards and editors: small results, human-readable, trivially implementable. It is the wrong shape for four hundred million rows. For that, amber-arrow offers two streaming paths that never serialise to text at all:

import amber as am, amber_arrow as aa

am.gentq(400_000_000)
for batch in aa.query("select from trades", batch_rows=1_000_000):
    process(batch)            # 400 batches, 0 copies — each is a window on one export

python -m amber_flight --home ../amber --port 8815 --eval "gentq 10000000"

import pyarrow.flight as fl
c = fl.connect("grpc://localhost:8815")
t = c.do_get(fl.Ticket(b"select vwap:wavg[sz;px] by sym from trades")).read_all()

A Flight ticket is an Amber query. There is no query language of its own to learn and nothing to translate; bare qSQL works exactly as it does at the amber> prompt. See amber-arrow for the batching semantics and the lifetime rules.

Security

Testing against a real server

cd amber-arrow
./build.sh --amberd
python -m pytest tests -q        # 31 tests, against a real amberd on a real socket

The suite covers the full protocol, including that two connections see each other's writes, which is the property the single-threaded shared-state model exists to provide. The Go backend has its own 26 tests against a real amberd, in pkg/amberd, written with nothing but the standard library so a protocol bug reproduces without a Grafana instance anywhere near it.