amber-jupyter

A Jupyter kernel for the Amber columnar time-series engine — with a zero-copy handoff to Python in the same process.

pip install amber-jupyter one process 18 tests over real ZMQ GitHub →

notebook

[1]  gentq 10000000

[2]  select vwap:wavg[sz;px], n:#px by sym from trades

[3]  %%python
     import numpy as np
     px = A.trades["px"]                                # NumPy view
     px.ctypes.data == A.raw("trades").column(2).handle()
     # True

A.trades["px"] is not a copy, not a serialisation, not a round-trip through a Parquet file. It is the memory the engine allocated when it built the column, and it stays valid for exactly as long as the array does.

Install

pip install amber-jupyter          # pulls in `amber` (the engine) and ipykernel
python -m amber_kernel.install --sys-prefix
jupyter lab

Then pick Amber in the launcher.

Use --sys-prefix inside a virtualenv or conda env

A user-level kernelspec points at whichever interpreter installed it, so a notebook opened from a different environment silently starts the wrong Python, imports a different (or no) amber, and fails in a way that has nothing to do with anything you typed. The spec pins sys.executable, so the kernel always starts the interpreter it was installed from.

python -m amber_kernel.install --uninstall removes it.

Design

Cells are Amber. `%%python` is the escape hatch.

The inverse — a Python kernel with an %%amber magic — was the other option, and it inverts the emphasis. In a notebook about market data, the Amber is the content and the Python is the tool you reach for at the edges. So Amber gets the unmarked case, Out is Amber-shaped, and the rich display is a data grid rather than a repr.

One process, not two

The modular design is a Python kernel and an Amber kernel talking over a socket or a shared file. It would be cleaner on a diagram and it throws away the only thing worth having: at 400 million rows, moving the data is the entire cost, and any design that moves it has already lost.

So there is one process, one engine, one heap, and the boundary between Amber and Python is a pointer. %%python cells run in the same interpreter that hosts the engine; A.trades["px"] goes through the buffer protocol, which pins the Amber refcount for as long as the NumPy array exists.

The `A` namespace

Expression	What it gives you
`A.trades`	the global, converted — a `Table` whose numeric columns are NumPy views
`A.raw("trades")`	the unconverted `Value`, for handing straight back to the engine
`A["signal"] = arr`	bind a global from NumPy, pandas, a list, a dict …
`A.q("select … from t")`	run a qSQL query from Python
`dir(A)`	every global currently defined
`am`	the whole `python-amber` module

A is a live proxy, not a snapshot: the engine's namespace changes as other cells run, and a proxy that cached would show you the notebook as of whenever it was built.

Magics

Magic	Effect
`%%python`	run the cell as Python, in this process. State persists across cells.
`%%amber`	explicit — cells are Amber by default
`%%time`	wall-clock the cell
`%load file.k`	load a script, as `\l` does at the REPL
`%ast EXPR`	the parse tree, as the engine's `\ast` prints it
`%disasm EXPR`	the compiled bytecode, as `\disasm` prints it
`%trace EXPR`	phase timings and the scratch arena's high-water mark

The three introspection magics print from C straight to file descriptor 1, so capturing them needs the descriptor redirected, not sys.stdout rebound — the C side never looks at Python's stdout object. There is a test for that, because it is the kind of thing that works in development and silently prints into the terminal in production.

Memory: `batch_rows`

A notebook cell that asks for four hundred million rows and gets them as one object has already lost. Reach for amber-arrow's streaming reader inside a %%python cell and set batch_rows to something your machine can hold:

%%python
import amber_arrow as aa

total = 0.0
for batch in aa.query("select from trades", batch_rows=1_000_000):
    total += batch.column("px").to_numpy().sum()     # one batch resident at a time
total

`batch_rows`	When to use it
`0` (default)	65,536 — the Arrow ecosystem's converged default. Per-batch overhead has disappeared and a handful of columns still fits in L2/L3.
`250_000 – 1_000_000`	Aggregating in NumPy or pandas per batch. Fewer Python-level iterations; still tens of megabytes per column at most.
whole table	Only when the result is small — a `by sym` aggregate, a mart, a day of bars. `t.to_pandas()` is the right call there.

Batches do not add memory pressure the way slices would

Every batch is a window onto one export, so raising batch_rows does not duplicate the data — it changes how much of the engine's existing buffer each Arrow object addresses and how many Python objects you create. The memory you save with a small batch_rows is whatever your loop body materialises, not the column itself.

Rendering

Amber's own table formatter (amfmt, the one the REPL uses) is beautiful in a terminal and wrong in a browser twice over: the ANSI colour codes render as literal escape sequences, and a fixed-width text block cannot be scrolled or read at 200,000 rows. So a notebook gets HTML, built to three rules:

Show the shape, not just the values. Every column header carries its Amber type tag — f float, j long, s symbol, t time. Key columns of a by result are marked.
Never render a million rows. A notebook that tries produces a 400 MB document that locks the browser tab. The grid caps its body at 500 rows and reports the true count in the footer, so the cap reads as a deliberate window rather than as the answer.
Work with the theme. Every colour is a --jp-* CSS variable with a literal fallback, so a table rendered under the light theme stays legible when you switch to dark without re-running the cell.

Every rich output also carries text/plain — the engine's own rendering, ANSI stripped — because notebooks get exported to scripts, diffed in git, and read over terminal clients, and in all three the HTML is useless.

Completion and inspection

Tab completion draws its candidates from the live engine namespace, not from a static vocabulary file — so a function you defined two cells ago completes, and so does a column of a table you just created. Shift-Tab on a name shows its type, its count, and its current value.

Try it

jupyter lab examples/quickstart.ipynb

examples/quickstart.ipynb is committed with its outputs, executed against a real kernel: two million synthetic trades, an as-of join against four million quotes, one-minute OHLCV bars, the zero-copy pointer check, a NumPy round-trip, a matplotlib chart, and the three introspection magics.

Tests

pip install -e ".[dev]"
python -m amber_kernel.install --sys-prefix
python -m pytest tests -q

18 tests, driven over ZMQ against a real kernel process rather than by calling do_execute in-process. Most of what goes wrong with a kernel — the kernelspec pointing at the wrong interpreter, the engine failing to boot in a fresh process, output landing on the wrong stream, a magic swallowing a reply — is invisible to a test that never starts a process.