Tutorial 03 · ~20 minutes

Interoperability: passing Arrow record batches from Amber to Python without copying

"Zero-copy" is a claim, and claims are cheap. This walkthrough ends with you asserting pointer equality between an Amber column, a NumPy array and an Arrow buffer — and then streaming four hundred million rows through Python with one of them resident at a time.

Setup

git clone https://github.com/BonucciAndrea/amber
git clone https://github.com/BonucciAndrea/python-amber
git clone https://github.com/BonucciAndrea/amber-arrow

(cd amber && chmod +x build.sh a && ./build.sh --shared)

python -m venv .venv && . .venv/bin/activate
AMBER_SRC=$PWD/amber pip install ./python-amber
pip install numpy pyarrow pandas
(cd amber-arrow && ./build.sh && pip install ./python)

Check you have one engine, not two

If the loader satisfies libamber_arrow.so's dependency with a second copy of libamber.so from elsewhere on the search path, you get a second engine — two heaps, two symbol tables, and handles from one that are meaningless in the other, with no error at any point.

import amber_arrow as aa
print(aa.library_path())     # tells you which libamber.so you actually got

Get a column and look at its address

import numpy as np
import amber as am

am.gentq(10_000_000)                       # 10M trades, 20M quotes — in-process

t   = am.q("select from trades")
raw = t.column("px", raw=True)             # an owned handle, unconverted
arr = np.asarray(raw)                      # PEP 3118 buffer protocol

print(arr.dtype, arr.shape)                # float64 (10000000,)
assert arr.ctypes.data == raw.handle()     # <- the engine's own buffer

raw=True skips conversion and hands back an am.Value. Conversion is the only step in the whole path that costs anything, so skip it whenever the next thing you do is hand the result back to the engine or wrap it in an array.

Understand why that is safe

amber_get_vector_ptr() returns the payload pointer. The buffer protocol wraps it with view->obj set to the owning Value, INCREF'd. NumPy stores that as the array's base:

arr.base is raw            # True — the Amber refcount cannot reach zero
del raw                    # the Value object goes, the refcount does not
arr[:5]                    # still valid

The same guarantee, through Arrow

import pyarrow as pa

at  = t.to_arrow()                                    # pyarrow.Table
buf = at.column("px").chunk(0).buffers()[1]
assert buf.address == np.asarray(t.column("px", raw=True)).ctypes.data

The ArrowArray's data buffer is the Amber column's payload, and its release callback drops the Amber refcount.

Symbol columns are the one exception

Amber stores interned int32 ids; Arrow's utf8 layout wants offsets and bytes. No pointer arithmetic bridges that, so symbol columns are materialised. Numeric columns — px, sz, time, bid, ask — are where the volume is, and they cross for free.

Stream a table you could not hold

import amber as am, amber_arrow as aa

am.gentq(400_000_000)                                    # 400M trades

total = 0.0
for batch in aa.query("select from trades", batch_rows=1_000_000):
    px = batch.column("px").to_numpy(zero_copy_only=True)
    total += float(px.sum())                             # 400 batches, 0 copies
print(total)

Every batch above points at the same engine-owned column buffers. Prove it:

>>> {b.column("px").buffers()[1].address for b in aa.stream(t, batch_rows=25_000)}
{140234099785792}                                        # one address, 80 batches
>>> _.pop() == am.Table(t).column("px", raw=True).handle()
True                                                     # the engine's buffer

Why a batch is a window, not a slice

The obvious way to stream a table in batches is to cut it into pieces. That copies every byte, once per stream — for a 400M-row table, that is the cost of the operation, and it defeats the point of using Arrow at all.

So the library never cuts anything. It exports the table once, and every batch is a set of ArrowArray structs whose buffers pointers are the parent's, with offset and length moved. Arrow's format defines exactly those semantics for primitive and utf8 layouts alike, so there is no per-type special case.

The one rule you have to know: batch 3 may outlive the stream

And batches 1 and 2 may be released before batch 3 is created. So the parent export is not owned by the stream. It is owned by a reference count that the stream holds one of, and each live batch holds one of, and it is torn down by whoever drops the last one.

Practically: you may keep a batch, put it in a list, return it from a function, or hand it to another thread, and the underlying memory stays valid. There is a test for exactly that pattern, because it is the one that would otherwise be a use-after-free.

keep = []
reader = aa.query("select from trades", batch_rows=250_000)
for i, b in enumerate(reader):
    if i == 3:
        keep.append(b)          # deliberately outlives the loop and the reader
del reader
keep[0].column("px").to_numpy(zero_copy_only=True)[:5]     # still valid

Choosing `batch_rows`

Value	Effect
`0` (default)	65,536 — the figure the Arrow ecosystem has converged on: large enough that per-batch overhead disappears, small enough that a handful of columns stays inside L2/L3.
`250k – 1M`	Fewer Python-level iterations when your loop body is doing real work per batch.
whole table	`aa.table_to_arrow(t)`. Right for a `by sym` aggregate or a day of bars; wrong for the tape.

Raising batch_rows does not duplicate data — it changes how much of the engine's existing buffer each Arrow object addresses. What you save with a small value is whatever your loop body materialises.

Handing the result to Polars or DuckDB

import polars as pl, duckdb

df = t.to_polars()                                   # via Arrow, numerics zero-copy
duckdb.sql("select sym, avg(px) from at group by 1")  # `at` is the pyarrow.Table

Both consume the Arrow C Data Interface, so neither of them copies the numeric columns either. Nothing in this path links libarrow — src/ar.c is about ninety lines of C producing the standard structs, and libamber_arrow.so adds the stream on top.

Going the other way

am.set("signal", np.random.randn(1_000_000))         # copies — always
am.eval("avg signal")

t2 = aa.arrow_to_table(at)                           # copies — always

Inbound always copies, and that is not a bug being worked around

Amber stores a 32-byte object header immediately before every vector payload, so it cannot adopt a foreign allocation as a native column. The asymmetry matches where the volume is: query results are large and query parameters are small.

Fix the epoch before you plot

Amber's timestamps count nanoseconds from 2000-01-01. pandas and Arrow count from 1970. Nothing converts for you, on purpose — a silent thirty-year shift is worse than an honest integer.

import pandas as pd

AMBER_EPOCH_NS = 946_684_800_000_000_000

ns  = np.asarray(t.column("time", raw=True))
idx = pd.to_datetime(ns + AMBER_EPOCH_NS, utc=True)
px  = np.asarray(t.column("px", raw=True))
pd.Series(px, index=idx).resample("1min").ohlc()

Full details on Temporal mechanics.

Over the network: Arrow Flight

Same batches, same absence of copying, now over gRPC.

python -m amber_flight --home ../amber --port 8815 --eval "gentq 10000000"

import pyarrow.flight as fl
c = fl.connect("grpc://localhost:8815")

for info in c.list_flights():        # every global that IS a table, with real schemas
    print(info.descriptor.path, info.total_records)

t = c.do_get(fl.Ticket(b"select vwap:wavg[sz;px] by sym from trades")).read_all()

A ticket is an Amber query. The default embedded backend runs the engine in the Flight server's own process, so a batch handed to gRPC points at the engine's column buffers — the result is never copied between evaluation and the socket. The remote backend fronts an amberd on another host and does copy and parse; reach for it only when the engine genuinely is somewhere else.

Verify the whole thing

cd amber-arrow && python -m pytest tests -q        # 31 tests
cd ../python-amber && python -m pytest tests -q

The suites assert pointer equality, not value equality — that every batch shares one buffer address, that the address is the engine's, that a batch survives its stream's release, and that the schema handed to a consumer is independent of the stream.