Zero-copy interop: bridging Amber memory buffers directly into Apache Arrow & Python

"Zero copy" is the cheapest claim in data infrastructure. Almost every system says it, and almost every system means something slightly different by it — usually "we avoid one copy that a naive implementation would make", which is fine but is not what the phrase says.

So here is a falsifiable version. If a query result crosses from Amber into NumPy without a copy, then the address NumPy reports and the address the engine allocated are the same integer. Not similar. The same. And you can print both.

>>> col = am.q("select from trades").column("px", raw=True)
>>> arr = numpy.asarray(col)
>>> hex(arr.ctypes.data) == hex(col.handle())
True

That assertion is in the test suite, along with its Arrow equivalent:

assert arr.ctypes.data      == raw.handle()
assert arrow_buffer.address == raw.handle()

The suites assert pointer equality, not value equality, because value equality is what a copy also satisfies. This post is about what it takes to make that assertion true and keep it safe.

Why this matters more than it sounds like it does

At small sizes it does not matter. Copying a thousand doubles is eight kilobytes and a memcpy; nobody will ever notice.

At four hundred million rows it is the entire operation. A single float64 column of 400M rows is 3.2 GB. Copying it once costs you the memory bandwidth to read 3.2 GB and write 3.2 GB, plus 3.2 GB of resident memory you now hold twice. Do that for four columns and you have turned a query into a memory-pressure event.

The relevant question is not "how fast is the copy". It is "does the pipeline have a step whose cost scales with the data and produces nothing". Serialisation is such a step. So is transposition. So is parsing. A design that includes one has already decided its ceiling.

The three boundaries

Amber crosses into three foreign worlds, and they are not equally easy.

The asymmetry in the third row is the honest part, and I will come back to it.

Boundary one: the buffer protocol

Python's buffer protocol is the mechanism by which one object exposes raw memory to another without either knowing about the other's type. NumPy consumes it; so does memoryview; so does pyarrow.py_buffer.

The engine side is one call:

const void *px = amber_get_vector_ptr(column, &type, &n, &bits);

That returns the payload pointer — the actual bytes the engine wrote when it evaluated the query. The extension module fills in a Py_buffer pointing at it and, critically, sets view->obj to the owning Value object with an INCREF.

That one field is the whole safety story. NumPy stores it as the array's base:

arr.base is raw            # True
del raw                    # the Python name goes; the refcount does not reach zero
arr[:5]                    # still valid — the engine value is pinned by the array

So the lifetime rule is: the engine's buffer lives exactly as long as anything Python is holding that points into it. You cannot get a dangling array by dropping the handle, because the array is a handle.

Boundary two: the Arrow C Data Interface

Arrow's C Data Interface is a pair of structs — ArrowSchema and ArrowArray — with a documented memory layout and a release callback. Two libraries that both understand those structs can hand data to each other without either linking the other.

The engine implements the export in about ninety lines, in src/ar.c, with no libarrow linkage at all:

p:arrow.export t                    / table  -> (schemaAddr; arrayAddr)
arrow.import p                      / and back

The ArrowArray's data buffer is the Amber column's payload, and the release callback drops the Amber refcount. Same discipline as the buffer protocol, different vocabulary.

That is exactly the right amount of Arrow for a ten-thousand-line C runtime to own. It answers "hand me this table" completely, and it stops there deliberately.

The one column type that cannot be zero-copy

Symbols. Amber stores a symbol column as interned int32 ids into a global symbol table — four bytes a row, integer comparisons, which is the reason grouping and joining on sym is fast. Arrow's utf8 layout wants an offsets buffer and a bytes buffer.

There is no pointer arithmetic that turns one into the other. So symbol columns are materialised, and the documentation says so rather than quietly folding it into an average. It matters less than it sounds: in a tick table the symbol column is one of nine, and the numeric columns are where the volume is.

Boundary three: the direction that always copies

Amber stores a 32-byte object header immediately before every vector payload. Given a pointer to a column's data, the engine finds its type, count, refcount and attribute by walking backwards.

That layout is why the outbound direction is free — the payload is already exactly what a consumer wants, contiguous and headerless from the consumer's point of view. And it is why the inbound direction cannot be: a NumPy array allocated by NumPy has no thirty-two bytes of Amber header in front of it, and there is no way to retrofit them without moving the data.

am.set("signal", np.random.randn(1_000_000))     # copies. always.

This is not a limitation anyone is working around. It matches where the volume is: query results are large, query parameters are small. A design that made inbound free at the cost of making outbound expensive would be strictly worse for every workload anyone actually has.

Streaming: the problem the engine deliberately does not solve

One export gives you the whole table. That is the right answer until the table is 400 million rows, at which point the consumer needs batches — and a Flight server, a Polars scan and anything with a memory budget all need exactly that.

The obvious implementation is to cut the table into pieces. That copies every byte, once per stream. For a 400M-row table that is the cost of the operation, and it defeats the point of using Arrow at all.

So amber-arrow never cuts anything. It exports the table once, and every batch is a set of ArrowArray structs whose buffers pointers are the parent's, with offset and length moved. Arrow's format defines exactly those semantics for primitive and utf8 layouts alike, so there is no per-type special case to get wrong.

You can watch it happen:

>>> {b.column("px").buffers()[1].address for b in aa.stream(t, batch_rows=25_000)}
{140234099785792}                                        # one address, 80 batches
>>> _.pop() == am.Table(t).column("px", raw=True).handle()
True                                                     # the engine's buffer

Eighty batches. One address. The set has one element because every batch is looking at the same memory.

The lifetime problem this creates

Batch 3 may outlive the stream, and batches 1 and 2 may be released before batch 3 is created.

That sentence is the whole design constraint. A consumer is entitled to keep a batch, put it in a list, return it from a function, or hand it to another thread — Arrow's contract says a RecordBatch is a value, and values do not stop being valid because you closed something else.

So the parent export is not owned by the stream. It is owned by a reference count. The stream holds one. Each live batch holds one. Whoever drops the last one tears it down.

keep = []
reader = aa.query("select from trades", batch_rows=250_000)
for i, b in enumerate(reader):
    if i == 3:
        keep.append(b)          # deliberately outlives the loop and the reader
del reader
keep[0].column("px").to_numpy(zero_copy_only=True)[:5]     # still valid

There is a test for exactly that pattern, because it is the one that would otherwise be a use-after-free, and a use-after-free that only fires when a consumer holds a batch a little too long is the worst kind: intermittent, workload-dependent, and impossible to attribute.

The failure mode nobody warns you about: two engines

This one cost real debugging time and is worth stating loudly.

libamber_arrow.so links libamber.so. So does the amber Python package's extension module. If the dynamic loader satisfies those two dependencies with different copies of the library, you get two engines in one process: two heaps, two symbol tables, two global namespaces.

Nothing errors. Nothing can error, because nothing is technically wrong — both libraries loaded, both initialised, both work. What happens instead is that a handle produced by one is meaningless to the other, and you get a segfault or, worse, plausible garbage, at a call site that has nothing to do with the actual problem.

Two mechanisms prevent it:

• The library records a SONAME. That is what lets the loader recognise an already-loaded copy and reuse it rather than mapping a second one.
• amber_arrow loads libamber.so from inside the installed amber package, with RTLD_GLOBAL, before it loads libamber_arrow.so. Getting the right one in first is this module's actual job.

And one diagnostic: amber_arrow.library_path() tells you which one you got. Call it first when anything is strange.

Why only amber_* is exported

Amber's internal C is written in a terse K-derived idiom. The engine's own globals are called mr, su, us, err, run, add, sub, pk, cpl.

Inside one static binary those names are perfect — short, consistent, and the compiler resolves them all at link time. Inside a shared library loaded next to NumPy, libarrow and libpython, a global symbol called add is a loaded gun.

So src/libamber.map is an export map that makes everything except amber_* and am_ext_* genuinely absent from the dynamic symbol table. Not private by convention — absent. It cannot be bound to by accident and it cannot interpose on a host's symbol of the same name.

There is a second, subtler consequence of building a dlopen-able library. The engine's allocator uses initial-exec TLS on its hot path, which is the fastest model and requires the loader to size the static TLS block before main(). A dlopen'd library cannot use it — it would have to borrow from glibc's small surplus reserve, which fails nondeterministically, with cannot allocate memory in static TLS block, depending on what else the host imported first. So the shared build drops to global-dynamic TLS, and the executable keeps initial-exec, and the two are compiled as separate object sets so neither pessimises the other.

Where the copies actually are, for real

An honest inventory of a full pipeline — engine to Grafana panel, engine to Polars, engine to a Flight client:

The remote Flight backend is in that table on purpose. It is slower and it says so: "sometimes the engine genuinely is somewhere else — a tickerplant, an HDB node — and a slower path beats no path. Do not reach for it when the engine could have been in-process."

The one conversion that is not free and should not be automatic

Amber timestamps are nanoseconds since 2000-01-01. pandas and Arrow count from 1970. Table.from_pandas maps datetime64 columns to int64 nanoseconds without re-basing the epoch.

That is a deliberate refusal. Silently shifting your data by thirty years is worse than handing you an integer and making you say what it means. The constant is 946_684_800_000_000_000; add it going out, subtract it coming in, and put it in a named variable so the next person can see it.

The libraries: python-amber and amber-arrow. The step-by-step version of this post is the Arrow interop tutorial; the C API behind it is documented on the C API page.