Amberv1.9.6
Home / Tutorials / Tick store
Tutorial 01 · ~15 minutes

Building a 5-million-row in-memory tick store in 60 seconds

By the end of this you will have five million trades and ten million quotes in memory, joined as-of, bucketed into one-minute bars, and you will know exactly how much RAM it cost.

What you need

git clone https://github.com/BonucciAndrea/amber && cd amber
chmod +x a build.sh
AMBER_NATIVE=1 ./a           # build machine-tuned, then open the REPL

Everything below is typed at the amber> prompt.

1

Generate the session

gentq is in fin.k, auto-loaded at startup. It sets two globals: trades and quotes, with realistic microstructure — an efficient price, a spread state, and prints that sit inside the standing quote.

amber>
\ts gentq 5000000
2381 1258291200        / milliseconds, bytes

\ts times an expression and reports the space it used. About 2.4 seconds and 1.2 GB for five million trades and ten million quotes.

2

Look at what you got

amber>
count trades                       / 5000000
count quotes                       / 10000000
meta trades
c     | t a
------|-----
sym   | s
time  | j
px    | f
sz    | i

The a column is empty — no attributes yet. That is the single most important thing to fix, and it is the next step.

3

Sort and stamp the attributes

The as-of kernel binary-searches each symbol group's time slice. For that to be legal the rows must be sorted by (sym, time) and time must carry the `s attribute. xasc does both.

amber>
trades:xasc[`sym`time; trades]
quotes:xasc[`sym`time; quotes]
meta trades
c     | t a
------|-----
sym   | s p
time  | j s
px    | f
sz    | i

sym picked up `p (parted — equal values are contiguous) and time picked up `s (sorted). Together those are what make the join O(log n) per row instead of O(n).

Want O(1) symbol slicing too?

Add the grouped attribute and the group index: `ga trades`sym, then bysym / symrows / gidx from fin.k give you a per-symbol row range without a scan.

4

The as-of join

Every trade gets the quote that was standing when it printed. This is TAQ, and it is the operation the whole layout exists to serve.

amber>
\ts m:aj[`sym`time; trades; quotes]
count m                            / 5000000
5#m
704 402653184

sym  time                px      sz  bid     ask
-------------------------------------------------
AAPL 34200000000000      187.31  100 187.30  187.32
AAPL 34200004120000      187.31  200 187.30  187.32
AAPL 34200011870000      187.32   50 187.31  187.33
AAPL 34200019330000      187.32  300 187.31  187.33
AAPL 34200026010000      187.31  100 187.30  187.33

704 milliseconds for five million binary searches over a ten-million-row quote table. The kernel is in src/a.c, marshalled from amber.k, and it runs off a thread-local 16 MB bump arena so malloc jitter stays out of the path.

5

Microstructure columns

amber>
m:update mid:0.5*bid+ask from m
m:update side:tsign m from m               / Lee-Ready: +1 buy, -1 sell
m:update eff:effspread m from m            / 2 * |px - mid|
select n:#px, eff:avg eff, bps:10000*avg eff%mid by sym from m
sym  | n       eff       bps
-----|---------------------------
AAPL | 1284203 0.0121    0.646
MSFT | 1091177 0.0184    0.447
GOOG |  918884 0.0102    0.722
...
6

One-minute VWAP and OHLCV bars

The headline query, bare at the prompt:

amber>
\ts select vwap:wavg[sz;px] by time:1m xbar time from trades
151 8388608

time         vwap
---------------------
09:30:00.000 187.2841
09:31:00.000 187.3106
09:32:00.000 187.2955
09:33:00.000 187.4012
..
[390 rows x 2 cols]

And the full OHLCV, spelled out with qby so you can see the shape:

amber>
tb: +@[+trades; ,`time; minbar[1]@]        / snap time onto 1-minute buckets
b: qby[tb; `sym`time;
       `o`h`l`c`v!({first x`px};{max x`px};{min x`px};{last x`px};{sum x`sz})]
5#b

Or, if you just want the standard bars, fin.k has it in one call:

amber>
b:bars[1; trades]                          / 1-minute OHLCV
candle b                                   / Unicode candlesticks, in colour
7

Measure the footprint

amber>
\v
+-------------+---------------+----------------+---------+
| Name        | Type          | Shape / Length | Memory  |
+-------------+---------------+----------------+---------+
| trades      | Table         | 5000000 x 4    | 120.0 MB|
| quotes      | Table         | 10000000 x 4   | 320.0 MB|
| m           | Table         | 5000000 x 8    | 280.0 MB|
| b           | Table         | 390 x 6        | 18.7 KB |
| ...         | ...           | ...            | ...     |
+-------------+---------------+----------------+---------+

\v walks nested structures recursively, so a table's number is real rather than a shallow guess. It also lists the REPL library's own internals (repl.*, PAL, GB) — scan for the names you defined, or \d yourns first to narrow the namespace.

8

Profile a single query

amber>
\trace aj[`sym`time; 100000#trades; quotes]
+-------------------------------------------------------+
| Parse         454ns  [                    ]   0.1%    |
| Arena          38ns  [                    ]   0.0%    |
| Execute      14.1ms  [■■■■■■■■■■■■■■■■■■  ]  91.2%    |
| Format        1.4ms  [■                   ]   8.7%    |
+-------------------------------------------------------+
| Total: 15.5ms     Arena peak: 32 B                    |
+-------------------------------------------------------+

Only expressions that reach an arena-backed kernel (aj, wj, `csvr, \ast) report a non-zero arena peak. Arena peak is a true high-water mark taken from arena_peak().

9

Persist it

Two options, depending on whether you want it human-readable or byte-exact.

amber>
/ portable Amber text — readable, version-independent, not memory-mapped
splay["/tmp/store/trades"; trades]
t2:dload "/tmp/store/trades"

/ compact binary — byte-exact including attributes, nulls and infinities
`:/tmp/trades.bin 1: -8!trades
t3:-9! read1 `:/tmp/trades.bin
t3~trades                                  / 1b

The binary form is what amber-tick's partitioned store writes, one file per column, so a query for px and sz reads two files and never opens the other seven.

Where the 60 seconds went

StepmsNotes
gentq 5000000238215M rows of realistic microstructure
xasc × 2~1900radix sort; sets `p and `s
aj7045M branch-free binary searches
1-minute VWAP151group + weighted average on raw column vectors
ema (50-period, tacit)0.1straight into the C kernel

Scale it up

Nothing above changes at ten times the size except the numbers. For a store that outlives the session — 38 million prints, 89 million quotes, ten-level book snapshots, LULD halts, and twenty-one stylized facts checked on every generated store — use amber-tick:

python -m amber_tick.generate --out store --symbols 500 --sessions 5 --rows 100M
python tests/validate.py --store store       # 21 stylized facts
PreviousTutorials