Tutorial 01 · ~15 minutes

Building a 5-million-row in-memory tick store in 60 seconds

By the end of this you will have five million trades and ten million quotes in memory, joined as-of, bucketed into one-minute bars, and you will know exactly how much RAM it cost.

What you need

git clone https://github.com/BonucciAndrea/amber && cd amber
chmod +x a build.sh
AMBER_NATIVE=1 ./a           # build machine-tuned, then open the REPL

Everything below is typed at the amber> prompt.

Generate the session

gentq is in fin.k, auto-loaded at startup. It sets two globals: trades and quotes, with realistic microstructure: an efficient price, a spread state, and prints that sit inside the standing quote.

amber>

\ts gentq 5000000

2381                   / milliseconds

\ts is accepted as a spelling of \t, which times an expression, and reports the elapsed milliseconds. About 2.4 seconds for five million trades and ten million quotes; use \v if you also want the workspace footprint.

Look at what you got

amber>

count trades                       / 5000000
count quotes                       / 10000000
meta trades

╭──────┬───┬───╮
│ c    │ t │ a │
├──────┼───┼───┤
│ sym  │ s │ p │
│ time │ i │ s │
│ px   │ f │   │
│ sz   │ i │   │
╰──────┴───┴───╯

The a column is already populated: gentq hands back a table laid out the way the as-of kernel wants it. The next step is what you would have to do yourself for data that arrived from anywhere else.

Sort and stamp the attributes

The as-of kernel binary-searches each symbol group's time slice. For that to be legal the rows must be sorted by (sym, time) and sym must carry `p (parted, so equal values are contiguous). xasc orders the rows and stamps the leading column `s, so partcol turns that into `p. This is exactly what gentq itself does. time is now sorted within each symbol but not across the whole column, so it carries no attribute: since 2.7 `s means the whole column is sorted, as in q, and Amber checks it.

amber>

trades:partcol[`sym; xasc[`sym`time; trades]]
quotes:partcol[`sym; xasc[`sym`time; quotes]]
meta trades

╭──────┬───┬───╮
│ c    │ t │ a │
├──────┼───┼───┤
│ sym  │ s │ p │
│ time │ i │   │
│ px   │ f │   │
│ sz   │ i │   │
╰──────┴───┴───╯

The sort and the `p are what make the join O(log n) per row instead of O(n): each symbol is one contiguous run, and its times are in order inside it.

Want O(1) symbol slicing too?

Add the grouped attribute and the group index: `ga trades`sym, then bysym / symrows / gidx from fin.k give you a per-symbol row range without a scan.

The as-of join

Every trade gets the quote that was standing when it printed. This is TAQ, and it is the operation the whole layout exists to serve.

amber>

\ts m:aj[`sym`time; trades; quotes]
count m                            / 5000000
5#m

704

╭──────┬──────────────┬────────┬─────┬────────┬────────┬─────┬─────╮
│ sym  │         time │     px │  sz │    bid │    ask │ bsz │ asz │
├──────┼──────────────┼────────┼─────┼────────┼────────┼─────┼─────┤
│ AAPL │ 09:30:00.000 │ 187.31 │ 100 │ 187.30 │ 187.32 │ 300 │ 300 │
│ AAPL │ 09:30:00.004 │ 187.31 │ 200 │ 187.30 │ 187.32 │ 300 │ 300 │
│ AAPL │ 09:30:00.011 │ 187.32 │  50 │ 187.31 │ 187.33 │ 100 │ 200 │
│ AAPL │ 09:30:00.019 │ 187.32 │ 300 │ 187.31 │ 187.33 │ 100 │ 200 │
│ AAPL │ 09:30:00.026 │ 187.31 │ 100 │ 187.30 │ 187.33 │ 200 │ 100 │
╰──────┴──────────────┴────────┴─────┴────────┴────────┴─────┴─────╯
[5 rows x 8 cols]

704 milliseconds for five million binary searches over a ten-million-row quote table. The kernel is in src/a.c, marshalled from amber.k, and it runs off a thread-local 16 MB bump arena so malloc jitter stays out of the path.

Microstructure columns

amber>

m:update mid:0.5*bid+ask from m
m:update side:tsign m from m               / Lee-Ready: +1 buy, -1 sell
m:update eff:effspread m from m            / 2 * |px - mid|
select n:#px, eff:avg eff, bps:10000*avg eff%mid by sym from m

╭──────┬─────────┬────────┬───────╮
│ sym  │       n │    eff │   bps │
├──────┼─────────┼────────┼───────┤
│ AAPL │ 1284203 │ 0.0121 │ 0.646 │
│ MSFT │ 1091177 │ 0.0184 │ 0.447 │
│ GOOG │  918884 │ 0.0102 │ 0.722 │
│   .. │      .. │     .. │    .. │
╰──────┴─────────┴────────┴───────╯

One-minute VWAP and OHLCV bars

The headline query, bare at the prompt:

amber>

\ts select vwap:wavg[sz;px] by time:minbar[1;time] from trades

151

╭──────────────┬──────────╮
│         time │     vwap │
├──────────────┼──────────┤
│ 09:30:00.000 │ 187.2841 │
│ 09:31:00.000 │ 187.3106 │
│ 09:32:00.000 │ 187.2955 │
│ 09:33:00.000 │ 187.4012 │
│           .. │       .. │
╰──────────────┴──────────╯
[390 rows x 2 cols]

And the full OHLCV, spelled out with qby so you can see the shape:

amber>

tb: +@[+trades; ,`time; minbar[1]@]        / snap time onto 1-minute buckets
b: qby[tb; `sym`time;
       `o`h`l`c`v!({first x`px};{max x`px};{min x`px};{last x`px};{sum x`sz})]
5#b

Or, if you just want the standard bars, fin.k has it in one call:

amber>

b:bars[1; trades]                          / 1-minute OHLCV
candle b                                   / Unicode candlesticks, in colour

Measure the footprint

amber>

\v

+-------------+---------------+----------------+---------+
| Name        | Type          | Shape / Length | Memory  |
+-------------+---------------+----------------+---------+
| trades      | Table         | 5000000 x 4    | 120.0 MB|
| quotes      | Table         | 10000000 x 4   | 320.0 MB|
| m           | Table         | 5000000 x 8    | 280.0 MB|
| b           | Table         | 390 x 6        | 18.7 KB |
| ...         | ...           | ...            | ...     |
+-------------+---------------+----------------+---------+

\v walks nested structures recursively, so a table's number is real rather than a shallow guess. It also lists the REPL library's own internals (repl.*, PAL, GB). Scan for the names you defined, or \d yourns first to narrow the namespace.

Profile a single query

amber>

\trace aj[`sym`time; 100000#trades; quotes]

+-------------------------------------------------------+
| Parse         454ns  [                    ]   0.1%    |
| Arena          38ns  [                    ]   0.0%    |
| Execute      14.1ms  [■■■■■■■■■■■■■■■■■■  ]  91.2%    |
| Format        1.4ms  [■                   ]   8.7%    |
+-------------------------------------------------------+
| Total: 15.5ms     Arena peak: 32 B                    |
+-------------------------------------------------------+

Only expressions that reach an arena-backed kernel (aj, wj, `csvr, \ast) report a non-zero arena peak. Arena peak is a true high-water mark taken from arena_peak().

Persist it

Since 2.7 a splayed table is binary and maps straight back into memory; one value can also go in one file.

amber>

/ splayed: one binary file per column, symbols in /tmp/store/sym, mapped on load
`:/tmp/store/trades/ set trades
t2:get `:/tmp/store/trades

/ Amber's own compact binary, byte-exact including attributes, nulls and infinities
`"/tmp/trades.bin" 1: `ser trades         / x 1: y writes bytes
t3:`des 1: `"/tmp/trades.bin"             / 1: x reads them back
t3~trades                                  / 1b

The binary form is what amber-tick's partitioned store writes, one file per column, so a query for px and sz reads two files and never opens the other seven.

Where the 60 seconds went

Step	ms	Notes
`gentq 5000000`	2382	15M rows of realistic microstructure
`xasc` × 2	~1900	radix sort; sets `p and `s
`aj`	704	5M branch-free binary searches
1-minute VWAP	151	group + weighted average on raw column vectors
`ema` (50-period, tacit)	0.1	straight into the C kernel

Scale it up

Nothing above changes at ten times the size except the numbers. For a store that outlives the session, 38 million prints, 89 million quotes, ten-level book snapshots, LULD halts, and twenty-one stylized facts checked on every generated store, use amber-tick:

python -m amber_tick.generate --out store --symbols 500 --sessions 5 --rows 100M
python tests/validate.py --store store       # 21 stylized facts