Tutorial 02 · ~40 minutes

Setting up an end-to-end Grafana & Amber dashboard pipeline

Five moving parts: a store, a set of derived marts, an amberd with the store attached, a Go datasource plugin, and Grafana with everything provisioned. At the end you have three dashboards where every panel is a one-line call into an Amber script you can also run at the prompt.

Prerequisites

Docker with Compose v2, or a Grafana ≥ 10 you installed yourself plus a Go toolchain
Python 3.10+, and ~12 GB of free disk for a 100M-row store
The engine, the Python bindings, and amber-arrow (for amberd)

Build the engine as a shared library

git clone https://github.com/BonucciAndrea/amber
git clone https://github.com/BonucciAndrea/python-amber
git clone https://github.com/BonucciAndrea/amber-arrow
git clone https://github.com/BonucciAndrea/amber-tick

(cd amber && chmod +x build.sh a && ./build.sh --shared)

python -m venv .venv && . .venv/bin/activate
AMBER_SRC=$PWD/amber pip install ./python-amber ./amber-arrow/python
pip install numpy pyarrow pandas
(cd amber-arrow && ./build.sh --amberd)

Generate a store

cd amber-tick
python -m amber_tick.generate --out store --symbols 500 --sessions 5 --rows 100M

About four minutes on two cores and roughly 10 GB. The generator solves for the arrival intensity that hits your --rows target analytically, before drawing a single random number, so scaling changes the message rate only — the cross-sectional distribution, the clustering and the trade-quote consistency are invariant.

python tests/validate.py --store store      # 21 stylized facts, all passing

Build the marts — this is not optional

python scripts/build_marts.py --store store --qhome q

A dashboard cannot re-run an as-of join on every refresh

Ten million prints joined against thirty million quotes is a 700-millisecond kernel call. A panel refreshing every ten seconds, times six panels, is not a dashboard — it is a denial of service against your own engine. build_marts.py derives four small tables once, using the same taq.* functions the ad-hoc queries use, and they come out about 145× smaller than the tape.

Mart	What it holds
`bars1m`	per symbol per minute: OHLCV, VWAP, buy/sell split
`symday`	per symbol per session: volume, notional, effective spread, price improvement
`venueday`	per venue per session: share of volume, execution quality
`mktminute`	the consolidated tape per minute: notional, prints, spread

Fill the `$sym` dropdown

python grafana/refresh_symbols.py --store store

The Amber datasource does not implement Grafana's variable-query API, so the symbol list is baked into the dashboard JSON. This script rewrites it from the store's actual security master.

Start `amberd` with the store attached

The daemon needs the engine's stdlib, the amber-tick Amber library, and the store root.

amberd --home ../amber --port 5012 \
       --load q/amber-tick.k \
       --eval 'tk.root:"'"$PWD"'/store"; tk.init[]' \
       --mode jsonc -v

Check it before you go near Grafana:

printf '\\ping\n' | nc localhost 5012
printf '\\mode jsonc\nselect from mktminute where time<34260000000000\n' | nc localhost 5012

If those return, the rest of the stack is a configuration problem and nothing else.

Build the datasource plugin

git clone https://github.com/BonucciAndrea/grafana-amber-datasource
cd grafana-amber-datasource

go mod tidy && mage -v            # backend  -> dist/gpx_amber_<os>_<arch>
npm install && npm run build      # frontend -> dist/

mage is the Grafana plugin SDK's build entry point. It links github.com/grafana/grafana-plugin-sdk-go, so the module proxy must be reachable — behind a corporate proxy, set GOPROXY first.

cp -r dist "$GRAFANA_PLUGINS/bonucciandrea-amber-datasource"

Allow the unsigned plugin

A locally built plugin has no signature, and Grafana silently declines to register it. This is the single most common reason the datasource "does not exist".

grafana.ini

[plugins]
allow_loading_unsigned_plugins = bonucciandrea-amber-datasource

or, as an environment variable

export GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=bonucciandrea-amber-datasource

Restart Grafana and confirm:

grep -i amber /var/log/grafana/grafana.log | head
# "Plugin registered" logger=plugin.loader pluginID=bonucciandrea-amber-datasource

Provision the datasource

provisioning/datasources/amber.yaml

apiVersion: 1

datasources:
  - name: Amber
    type: bonucciandrea-amber-datasource
    uid: amber
    access: proxy
    isDefault: true
    jsonData:
      host: 127.0.0.1        # 'amberd' inside docker compose
      port: 5012
      timeoutSeconds: 60
      maxConnections: 8
      defaultTimeUnit: ns-2000

The host value is environment-specific

Inside Compose it is the service name amberd. Outside Compose that name does not resolve — use 127.0.0.1, or host.docker.internal if Grafana is in a container and amberd is on the host.

Provision the dashboards

provisioning/dashboards/dashboards.yaml

apiVersion: 1

providers:
  - name: amber-tick
    type: file
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false

Copy amber-tick/grafana/dashboards/*.json into that path. Three dashboards: Market overview, Symbol microstructure and Execution quality.

Or do steps 6–9 in one command

cd amber-tick/grafana
AMBER_TICK_STORE=$PWD/../store ./run.sh      # http://localhost:3000

run.sh checks the things that otherwise fail as something else — a missing store, missing marts, a relative path Compose resolves differently from your shell, and access to the Docker daemon — then brings up three containers: an amberd built from source with the store attached, a one-shot container that builds the plugin into a shared volume, and Grafana 11.3 with everything provisioned.

Writing a panel

The query editor takes the qSQL source, the reply format, the time unit, the time column, a row cap, and whether to substitute the panel's time range.

Market overview — consolidated notional per minute

select
  notional: sum notional
by time: $__interval_ms xbar time
from mktminute
where time within ($__fromNs; $__toNs)

Symbol microstructure — 1m bars vs VWAP

select
  o, h, l, c, vwap
by time
from bars1m
where sym = `$"$sym", time within ($__fromNs; $__toNs)

Execution quality — effective spread by venue

select
  effbps: 10000 * wavg[notional; eff % mid],
  pibps:  10000 * wavg[notional; pi % mid]
by venue
from venueday
where date within ($__fromNs; $__toNs)

Every panel query runs at the prompt too

Because each one is a call into q/grafana.k, you can paste it into ./amber with the store loaded and get the same answer. That is the whole point of keeping the query surface in Amber rather than in dashboard JSON.

amber>

\l q/amber-tick.k
tk.root:"/path/to/store"
tk.init[]
show 5#gd.marketspan[]

Time macros

Macro	Value
`$__from` / `$__to`	milliseconds of day
`$__fromNs` / `$__toNs`	nanoseconds since 2000-01-01
`$__interval_ms`	the panel's step, for `xbar`

The store's time columns are ns-2000, so $__fromNs / $__toNs are the ones you want. Using the wrong pair does not error — it returns an empty panel or a chart thirty years adrift. See Temporal mechanics.

Two failures that look like something else

Panels empty, datasource tests green

The marts are missing. Every panel reads bars1m, symday, venueday or mktminute, never the raw tape. Run scripts/build_marts.py. The connection test only proves amberd answers \ping.

"permission denied … docker.sock"

Your user is not in the docker group. It has nothing to do with this stack. sudo usermod -aG docker "$USER" && newgrp docker. On a snap-installed Docker you must sudo addgroup --system docker first — the snap does not create the group, is confined to $HOME and /media, and ships its own Compose.

Adding the live pipeline

Once the historical dashboards work, point them at a moving tape:

python -m amber_tick.runtime.tickerplant --port 5010 --log tplog --date 9729 &
python -m amber_tick.runtime.rdb --port 5010 --store store --qhome q &
python -m amber_tick.runtime.feed --store store --part 2026.08.21 --speed 60

The tickerplant writes its sequenced log before it publishes, so a subscriber can never see a message the log does not already contain — recovery is a replay from the last acknowledged sequence number. Point a second datasource at the RDB's port, or put the gateway in front of both and let it fold RDB and historical results with an explicit combine rule (you cannot average five partitions' averages and get a VWAP).

Editing the dashboards

python grafana/build_dashboards.py     # rewrites dashboards/*.json

Panels are generated, not hand-edited as JSON. Colour follows one rule: identity gets a categorical hue from a fixed, colour-vision-validated order; magnitude gets a single hue. No chart puts more than three identity colours on screen at once.