Grafana Amber datasource

Live dashboards over the Amber engine. Panels are bare qSQL, results are column-oriented on the wire, and the time axis knows that Amber counts from 2000-01-01.

bonucciandrea-amber-datasource Go backend Grafana ≥ 10.0 26 protocol tests GitHub →

Architecture

The plugin keeps a deliberate separation. pkg/amberd holds the wire protocol, the columnar decode, the temporal conversions and the connection pool, using only Go's standard library — and it has 26 tests that run against a real amberd on a real socket. pkg/plugin holds the Grafana-specific integration.

Why the split matters to you

A protocol bug reproduces in go test ./pkg/amberd/ with no Grafana anywhere near it. Debugging a decode problem does not require standing up a dashboard stack.

Prerequisites

The Amber engine built as a shared library — ./build.sh --shared
amberd, which ships with amber-arrow — ./build.sh --amberd
A Go toolchain and Mage
Node.js and npm, for the frontend
Grafana 10.0 or newer

1 — Build the plugin

git clone https://github.com/BonucciAndrea/grafana-amber-datasource
cd grafana-amber-datasource

# backend — compiles gpx_amber_<os>_<arch> into dist/
go mod tidy && mage -v

# frontend — TypeScript / React into dist/
npm install && npm run build

mage -v is the Grafana plugin SDK's build entry point: it cross-compiles the backend binary for every supported platform and writes the results into dist/ with the names plugin.json's executable field expects.

The build needs the Go module proxy

The backend is a real Grafana plugin and links github.com/grafana/grafana-plugin-sdk-go. Behind a corporate proxy, set GOPROXY before mage (or add it to Dockerfile.plugin if you are using the containerised stack). There is no way around it.

2 — `plugin.json`

The plugin's identity. The id is what every later configuration step refers to, and it must match exactly.

src/plugin.json

{
  "$schema": "https://raw.githubusercontent.com/grafana/grafana/main/docs/sources/developers/plugins/plugin.schema.json",
  "type": "datasource",
  "name": "Amber",
  "id": "bonucciandrea-amber-datasource",
  "metrics": true,
  "backend": true,
  "alerting": false,
  "annotations": true,
  "logs": false,
  "executable": "gpx_amber",
  "info": {
    "description": "Query the Amber columnar engine from Grafana. Bare qSQL over amberd's TCP port.",
    "author": { "name": "the Amber authors" },
    "keywords": ["amber", "tick", "time series", "columnar"],
    "logos": { "small": "img/logo.svg", "large": "img/logo.svg" },
    "version": "1.0.0",
    "updated": "2026-08-25"
  },
  "dependencies": {
    "grafanaDependency": ">=10.0.0",
    "plugins": []
  },
  "routes": []
}

Field	Why it is set this way
`"backend": true`	All the logic — protocol, decode, temporal conversion — is in Go. The frontend only builds queries.
`"executable": "gpx_amber"`	Grafana appends `_<os>_<arch>` and launches that binary. `mage` produces exactly those names.
`"metrics": true`	The datasource can back any panel, not only a time series.
`"alerting": false`	Deliberate: `amberd` is single-threaded and unauthenticated. Alert rules would hammer it on their own schedule.

3 — Install the plugin into Grafana

cp -r dist "$GRAFANA_PLUGINS/bonucciandrea-amber-datasource"

$GRAFANA_PLUGINS is usually /var/lib/grafana/plugins on a package install, or <grafana>/data/plugins on a tarball install.

4 — Allow the unsigned plugin

Grafana refuses unsigned plugins by default

A locally built plugin has no signature. Without this step the plugin is present on disk, absent from the datasource list, and the only clue is one line in the Grafana log.

Pick one of the three forms — they are the same setting.

grafana.ini

[plugins]
allow_loading_unsigned_plugins = bonucciandrea-amber-datasource

environment variable

export GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=bonucciandrea-amber-datasource

docker-compose.yml

services:
  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: bonucciandrea-amber-datasource
      GF_USERS_DEFAULT_THEME: dark
    volumes:
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
      - plugin-dist:/var/lib/grafana/plugins/bonucciandrea-amber-datasource:ro
    ports:
      - "3000:3000"

Restart Grafana, then confirm it loaded:

grep -i "amber" /var/log/grafana/grafana.log | head
# ... "Plugin registered" logger=plugin.loader pluginID=bonucciandrea-amber-datasource

5 — Start `amberd`

amberd --home /path/to/amber --port 5012 \
       --load /path/to/amber-tick/q/amber-tick.k \
       --eval 'tk.root:"/srv/store"; tk.init[]' \
       --mode jsonc -v

6 — Provision the datasource

provisioning/datasources/amber.yaml

# amberd has no authentication: anything that can open the socket can evaluate
# arbitrary Amber. Keep it on loopback (or inside the compose network, as here)
# and never expose the port.
apiVersion: 1

datasources:
  - name: Amber
    type: bonucciandrea-amber-datasource
    uid: amber
    access: proxy
    isDefault: true
    jsonData:
      # Inside docker compose this resolves to the amberd service. Running
      # Grafana outside compose against an amberd on the host: use
      # host.docker.internal on macOS/Windows, or the host address on Linux.
      host: amberd
      port: 5012
      timeoutSeconds: 60
      maxConnections: 8
      # The store's `time` columns are nanoseconds since 2000-01-01 -- Amber's
      # own timestamp epoch, not the Unix one. Every panel sets this
      # explicitly too, but the default belongs here so a new panel is right
      # before anyone touches it.
      defaultTimeUnit: ns-2000

Config field	Meaning
`host` / `port`	where `amberd` is listening
`timeoutSeconds`	per-query deadline. `amberd` runs one query at a time, so this is also how long a slow query can block a panel.
`maxConnections`	connection pool size. Panels then wait on the engine rather than on each other's sockets.
`defaultTimeUnit`	`ns-2000` (nanoseconds since 2000-01-01), `ms-day` (milliseconds of day), or `ms-unix`. Explicit because a column of each overlaps in magnitude and cannot be told apart by value.
`baseDate`	optional override for time-only columns, so a `ms-day` column can be placed on a real calendar day.
`maxRows`	row cap, as a safety valve against a panel that forgets a `where`.

7 — Create a panel

The query editor takes six things: the qSQL source, the reply format, the time unit, which column is the time axis, a row cap, and whether to substitute the panel's time range.

panel query

select
  t:time,
  notional:sum px*sz
by time:$__interval_ms xbar time
from bars1m
where time within ($__fromNs; $__toNs)

Macros

Available when Apply time range is enabled. They come in both epochs, named so you cannot pick the wrong one by accident.

Macro	Substituted with
`$__from` / `$__to`	the panel range as milliseconds of day
`$__fromNs` / `$__toNs`	the panel range as nanoseconds since 2000-01-01
`$__interval_ms`	the panel's step, for bucketing with `xbar`

Editor features

Live completion from the engine — table names and their real column names, asked for at edit time, not read from a static list.
Real-time syntax validation via the engine's \parse, which runs the real parser and never touches the compiler or the evaluator. Nothing you type is executed.
Verbatim engine diagnostics on failure — the error[E0101] report you would have seen at the prompt, in the panel.

Design decisions worth knowing

Column-oriented throughout

A Grafana data frame is a set of columns and Amber stores a table as a dictionary of columns. Going through row-oriented JSON would transpose twice and add a per-row object header for every tick. jsonc is the absence of a pessimisation, not an optimisation.

Temporal handling is explicit and tested

Conversion between Amber's epoch and Grafana's is done in Go, covered by boundary tests, and disambiguated by column name where two units could plausibly apply.

Nulls stay null

Amber nulls map to nullable Grafana fields, which preserves chart continuity instead of drawing a line through zero.

Pooled connections, one engine

amberd is single-threaded by design. Pooling means several panels queue on the engine rather than on each other's sockets — and the fix for a slow dashboard is a mart, not more connections.

Quick start with Docker

scripts/dev.sh          # orchestrates engine, plugin and Grafana in Docker

Or, from amber-tick, the full three-container stack with the tick store attached and three dashboards provisioned:

cd amber-tick
python -m amber_tick.generate --out store --symbols 500 --sessions 5 --rows 100M
python scripts/build_marts.py --store store --qhome q
python grafana/refresh_symbols.py --store store    # fills the $sym dropdown

cd grafana && ./run.sh                             # http://localhost:3000

Troubleshooting

Symptom	Cause and fix
The datasource does not appear in the list	`allow_loading_unsigned_plugins` is missing or has the wrong id. It must be exactly `bonucciandrea-amber-datasource`.
Panels are empty but the datasource tests green	The marts are missing. Every panel reads `bars1m`, `symday`, `venueday` or `mktminute`, never the raw tape. Run `scripts/build_marts.py`.
The `$sym` dropdown is empty	The Amber datasource does not implement Grafana's variable-query API, so the list is baked into the dashboard JSON. Run `python grafana/refresh_symbols.py --store store`.
Provisioned datasource cannot reach `amberd` outside Compose	The provisioning file points at the Compose service name `amberd`, which does not exist outside Compose. Change `host` to `127.0.0.1`.
`permission denied … /var/run/docker.sock`	Your user is not in the `docker` group. `sudo usermod -aG docker "$USER" && newgrp docker`. On a snap install you must `sudo addgroup --system docker` first — the snap does not create it.
Everything looks right, times are thirty years off	The panel's time unit is `ms-unix` against a `ns-2000` column. See Temporal mechanics.

Tests

go test ./pkg/amberd/ -v      # protocol decode, temporal boundaries, errors, concurrency
npm run typecheck             # frontend