feat(monitoring): Supabase host metrics relay and dashboard row

Superseded by !442 (closed) — proposed to close. This MR conflicts with main and with !435 (closed), and !442 (closed) contains all of it with the conflicts resolved. Its code is in !442 (closed), with the round-2 cuts applied there: the awk YAML parser, the static scrape job and its relabel toggle are replaced by a scrape file that mon targets add writes with the target's labels, the relay's Handler(enabled bool) lost its argument, and the Docker scrape test CI skipped is replaced by a test on the real VictoriaMetrics binary that CI runs. Closing is Nik's call; nothing here needs to merge separately.

Update 2026-09-30 (review findings, platform-all!884 auth).

• Labels. The Supabase scrape target now carries the pgwatch cluster and node_name: sources-generator, which already reads instances.yml for pgwatch, writes prometheus/supabase-host-metrics.json (file_sd) with the tags of the one enabled Supabase target (db.<ref>.supabase.co or *.pooler.supabase.com). With none or several, the target has no labels. It parses the file the way the CLI writes it (js-yaml folds a long conn_str into a block scalar) and escapes the same in BusyBox awk (bash:5.2), mawk and BSD awk. The row queries now filter on the dashboard's cluster and node selectors, and the test asserts that they do, where it used to assert the opposite.
• Instance secret (platform-all!884). The RPC now also requires the box's own secret. The relay reads instance_secret from .pgwatch-config (the file only, never the environment) and sends it in the instance-secret header, next to access-token, and never in the body. It is never logged. Without it the relay makes no call, answers 503 instance_secret_missing, and logs once an hour that the instance needs re-provisioning. A golden request test pins the headers and body.
• Superseded for the dashboard. The unified Host row that reads host_* for all providers is in !442 (closed) (see there for merge order).

Required provisioning change (postgres_ai-ansible, not in this MR). SI already receives instance_secret as an extraVar from supabase_monitoring_provision. In roles/postgres_ai/tasks/main.yml, after postgres_ai_project_dir is resolved and after mon local-install has created .pgwatch-config:

- name: Write the instance secret for Supabase host metrics (platform-all!884)
  ansible.builtin.lineinfile:
    path: "{{ postgres_ai_project_dir }}/.pgwatch-config"
    regexp: '^instance_secret='
    line: "instance_secret={{ instance_secret }}"
    create: false
  no_log: true
  when: (instance_secret | default('') | length) > 0

- name: Turn on Supabase host metrics
  ansible.builtin.lineinfile:
    path: "{{ postgres_ai_project_dir }}/.env"
    regexp: '^PGAI_SUPABASE_HOST_METRICS='
    line: "PGAI_SUPABASE_HOST_METRICS=true"
  when: (instance_secret | default('') | length) > 0

The relay rereads .pgwatch-config on every credential fetch, so the secret needs no restart. The .env flag is read at container start: run docker compose up -d sink-prometheus instance-jobs in the project directory after the second task, with instance-jobs enabled for the box. regexp replaces an existing line: the config reader takes the first instance_secret= line, so a duplicate must never be appended. Boxes provisioned before !884 have no secret and need re-provisioning.

What

Host metrics (CPU, memory, disk, IOPS, network) for Supabase projects, with no key pasted by the customer and no key on disk.

• instance-jobs gains an internal relay (GET /supabase/metrics on :9188, compose network only). It asks the platform RPC supabase_host_metrics_credential for the project's secret key, keeps it in memory for at most 1 hour, and scrapes https://<ref>.supabase.co/customer/v1/privileged/metrics with it.
• VictoriaMetrics scrapes the relay every 60 s and keeps only the node_exporter host families.
• Dashboard 01 gets a collapsed "Host (Supabase)" row: CPU busy %, memory available, filesystem available, disk IOPS, network bytes/s.
• The relay is off unless PGAI_SUPABASE_HOST_METRICS=true.

Platform half: https://gitlab.com/postgres-ai/platform-all/-/merge_requests/884 (RPC, re-consent button).

Key handling

• The key is held in memory only. It is never logged or written to a file, env or argv, and error text never includes it. The relay replies with a fixed Content-Type, so no upstream header is passed through.
• metrics_url must be https://<20 a-z>.supabase.co/customer/v1/privileged/metrics and match project_ref. Anything else is refused, and redirects are not followed.
• Credential fetches are limited to one per 5 minutes and run detached from the scrape request (10 s timeout). On a 401/403 the relay refetches once. The last exposition is served from memory for 30 s, so compose-network peers cannot drive upstream scrapes.
• The org API token goes in the access-token header only, like Poll, so it never becomes a logged bind parameter.

Tests

• Red/green commits. Go unit tests use httptest fakes: log and header leak checks, 401 refetch, cooldown, TTL, URL validation, consent_needed → 503, and a byte-for-byte passthrough golden.
• The golden testdata/supabase_metrics.prom was recorded on 2026-09-27 from a real Free-plan Nano project and trimmed to host families. The ref is replaced.
• Live test (-tags supabase_live, needs SUPABASE_TEST_ACCESS_TOKEN + SUPABASE_TEST_PROJECT_REF): it passed against a real project in the pgai-ci-test org. The relay returned 200 with all five host families and the key did not appear in logs.
• go test ./..., go vet (both tag sets), gofmt, pytest tests/grafana_dashboards (392 passed), compose config check and VictoriaMetrics -promscrape.config dry run all pass.

Known limits

• When the feature is off, the scrape target is dropped by relabeling, so no up{job="supabase-host-metrics"} series exists on those boxes.
• One project per box, which is how the platform provisions Supabase monitoring. Series carry supabase_project_ref and supabase_identifier (unique per host, so read replicas stay separate), not cluster/node_name.

Part of postgres-ai/platform-all#296, postgres-ai/internal#354.

🤖 Generated with Claude Code

Full-stack acceptance (2026-09-28)

We ran instance-jobs (this branch), VictoriaMetrics v1.140.0 and Grafana 12.3.2 against the live Free-plan project for 11 min 11 s. A local stand-in answered the platform RPC, because the OAuth app does not have the scope yet.

• up{job="supabase-host-metrics"} stayed at 1 for 11 of 11 scrapes, with 1 credential RPC call and 0 restarts.
• All six "Host (Supabase)" queries returned live series through Grafana /api/ds/query. Sample values: CPU 3.6 %, memory available 206 MiB, filesystem available 2.95 GiB, IOPS 58/s, network rx 400 B/s, tx 1.08 KB/s.
• Key leak check: 0 matches across all container logs and every VictoriaMetrics label value. https://claude.ai/code/session_01GRNbWXUQJebE3vqWjAgTFh

Rendered evidence (2026-09-28, SHA 331344f9)

Environment: VictoriaMetrics v1.140.0, Grafana 12.3.2 (provisioned from config/grafana/dashboards on this branch), headless Chrome 1600x1000, dark theme. Data: the relay's golden exposition testdata/supabase_metrics.prom (recorded from the Free-plan Nano project) imported into VictoriaMetrics with job="supabase-host-metrics" over 25 minutes, once for the primary and once as a read replica (supabase_identifier=...-replica-1) to show that replica series stay separate.

Feature on, row expanded (all 5 panels, legends by supabase_identifier), footer above the row:

Feature on, row collapsed, footer above the row header:

Feature off (empty store), row expanded: No data in every panel, no up target exists (see test_supabase_target_only_when_enabled):