Unified host metrics (RDS, Supabase, ClickHouse)

Nikolay Samokhvalovrequested to merge
feat/unified-host-metrics into main

What

RDS, Supabase and ClickHouse Managed Postgres host metrics share one vocabulary, one dashboard row and one way to get their labels.

  • One vocabulary. A vmalert service evaluates config/prometheus/host_rules.yml against sink-prometheus and writes the results back. The rules map Supabase node_* (the primary only) and ClickHouse PostgresServer_* to the host_* names, units and {cluster, node_name} labels that rds-host-stats writes. vmalert runs under the host-metrics compose profile, which mon targets add/remove keep in COMPOSE_PROFILES only while a ClickHouse or Supabase scrape file exists (a stack with no such target, or RDS only, runs no vmalert). It is pinned to the stack's VictoriaMetrics release (v1.140.0), runs read-only with no capabilities, listens on loopback only and has no notifier. docs/host-metrics.md is the single table of names, units and sources.
  • One row. Dashboard 1 has one collapsed Host row, the last on the page, with five panels (CPU utilization, Memory, Storage, Disk IOPS /s, Network /s). They read host_* only, with the cluster and node selectors, and each charts a family every provider writes. A self-managed node sees one closed row.
  • One way to label. mon targets add (and mon local-install --db-url, which goes through it) copies the target's cluster/node_name tags into what the provider's collector reads:
    • ClickHouse: the scrape file host-metrics/clickhouse-<name>.yml (unchanged);
    • Supabase: the scrape file host-metrics/supabase-<name>.yml for the instance-jobs relay, while PGAI_SUPABASE_HOST_METRICS is true (environment or .env), removed when it is not. The relay serves one project, so a box has one such job, and it keeps only that project's series;
    • RDS: rds-host-stats pushes past samples and is not scraped, so it gets RDS_DB_INSTANCE_IDENTIFIER, AWS_REGION, PGAI_CLUSTER and PGAI_NODE_NAME in .env, parsed from the instance endpoint, and is recreated if it is running. A cluster, reader or proxy endpoint names no instance and sets nothing.
  • Only what a panel reads is collected. rds-host-stats no longer requests disk latency, queue depth, burst and EBS balance, replica lag or Performance Insights DB load: GetMetricData bills per metric, so a poll drops from 13 to 7 metrics (Aurora 14 to 10), about $3.02 instead of $5.60 per instance and month. The three Supabase rules for the same families are gone too.

Merge path

This MR is the single mergeable one for the host metrics work. It is main plus !434 (closed) → !436 (closed) → !440 (closed) (merged in that order and kept current) plus the contents of !435 (closed) and !437 (closed), with their conflicts against main and each other resolved here. It merges cleanly into main (e61a8f3).

  • Merge !434 (closed), then !436 (closed), then !440 (closed) (stacked, each retargets to main in turn), then this MR. Merging only this MR brings all of them in.
  • !435 (closed) and !437 (closed) each conflict with main and with each other and are fully contained here: superseded by this MR, proposed to close.
  • !433 now targets this branch and merges on top of it.
  • The mon update PGAI_TAG change moved from !440 (closed) to its own MR, !444. It is not in this branch.

Round 3 (review of 2026-09-29, round 2)

  • Labels: the 80-line awk YAML parser in generate-pgwatch-sources.sh, the static supabase-host-metrics job with its file_sd target and %{PGAI_SUPABASE_HOST_METRICS} relabel toggle, and the manual RDS .env step are replaced by the mon targets add path above. sink-prometheus no longer needs PGAI_SUPABASE_HOST_METRICS. The relay's Handler(enabled bool), only ever called with true, lost its argument.
  • Cuts from !435 (closed) (six unused series and PI, with the @aws-sdk/client-pi dependency) and from this MR (three Supabase rules and their test cases). The recorded AWS replies are trimmed, not re-recorded.
  • The Docker scrape test that CI skipped is replaced by cli/test/supabase-scrape.integration.test.ts, which runs the real VictoriaMetrics v1.140.0 binary in CI (job renamed cli:host-metrics-scrape:tests): it loads the scrape file the CLI writes, scrapes a relay stand-in, keeps only the node_* families of the target's project, and labels them with the target's cluster/node_name.
  • Scrape-file helpers shared by ClickHouse and Supabase moved to cli/lib/host-metrics.ts.
  • From an independent security review of this round: targets remove no longer fails on a target name like prod.db; a relay credential for another project cannot label its series with this target; a running rds-host-stats is recreated with the values just written, passed explicitly because an exported variable would win over .env; a failed recreate on remove still applies the removal. Declined: migrating the static Supabase job on existing installs (it was never released), and the mon restart hint after mon update (that is main's text; !444 changes it).
  • Own delta against main: code +1572/−137 (was +1662/−100), tests +6984/−1 (was +8357/−1, mostly the trimmed replays).

Round 5 (review of 2026-09-30, round 4)

Commits 351cedd, 16dc9a0, f3d8e9b. readEnvValue drops an inline comment as compose does and syncVmalert reuses it; with the relay on, a Supabase target name that cannot be a file name is refused before the target is saved ("choose another name"); targets add says when the instance-jobs profile is off; targets remove reads the RDS host as add does; maskConnectionString hides a password with / or @ and sslpassword=. Rule tests for a missing, flat or replica-only clock; the scrape test feeds node_time_seconds; the scrape CI job fails without the binary. Docs and .env.example name the profile and the 7-metric poll.

CLI at f3d8e9b, docker replaced by the logging stand-in from host-metrics-providers.test.ts (no stack, no provider; commands as shown):

mr442-round5-cli

Round 6 (review of 2026-09-30, round 5)

Commit ba74624. maskConnectionString: the password runs to the last @, so digits followed by # or ? are hidden, and a host:port/...@... URL with no userinfo is printed as it is (a password made of digits and then / reads the same way and is the residual). An exported empty PGAI_SUPABASE_HOST_METRICS overrides .env, as it does for compose. readEnvValue takes single quotes and an escaped ". SUPABASE_TARGET_NAME_RE. Tests: seven .env shapes of the flag, three of COMPOSE_PROFILES, exported "" against .env true, targets remove for a conn_str new URL rejects (exit 0, the RDS keys stay: add never wrote them), the two mask shapes.

Round 7 (review of 2026-09-30, round 6)

Commit 12afd5f. maskConnectionString hides query passwords first, then a password to the last @ a / follows (@host/db), else to the first @ with no ?/# before it: a /-first password is hidden and ?application_name=me@corp is left alone (residual: digits then / with no /db path). .env inline comments start at # only, as compose reads them; readComposeProfiles shares the parser, so mon health agrees with targets add. Tests: five more .env flag shapes, COMPOSE_PROFILES rows with a negative, the mask shapes.

CLI at 12afd5f (working tree of the commit), docker stand-in (inspect exits 1, all else 0; no stack, no provider):

mr442-round7-cli

Round 8 (review of 2026-09-30, round 7)

Commit bd4bdfb. maskConnectionString finds the password's end procedurally on a copy with query values blanked. Without a /db path a password with ? or # now runs to the last @ (hidden, not printed): #S3cret, ?S3cret, Ab/S3cret#x, a?b@S3cret, S3cret?password=. A [ first is an IPv6 host and digits then /, ,, ? or # are a port, printed as given (the residual, in the code comment). The query mask runs after the userinfo mask, so ?password= inside a password keeps the host; a query value runs to &, so a # after it is hidden; pwd counts; the key is tested in a callback, so a 100 KB key with no = takes 1 ms (was 1.5 s). readRaw says why it is not parseEnvValue (the value is written back as it was). Tests: 11 rows red on 12afd5f, green at head; COMPOSE_PROFILES= # c pinned as # c, which is how compose 5.0.2 reads it (docker compose config).

CLI at bd4bdfb (the reviewer's round-7 shapes and 100 KB timings; no stack, no provider):

mr442-round8-cli

CLI at ba74624, docker replaced by a stand-in that exits 0 (no stack, no provider):

mr442-round6-cli

Tests (red, then green)

  • cli/test/host-metrics-providers.test.ts: mon targets add for Supabase writes the golden supabase-scrape.golden.yml, a second target takes the job over, the flag is read from .env, flag false or empty removes it, targets remove drops it; for RDS, .env gets the instance, region and labels and loses them on remove, cluster/reader/proxy endpoints set nothing, a running rds-host-stats is recreated with the new values even when stale ones are exported, and an unknown compose state is reported.
  • cli/test/supabase-scrape.integration.test.ts and clickhouse-scrape.integration.test.ts: real VictoriaMetrics binary.
  • tests/host_rules/host_rules.test.yml (vmalert-tool unittest, CI job quality:host-rules): SUCCESS.
  • tests/grafana_dashboards: 405 pass (including test_host_row.py: one collapsed Host row, per-panel families, every panel has data from every provider, the doc table equals what the providers write).
  • rds-host-stats: 39 pass; bunx tsc --noEmit clean. instance-jobs: go test ./... and go vet -tags supabase_live pass. docker compose config validates with the rds and instance-jobs profiles.
  • Full CLI suite at f3d8e9b (2026-09-30, fix box): 2173 pass, 16 skip, 9 fail. 7 are mon local-install cases in init.test.ts that fail identically at 41a8395 on that box, which runs another stack under the fixed container names; the other 2 are 5 s timeouts that pass when their files run alone. At 41a8395 on the author's machine: 2124 pass, 66 skip, 1 fail (the macOS-only CHECKUP_ALLOW_HTTP case).
  • Red commits this round: 822a34a, 1b8abb9, 5b5b3d1, 117f245 (each followed by its fix).

Real runs

  • ClickHouse Managed Postgres (r6gd.medium, PostgreSQL 17.11), 2026-09-30: the cloud suite in clickhouse.cloud.test.ts passed from this branch: provider detected, scope printed, prepare-db --verify, and the service's /prometheus endpoint served every documented series. The service was created and deleted by the test (list empty afterwards).
  • table_describe as SECURITY INVOKER and the explain_generic removal were checked on another real ClickHouse service the same day (see !434 (closed) and !433); it returns 404 after deletion.
  • Local stack at 41a8395 (2026-09-30, images of pipeline 2896968060, fresh compose project) with one real ClickHouse Managed Postgres service (PG 17, test org, deleted afterwards): mon local-install -y creates no vmalert; mon targets add <ClickHouse URL> r4-ch writes host-metrics/clickhouse-r4-ch.yml, puts host-metrics in COMPOSE_PROFILES and starts vmalert with groups host-clickhouse, host-supabase; within ~4 minutes up{job="clickhouse-r4-ch"} = 1 and the 10 host_* series of the host-clickhouse group exist with {cluster="default", node_name="r4-ch"}; mon targets remove r4-ch stops and removes vmalert and the profile. Not a render of the Host row: see "What we did not check".
  • Local stack acceptance from round 2 (stub providers, Grafana 12.3.2) predates the label change; the label path is now covered by the real VictoriaMetrics tests above.

What we did not check

  • Real RDS or Supabase host metrics end to end from this branch (the RDS poller change is a removal of queries; the Supabase job was checked against the real VictoriaMetrics binary with the recorded exposition).
  • The ansible change (postgres_ai-ansible !49 (closed)) against the new Supabase job: provisioning runs mon local-install --db-url with PGAI_SUPABASE_HOST_METRICS in the environment, which writes the job; turning the flag off takes a re-run of mon local-install or mon targets add.
  • The Helm chart: it shares the dashboard but has no vmalert or collectors, so its Host row is empty. mon health does not report vmalert.
  • A render of the Host row at the current head for Supabase, ClickHouse or RDS (the round-4 blocker): it needs a real project/service/instance per provider, and the round-5 to round-8 fix runs had no AWS, Supabase or ClickHouse credentials. The round-5 to round-8 shots above are CLI output with a docker stand-in, not a stack.
  • Tags edited by hand in instances.yml reach the host metrics labels only after mon targets add is run again for that target.

Part of postgres-ai/internal#354.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GRNbWXUQJebE3vqWjAgTFh

Edited by Maya P

Merge request reports