feat(rds-host-stats): poll RDS host metrics

Superseded by !442 (closed) — proposed to close. This MR conflicts with main and with !437 (closed), and !442 (closed) contains all of it with the conflicts resolved. Its code is in !442 (closed), with the round-2 cuts applied there: the six series no panel reads and Performance Insights are no longer requested (7 GetMetricData metrics per poll instead of 13), and PGAI_CLUSTER/PGAI_NODE_NAME now come from mon targets add. Closing is Nik's call; nothing here needs to merge separately.

Update 2026-09-30 (review findings).

• CPU cap. The compose default is now RDS_HOST_STATS_CPUS=0.5, not 0.1. We measured the release image against real AWS endpoints (invalid credentials, so every call paid TLS and SigV4 and then failed auth). At 0.1 CPU the first cold call took 1.2 s instead of 0.28 s. From 0.25 CPU up, call latency no longer depended on the cap. QA had seen every poll hit the 5 s connect timeout at 0.1 on a smaller VM, and 0.5 leaves margin for a slower vCPU and a full 1 MB Enhanced Monitoring page. test/compose.test.ts pins the default.
• No crash loop. A missing or invalid variable (RDS_DB_INSTANCE_IDENTIFIER, PGAI_CLUSTER, PGAI_NODE_NAME, AWS_REGION, the role pair, the interval) is now logged once, and the service idles until the container is restarted. Compose defaults these to empty and restarts on any exit, so exit 2 used to loop. Verified with the image: restarts=0, one log line. The intended exit 1 after a stalled poll is unchanged.
• Dashboard. The "Host stats" row is now collapsed and last, with its panels inside it, so boxes without host_* see one closed row instead of ten empty panels.
• Superseded for the dashboard. The unified Host row for all providers is in !442 (closed) (see there for merge order).

Part of https://gitlab.com/postgres-ai/internal/-/issues/354 (lane H, feature plan (b): RDS/Aurora host stats). Companion: postgres-ai/platform-all!883 (the customer-side read-only role).

What

rds-host-stats/ is a Bun + TypeScript service that runs on the monitoring VM beside one RDS or Aurora PostgreSQL instance. Every 60 s it:

• reads CloudWatch AWS/RDS metrics (CPU, freeable memory, free storage, read/write IOPS and latency, queue depth, network, burst and EBS IO balance, replica lag),
• reads Performance Insights db.load.avg when PI is on,
• reads Enhanced Monitoring OS metrics from RDSOSMetrics when EM is on (CPU and iowait, load, memory total/free/cached, swap used, largest process RSS),
• writes only samples newer than the last one per series as host_* series, labelled with the pgwatch cluster and node_name, through the VictoriaMetrics import API.

Aurora is detected from the engine: AuroraReplicaLag (ms to s), FreeLocalStorage, and cluster-level VolumeBytesUsed and VolumeRead/WriteIOPs (per 5 min, divided by 300).

With RDS_ROLE_ARN and RDS_EXTERNAL_ID it assumes the customer's read-only role (1 h sessions) from the VM's base credentials. It ships as an image (MR and release builds) and as compose service rds-host-stats under profile rds: read-only root, no capabilities, half a CPU (RDS_HOST_STATS_CPUS=0.5), 128 MiB.

Dashboard 1 gets a "Host stats" row with 10 panels over host_*, scoped to $cluster_name/$node_name: CPU (CloudWatch, OS, I/O wait), memory, storage, disk IOPS /s, disk latency, queue depth, network /s, Performance Insights DB load, replica lag, and burst / EBS I/O balance. Series that a provider does not report (for example, the Aurora-only ones on RDS) stay empty.

Customer-side cost: GetMetricData is $0.01 per 1,000 metrics, so 13 metrics every minute come to about $5.60/month per instance. Logs and PI Standard reads are free.

TDD

• Red b944c9e: we recorded real DescribeDBInstances, GetMetricData, GetResourceMetrics and GetLogEvents responses from a db.t4g.micro (PG 18.6, with and without Enhanced Monitoring) and an Aurora Serverless v2 writer in the test account. The replay fails on any request that differs from the recorded one, so request shape is part of the golden. expected.prom is the hand-derived import text. Result: 6 fail, 1 skip. Live e2e: 1 fail.
• Green 41c5739: 6 pass, e2e skipped offline, typecheck clean.
• Review fixes, each red then green:
  • 8452e7f/9420e3d: a datapoint that shows up late behind a newer one is still written. Dedupe is now per (series, timestamp), pruned after two windows.
  • c425c25/16e189f and 0f0e7a3/59051ee: a stalled VictoriaMetrics write or AWS call fails instead of hanging the loop. We added timeouts and throwOnRequestTimeout. The service also refuses a role ARN without an External ID, and the reverse.
  • a39806a/28d48e3: a response body that stalls after its headers fails the poll through a 30 s deadline for the whole poll.
  • 1467390/41e444d: under Bun, neither aborting nor destroying the SDK client closes a socket that is stuck mid-body. After a poll timeout the service exits with status 1, and the compose restart policy starts it clean. The test runs the real binary against a stalling endpoint.
• Dashboard: red c938504^ (4 failed), green c938504. Units and legend sorts pinned in 1467390, fixed in ff1a0e2 (binBps; Mean for rates, Last for levels). tests/grafana_dashboards: 394 passed.
• Final before review: 17 pass, 1 skip (live e2e), typecheck clean.
• Review round 1 fixes, 34fb53e:
  • CloudWatch and PI are queried up to the last minute boundary, and a bucket is written only once a further period has passed after it closed. A partial Average written early was frozen by the per-timestamp dedupe. Goldens regenerated: only the newest 60 s bucket of each fixture is dropped (9–10 rows per case, nothing else changes). New replay cases: partial newest bucket, unaligned clock (17 s past the minute sends the recorded requests unchanged), future PI point, PI failure keeps the CloudWatch samples, malformed Enhanced Monitoring event is skipped and never interpolated, synthetic Aurora reply asserting ×0.001 and ÷300.
  • The real binary against a mock AWS + import endpoint (RDS_POLL_INTERVAL_SECONDS=2): a 500 from VictoriaMetrics re-sends the same sample on the next tick; a missing instance logs samples written: 0 and stays up.
  • Fixtures redacted (account ID in ARNs, VPC/subnet/SG IDs, KMS key IDs); test/record.ts applies the same.
  • Dockerfile: VERSION/BUILD_TS ARGs, OCI labels, /VERSION, /BUILD_TS; built locally with the args (labels verified with docker inspect) and without them (fails as intended).
  • Dashboard 1: replica lag and burst / EBS I/O balance panels, spanNulls: 180000, CPU and Memory panels 15 units tall so all six Memory legend rows show. Tests assert families per panel, the spanNulls threshold and legend heights. tests/grafana_dashboards: 398 passed.
  • Result: bun test 36 pass, 1 skip; tsc --noEmit clean.
• Review round 2 fixes, fb425a9:
  • Performance Insights points get the same ingestion margin as CloudWatch: a point stamped at the query end (a minute that closed moments ago) waits for the next poll. Goldens lose one host_db_load row per fixture.
  • Enhanced Monitoring: an event whose timestamp does not parse is reported like any other malformed one; a malformed event is reported once per state, not on every tick; a null or rss-less processList row no longer drops host_os_process_max_rss_bytes for the whole window.
  • RDS_POLL_INTERVAL_SECONDS is capped at 300 s, so Aurora's 300 s volume buckets cannot be skipped; the README states the 13 min / 5 min spans.
  • Recorder redaction covers multi-region KMS keys (key/mrk-…), with a unit test.
  • Dashboard 1 "Burst and EBS I/O balance" uses a soft 0..100 axis like the other percent panels; a test asserts no hard min/max on percent host panels.
  • New tests: Enhanced Monitoring source failure keeps the CloudWatch samples; malformed timestamp; reported once across polls; processList rows; a 300 s volume bucket 3 min old is withheld while a 60 s one is written; interval 1.5, 60s, 301; the mock server binds loopback. Red against 34fb53e: 18 of 37 fail. Green: bun test 43 pass, 1 skip; tsc --noEmit clean; tests/grafana_dashboards 400 passed.

Real run (2026-09-28, test account, us-east-1)

Check	Result
Live e2e: db.t4g.micro, PG 18.6, EM 15 s, PI Standard → local VictoriaMetrics 1.140	Pass. 648 samples. 20 families: all 9 common CloudWatch, host_disk_free_bytes, host_ebs_io_balance_percent, host_db_load, and all 8 host_os_*
Live e2e rerun on 59051ee, a fresh cloud-rig instance (EM 60 s, PI)	Pass. Same 20 families, per-run label. The later commits add only the whole-poll deadline, which the replay tests exercise
Service loop against the Aurora Serverless v2 writer (0.5 ACU, EM 15 s, PI), 150 s	3 polls (630, 42, 34 samples). 18 families: 9 common, host_db_load, 8 host_os_*
Full monitoring stack (compose project of its own on local Docker: config-init with this branch's ./config, sink-postgres, VictoriaMetrics, pgwatch ×2, Grafana 12.3.2, and the rds-host-stats image built from this branch), prepare-db monitoring role, public db.t4g.micro whose SG admits only this host's /32, sslmode=require, light pgbench, 20 min	20 polls, no errors. VictoriaMetrics: 20 host_* families (18–20 samples each, 20 min span) next to 208 pgwatch_* families (16 of them pg_stat_statements) for the same cluster/node_name. Grafana: all 8 Dashboard 1 host panels return data through the provisioned datasource. The 3 empty series are Aurora-only: storage 1/3, IOPS 2/4, and everything else full
Teardown	Every instance, the cluster and the rig security group deleted, with no final snapshots. Stack volumes, .env and instance files removed. Reaper dry-run empty; 0 instances and 0 clusters left
Cost	About $0.08 in total: RDS about 45 + 7 + 26 min ($0.02), Aurora about 45 min at 0.5 ACU ($0.045), plus cents of Logs

Screenshots

Round 4 (5c4b768, collapsed row)

Environment: Grafana 12.3.2 and VictoriaMetrics v1.140.0 in Docker on the fixing host, dashboards provisioned from this branch's config/grafana/dashboards at 5c4b768 (the row JSON is unchanged since b8c34e5), datasource uid P7A0D6631BB10B34F. Series come from the replay golden test/fixtures/rds-postgres/expected.prom imported via /api/v1/import/prometheus; a pgwatch_db_size_size_b stub per node populates the dashboard variables. No AWS credentials are available on this host, so these are fixture renders. Captured with Playwright (system Chrome), 1600 px wide, time range 05:38–05:54 UTC on 2026-09-28.

1. node-03, a node with no host_* series, scrolled to the bottom of Dashboard 1: the "Host stats" row is the last panel and stays collapsed, "(10 panels)".

2. node-01, the rds-postgres fixture: the same row after clicking its title, with its 10 panels.

3. "DB load (Performance Insights)" with a non-zero series. Both recorded fixtures carry host_db_load = 0 throughout and there is no AWS access here, so this is the rds-postgres fixture re-imported as node-02 with host_db_load replaced by synthetic values between 0.4 and 1.6. It shows the panel's rendering only, not real Performance Insights load.

Round 3 (fb425a9, row expanded by default)

Dashboard 1 "Host stats" row at fb425a9, Grafana 12.3.2 + VictoriaMetrics 1.140.0 provisioned from this branch's config/grafana, datasource uid P7A0D6631BB10B34F, fed with the replay goldens (expected.prom) that the poller produced from the recorded AWS responses. No AWS credentials are available on the fixing host, so these are fixture renders, not re-renders of the 20-minute real run. Captured with a 60 s render budget so every panel has loaded. Series absent from a fixture show "No data": Storage, Replica lag and Burst / EBS balance on Aurora (young cluster, no replica, gp3), Replica lag on RDS. On RDS, Burst / EBS balance draws the EBS I/O balance at 99 % (gp3, so no BurstBalance series).

RDS db.t4g.micro fixture (rds-postgres, EM 15 s, PI), var-node_name=node-01, 05:38–05:54 UTC:

Aurora Serverless v2 writer fixture (aurora-postgresql, EM 15 s, PI), imported as node_name=aurora-01, 06:02–06:18 UTC:

Review

We ran the security, bugs, tests and guidelines checklists in seven rounds. Everything the rounds found is fixed above. The final round on 41e444d found nothing in any checklist.

We chose not to do two things:

• Follow GetLogEvents pagination. Each poll reads the newest page, up to 1 MB; one page covers under 60 s only at a 1 s Enhanced Monitoring interval with a large process list. The README says that at Enhanced Monitoring intervals of 1–5 s, OS samples missed during downtime are not backfilled.
• Unit-test the STS credential factory with a mock.

The MR is set to squash on merge, so the pre-redaction fixture blobs from 41e444d and earlier do not reach main.

What we did not check

• BurstBalance (gp3 has none) and ReplicaLag (no replica) returned no data, so those mappings rest on AWS docs.
• Aurora VolumeBytesUsed, VolumeRead/WriteIOPs and FreeLocalStorage did not appear within 45 min of cluster creation. The mapping follows AWS docs, and the recorded Aurora fixture has those queries with empty results.
• Assuming a customer role with an External ID end to end: the test user cannot create IAM roles. The live run used the default credential chain.
• The image itself: there is no Docker here. CI builds it.
• Wiring the role ARN into provisioning (console field, rds_monitoring_provision, the ansible role that writes the compose env) is a separate change. The Dashboard 1 "Host stats" row is in this MR.

🤖 Generated with Claude Code