fix(reporter): first run after 3 min, not 30

Tanyarequested to merge
fix/reporter-first-run-delay into main

Closes #423 (closed)

Problem

A new database gets its first checkup report 30 min after the stack starts: the reporter sleeps REPORTER_INITIAL_DELAY_SECONDS (default 1800) before run 1. mon deploy shows the same 30 min as first_checkup_eta.

Fix in one line

Run 1 starts after a 180 s minimum delay, then waits until every enabled target has data (at most 30 min). The data wait is the real bound, not the 180 s.

Why 180 s is a minimum, not a bound

Run 1 needs settings and db_stats samples per target. On a normal start they are in VictoriaMetrics after about 105 s (first pgwatch fetch 15 s + second sample 30 s + scrape 30 s + -search.latencyOffset 30 s). Collection has no fixed bound: some metrics run for minutes, and a failed first connect retries after 120 s. So after 180 s the script polls Prometheus every 15 s until each target has data, for at most REPORTER_READINESS_TIMEOUT_SECONDS (1800 s), then runs anyway.

Changes

  • config/scripts/postgres-reports.sh: first delay 1800 -> 180 s; then the data wait (python -m reporter.wait_ready); a failed run is retried after REPORTER_RETRY_SECONDS (1800 s) instead of exiting (the container restarted and ran again every few minutes).
  • reporter/wait_ready.py (new): reads enabled targets (cluster, node_name) from the pgwatch sources.yml; polls count by (cluster, node_name) of pgwatch_settings_configured and pgwatch_db_stats_in_recovery_int; always exits 0.
  • reporter/postgres_reports.py: when no cluster has data, the local fallback report is generated but not uploaded (it would read as "no issues").
  • cli/bin/postgres-ai.ts: mon targets add restarts postgres-reports, so a target added later gets its first report after the same delay and wait, not the next day. targets remove does not restart it (a removed node stays visible for 3 h; a run would report it again).
  • cli/lib/connect.ts: first_checkup_eta = ready + 3 min (the earliest time; a slow first collection makes it later).
  • .gitlab-ci.yml, cli:node:full:dind: real-stack evidence. Happy path: after targets add, run 1 has A002/A003 data per node. Non-happy paths, branch script in extra reporter containers with short delays: (a) unresolvable target host, 60 s limit; (b) empty VictoriaMetrics, upload on: no upload; (c) upload to a stub platform API; (d) Prometheus down: retry; (e) no sources file: no wait. targets remove must leave the StartedAt of postgres-reports the same. The job fails if a case's log lines are missing.
  • docker-compose.yml, CHANGELOGs: defaults and override list.

New env vars (postgres-reports): REPORTER_READINESS_TIMEOUT_SECONDS (1800, 0 = no wait), REPORTER_READINESS_POLL_SECONDS (15), REPORTER_RETRY_SECONDS (1800), REPORTER_SOURCES_PATH (/postgres_ai_configs/pgwatch-prometheus/sources.yml). REPORTER_INITIAL_DELAY_SECONDS default is now 180.

Not changed

  • Later runs stay daily (REPORTER_INTERVAL_SECONDS=86400).
  • Helm (own CronJob with --cluster) is separate.

Trade-off

Run 1's query reports (K, M, N) cover only the minutes since start. They also floor the window end to the hour today, so run 1 often has no query data either way; !446 changes that. The next full report is 24 h later.

Tests

  • tests/reporter/test_postgres_reports_script.py: test_first_run_waits_three_minutes_by_default, test_first_run_delay_env_override, test_first_run_waits_for_data_once_then_runs_daily, test_readiness_wait_can_be_turned_off, test_failed_run_is_retried_after_a_back_off.
  • tests/reporter/test_wait_ready.py: test_read_targets_takes_enabled_cluster_and_node, test_read_targets_missing_file_is_empty, test_read_targets_unquotes_before_it_strips_comments, test_a_target_needs_both_settings_and_db_stats, test_waits_until_every_target_has_data, test_gives_up_at_the_time_limit, test_no_targets_does_not_wait, test_main_passes_url_timeout_and_poll (main and python -m).
  • tests/reporter/test_main_cli.py: test_main_never_uploads_the_local_fallback.
  • cli/test/connect.test.ts: the ETA is 3 min ahead; first_checkup_eta uses the reporter's default initial delay ties FIRST_CHECKUP_DELAY_MS to the script default.
  • cli/test/connect.integration.test.ts (CI cli:clickhouse-like:tests): the real mon deploy against a fake platform and a real Postgres. It prints first_checkup_eta and the end of the run, and fails unless the ETA is the end + 3 min.
  • cli/test/host-metrics-providers.test.ts: targets add restarts postgres-reports after the collectors; remove does not; the compose call order.
  • CI cli:node:full:dind: the real-stack cases above.

Not covered: that the real first report lands near the ETA. That needs a hosted box.

🤖 Generated with Claude Code

Edited by Maya P

Merge request reports