feat(ingest): Serena memory migration and proactive injection parity

James Hebdenrequested to merge
jhebden/opencode-memory:feat/serena-memory-integration into master

Problem

Two issues observed in practice:

  1. Serena memory silo: Serena accumulates project knowledge in .serena/memories/*.md files that the memory system never sees. When a session uses Serena's tools, those learnings stay in Serena's flat-file store rather than being ingested into the semantic memory DB where they can be recalled, linked, and proactively injected across future sessions.

  2. Proactive injection imbalance: Serena's read_memory / write_memory tools get invoked automatically and preferentially because they appear directly in Serena's tool surface. The opencode-memory plugin injects context proactively, but when Serena's memory tools have already fired, the model treats that as sufficient and the richer semantic memory system gets bypassed.

Scope

1. SerenaObserver — Python (src/opencode_memory/ingestion/serena.py)

  • Watches .serena/memories/ directories under registered project roots
  • Parses each .md file: filename → what, body → content, category inferred from content heuristics
  • Tracks ingest progress by file mtime high-water mark — idempotent on re-run
  • Source key: serena:<project>:<filename> for deduplication
  • Wired into the daemon's polling loop alongside the existing OpenCode DB and Claude Code observers

2. SerenaObserver — Rust (rust/crates/memory-core/src/ingestion/serena.rs)

  • Full parity with the Python observer — same parsing logic, same mtime tracking, same source key scheme
  • Added as a third polling arm in run_daemon alongside OpenCodeDbObserver and ClaudeCodeObserver

3. Historical migration — Python and Rust

  • run_serena_ingest in historical_ingest.py + memory ingest-serena CLI subcommand (Python)
  • run_ingest_serena in commands.rs + IngestSerena subcommand in main.rs (Rust), mirroring run_import_claude_native
  • Both dedupe against existing memories and preserve original file mtime as created_at

4. Plugin: Serena-aware proactive injection (plugin/src/index.ts)

  • tool.execute.after records calls to the Serena memory/knowledge tool set (serena_read_memory, serena_write_memory, serena_list_memories, serena_find_symbol, serena_search_for_pattern, serena_find_referencing_symbols, serena_get_symbols_overview, serena_initial_instructions, serena_onboarding)
  • Relevant argument values (memory_name, topic, name_path_pattern, substring_pattern, relative_path, content) are queued per session
  • chat.message consumes (and clears) the queued terms, folding them into the existing getProactiveContext query — one enriched call, no second round-trip, no double-injection
  • messages.transform (thinking-aware path) peeks the same queued terms without consuming them

5. Serena → memory mirror (plugin/src/index.ts + memory-client.ts)

  • serena_write_memory calls are now also fire-and-forgot into memory_remember (category fact), so Serena onboarding facts become recallable immediately rather than waiting on the next daemon poll cycle
  • New MemoryClient.remember() and MemoryClient.sessionStart() HTTP wrappers
  • system.transform now calls memory_session_start on the first turn (fire-and-forget) so session-coordination tools see the session from turn one

6. Boot context parity note

  • Added a note explaining the two memory systems' complementary roles: Serena = project-local structured knowledge; GHMEM = cross-session semantic recall, decisions, blockers, queue, workflow.

What this is NOT

  • Not replacing Serena's memory tools
  • Not blocking Serena tool calls — the plugin supplements, never intercepts
  • Not adding new MCP tools

Testing

  • 25 Python + 22 Rust unit tests for SerenaObserver (parse, dedup, mtime tracking)
  • CLI dispatch test in Rust for ingest-serena
  • 4 plugin tests (serena-aware.test.mjs): query enrichment, one-shot consumption, non-Serena tools ignored, multi-call accumulation
  • Full existing suites re-verified green: 1063 memory-server + 2236 memory-core Rust tests, 250 plugin tests, 5591+ Python tests (excluding pre-existing environment-flaky failures unrelated to this change)
  • Not yet covered by automated tests: the system.transform auto session_start call and the serena_write_memory → memory_remember mirror (both fire-and-forget, added most recently) — manual verification only so far

Rebased onto latest master (no conflicts, CI green).

Edited by James Hebden

Merge request reports