Dreaming

Quiet-hours consolidation of the Project Ledger.

What Dreaming is

Dreaming is Aira's scheduled, quiet-hours pass over its own Project Ledger. While the team is away, Aira re-reads the evidence-backed knowledge it has accumulated and performs the global-scale hygiene and reasoning that cannot happen at ingestion time: it deduplicates and merges near-identical atoms, prunes stale evidence, cleans up duplicate atom wording, and resolves contradictions between conflicting claims. Every change Aira wants to make is written as a proposal for human review first; nothing is applied silently. Only after a project has earned a track record (the graduation gate) can an operator turn on autonomous application of the safe, high-confidence changes — and even then every non-obsolete applied change is evidence-backed, audited, and reversible for 48 hours.

The one-line mental model: ingestion writes the Ledger atom-by-atom; Dreaming steps back and curates the whole Ledger as a body of knowledge.

Why it exists

A trustworthy, evidence-backed Ledger is the product — answers are only as good as the knowledge behind them, and "day 30 better than day 1" collapses if the Ledger rots. Some rot is structurally impossible to fix at ingestion time and only visible globally:

• Cross-source duplication — the same fact arrives from four sources as four near-identical atoms.
• Contradictions — two engineer-authored tasks give opposite instructions ("use PostgreSQL" vs "don't"); the conflict only exists when you look at both.
• Staleness vs moved ground truth — an atom was true when captured but the source has since changed.
• Unanalyzed backlog — sources ingested but never run through analysis.

These are cross-cutting, latency-unconstrained problems — exactly what a batch consolidation pass on a stronger model, run during quiet hours, is for. Dreaming is the consolidation layer over the Ledger primitives, not a new subsystem.

The dream cycle

A dream run is a resumable, three-stage pipeline (services/dreaming/engine.py). Each stage records its state in the DreamRun.stages JSONB, so a run that pauses (budget, circuit breaker, failure) resumes from where it stopped rather than restarting.

flowchart TD
  T["Stage 1 · Triage (deterministic)<br/>scan atoms · fingerprint · flag unanalyzed sources"]
  C["Stage 2 · Consolidate<br/>cosine-near pairs → one datamarked verdict<br/>(merge / not_duplicate / normalize) → cluster collapse"]
  R["Stage 3 · Resolve<br/>open contradictions → datamarked verdict<br/>(keep_a / keep_b / both_valid / needs_human)"]
  P["Propose-only: MergeOps (review_status = pending)<br/>+ metrics · cost · stop_reason on the DreamRun"]
  Q["Morning review queue (HITL)<br/>approve / reject / bulk → atomic content-validated apply · 48h rollback"]
  T --> C --> R --> P --> Q
  classDef stage fill:#ede9fe,stroke:#7c3aed,stroke-width:1px,color:#4c1d95
  classDef gate fill:#dbeafe,stroke:#2563eb,stroke-width:1px,color:#1e3a8a
  class T,C,R stage
  class P,Q gate

• Stage 1 — Triage (deterministic, no LLM): scans the project's atoms, computes fingerprints, identifies what needs attention, and flags sources that were never analyzed. Cheap and bounded.
• Stage 2 — Consolidate: finds candidate near-duplicate pairs by embedding cosine similarity (above COSINE_THRESHOLD), then for each pair makes one datamarked LLM call returning a verdict — merge, not_duplicate, or normalize (here normalize means "same fact, cleaner wording" — it rewrites the duplicate's canonical body via proposed_body; it is not temporal-reference normalization). Confirmed duplicates are handled cluster-aware: a union-find groups the pair-wise duplicates, then proposes cluster-collapse operations so each cluster can later be collapsed into one canonical atom in the apply phase (POST /proposals/{op_id}/approve, POST /runs/{id}/apply-cluster; losers → superseded, evidence/links re-pointed on apply), rather than emitting O(N²) pair-wise merges.
• Stage 3 — Resolve (contradiction sweep): scans open LedgerContradiction rows, gathers the two conflicting atoms and their evidence, optionally consults the Guardian drift framework for a source-of-truth signal where a validator applies, and makes one datamarked LLM call returning keep_a, keep_b, both_valid, or needs_human. In current M2.2 behavior, no atom-level adjudicator is registered, so Guardian returns no signal and resolution is based on evidence + LLM alone. A confident verdict at or above CONFIDENCE_FLOOR becomes a propose-only resolve_contradiction proposal; everything below the floor, both_valid, or needs_human is escalated (left open for a human) rather than auto-decided.

Datamarking (prompt-injection defense). Every LLM call in Stages 2 and 3 wraps all Ledger/source content in datamark sentinels so an instruction embedded in an atom body or evidence snippet is treated as data, not as a command. A dedicated injection-regression suite proves it (disabling the datamark makes the test fail), and it is a hard precondition for auto-apply graduation.

Propose → Review → Apply

Dreaming never mutates the Ledger inside the engine. The stages only write proposals — MergeOp rows with review_status = pending. The op kinds are merge, obsolete, update, and resolve_contradiction. They surface in the morning-review queue (GET /proposals), where a reviewer can approve or reject a single proposal, bulk approve/reject many at once, or collapse a whole duplicate cluster atomically.

Apply is atomic and content-validated: it locks the affected atoms, verifies they still match the proposal's content snapshot (it validates by content, not by updated_at — a hard lesson from production, where a background job bumping updated_at in lockstep defeated timestamp-based staleness), then performs the change in one transaction and writes a LedgerHitlEvent capturing before_state and after_state.

Rollback. Because non-obsolete applies record before/after state, those changes are reversible for ROLLBACK_WINDOW_HOURS (default 48h) via GET /hitl-events and POST /hitl-events/{event_id}/rollback, which re-activates superseded atoms, re-points evidence back, and re-opens resolved contradictions. obsolete applies are intentionally non-rollbackable because they orphan user-facing insight/artifact state outside the HITL snapshot.

Governance and graduation

Dreaming launches propose-only for every project (hitl_mode = review_required). Autonomous application is a privilege a project earns and an operator grants — never a default.

The graduation gate (services/dreaming/graduation.py, compute_graduation_state). A project becomes eligible for auto-apply only when all three hold:

• at least GRADUATION_MIN_CYCLES (default 10) qualifying dream cycles, and
• false-prune rate below GRADUATION_MAX_FALSE_PRUNE (default 2%), and
• HITL acceptance rate at or above GRADUATION_MIN_ACCEPTANCE (default 90%).

A qualifying cycle is one that cleared a minimum-evidence floor (MIN_QUALIFYING_PROPOSALS, default 1, or MIN_QUALIFYING_ATOMS_SCANNED, default 50) — a quiet no-op night does not count toward graduation. Acceptance is approved ÷ (approved + rejected) proposals; false-prune is rolled-back/reverted applies ÷ total applies.

Operator-confirmed switch. Meeting the gate only flips auto_apply_eligible. It never flips behavior. A human Owner/Admin must explicitly enable it (POST /graduation/enable, which refuses with 409 if the project is not eligible) — and can disable it at any time (POST /graduation/disable, always allowed, reversible). The gate measures; the human authorizes.

Guarded auto-apply. When a project is graduated and enabled, the engine auto-applies only proposals at or above CONFIDENCE_FLOOR (default 0.85), through the same atomic content-validated path used for HITL approvals. obsolete op types are carved out of auto-apply entirely (_NON_AUTO_APPLY_OP_TYPES): even at confidence = 1.0 they always route to HITL, because an obsolete apply flips orphaned insights/artifacts and is not rollback-safe. Everything below the floor, plus every escalation, still goes to the morning queue. All other guards stay in force: per-cycle circuit breakers, the token budget, datamarking, the advisory lock, and the 48h rollback (auto-applied changes are equally reversible).

Scheduling

• Nightly — the NightlyDreamScheduler loop (services/dreaming/nightly_scheduler.py), not the Scheduler agent, enqueues a per-project dream run inside the project's timezone-aware quiet-hours window. It owns no new engine path — it enqueues the same dream_run job a manual trigger does. It is on by default per project (opt-out via nightly_dreaming_enabled), idempotent (one run per day), and fail-loud on misconfiguration. Nightly runs use the same dream job path as manual triggers; they are propose-first unless the project is graduated and auto-apply is enabled, in which case eligible proposals are auto-applied at completion.
• Manual "Dream now" — POST /runs triggers an on-demand run for review (used for the supervised first-run backfill before nightly scheduling is enabled).

Safety model

Data model and API surface

Core tables. ledger_dream_runs (DreamRun: status, resumable stages, stage_tokens, total_cost_usd, stop_reason, and per-stage counters — atoms scanned, candidate pairs, proposals created, contradictions scanned, resolutions proposed, escalated-to-HITL); merge_ops (proposals, with review_status and a payload carrying the originating dream_run_id); ledger_hitl_events (before_state/after_state + reverted_at for rollback); ledger_contradictions (the Stage-3 input/output). Per-project governance state (hitl_mode, auto_apply_eligible, auto_apply_enabled + audit) lives in project settings.

Endpoints (under /api/v1/projects/{id}/dreaming):

• PM users can read: GET /graduation, GET /runs, GET /runs/{id}/report, and GET /metrics.
• HITL/review/rollback/apply routes are Owner/Admin-only.

Configuration reference

All knobs are AIRA_* environment variables on the agent service.

User-facing surfaces

• Dream Report (/dreaming) — the "what did Aira figure out overnight?" view: a hero summary, the morning-review queue as the primary action, the list of what changed (applied / pending / escalated), cost, and graduation progress (with the guarded "Enable auto-apply" affordance when eligible).
• Benchmark dashboard (/dreaming/metrics) — graduation progress plus time-series trends (acceptance, false-prune, cost per cycle) and a per-run history.
• An in-app notification announces a finished run's report.

Boundaries

Dreaming is deliberately scoped. Out of scope (today): risk detection and dependency mapping, artifact regeneration, multi-project dreaming, adaptive scheduling, and channel delivery (Telegram/email) of the Dream Report. Explicitly not part of Dreaming: any coupling to development/CI infrastructure — Dreaming operates only on the project's own Project Ledger.