Signals harness: final architecture

One harness, one PR · target design, nothing built yet · 2026-10-22

already in Olivnew shared coresafety gate

1 · Data already in Oliv (the harness only reads it)

EmailsGmail / Graph pull + association

Meetingstranscripts, post-call

Slack messagesbackground context

TicketsClickUp / Jira

Deals · Accounts · Fieldscurrent values, owners, CRM link

↓ reads Baserow tables only · holds no provider credentials

2 · Per-org config (Baserow rows; turning a signal on is a row, not a deploy)

Source Configorg × source: on/off, filters, lookback, when an item counts as ready

Signal Configorg × signal: mode off / dry-run / live, settings, each output → destination + publish policy, model/effort, $ cap per night

↓

3 · Schedule (one worker pool, `oliv-celery-harness`)

beatnightly run + weekly full sweep

dispatchevery org × signal whose mode isn't off

pick subjectsdeals or accounts with new/changed items since the watermark (with overlap)

fenced leaseone run per (org, signal, subject); a second delivery exits; kill switch

↓ one task per subject

4 · Per-subject run (same steps for every signal)

Read sources → items {ref, deal/account, occurred_at, changed_at, text, hash, ready}
emails by sent date or associated time · meetings re-scanned 14 days · Slack/tickets by update time
Completeness gate: are all enabled sources complete for the window?
no → no decision this run, old output stays, watermark doesn't move
Ledger diff (S3 hash per item): only new or changed items go on
nothing changed → no model call · transcript not ready → waits as “pending”
plugin.prompt → model runner (Codex turn, shared cached prefix per org so repeat calls are cheap) → facts, each with {ref, quote}
Quote check: drop any fact whose quote isn't in the cited email/meeting (numbers and dates too)
plugin.decide: pure rules, no I/O → field values / child rows / card values (shown on the dashboard)
Publisher: applies each output's policy; lease checked before every Baserow write, retries included; field edited by a person since the read → held; readback after write
Ledger written last, then the watermark moves
crash before that → next run repeats; publish finds nothing new to write

↓

5 · Where results land (publish policy, set per output in Signal Config)

Policy	Writes	Goes to CRM?
`card`	AI-owned Deals columns (e.g. Competition Status / Description / Reasoning)	never
`rows`	AI-owned child rows (e.g. one per deal × competitor); rows a person made or hid are never touched	never
`if-empty`	AI field only when the live value is empty	yes
`overwrite`	AI field when the value changes; held if a person edited it since the read	yes
`review`	the field's AI Recommended column only	after a person accepts

Signals (plugins) in this PR

Plugin	Subject	Reads	Outputs	First org
`field_extraction`	account + its deals	emails (meetings later)	CRM AI fields via `if-empty` / `overwrite` / `review`, chosen per field type	Qu, then Whatfix
`competitors`	open deal	meetings + emails	`card` (health, description, reasoning) + `rows`	Onyx (ws1197)

A plugin is one small module: prompt(), output_schema, settings_schema, pure decide(). Everything that touches the network or Baserow lives in the core. A new signal (e.g. escalation reasons, next steps) = a plugin + a config row.

Code layout

services/signals/
  dispatch.py      beat → org × signal → subjects
  lease.py         fenced lease, kill switch
  sources/         email, meeting, slack, ticket readers + completeness
  ledger.py        S3 item hashes, pending, watermark
  runner.py        Codex turn, cached prefix, cost per run
  verify.py        quote check
  publish/         card, rows, field policies (if-empty, overwrite, review), readback
  runs.py          status rows in Runs (Type = signal)
  plugins/
    field_extraction/
    competitors/
jobs/tasks/signals.py   Celery entry points (harness pool)

State

Store	Holds
S3 `oliv-harness/signals/<ws>/<signal>/<subject>.json`	item ledger (ref → hash), pending items · no expiry
Baserow `Runs`, Type = `signal`	one row per org × signal × day: complete / partial / failed, counts, cost, errors (needs the `signal` option added per workspace)
Baserow `Source Config`, `Signal Config`	what's on, where outputs go, policies, caps
Redis	leases, publish once-keys, kill switch

Done = both old PRs can close

Competitors parity: dry run on the same Onyx deals; card values and rows match the old competitors code exactly
Field extraction parity: replay the Qu and Whatfix shadow sets; same values, same or fewer wrong, cost no higher
Safety tests carried over: two workers on one subject, lease lost mid-write, retries after takeover, crash at each step, person edits a field mid-run
Old flags and config/deal_competitors.json become Signal Config rows; nothing else reads them