Note CRDT sync

Note CRDT sync

Canonical implementation doc for study note / journal body sync on web (v1). Product rationale and scope table: Sync and Offline.

Library: Automerge

Package: @automerge/automerge on web (ui/) and in api/scripts/crdt/ for server-side merge (same major version in api/package.json and ui/package.json).

Document shape: { body: string } — a rich text envelope JSON string in body (CRDT_SCHEMA_VERSION = 2). Legacy markdown in body is migrated on read; markdown export is derived for markdown_body only.

Why Automerge (not Yjs): one library on web and server merge path; structured document model; automerge-kotlin path for mobile later. Yjs was rejected for v1 to avoid incompatible merge semantics across clients.

System context

flowchart LR
  subgraph clients [Clients]
    WebUI[Vue3_Automerge]
    MobileLater[KMP_later]
  end
  subgraph api [API]
    GQL[GraphQL_pull_push]
    Laravel[Laravel_PHP]
  end
  subgraph mergeWorker [CRDT merge worker]
    Merge[Node_Automerge]
  end
  subgraph data [Data]
    PG[(notes_crdt_state)]
  end
  WebUI --> IDB[(IndexedDB)]
  WebUI --> GQL
  GQL --> Laravel
  Laravel -->|HTTP_or_queue| Merge
  Laravel --> PG
  Merge --> Laravel
  MobileLater -.-> GQL

• Web: Vue 3 loads/saves Automerge state in IndexedDB; sync via GraphQL when online.
• API (Laravel): GraphQL, auth, Postgres — no Automerge in PHP.
• CRDT merge worker (Node): small service (almost a microservice) that runs Automerge merge; Laravel calls it instead of embedding Node in the PHP image.
• Postgres: notes.crdt_state (base64 snapshot) and notes.crdt_version (heads / version metadata) are source of truth after push.
• Mobile: deferred until after web Wave 2 checkpoint (see Mobile timing).

Sync sequence (two tabs)

v1 uses polling (no GraphQL subscriptions). Typical push then pull on another tab:

sequenceDiagram
  participant TabA as BrowserTabA
  participant IDB as IndexedDB
  participant API as LaravelGraphQL
  participant Merge as NodeAutomergeMerge
  participant TabB as BrowserTabB

  TabA->>IDB: save local Automerge state
  TabA->>API: pushNoteUpdates(changes)
  API->>Merge: merge server snapshot + changes
  Merge-->>API: merged snapshot + heads
  API-->>TabA: new crdtVersion + state
  TabB->>API: pullNoteUpdates(sinceState)
  API-->>TabB: state if version differs
  TabB->>TabB: merge into local doc

Note record fields

Note row on Postgres:

flowchart TB
  subgraph noteRow [notes row]
    id[id_UUID]
    crdt_state[crdt_state_base64]
    crdt_version[crdt_version_heads]
    markdown_body[markdown_body_export]
    deleted_at[deleted_at_soft_delete]
  end
  crdt_state -->|source_of_truth| Body[doc.body]
  Body -->|after_each_push| markdown_body
  deleted_at -->|plus_CRDT_tombstone| Tombstone[doc.deleted]

Field	Role
id	Client-generated UUID (optional on createNote in Wave 2) — stable across devices
crdt_state	Base64 Automerge snapshot — source of truth for body
crdt_version	Server heads / version string — cheap pull comparison
markdown_body	Denormalized export of doc.body for lists, search previews, legacy reads
deleted_at	Laravel soft delete plus CRDT tombstone in document when deleted

Wire format

Automerge binary snapshots are transported as base64 in JSON/GraphQL.

Merge script contract (api/scripts/crdt/merge.mjs):

• stdin: { "serverState": "<base64|null>", "incomingState": "<base64>" }
• stdout: { "state": "<base64>", "version": "<heads>", "body": "<envelope-json>", "deleted": <bool> } — body is the merged envelope string; markdown_body is derived from it on persist.

v1 dev: Laravel NoteCrdtService invokes api/scripts/crdt/merge.mjs as a subprocess (Node on PATH in Dockerfile.dev) for fast iteration.

Production target: the same merge logic lives in a dedicated Node worker (HTTP or queue consumer). Laravel sends { serverState, incomingState } and receives { state, version, body, deleted } — languages stay separate; the PHP runtime image does not need Node.

Client schema version: CRDT_SCHEMA_VERSION = 2 in ui/src/lib/crdt/ — bump when document shape changes; show recovery UI on mismatch.

GraphQL pull / push (Wave 2)

Schema additions in api/graphql/schema.graphql (planned):

Operation	Purpose
pullNoteUpdates(noteId, sinceState?)	Return full crdt_state + crdt_version when server version differs from client
pushNoteUpdates(noteId, incomingState)	Merge via merge.mjs, persist snapshot, return new version + state
createNote	Optional client id for offline-first create
deleteNote	Soft delete + CRDT tombstone

Scalar: Bytes (or equivalent) for opaque CRDT payloads in GraphQL.

Auth: same as other note mutations — user owns note (ABAC).

Not in v1: GraphQL subscriptions for live multi-tab sync; tabs poll or refetch after push.

Server merge

On every successful pushNoteUpdates:

1. Load current crdt_state / crdt_version from Postgres (or empty for new note).
2. Call the merge implementation (dev: subprocess merge.mjs; prod: merge worker).
3. Save merged state, version, and refresh markdown_body from merged body.
4. Return merged state to client.

Concurrent pushes from two devices merge deterministically via Automerge (no last-write-wins).

Merge worker (production)

Treat the worker as a small Node microservice, not part of the Laravel monolith:

Concern	Laravel API	Merge worker
Runtime	PHP 8.4	Node 22
Responsibilities	GraphQL, Sanctum, policies, DB writes	Automerge load / merge / save only
Interface	NoteCrdtService abstraction	POST /merge (or Redis job payload) with same JSON as merge.mjs stdin/stdout
Deploy	Existing api image (PHP-only)	Separate container in Compose / K8s

Benefits: PHP image stays lean; merge can be scaled or restarted independently; one Automerge version shared with ui/ and the worker codebase (extract merge.mjs logic into a tiny package the worker runs).

Wave 2 can keep the subprocess adapter; swap the adapter to HTTP/queue when the worker ships — no change to GraphQL or clients.

markdown_body rules

Case	Rule
Note has crdt_state	markdown_body is derived on each server merge from doc.body; do not treat as editable source of truth
updateNote with CRDT present	Reject markdownBody input when CRDT is active (bootstrap-only paths excepted)
New note without CRDT yet	Allow markdownBody on create for migration/bootstrap; first push establishes CRDT
List / session UI previews	Read markdown_body (kept in sync on merge)

Web client offline outbox

stateDiagram-v2
  [*] --> Editing: load_from_IDB_or_pull
  Editing --> LocalSaved: debounce_write_IDB
  LocalSaved --> Queued: offline_or_push_failed
  LocalSaved --> Synced: push_ok
  Queued --> Synced: online_drain_outbox
  Synced --> Editing: user_edits
  Editing --> Recovery: schema_mismatch
  Recovery --> [*]: copy_text_manual

Modules (Wave 1–3):

Path	Role
ui/src/lib/crdt/doc.ts	Create/load Automerge doc, apply changes
ui/src/lib/crdt/storage.ts	IndexedDB persistence
ui/src/lib/crdt/sync.ts	Outbox drain, pull/push (Wave 2)
ui/graphql/notes.graphql	Operations for codegen

Conflict UX: Automerge merges edits automatically. On schema mismatch, show explicit “sync problem — copy your text” recovery (Sync and Offline).

Mobile timing

Defer KMP implementation until after the web Wave 2 checkpoint (API NoteCrdtTest green + two-tab pull/push spike).

Until then:

• Document only here: future Apollo Kotlin pull/push, SQLDelight local CRDT cache, automerge-kotlin integration.
• Do not add Apollo, SQLDelight, or automerge-kotlin in parallel with Wave 1–3 web work.

Follow-up (post-checkpoint): Wave 4 — mobile GraphQL ops, SQLDelight cache, journal UI.

Deployment

Environment	Merge execution
Dev / CI	Subprocess merge.mjs in API container (Dockerfile.dev includes Node 22)
Production	Dedicated merge worker container — Laravel calls worker; production api image stays PHP-only

Operator wiring (Compose / Kubernetes): add a crdt-merge (name TBD) service built from api/scripts/crdt/ (or a slim services/crdt-merge/ package), internal network only, env for timeouts. Document URL in stack.env.example when implemented.

Rejected for production: installing Node inside the main API image — couples runtimes and enlarges every API deploy for one operation.

Scripture references in note bodies (Wave 3)

Note body stores scripture citations as TipTap scriptureRef inline atoms (path, label). Legacy [[ref:…]] markdown tokens import via fromMarkdown(); see Rich text format.

• Book ids: USFM-style (Jn, Rom, 1Co, Ps). Free-form text (e.g. John 3:16) normalizes to refs on editor blur.
• Chapter-only: Citations of a full chapter use path without a verse segment (including 1–N spans per KJV canon).
• Validation: Client-only, KJV verse counts (ui/src/lib/bible/canon.kjv.ts); warnings are non-blocking — save/sync still proceeds.
• Preview / list cards: Refs expand to English labels; preview links use Logos ref.ly (https://ref.ly/{bookId}{chapter}.{verses}, e.g. Rom.1.1 → https://ref.ly/Rom1.1).
• Usage indexing: Deferred — do not sync bible_references on push; see Architecture Questions.

Editor polling: While the note editor is open, the client pulls about every 45s and on visibilitychange / online (in addition to debounced push after local edits).

Out of scope (this effort)

• Mentorship message CRDT
• Mobile KMP code (this effort)
• GraphQL subscriptions (v1 polling)
• Yjs

Related

• Rich text format — envelope schema, TipTap nodes, markdown interchange
• Sync and Offline — product decision and scope
• Domain Model — studies, log entries, and notes
• Architecture (planning) — monorepo stack overview